Author: Dan Gentry

I have the first application upgraded from v1.0 to v1.1 of Django (spending most of my time with the environment, not with code), so I’m ready to move on to v1.2. My quick review of the release notes shows just a couple of things I have to change, and may I’d like to change. (Stay in scope, Dan!)

CSRF protection – requires a tag to be added to every form, plus changes to the middleware settings.
Multiple Database Declaration – allows the use of more than one database. Not a required change now, but since the old code will be deprecated soon enough, I thought I should get the change made while it is fresh in my mind.

There are so many posts on the web covering how to get the Python Imaging Library (PIL) to work on Linux, and they almost all say the same thing: install libjpeg6.2 and then recompile/install PIL.

What most fail to mention is that one apparently needs to also remove the previous PIL install before reinstalling. I don’t know (and don’t care to research it), but I think that the install sees the already compiled code and skips that step. Wonder if there is an option that forces new compiles.

So, find wherever your distribution stores the Python site-packages/dist-packages and remove the PIL directory and PIL.pth file before running install. Just to be thorough (and maybe a little anal retentive), I also removed the unpacked directory of PIL source.

Also, and this is the BIG PIECE OF NEWS to me, for those using Ubuntu 11.04 like me, libjpeg6.2 won’t work. Instead, use libjpeg8. In the Synaptic Package Manager, a quick search for libjpeg will show both versions. Leave libjpeg6.2 alone, but be sure to remove any dev libraries for that version. Then, mark anything that looks remotely connected to libjpeg8 and apply.

Now that your old version of PIL is gone, and the libs are squared away, it is time to install PIL. I used version 1.1.7. Finally, success!

(Well, except that I seem to have not installed the library that enables PNG use. Some other day maybe.)

UPDATE:
To enable PNG support in Ubuntu 11.04, use these commands to find the zlib library BEFORE installing PIL:

$ cd /usr/lib
$ sudo ln -s i386-linux-gnu/libz.so libz.so

ANOTHER UPDATE:
Using version 12.04 of Ubuntu, I also had to create a symbolic link for libjpeg – AFTER following the instructions above and BEFORE installing PIL:

$sudo ln -s i386-linux-gnu/libjpeg.so libjpeg.so

 

My quick review of the 1.1 release notes only found a few things to update to move my v1.0 application up a version.

• Unmanaged Models – will be very helpful with my latest app that uses legacy tables
• Admin URLs – uses include syntax
• Reverse Admin URLs – I’ve already coded this in two apps, but had to comment it

Also, I’m looking at the {% empty %} clause of a {% for %} loop in templates. It will be a lot cleaner than the current method of checking for data in the data source, although I did promise myself not to do any refactoring. We’ll see how that goes.

As the title of this post reveals, I’m a little behind in Django versions on my app server. There are several reasons why:

No virtualenv

My server admins, who are very competent, don’t have experience with virtualenv (that I know of). Without this tool on the server, it is impossible to upgrade one app at a time, or even introduce a new application using a newer version. The app I just deployed on Friday is using 1.0.4.

Other App Work

I’ve been either writing new apps or adding functionality to existing apps pretty much constantly for the past 18 months. No time to freeze features and upgrade.

Other Work

My main job function is as a manager, so I can’t sequester myself inside Eclipse for a month to get things up to date

Besides being vulnerable to bugs that have long since been fixed, I am also unable to use any of the new features that have come out in the past three years. Aggregation functions, CSRF, class based views, and others tempt me with their usefulness, but I can’t take advantage of them.

So, the project to upgrade has started. First step was to review the release notes for the three versions that succeed 1.0 to refresh my self on the changes – especially the backward incompatible ones. I’m not looking to do a lot of refactoring unless the change is required for the app to run under v1.3. That will come later during work on the next release of each app. I have a pretty short list of things to check and update before I even try to run in the latest version of the framework.

I’m currently in step two, which is to begin to modify the first app to run under v1.3. Since I already have virtual environments setup for each app, its no problem to switch things around as I work. Also, I’m adding a new branch in SVN for the 1.3 work in case any maintenance needs come along.

Third step, or more likely concurrently with step two, will be to work with the sysadmins to setup virtualenv on the DEV server. We will isolate each application (there are four) in its own environment and then I can update and test the code for each at my own pace as changes are ready. I think that once they see the benefits of separating the dependencies of each program, they will accept it wholeheartedly.

Lastly, we will setup the virtual environments on the QA and PROD servers and move things forward. My users (and customers) will hopefully not notice as this happens.

Wish me luck!

Today I wrote a custom renderer for a radio button field. What I wanted to do was render the control as a table, and expand the label to multiple columns. Took me a long time and much research to figure this out.

Rather than post my custom work, I’m going to work on a generic version – sort of an .as_table option – and post that when it is ready.

Still, had to mention it here because I’m pretty darned proud of myself. 🙂

My application includes some existing tables from another, non-Django application. One of them does not have an atomic primary key, and I’m not in a position to change that. Therefore, I’m using one raw SQL to access the data.

 

Where I run into trouble is that the query returns data in columns, but not objects. That means that I can’t reference an object attribute using dot notation in my templates or views. Instead, I have to include each attribute that will be needed as a query column. This will limit what I or someone else can do in templates later without a mod to the query.

In an earlier application with a similar situation, I took the results from the query and built a dictionary of objects. Clumsy, hard-coded, and likely slow, but it did give me what I needed.

I’ve looked around the web, but haven’t found any practices to follow, so I hope my many readers (smirk) can offer some help.

Have you tackled this dilemma in the past? Or maybe read where someone else has discussed a possible solution? (I’ll also take wild guesses). Please leave a comment below or drop me a line. I’ll summarize the responses and my solution(s) in a future post.

Thanks!!

UPDATE:

Here’s how I’m doing it now:

def available_classes_query(username=None):
    """Run a simple query and produce a generator
    that returns the results as a bunch of dictionaries
    with keys for the column values selected.
    """
    from itertools import izip
    from django.db import connection

    query_string = """select tu.id as traininguser, cl.id as trainingclass, ts.id as trainingschedule
                                 -- big chunk of SQL cut for readability --"""

    cursor = connection.cursor()
    cursor.execute(query_string, [username])  ## parameter for query
    col_names = [desc[0] for desc in cursor.description]
    while True:
        row = cursor.fetchone()
        if row is None:
            break
        row_dict = dict(izip(col_names, row))

        try:
            row_dict['traininguser'] = TrainingUser.objects.get(id=row[0])
        except TrainingUser.DoesNotExist:
            row_dict['traininguser'] = None
        try:
            row_dict['trainingclass'] = TrainingClass.objects.get(id=row[1])
        except TrainingClass.DoesNotExist:
            row_dict['trainingclass'] = None
        try:
            row_dict['trainingschedule'] = TrainingSchedule.objects.get(id=row[2])
        except TrainingSchedule.DoesNotExist:
            row_dict['trainingschedule'] = None

        yield row_dict
    return

ANOTHER UPDATE:

I posted this question on the Django Users group of Google Groups, and the very obvious answer was suggested – use the .raw() method of the manager.  This was added in Django 1.2, and I remember reading about it.  The RawQuerySet returned by this method will contain objects – even though the SQL provided will only return columns from the row.  Pretty slick.

The reason I didn’t find this is because I’m still using version 1.0.4, so I limited my searches to the docs from that version.  Yet one more reason to upgrade.

Today I’m working on an XML feed of events into an enterprise calendar program.  One of the requirements is the base64 encoding of accompanying images. My system gives the option to store the URL to a photo, not to store the image locally.  A bit of a shortcut I know, but I didn’t want to mess with the bureaucracy of the server team to work out the storage a backup of the images.  Luckily, it’s no big deal to Python either way. For a locally stored images, one can convert to base64 pretty easily, given the file name.  Try this:

import base64

image = 'example.jpg'
image_64 = base64.encodestring(open(image,"rb").read())

Not much different working with a URL, as urllib provides the urlopen method to present an online resource with a .read() function:

import base64, urllib

image = urllib.urlopen('http://docs.python.org/_static/py.png')
image_64 = base64.encodestring(image.read())

In this example, I’m using the location of the little Python symbol on the top left of most python.org pages.  Even this small graphic converts to a lot of base64 characters.  Here is the result:

iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A
/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9gEGxE4IQYzJ14AAAI3SURBVDjL
ZZNPSFVBFIe/e9+zd3silBCl0SZoU4s2rVq0EB5tQip4UNvATVGu3QRBiyAi2iltWkgbF5EgRhFF
RpiWtrWIzDIV1Pzz7p15M2fmtvDevOqBw8DM9zvnN8ycgF3R/eDtM2mac96ZdrFNxBikqbRV+vHH
/ut9gAZczoe7C3gnF0f6au1OLM5avFi8d1Ea+JvAMSAq8nsKOGs5f2cYJ3Y7rc2PO4BqkS8DdD98
f9tbe1ysCoxOBo1qlEXHJWcM4b5KPU19zleA0o4Clx99eO3EdqVewHsCoFRugUoVghJO7A6H6Vx9
wdtYi27cr5x6dy/03nVtWTU7bWeZh6jNUcAiCaFTURl9A+gs56AviHzh3mnqtdPxm6knfQPLU7Ua
okASQq/agY7yDrG16Mba6Pz48NP56VdrgAApYObGaicPtkovToFLQBKA/WUxTe3FRk4san15aGKg
d3Dj560rrdGJS6FT0X9YYvLuiMKL1kAQOpHZ3PqfyZfP41+9PW1VfzX0RXFSECfgNEmSTgImdDru
F2O0E8vvqZG1auQubAsKooIYYHpGvwA2g+xndQBHgWa6cG0ih5cW/w6VvEq3nChwCoBvs+bL2Z7V
ceBHGTDAIrABpMVuhw+4OiLgLIglOLPYBTQAlfErIeCzjRVg1dtEb1kt5Omv+DTV2YssAN+zNdkz
C42N9brV8WdvYp07seOdM2Of1F3AAknW0AJpwN6IgEPAEaANaMlcbmZdl7KRBuAfAb+v//yMAJoA
AAAASUVORK5CYII=

My application doesn’t put any limitations on the size of the image being linked, so I may have to add logic to resize it to something sensible for the receiving program.  I surely don’t need a 5MB image in my XML stream!  Watch this space for updates. Finally, I need to put this converted data into the XML output.  Template tags to the rescue! In tempplatetags/filters.py:

@register.filter
def b64(url):
    import base64, urllib
    if url:
        try:
            i = urllib.urlopen(url)
            return base64.encodestring(i.read())
        except:
            return None
    return None

This tag will try to open the url and convert the contents.  Not much error checking here.  If it doesn’t work, the value of None is returned. An this is an example template file, called ucal.xml:

{% load filters %}<!--?xml version="1.0" encoding="utf-8"?-->

...

{% if e.photo_link %}{{ e.photo_link|b64 }}{% endif %}

(Note that the {% load filters %} directive is on the same line as the xml version declaration.) The b64 filter takes the supplied URL and returns the base64 encoded version.

UPDATE:

Learned that the calendar can also just take a URL to the photo as input. Oh well, at least I got to learn something new.

I’ve been working on a project (hereafter called ‘The Project’) for almost two years, although not much lately.  When I switched the platform from a customer hosted solution to a SAS application, the complexity increased greatly, and the estimated time to finish (if I were actually working on it) has extended past any acceptable time frame.  Plus, I’m finding more and more companies that have already put up applications to fill this need – plenty of competition.

I have interests in other Django tasks, including a possible reusable application, conversion of previous code to a newer Django version, and a personal project.  However, my guilt on not focusing on The Project has kept me from these tasks.

Therefore, I am officially announcing that The Project is being put aside – maybe temporarily, maybe forever.  I’ll certainly keep the code around – lots of useful snippets there, but no more active development.

The Project is dead, long live the new projects!

I’m playing around today with Django and Oracle, connecting my application to data from another application.

First thing I realized is that I cannot use a table or view that does not have a single column primary key.  Without a primary key, the Django Admin system complains loudly.  For those that do have a good key, it’s pretty easy to declare it to Django:

id_number = models.CharField(max_length=10,primary_key=True)

This will override Django’s default behavior of using ‘ID’ as the primary key field name.

Next, there’s a specific way that one declares the schema and table/view name for the existing structure.  Each has to be in quotes, with the entire name also in quotes.  Something like this:

    class Meta:
        db_table = u'"SCHEMA"."TABLE"'

Something else that I needed to do was to not allow changes to the external data.  This is enforced by Oracle security, but I also wanted to make it clear in my code that saves and deletes are not going to happen.  To do this, I create an abstract class upon which my other models are based, which includes the redefined save() and delete() methods that raise a NotImplemeted error.  These produce a nice, big error page when DEBUG=TRUE.

class AdvanceBase(models.Model):
    """
    Abstract base class for accessing tables and views in the external schema
    """

    class Meta:
        abstract = True

    def save(self): # No saving allowed
        raise NotImplementedError

    def delete(self): # No deletes allowed
        raise NotImplementedError

I decided to try setting up a relationship between two of the external tables.  This wasn’t really different than any other foreign key, with one exception.  I wanted to rename the field to follow Django conventions.  Orginally called SCHOOL_CODE, I changed it to SCHOOL – same as the foreign key object.  In order to still map to the correct DB column, the db_column attribute is used to declare the original name:

school = models.ForeignKey(School,blank=True,null=True,db_column='SCHOOL_CODE')

One more experiment – using a manager.  In this case I wanted to filter a table to include only living people.   The fact that I was using an external table did not change the method for this:

class PersonManager(models.Manager):
    def get_query_set(self):
        return super(PersonManager, self).get_query_set().filter(status='A')

Something else I learned was that the Django Admin will use the first manager listed for a model.  By placing the reference to the PersonManager ahead of the normal objects, the Admin only showed living people.

I’m sure there will be more later as I dig deeper into using external data.  I have an actual application related goal driving my interest, so I should get this solved soon.

As always, leave your comments below.

I’ve been fighting a problem in my Django project for a while, and the solution I came up with seemed like I was settling for practicality over elegance.  This post is an attempt to rationalize my choices.

First, I love the idea of reusable applications.  I have incorporated a few from open source, including django photologue for the gallery, and the ease of use was amazing.  While I know that my work may not be general enough for public use, I do hope to be able to reuse some code in different parts of my project.  And who knows, maybe something useful to the world will come from it.

So, I divided my project into what seemed to be mostly independent applications, and started coding.

My problem came when I started heavy use of the @property decorator in my models.  Being able to reference a function using the dot notation is very handy (even though they aren’t available in functions that reference fields internally – like the ORM).  However, I found myself referencing models and fields in these two certain applications often, and I ended up with a circular import error.

I’d like to take a moment to point out that my Internet search on circular imports did not yield much help.  Several sites merely pointed out that a circular import was the result of poor coding, but offered no advice.  Finally, one post suggested moving the import to the end of models.py.  This solved the problem in my laptop development environment, but the same code would not run on my Dreamhost server (maybe it is importing in a different order).

Next, I tried to rewrite my property functions to reference the models from the other application only through the ORM – to avoid the need for an import.  This helped on the less complicated functions, but I couldn’t do it for the really useful, complicated ones.

My solution was to combine these two applications together.  The two are obviously very connected, and it was easy to make the case.  However, one of them could have been used in a more general setting, and now it is too specific to be presented to the public.  I guess I’ll have to earn my open source cred with something else.

The important thing here is that I am past this and am once again able to work on features instead of back end problems.

Am I being too theoretical? Are properties not the way to go? The comment section is waiting for your thoughts.