Django on Dreamhost

Dreamhost LogoThere are many posts that outline the steps to deploy a Django app onto a Dreamhost shared server. This is my experience.

I used a couple of pages as references, both to help me know what to do and to learn what not to do. Installing django 1.3 on DreamHost on oscarcp.com helped with the virtualenv setup, and the Passenger and WSGI section on the Dreamhost Wiki page on Passenger showed how to use the correct python interpreter.

Python

I’ve previously setup the Python 2.6.5 interpreter to be my default on Dreamhost (see this post), so I’m not sure what version they offer currently as the default. Be sure that the version you are running is supported by your desired Django version.

Setup Passenger for you Domain

It’s an option on the Dreamhost panel. When editing your domain, check the box labeled “Passenger (Ruby/Python apps only)“. This will add a /public directory in the domain root. (It may take a few minutes for this work to finish.)

Install Virtualenv

There is a version of virtualenv available on Dreamhost which will work fine. However, I wanted the latest, so I downloaded and installed it. First, I opened a terminal window and ssh’ed to my Dreamhost account. I have a downloads folder setup, so I switched to that before running the download. (As of this writing, 1.6.4 is the latest released version.)

wget http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.6.4.tar.gz
tar -xvf virtualenv-1.6.4.tar.gz
python setup.py install

Create the Virtual Environment for the Project

My naming convention for virtual environments is to use the project name followed by ‘env’. Pretty creative, huh? Since my project is an art gallery, I called it ‘galleryenv’.

Before running the command, switch to the root directory for the domain. This is the folder containing the /public directory created by the Passenger selection earlier. The Dreamhost default is to name this the same as the domain, but yours may be different.

virtualenv envname --no-site-packages

The –no-site-packages argument is pretty important, as it instructs virtualenv to ignore all python packages installed in the base interpreter. Everything you need for the project should be installed in this new environment. This means that your project will be insulated from changes to the base install, and will allow different projects/environments to use different versions of the same packages.

Next, activate the interpreter in the virtual environment you just created. Basically, you’re switching to this new copy of Python rather than the default for your account.

source envname/bin/activate

Your command prompt should now be prefixed with the name of the environment.

Run Django Setup Script? I Say No

Dreamhost provides a script, django-setup.py, that will create your settings file with database settings and the passenger_wsgi.py file for your application. This works pretty well for a new project, but most of the work has to be backed out when an existing application using a different directory structure or settings is being setup. Since I develop my projects on a local development machine (as should you), this isn’t much help. Instead, there are just a few manual steps to perform that will get things ready to go.

Also, you’ll notice that we haven’t install Django yet. Since we used the –no-site-packages option when creating the virtual environment, any Django version installed on your account earlier will not be accessible while you are using the new interpreter. The Dreamhost script won’t run unless Django and the MySQL library for Python are installed. No worries! We’ll get to that soon enough.

Bring On the Source code

At this point, it is time to load the source for your Django project onto the server. My convention is to create a directory with the project name in the root directory for the domain – so mine would be called ‘gallery’. You can put these wherever you like, just remember the path for a later step.

I use the Subversion version control system, also provided by Dreamhost, to store my code, so I would use the ‘svn copy’ command to install my source code. Any VCS or DVCS is fine, or the age old FTP approach will work as well.

PIP and Requirements

I’ve just started using PIP recently, although it has been around the Python world for a while. PIP is a general Python install tool, and we’re going to use it to quickly install the libraries we need for our application. Since it is installed with virtualenv (at least in 1.6.4), it is already setup and ready to go.

Before this step, ensure that the virtualenv interpreter is still activated – see the source command above. The goal is to install the needed Python libraries only for this environment and not for use by the default Python install.

PIP offers the option to include a list of libraries to be installed in a batch run. Following convention, I’ve included a file with my application called requirements.txt. Here’s what mine looks like:

Django<1.4
PIL
django-photologue

First, I want to install the latest release of Django 1.3, so I ask for the most recent less than v1.4. Next, I need to use the Python Image Library, or PIL. (This library has operating system dependencies that will need to be satisfied before the install.) Finally, the package django-photologue is requested. Prepare your own requirements in this format.

To run PIP with your requirements, use the following syntax:

pip install -r requirements.txt

It's just that easy! Watch the output carefully for any errors or warnings.

Since I'm using MySQL for my database on the server, I also need the mysql-python library.

pip install mysql-python

Passenger Configuration

The django-setup.py script I mentioned earlier builds a file called passenger_wsgi.py that links Apache to the django application. For your install, you can either run the script, or use my example as a starting point. Either way, some edits will be needed to use the virtual environment. Here's is my file:

import sys, os
INTERP = os.path.join(os.environ['HOME'], 'gallery.gentryart.us', 'galleryenv', 'bin', 'python')
if sys.executable != INTERP:
    os.execl(INTERP, INTERP, *sys.argv)
sys.path.append(os.getcwd())
sys.path.append(os.path.join(os.getcwd(),'gallery'))
os.environ['DJANGO_SETTINGS_MODULE'] = "gallery.settings"
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

The second line of the script sets a variable (INTERP) to the path of the virtual environment's python interpreter. Basically, the os.path.join command builds the path using the directory names we created earlier. Starting at the home directory, we add in the domain name and the environment name, followed by 'bin/python'. Replace my gallery references with your application's directories.

Lines 3 and 4 will switch to the new python interpreter referenced by the variable.

The sys.path.append lines include our application directory in the search path used by Python, and the next line points to our settings.py file. In these lines, replace 'gallery' with the path to your application source code.

passenger_wsgi.py should be placed in the root directory for the domain. Feel free to copy my example and edit to taste.

Static Media

Better bloggers than I have written about serving static media, and this is just one of many ways to do it. Since I store these files (CSS, JS, images) in version control, they are installed in the application directory created earlier. A simple symbolic link will point Apache to the correct files.

First, switch the the /public directory located in the root directory for the domain. Then, setup the symbolic link to refer to the files. Since my files are located in a directory called 'static', this command takes care of it.

ln -s ../gallery/static static

Next, setup a link for the Django admin media.

ln -s ../galleryenv/lib/python2.6/site-packages/django/contrib/admin/media admin_media

One Last Thing

Since I had already been running my app on this domain using a pre-passenger method for hosting Django, I had a leftover (and hidden) .htaccess file that was messing things up. A quick note to the Happy Dreamhost Support Team and they found and fixed the problem. No .htaccess settings are needed for this type of configuration.

Ready To Go

Hopefully, everything is working for you at this point. In my case, Django came up and told me my database settings were wrong, but I see that as progress. The site is now working fine.

Good luck!

http://gallery.gentryart.us

Database Not Available

The Situation:

During certain processes, such as fiscal year end and software upgrades, my Oracle database is placed in restricted session. As expected, an attempt to use the Django application results in the error “ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege”. This error is thrown from deep within the ORM code.

The Question:

How can I grab this error within my application to be able to send a meaningful error message back to the user instead of the 500 page? I’d also like to stop the email sent to the administrator (me) each time this error is triggered.

TDD Works Better Up Front

For the Django app I deployed last month, I had started the process using a Test Driven Development (TDD) approach. In a nutshell, it means that tests are written before a feature is coded. This went well at the beginning, but as crunch time neared, the tests were thrown by the wayside and code was tossed in as needed.

Now that I’m in this Django upgrade project, I decided to go back and complete the tests before starting the upgrade work. It is now three weeks later, and I still have much to do (working on my own time – not during work hours). However, I’m really enjoying the satisfaction that passing a well written test can bring.

My hope is that my next project will be more TDD compliant, as I am learning the skills now that I need to write quality tests quickly.

Next Application Please

For my Django upgrade project, I see several steps for each of the four applications:

  • Review the code and update on my laptop
  • Pass to the designer for any changes he needs to make (probably not many related to the Django version change)
  • Move the code to the DEV server to test Apache and Oracle compatibility
  • Deploy to QA and test the heck out of it.
  • Deploy to production and hope nobody notices

The first time I work on the DEV server, there will also be virtualenv things to work out.

Application #1 has gone through the first step. so I’m now starting work on app #2. It’s a little bigger, so this may take a little longer. Keep in mind that this first step is all being done in my spare time.

Work has been pretty crazy, so I haven’t had time to talk with the server admins about virtualenv or Oracle 11g. Hope to get to it soon.

And As If Version Upgrades Weren’t Enough…

In addition to the Django version upgrades I’m in the middle of, I also am facing a scheduled upgrade of the Oracle version used by my apps. They share an Oracle instance with the enterprise application that runs the business, so these smaller systems upgrade on the big guy’s schedule.

Currently, all run on Oracle 10g, and the DB platform upgrades to 11g in July. Shouldn’t be a big deal, but I don’t have enough experience with the Oracle client running on the web servers, or with the Python interface to Oracle.

This project starts next week.

Django Framework 1.3 Fixes

Looking through the release notes for v1.3, I don’t see any backwards incompatible issues that affect me. As for new features, there is an index that is now used on the session table that I can add, and the much heralded class-based views are now available.

Looking through the deprecation list, I did find a couple of things to check. First, an upcoming change to the url template tag warrants a look through my code, and the eventual removal of the function-based generic views means that I can either fix them now or during the next upgrade.

Hopefully, this will be quick, not only on my first, uncomplicated application, but the more complicated ones I’ve written more recently.

Django Framework 1.2 Changes

I have the first application upgraded from v1.0 to v1.1 of Django (spending most of my time with the environment, not with code), so I’m ready to move on to v1.2. My quick review of the release notes shows just a couple of things I have to change, and may I’d like to change. (Stay in scope, Dan!)

CSRF protection – requires a tag to be added to every form, plus changes to the middleware settings.
Multiple Database Declaration – allows the use of more than one database. Not a required change now, but since the old code will be deprecated soon enough, I thought I should get the change made while it is fresh in my mind.

Python Imaging Library and JPEG Files

There are so many posts on the web covering how to get the Python Imaging Library (PIL) to work on Linux, and they almost all say the same thing: install libjpeg6.2 and then recompile/install PIL.

What most fail to mention is that one apparently needs to also remove the previous PIL install before reinstalling. I don’t know (and don’t care to research it), but I think that the install sees the already compiled code and skips that step. Wonder if there is an option that forces new compiles.

So, find wherever your distribution stores the Python site-packages/dist-packages and remove the PIL directory and PIL.pth file before running install. Just to be thorough (and maybe a little anal retentive), I also removed the unpacked directory of PIL source.

Also, and this is the BIG PIECE OF NEWS to me, for those using Ubuntu 11.04 like me, libjpeg6.2 won’t work. Instead, use libjpeg8. In the Synaptic Package Manager, a quick search for libjpeg will show both versions. Leave libjpeg6.2 alone, but be sure to remove any dev libraries for that version. Then, mark anything that looks remotely connected to libjpeg8 and apply.

Now that your old version of PIL is gone, and the libs are squared away, it is time to install PIL. I used version 1.1.7. Finally, success!

(Well, except that I seem to have not installed the library that enables PNG use. Some other day maybe.)

UPDATE:
To enable PNG support in Ubuntu 11.04, use these commands to find the zlib library BEFORE installing PIL:

$ cd /usr/lib
$ sudo ln -s i386-linux-gnu/libz.so libz.so

ANOTHER UPDATE:
Using version 12.04 of Ubuntu, I also had to create a symbolic link for libjpeg – AFTER following the instructions above and BEFORE installing PIL:

$sudo ln -s i386-linux-gnu/libjpeg.so libjpeg.so

 

Django Framework 1.1 Updates

My quick review of the 1.1 release notes only found a few things to update to move my v1.0 application up a version.

  • Unmanaged Models – will be very helpful with my latest app that uses legacy tables
  • Admin URLs – uses include syntax
  • Reverse Admin URLs – I’ve already coded this in two apps, but had to comment it

Also, I’m looking at the {% empty %} clause of a {% for %} loop in templates. It will be a lot cleaner than the current method of checking for data in the data source, although I did promise myself not to do any refactoring. We’ll see how that goes.

Upgrading Django – 1.0 to 1.3

As the title of this post reveals, I’m a little behind in Django versions on my app server. There are several reasons why:

No virtualenv
My server admins, who are very competent, don’t have experience with virtualenv (that I know of). Without this tool on the server, it is impossible to upgrade one app at a time, or even introduce a new application using a newer version. The app I just deployed on Friday is using 1.0.4.
Other App Work
I’ve been either writing new apps or adding functionality to existing apps pretty much constantly for the past 18 months. No time to freeze features and upgrade.
Other Work
My main job function is as a manager, so I can’t sequester myself inside Eclipse for a month to get things up to date

Besides being vulnerable to bugs that have long since been fixed, I am also unable to use any of the new features that have come out in the past three years. Aggregation functions, CSRF, class based views, and others tempt me with their usefulness, but I can’t take advantage of them.

So, the project to upgrade has started. First step was to review the release notes for the three versions that succeed 1.0 to refresh my self on the changes – especially the backward incompatible ones. I’m not looking to do a lot of refactoring unless the change is required for the app to run under v1.3. That will come later during work on the next release of each app. I have a pretty short list of things to check and update before I even try to run in the latest version of the framework.

I’m currently in step two, which is to begin to modify the first app to run under v1.3. Since I already have virtual environments setup for each app, its no problem to switch things around as I work. Also, I’m adding a new branch in SVN for the 1.3 work in case any maintenance needs come along.

Third step, or more likely concurrently with step two, will be to work with the sysadmins to setup virtualenv on the DEV server. We will isolate each application (there are four) in its own environment and then I can update and test the code for each at my own pace as changes are ready. I think that once they see the benefits of separating the dependencies of each program, they will accept it wholeheartedly.

Lastly, we will setup the virtual environments on the QA and PROD servers and move things forward. My users (and customers) will hopefully not notice as this happens.

Wish me luck!