Simple Search in Django

I started programming (in BASIC, Pascal, and COBOL) before the Web was available, and it’s amazing to see how much the craft has changed with instant access to thousands of references. However, sometimes I throw up a search and get back the perfect solution, and I’m frustrated that I couldn’t figure out something so simple. It’s just too easy to search instead of think.

My example today is an easy search in Django. I have read about Haystack, Solr, and other search engines that require a daemon, special database structures, and other special things, but these are overkill for a search on a relatively small dataset.

My search returned this post from Toast Driven that outlined a way to append together a few Q objects to return a search result that uses multiple terms and multiple fields. Genius!

Thanks to Daniel at Toast Driven.

Apple Sucks!

(a little side rant that I had to get out)

My iPhone has always on Internet access. Also, I am required to setup an account with Apple to even run iTunes in order to copy files to my iPhone.

So, with the potential of storing my settings and information in three places (phone, laptop, cloud), why is the only copy of these settings that I can use on the laptop?

I can’t figure out why I need a laptop at all. Why can’t the phone connect to the iTunes service directly to ‘sync’. Nope, podcast subscriptions and music (unless I purchase a song) have to be copied to the laptop first, and then copied through a little wire to the phone. Very 20th century.

Yesterday, I was trying to switch to a new computer for my syncing and downloading, and it was not easy. Just plugging in the phone didn’t work, as it wanted to overwrite the phone contents with the data in the new iTunes installation – AN EMPTY ONE THAT I HAD JUST INSTALLED! Plus, it wanted to erase my existing music, podcasts, and videos from the phone! Why can’t I copy the phone info into the new iTunes? Or a least leave it alone? Why aren’t my settings, information, subscriptions, etc., saved to and accessible from the cloud?

A quick web search found many procedures to try when transferring to a new computer, but none were easy, and I doubt if most users would be able to do it. I’ll figure it out tonight, but from a company that values design and ease of use, this is an abomination.

The Big Picture

Part 1 of 2 posts discussing my plans for an application. In this installment, I talk about phases of development and the approach to take.

My project is actually a collection of several separate, but related applications, and all share some common data elements. The conventional wisdom would be to develop the central functions first, and then bolt-on the peripheral applications after. However, real life often calls on us to do things the hard way. One of the add-on programs is needed in three months, while the basic system need is not urgent.

The central function is currently provided by an older, difficult to support application. It provides enough functionality to get the job done, but isn’t complete. Extending this system to provide the new features is unwarranted for a couple of reasons. First, the development environment (language, database, etc.) is outdated, and I have no one on staff that can work quickly in that language. Second, the platform on which it is deployed is a little shaky and certainly doesn’t meet modern expectations for scalability and security. In other words, we’re going to have to replace it more likely sooner than later, and we will have a tough time supporting it until then.

So with that background, here is my dilemma. As I work on the peripheral application, how much should I worry about later integration with the central system? I certainly don’t have the time to develop all of the central functionality needed, which would basically require that I design and write the central application. Creating what would be a stand-alone application for this peripheral would be quicker, but the integration work later would be extensive.

I’m reminded of a bridge that was built near my house. The big plan calls for a major highway to be built connecting this bridge to a new route around the city. But the construction (or even the funding plans) for this new highway hadn’t started, and there was no clear idea when or if it would. So, an approach that connected the bridge to an existing road was built, even though it would have to be torn out when the new highway came through. Now, nine years a later, the new highway is still a dream, while the bridge gets much traffic. It would have been a shame to leave it either unconnected or poorly connected to the highway grid for this long.

My plan calls for a (mostly) independent development of this add-on application. I will take the bigger picture into account when laying out the data structures, but I am not going to worry about writing all of the functionality in a generic way that will work in all cases. Focus will be on the current goal.

When it comes time to connect to the central application, some rewrite will be necessary, but my users will have a solid system to use in the mean time.

What do you think? Am I on the right track, or am I heading for big troubles down the road? Please leave your comments below.

Didn’t Expect That

Had an invitee drop us a note the other day to tell us that she was frustrated that she wasn’t able to indicate that she would not be attending. A second person invited to the same event accepted the invitation, only to mention in the notes that she couldn’t make it.

I hadn’t thought about including a decline option on a registration site, but it’s been added to the feature list for the future.

Changing Hosts in Dreamhost

Dreamhost logoLast night, Dreamhost moved my account to a new host. Not a big deal, and it’s certainly understandable, but it did break my one running Django app – the Gentryart Gallery. Here’s what I did to fix it.

First, as suggested in the Dreamhost Server Moves Page, I recompiled Python to create a 64bit version. While I was at it, I upgraded from v2.5.2 to v2.6.5. This great page by Ryan Kanno outlined the steps. I just changed the version number. I also deleted my old installation of Python so there wouldn’t be any chance of confusion there.

Now, if I would have used the same python version and kept everything in the same locations, I would have been done. Since I didn’t, I had a little more to do.

Next, I needed to install all of the python packages I use. Here’s the list:

Following the famous Jeff Croft post, I updated the admin_media shortcut to point to the new Django files.

And the gallery is back in business!

Reporting in Django

The Django application I’m developing has a reporting need: paper (PDF) output of reports, both 8 1/2 X 11 and special forms (labels, name tags, etc.). So, I’ve begun evaluating my reporting options.

Unfortunately, I haven’t found very many methods to examine, probably because of my requirements:

Python based
Since I’m using Python/Django for the rest of the work, it would make sense to use Python for this component as well. Other folks that may someday support the application will appreciate having a single skill set to maintain.
Contained in the Application
I’ve found at least one system that runs as a separate service on a server, but that is more complexity than I would like to introduce into a client’s technology stack. This application will appeal to small offices that likely don’t have strong tech support. Adding a separate service (in this example, Java/Tomcat) on top of Django, MySQL and Apache is more than many can do, and I certainly don’t want to get into the server admin business on their behalf.
Open Source
Another issue that the clients won’t want to deal with is licensing to third parties. I’m using all open source tools to build and run the application, and it wouldn’t make sense to require a potential customer to license something before they could use it. Plus, would I have to become a reseller for the licensed software?
Integrated with Django
This is total ‘wish list’ stuff, but wouldn’t it be neat to have an interface similar to the Django templating system where I could bring in data from a view into a report ‘template’?

Looking through Google searches, I’ve found that the terms “reports” and “reporting” can mean different things to different people. Since I’m looking to generate paper, I have been looking at Geraldo, which uses the ReportLab library to generate PDF output. This application looks promising, but is still a work in progress. I’m finding that flexible formatting with variable length text is a difficult, and maybe impossible, thing to do.

I’d like to hear from others on this subject. Have you found a good package for reporting? Are you developing one? Please leave your thoughts in the comments.

Accessing the Value of a Form Field in a Template

(This is a ‘Note to Self’ post)

I’ve been trying for half the day to get radio buttons in a custom template to work. The buttons are generated in a for loop that builds a table row for each choice (iterating over a queryset called cls), including this code for the row’s radio button:

input type="radio" name="reg_choice_1" value="{{ cls.id }}"
 {% ifequal srform.reg_choice_1 cls.id %}CHECKED{% endifequal %}

The goal is to have the ifequal test tell me if the row’s cls.id matches the value set in the form. However, I didn’t know how to access that value. After many Google searches, perusal of the Django documentation, and even a failed attempt to follow the template rendering code, I was stumped. As a wild guess, I tried the .data attribute, and it worked! Here’s the updated (and functional) code:

input type="radio" name="reg_choice_1" value="{{ cls.id }}" 
{% ifequal srform.reg_choice_1.data cls.id %}CHECKED{% endifequal %}

I hope that I’ve put enough keywords in this post so that future stumped Djangonauts can find some help.

As always, leave your comments below.

Full URL in Django

The application I’m developing includes an XML feed, in which one of the items is a URL Unfortunately, using the {% url %} tag in the template only gives the path portion of the URL. Time for a little research.

Many web posts pointed me to the Sites framework included with Django, as it can return the domain for the site. I tried it out, and it worked, but I didn’t like how it would have to be implemented in my multiple server scenario (laptop, DEV server, QA server, Production). Each would have to have any entry in the sites table, and the SITE_ID in the settings.py file for each instance would have to point to the correct entry. This would be awkward at best, and would limit the possibility of making the application reusable.

A couple of bloggers mentioned using RequestSite instead. It shares the same API with the Sites class, but doesn’t use the sites table. Instead, the domain name is pulled from the request object. Here’s a sample of the view logic:

   from django.contrib.sites.models import RequestSite    
    site_name = RequestSite(request).domain
    
    return render_to_response('template.xml', 
                              {'entries': queryset, 'site_name' : site_name}, 
                              context_instance=RequestContext(request), 
                              mimetype='text/xml')

After pulling the domain from the request, it is passed along to the template in the context dictionary. In the template, I can reference the variable ‘site_name‘ in front of the {% url %} tag. However, before I have a full URL, I also need to know the protocol.

Another blog post demonstrated the is_secure attribute of the request object. Testing this attribute can help us determine is the protocol is http or https. The temple code for a full URL is:

{% if request.is_secure %}https{% else %}http{% endif %}://{{ site_name }}
{% url entry e.pk %}

Maybe a little clumsy, but it gets the job done without hard coding anything.

I’d love to hear your thoughts in the comments.

Is Short Sweet?

I was reading this post by Jacob Kaplan-Moss on Dynamic Form Generation, and this little side point caught my interest. He pointed out a more compact way to write a view for a create operation.

OLD WAY:

from django.shortcuts import redirect, render_to_response
from myapp.forms import UserCreationForm

def create_user(request):
    if request.method == 'POST':
        form = UserCreationForm(request.POST)
        if form.is_valid():
            do_something_with(form.cleaned_data)
            return redirect("create_user_success")
    else:
        form = UserCreationForm()

    return render_to_response("signup/form.html", {'form': form})

NEW WAY:

def create_user(request):
    form = UserCreationForm(request.POST or None)
    if form.is_valid():
        do_something_with(form.cleaned_data)
        return redirect("create_user_success")

    return render_to_response("signup/form.html", {'form': form})

Now, I’m all for keeping things concise, but I’m not sure if this new way is as clear. I want to keep my code readable for those that are tasked with maintaining it someday can understand it.

What do you think? Is the 2nd method OK when others need to read the code?

Things I Learned About Django Authentication and django_cas Today

After programming a new site on my laptop (standalone web server, Django standard authentication), I moved it to the DEV server (Apache, CAS for authentication). Some things didn’t work the same.

I had designed three security settings: Admin (can get to everything – superuser in Admin), Staff (can run reports, just a few permissions in Admin – marked as staff), and User (entry in Admin, but not staff and no permission, runs reports only).

The views that displayed the reports use the @login_required decorator.

Here’s the problem, and I saw this coming: any user who could sucessfully log in to CAS could run reports. Since I’m using a campus wide CAS database, this included any campus staff member or student. Not what I was going for.

I had forgotten that django_cas adds every sucessfully loged in user as a User record in the Django Admin. At that point, the user passes the @login_required test and can run those views.

My solution was two-fold. Mark the User level records as staff, and replace the @login_required decorator with @staff_member_required. Unfortunately, the system seemed to be using the Django internal login mechanism rather than CAS. A little code inspection showed that the decorator wasn’t using a URL redirect to login, and instead was calling the Admin login function directly. This bypasses both the django_cas middleware and the LOGIN_URL setting.

Next, I tried using the @user_passes_test decorator testing for user.is_staff in place of @staff_member_required (which I will likely never use again). This also worked in the standalone setting, but not with django_cas (kept throwing 500 errors). I then switched to the versions of the @user_passes_test and @login_required decorators provided with django_cas, but then I kept getting 403 errors.

TIME FOR A LUNCH BREAK!!

After a sandwich and a quick walk, my refreshed mind reminded me to check to see if django_cas had been updated from v2.0.2 that I had installed. Sure enough! Version 2.0.3 specifically updated the decorators. The @user_passes_test code had been updated to return a 403 only if a logged in user failed the test, and an unauthenticated user would get a proper login screen. Furthermore, @login_required had been removed completely and replaced with an import of the standard django supplied version. Even though I am currently unable to update the version on the DEV (or higher) servers, I saw what I needed to do.

First, I changed my import statements to grab the correct decorators for the environment.

import settings

if settings.USE_CAS:
    from django_cas.decorators import user_passes_test
else:
    from django.contrib.auth.decorators import user_passes_test
    
from django.contrib.auth.decorators import login_required

(Note that USE_CAS is a setting I’ve defined in the server specific settings file.)

This code pulls in the proper @user_passes_test decorator, and the standard @login_required decorator,

Next, for each veiw that needs protection, I implemented the decorators thusly:

@login_required   
@user_passes_test(lambda u: u.is_staff)
def view_staff(request):
    ...

Putting @login_required first forces a login if the user hasn’t, and then @user_passes_test checks for the staff attribute.

With the changes in django_cas v2.0.3, the call to @login_required will no longer be necessary. I hope to setup a project to update the version soon.