Unit Testing Forms in Django

This is a subject that doesn’t seem to be covered well by other sources, so I am recording my notes on testing a Django form. Credit to “fRui Apps” for this post that gives an example. (See the section Testing Forms)

My goal is to get the test as close to the code as possible. Rather than using the view test, or even Selenium, this approach will isolate the form’s functions without going through all of Django’s URL routing, middleware, etc. Save that for the functional tests.

In this first example, my form will pad the entered number with left hand zeroes. This is the quick test:

def test_id_format(self):  
    ## Use factoryboy to create a model instance  
    e = EndowmentFactory.create()
    ## Instantiate form  
    form = EndowmentEntityForm(data={'endowment': e.pk,  
                                     'id_number': '528989',  
                                     'entity_type': EntityType.ORIGINATOR,})
    ## call is_valid() to create cleaned_data  
    self.assertTrue(form.is_valid())  
    ## Test away!  
    self.assertEqual(form.clean_id_number(),'0000528989')

Note that I’m using factoryboy to create a model instance to work with. The form data includes an id_number of fewer than 10 characters. After I call is_valid() to populate cleaned_data, I can test the cleaned id_number for the proper format.

In this next example, I am testing for valid and invalid data. My application allows changing the status of an object, but certain rules apply. Here is a test confirming that a valid change is allowed to continue:

##  Test for valid action types
def test_action_type_valid(self):
    ## Use factoryboy to create a model instance
    e = EndowmentFactory.create(status='E')        
    ## Instantiate form
    form = UserActionForm(data={'endowment': e.pk,
                                'username': 'dgentry',
                                'action_type': 'A',})
    ## is_valid() should return True if no errors are found
    self.assertTrue(form.is_valid())

And this will test for an invalid status change:

##  Test for valid action types
def test_action_type_valid(self):
    ## Use factoryboy to create a model instance
    e = EndowmentFactory.create(status='A')        
    ## Instantiate form
    form = UserActionForm(data={'endowment': e.pk,
                                'username': 'dgentry',
                                'action_type': 'E',})
    ## We run is_valid() and expect False
    self.failIf(form.is_valid())
    ## check for the error
    self.assertIn(u'Invalid Action',form.errors['__all__'])

I’m sure that I will learn more about testing forms as I go. Stay tuned…

Lazy Reverse

Working in Django 1.3, I had some trouble with my view code.

Actually, the error was pretty vague, something about my urls.py containing no patterns. After much wailing and gnashing of teeth (during which I learned I really don’t know how to best use the debugger in Eclipse), I traced the problem down to three of my view classes. All three included a setting for the success_url attribute:

success_url = reverse('support_list')

Apparently, you can’t do that.

Once I found the offending line of code, something in my mind clicked on to remind me that I had hit this problem before. To solve it, I had moved the reverse statement into the get_success_url() function:

def get_success_url()
    return reverse('support_list')

Looking for a better solution, a quick Google Search brought up this post in in the django-users group.

I have a CreateView which I’d like to redirect to a custom success_url

defined in my URLconf. As I want to stick to the DRY-principle I just

did the following:

  success_url = reverse("my-named-url") 

Unfortunately, this breaks my site by raising an

“ImproperlyConfigured: The included urlconf doesn’t have any patterns

in it”. Removing success_url and setting the model’s

get_absolute_url() to the following works fine:

  def get_absolute_url(self): 
      return reverse("my-named-url") 

I could reproduce this with a brand new project/application so I don’t

think this has something to do with my setup.

Can anyone confirm this issue?

So others have hit the same problem. Looking further down the page, a more concise solution was offered:

 success_url = lazy(reverse, str)("support_list") 

I used this for my current project – a little cleaner.

In v1.4, we can use the ‘reverse_lazy()’ function in place of ‘reverse()’.

UPDATE:

Suddenly, I understand the reason for the SuccessURLRedirectListMixin in the popular Django Braces package. Using this mixin, one can declare the success url with a simple variable assignment (no lazy code required), and the mixin provides the get_success_url() function needed to make everything work.

BTW, I highly recommend using Braces with class based views. I have found several to be quite useful to speed up coding and to keep things clean and clear.

Quick Post – Django Factory Boy

I’ve been using factoryboy in my tests as of late. It’s a nice tool to create test data in a programmatic way – no fixtures.

A factory includes a Sequence class that can be used to create unique field values with each instance. The examples I’ve seen use something like this:

title = factory.Sequence(lambda n: 'Event ' + n)

Which will create successive records with titles like “Event 0”, “Event 1″, Event 2”, etc.

The variable n is a string type by default, but I wanted to use it as an integer – in this case to set a value to an Integer field:

capacity = factory.Sequence(lambda n: n * 10)

However, this instead put 10 ones in the field, instead of the number held in n times 10.

A quick perusing of the documentation here showed that the Sequence class takes a second parameter after the function that indicates the desired type for n. Adding int to the end of the call makes it work much better.

capacity = factory.Sequence(lambda n: n * 10,int)

Now the field holds a multiple of 10.

Want a more complicated example? Here I assign a date field that is n nubmer of days after the current date.

event_date = factory.Sequence(lambda n:(datetime.now() + timedelta(days=n)).date(), int)

Project Cancelled

"Project cancelled" on rubber stampI’ve been working on a project at work for the past couple of months. This has been a “work on it when I can” project, so I have probably spent 4 or 5 days total on it.

However, I learned last week that it will likely be cancelled. Since everything I work on is a chance to try new things and update my current patterns, I’d like to keep this code around. But I’m worried that it will end up neglected in a long forgotten version control repository branch.

The question of the day is: What do you do with abandoned code that may have some good ideas in it?

RelatedFieldWidgetWrapper

The RelatedFieldWidgetWrapper (found in django.contrib.admin.widgets) is used in the Admin pages to include the capability on a Foreign Key control to add a new related record. (In English: puts the little green plus sign to the right of the control.) In a new application of the FilteredSelectMultiple widget I discussed recently, I needed to add this functionality.

This wrapper is a little complicated, but it didn’t take too much Google searching to find an example.  A Google Code file from something called Crimson Online helped me understand the concept.

The wrapper takes three required parameters plus one optional:

  1. The widget to be wrapped. In my case the FilteredSelectMultiple widget
  2. A relation that defines the two models involved
  3. A reference to the admin site
  4. The Boolean can_add_related (optional)

For the first parameter, I used the widget with its parameters:

FilteredSelectMultiple(('entities'),False,)

The relation confused me at first, but the example made it easy.  The model linked to the form, in this case Item, provides it.

Item._meta.get_field('entities').rel

The third parameter, the admin site object, is used along with the relation to determine if the user has permission to add this related model.  The example used django.contrib.admin.site, which I found doesn’t work.  Instead, I captured the admin_site value from the ModelAdmin class in my admin.py and set it as a variable for the form.  Then I referenced that variable in the \__init__ method of the form to include it in the widget.  This means that the widget has to be assigned within the __init__ method of the form.

self.admin_site

The fourth parameter can be used to override the permission check performed by the widget.  However, when a user without the permission to add clicks that little green plus sign, an ugly 500 error is returned.

Here is the code used:

in admin.py:

class ItemAdmin(admin.ModelAdmin):
    form = ItemAdminForm

    def __init__(self, model, admin_site):
        super(ItemAdmin,self).__init__(model,admin_site)
        self.form.admin_site = admin_site # capture the admin_site

admin.site.register(Item, ItemAdmin)

in forms.py

from django.contrib.admin.widgets import FilteredSelectMultiple, RelatedFieldWidgetWrapper

class ItemAdminForm(forms.ModelForm):
    entities = forms.ModelMultipleChoiceField(queryset=None,label=('Select Entities'),)

    def __init__(self, *args, **kwargs):
        super(ItemAdminForm,self).__init__(*args, **kwargs)
        # set the widget with wrapper
        self.fields['entities'].widget = RelatedFieldWidgetWrapper(
                                                FilteredSelectMultiple(('entities'),False,),
                                                Item._meta.get_field('entities').rel,
                                                self.admin_site)
        self.fields['entities'].queryset = Entity.objects.all()

    class Media:
        ## media for the FilteredSelectMultiple widget
        css = {
            'all':('/media/css/widgets.css',),
        }
        # jsi18n is required by the widget
        js = ('/admin/jsi18n/',)

    class Meta:
        model = Item

UPDATE:

I’ve published a follow up post that covers using this wrapper outside of the admin application. See More RelatedFieldWidgetWrapper – My Very Own Popup.

 

Using the FilterSelectMultiple Widget

I will soon be starting a new project at work, and one of the mockups calls for a side by side control to select multiple choices in a many-to-many relationships.  The default control used by Django is the SelectMultiple widget.

 

However, this can get unwieldy when the number of choices grows.  Instead, I wish to use a filter select, as seen in the Django admin when assigning permissions to a user or group.

 

It didn’t take too much research to find a Django snippet as an example.  http://djangosnippets.org/snippets/2466/.  I pretty much copied his example, except for the CSS file called ‘uid -manage-form.css’.  This must be part of his application, because I couldn’t find any other reference to it online.

Here is the form code I ended up with:

from django.contrib.admin.widgets import FilteredSelectMultiple

class BookForm(ModelForm):
    authors = forms.ModelMultipleChoiceField(queryset=Author.objects.all(),
                                          label=('Select Authors'),
                                          widget=FilteredSelectMultiple(
                                                    ('authors'),
                                                    False,
                                                 ))
    class Media:
        css = {
            'all':('/media/css/widgets.css',),
        }
        # jsi18n is required by the widget
        js = ('/admin/jsi18n/',)

    class Meta:
        model = Book

As the example shows, I included the tag for the form media in the <head> section of the template, which inserts the CSS and JS calls.

<head>
    ...
    {{ bookform.media }}
</head>

Once my designer gets a hold of this, I’m sure it will look great.

UPDATE: For an example of using the RelatedFieldWidgetWrapper to include the capability to add a new record, see this post.

Django Templates for the Designer

There’s no need to know Python or Django to work well with Django templates. A good understanding of the way HTML is generated will take the designer most of the way.

A common shortcut in rendering a form in the template is to use {{ form.as_p }}.  This will include all of the included fields from the form, along with labels and errors.   A nice side effect is that changes to the fields – whether adding a new field, taking one away, or changing its format – are reflected immediately when the app is restarted without any modifications necessary for the template itself.  Here’s an example:

<form action="." method="post">{% csrf_token %} 
{{ form.as_p }} 
<input type="submit" value="Save" /></form>

However, without an understanding of the underlying form, and possibly a related model, it can be difficult for the non-developer to predict which fields will be included. When using a basic Form, one just needs to check the form definition to see the list of fields.  This form is a good example:

class UserSelectForm(Form): 
    user = ChoiceField(required=True)
    reg_date = DateField(required=False)

Just the two fields listed will be included in the generated HTML.

With a ModelForm, things get a little more complicated. Before viewing an example, let’s start with a quick set of rules:

  1. All fields from the related model will be included
  2. UNLESS, the field is marked as editable=False in the model definition
  3. The fields attribute in the ModelForm‘s Meta section can be used to limit the form to a subset of the fields
  4. The exclude attribue, also in the Meta section, is used to omit fields from the form
  5. Additional fields specified in the form that are not part of the model will also be included

For more information, see Creating Forms from Models in the Django documentation.

Here’s an example:

class TrainingScheduleForm(ModelForm):
    class_date = DateFieldMultiFormat(required=True)
    class_time = TimeFieldMultiFormat(required=True)
    send_email = fields.BooleanField(required=False)

    class Meta:
        model = TrainingSchedule
        exclude = ('training_class',)

The model attribute tells us to look at the TrainingSchedule model and include those fields that are editable.

class TrainingSchedule(ModelBase):   
    training_class = models.ForeignKey(TrainingClass,null=False,blank=False)
    class_date = models.DateField(null=False,blank=False)
    class_time = models.TimeField(null=False,blank=False)
    location = models.CharField(max_length=255,null=True,blank=True)
    campus = models.CharField(max_length=255,null=True,blank=True)
    capacity = models.IntegerField(blank=True,null=True)
    display_flag = models.BooleanField(default=True)

Then, we look at the exclude attribute to see that the training_class field is not included. Class_date and class_time are explicitly listed only because of a need to override the default field type, but the send_email is an extra field that is not part of the model. It is included as well. So, the final list of fields in the template is:

  • class_date
  • class_time
  • location
  • campus
  • capacity
  • display_flag
  • send_email

Using the {{ form.as_p }} shortcut, the HTML will include this snippet:

<p><label for="id_class_date">Class date</label><input type="text" name="class_date" id="id_class_date" /> </p>
<p><label for="id_class_time">Class time</label><input type="text" name="class_time" id="id_class_time" /> </p>
<p><label for="id_location">Location</label><input id="id_location" type="text" name="location" maxlength="255" /> </p>
<p><label for="id_campus">Campus</label><input id="id_campus" type="text" name="campus" maxlength="255" /> </p>
<p><label for="id_capacity">Capacity</label><input type="text" name="capacity" id="id_capacity" /> </p>
<p><label for="id_display_flag">Display flag</label><input checked="checked" type="checkbox" name="display_flag" id="id_display_flag" /> </p>
<p><label for="id_send_email">Send email:</label> <input type="checkbox" name="send_email" id="id_send_email" /></p>

Note that both the labels and fields have IDs just waiting to be styled. If there were any errors generated, those sections would also be included, with all of the ID and class goodness.

Shortcomings

Sometimes, the designer wants to change the order of the fields, put the errors in a different spot, assign other classes and IDs, and insert additional information in the template. In these cases, one can skip the shortcut, and specify the form pieces individually. Here is an example of a template with individual fields listed:

<form action="." method="post"> {% csrf_token %}

<div id="signup_details">

{{ form.non_field_errors }}

<p>{{ form.training_class.errors }}{{ form.training_class.label_tag }} {{ form.training_class }}</p>
<p>{{ form.training_schedule.errors }}{{ form.training_schedule.label_tag }} {{ form.training_schedule }}</p>

<fieldset id="new_schedule">
<legend>Create New Schedule</legend>
<p>{{ form.class_date.errors }}{{ form.class_date.label_tag }} {{ form.class_date }}</p>
<p>{{ form.class_time.errors }}{{ form.class_time.label_tag }} {{ form.class_time }}</p>
<p>{{ form.location.errors }}{{ form.location.label_tag }} {{ form.location }}</p>
<p>{{ form.campus.errors }}{{ form.campus.label_tag }} {{ form.campus }}</p>
</fieldset>

</div>
<div id="signup_flags">
<p>{{ form.send_email.errors }}{{ form.send_email.label_tag }} {{ form.send_email }}</p>
<p>{{ form.attended.errors }}{{ form.attended.label_tag }} {{ form.attended }}</p>
<p>{{ form.exempt.errors }}{{ form.exempt.label_tag }} {{ form.exempt }}</p>
</div>

<br clear="left" /><br />

<input type="submit" value="Save" />
<INPUT TYPE="BUTTON" VALUE="Cancel" onClick="history.go(-1)">
</form>

Note a couple of things. The tag {{ form.non_field_errors }} is where any form level errors would be displayed. Likewise, the {{ form.field_name.errors }} tag holds the place for individual field errors.

Once the tags are in the form, the designer can move them around and add additional HTML – notice the use of fieldset above.

See more information on the individual tags in Working With Forms.

Hope this helps a few designers figure out Django templates.  Please leave any comments or questions below.

A Better Way – Don’t!

In an earlier post, I talked about using form.save(commit=False) in a class based view, and I showed a way to do it.  However, I wasn’t looking far enough back to see the real question:

How can I add information not found on the form to the object before saving?

Thanks to a reply way down in this Google Groups post, I found what I needed, and I kicked my self for not seeing this earlier. Since I was using the default form logic to save the object, I could send the additional value to the form.

    ## Include the instance object before saving
    def form_valid(self, form):
        form.instance.institution = self.institution
        return super(EventCreateView,self).form_valid(form)

I like this solution better than what I provided in the earlier post because it is a better object oriented approach.

Of course, I have used the older example all over my applications, so I’ve got some refactoring to do.

Installing Oracle Client on Ubuntu 11.10

Oracle Logo(Another post written for personal documentation)

Get Software

Download these three packages from Oracle for the proper operating system (32 bit for me):

  • Instant Client Basic-Lite
  • Instant Client SDK
  • Instand Client SQLPlus

Unzip and copy to /opt/oracle/11_2/instantclient

Set LD_LIBRARY_PATH

Create /etc/ld.so.conf.d/oracle_instantclient.conf:

#Oracle client LD_LIBRARY_PATH setting
/opt/oracle/11_2/instantclient

Update cache:

sudo ldconfig -v

Symbolic Link to Library

ln -s libclntsh.so.10.1 libclntsh.so

Set ORACLE_HOME

export ORACLE_HOME=/opt/oracle/11_2/instantclient

Install Python Library

The library is called cx-oracle. Use your favorite installation method. (Don’t forget about your virtual environment!)

That’s It

Hope I remembered everything.

UPDATE

Found another post that outlines the procedure maybe a little better than I did, and includes notes on setting up tnsnames.ora. See Install Oracle Instant Client 11.1 and cx_Oracle 4.4 on Ubuntu 8.04 on Technoblog.

Using form.save(commit=False) in a Class Based View

UPDATE: I found a better way to do this.  See this post.

In this example, I once again hadn’t used a generic view for this create function, since I needed to add a value that is not in the form to the new object before the save.

    if form.is_valid():
        new_rec = form.save(commit=False)
        new_rec.institution = institution
        new_rec.save()
        return HttpResponseRedirect(reverse('cat_list'))

The value for institution comes from the user’s session area. The above is the accepted pattern to add the data to the object.

So how can we do this in a class based view?

An override of form_valid() in the class will handle this case.

    def form_valid(self, form):
        self.object = form.save(commit=False)
        self.object.institution = self.kwargs['institution']
        self.object.save()
        return HttpResponseRedirect(self.get_success_url())

(Putting the institution value in kwargs is the subject for another post. One could also use an object variable.)

UPDATE: I found a better way to do this.  See this post.

The reader who just wants to know how to do this can stop here. However….

Is this the best way to do this?

I’m not very skilled in the art of overriding methods, so I am a little concerned about the forward compatibility of this solution.
Let’s first see the upstream definitions of this method.
The method is defined in ModelFormMixin as follows:

    def form_valid(self, form):
        self.object = form.save()
        return super(ModelFormMixin, self).form_valid(form)

And above that we find a simpler method in FormMixin:

    def form_valid(self, form):
        return HttpResponseRedirect(self.get_success_url())

I like how the ModelFormMixin method is able to refer back to the original using the super function, but I don’t know how I can jump over the immediate ancestor to call back the original method. Is this possible? Does it matter?

I am slightly concerned that changes to the method in FormMixin that may come along in future versions may be overlooked with this solution. Sure, I have to check these things during an upgrade anyway, but I would like to perform this as “correctly” as possible.

Please leave your suggestions, thoughts and comments below.

UPDATE: I found a better way to do this.  See this post.