Unit Testing Blog Entry 1

This post has to do with some experiences I had with setting up testing that I hope the team will find useful. In the absence of a convenient place to blog I decided to post it here.

---

Initial setup

In trying to set up the testing infrastructure, I began with the simplest test I could write that I knew tested something useful: The code for auto creation of student/ta/instructor groups upon course creation. (I'm not trying to pick on anyone in any way, but it just serves a convenient running example ;-) ).

Here's the code (c2g/models.py):

def DefineUserGroupsForCourse(sender, **kwargs):
        instance = kwargs.get('instance')
        instance.student_group = Group.objects.create( name="Student Group for " + instance.handle + "_" + str(instance.institution.id))
        instance.instructor_group = Group.objects.create(name="Instructor Group for " + instance.handle + "_" + str(instance.institution.id))
        instance.tas_group = Group.objects.create(name="TAS Group for " + instance.handle + "_" + str(instance.institution.id))
        instance.readonly_tas_group = Group.objects.create(name="Readonly TAS Group for " + instance.handle + "_" + str(instance.institution.id))
            
pre_save.connect(DefineUserGroupsForCourse, sender=Course)

and here's the test I wrote (c2g/tests.py):

from django.test import TestCase
from c2g.models import Institution, Course
from django.contrib.auth.models import Group 

class C2GUnitTests(TestCase):
    
    def test_course_create(self):
        """
        Tests that course creation creates groups
        """
        numGroupsB4=len(Group.objects.all())
        i=Institution(title='TestInstitute')
        i.save()
        course1=Course(institution=i,title='gack',handle='test-course')
        course1.save()
        numGroupsAfter=len(Group.objects.all())
        self.assertEqual(numGroupsB4+4, numGroupsAfter)

Run it

(django-class2go)Jasons-MacBook-Air:django-project jbau$ ./manage.py test c2g
Creating test database for alias 'default'...
.
----------------------------------------------------------------------
Ran 1 test in 0.006s

OK
Destroying test database for alias 'default'...

Woohoo! It works.

Adding fixtures

Of course, you need actual data for a test to be meaningful. I decide to dump the database generated by a clean run of db_populate into a fixture and use that:

(django-class2go)Jasons-MacBook-Air:django-project jbau$ ./manage.py dumpdata --indent=2 > c2g/fixtures/db_snapshot.json 

new c2g/test.py, adding fixtures and a new test

class C2GUnitTests(TestCase):
    
    fixtures=['db_snapshot.json']
    
    def test_fixture_install(self):
        """
        Tests that fixtures were installed correctly
        """
        c = Course.objects.get(code='CS1234') 
        self.assertEqual(len(Course.objects.all()),1) #only 1 course in the test data
        self.assertEqual(c.title, u'Natural Language Processing')

Run the test, and an error that makes no sense pops up: (shortened)

(django-class2go)Jasons-MacBook-Air:django-project jbau$ ./manage.py test c2g
Creating test database for alias 'default'...
Problem installing fixture '/Users/jbau/projects/class2go/django-project/c2g/fixtures/db_snapshot.json': Traceback (most recent call last):
IntegrityError: Could not load c2g.Course(pk=1): (1062, "Duplicate entry 'Student Group for nlp-Fall2012_1' for key 'name'")

Why would there be a duplicate entry? The fixture is supposed to create a clean database for every test. I check and confirmed that db_snapshot.json was free of duplicates.

##Debugging

So I spent an hour or two trying to figure out why this didn't work. Some of it was just checking that the fixture file was clean, that I was loading it in with the right syntax, etc. But I was still pretty stumped, until I looked back at error messages in detail and found this line in the stack trace.

  File "/Users/jbau/projects/class2go/django-project/c2g/models.py", line 72, in DefineUserGroupsForCourse
    instance.student_group = Group.objects.create( name="Student Group for " + instance.handle + "_" + str(instance.institution.id))

Remember the code for auto-creating groups above. It registered a pre_save listener. Which means it would run every time save is called. This behavior could be creating the duplicate groups. Time to modify our test to see if this was the case.

class C2GUnitTests(TestCase):
    
    #fixtures=['db_snapshot.json']
    
    def test_multisave(self):
        """
        Tests saving a course more than once
        """
        numGroupsB4=len(Group.objects.all())
        i=Institution(title='TestInstitute')
        i.save()
        course1=Course(institution=i,title='gack',handle='test-course')
        course1.save()
        course1.title='hack'
        course1.save()
        numGroupsAfter=len(Group.objects.all())
        self.assertEqual(numGroupsB4+4, numGroupsAfter)

Run this (and with fixtures commented out), and I get the same error. Printed below (shortened)

(django-class2go)Jasons-MacBook-Air:django-project jbau$ ./manage.py test c2g
Creating test database for alias 'default'...
.E
======================================================================
ERROR: test_multisave (c2g.tests.C2GUnitTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbau/projects/class2go/django-project/c2g/tests.py", line 26, in test_multisave
    course1.save()
IntegrityError: (1062, "Duplicate entry 'Student Group for test-course_2' for key 'name'")

##Fixing the code and confirming the fix, with existing tests! Now that the bug is confirmed, it's time to make the fix. The auto-creation should test to see if a group has already been specified. Here's the new code:

def defineUserGroupsForCourse(sender, **kwargs):
    instance = kwargs.get('instance')
    if (not hasattr(instance,'student_group')):
        instance.student_group = Group.objects.create( name="Student Group for " + instance.handle + "_" + str(instance.institution.id))
    if (not hasattr(instance,'instructor_group')):
        instance.instructor_group = Group.objects.create(name="Instructor Group for " + instance.handle + "_" + str(instance.institution.id))
    if (not hasattr(instance,'tas_group')):
        instance.tas_group = Group.objects.create(name="TAS Group for " + instance.handle + "_" + str(instance.institution.id))
    if (not hasattr(instance,'readonly_tas_group')):
        instance.readonly_tas_group = Group.objects.create(name="Readonly TAS Group for " + instance.handle + "_" + str(instance.institution.id))

and we can re-use our previously created unit tests to check that everything now works. (I also added some tests, and had to fix another bug in the UserProfile signal listener, but I finally got this output)

(django-class2go)Jasons-MacBook-Air:django-project jbau$ ./manage.py test c2g
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 2.911s

OK
Destroying test database for alias 'default'...

##Conclusion What's the conclusion? That the testing process had uncovered a legitimate bug, even despite the confusing and indirect error messages. And that the testing framework is a useful place to try out hypotheses. This seems to argue that testing is a valuable exercise in improving code quality, despite its very real tediousness.