Python leaky unittests Gotcha - Too Many MySQL Connections

OpenWorm - a long way to go

Open Source Software Presentation

I am a software engineer (lapsed physicist) who likes working with open source technology. I spend most of my time writing code in Python but I am currently also learning Go.

Some time ago I noticed that some of my integration tests were failing with a MySQL Too many connections error. The tests were being run with Python's unittest package. Each integration test was opening a MySQL connection, but I couldn't understand the cause of this error - each test was run with a new TestInstance class and I (wrongly) assumed Python's garbage collector would come along and clear the old TestInstance, closing the MySQL connection in the process.

Heading over to Stack Overflow eventually gave me the answer (thanks Kara!)

It turns out that a reference to each test case is kept until the entire test suite is run, preventing the garbage collector from freeing this object

So the following TestCase will have opened 3 simultaneous MySQL connections by the time it terminates:

import unittest
import MySQLdb

class TestMySQLConn(unittest.TestCase):
    def setUp(self):
        self.db_connection = MySQLdb.connect(some_credentials)

    def test1(self):
        # a MySQL connection opens here
        pass

    def test2(self):
        # another MySQL connection opens here
        pass

    def test3(self):
        # yet another MySQL connection opens here
        pass

The best solution is to use a tearDown method in the TestCase:

def tearDown(self):
    self.db_connection.close()

This problem was eventually discussed and resolved in Python3.4

OpenWorm is a fantastic project which it has been my pleasure to work on for two years. In that time I have seen the community go from strength to strength and our prospects of making meaningful scientific contributions continue to grow.

It's recently been driven home to me how cautious one has to be when reading articles in the media about science. It all started with a video I released to our mailing list. This was intended as a very early proof of concept - showing that our method of simulating a worm muscular system as blocks of PCISPH contractile matter had the potential to reproduce undulutory thrust - an exciting discovery, but one filled with subtle caveats.

A few days later this BBC news article about OpenWorm surfaced. It's fantastic to see how much interest there is in our project, but the article was (while generally OK) inaccurate in a few ways. John Hurliman for instance is incorrectly attributed as project leader (OpenWorm has no "leader") and the project has been going on for much longer than suggested, the scientific claims were broadly correct though. Over time more articles were released, with the claims made becoming a bit outlandish, culminating in this. There were also some really good articles - the one on phys.org being my personal favourite.

We have released a blog post addressing our concerns about reporting.

openworm

I recently gave a workshop on Open Source Software development at the Malta Information Technology Agency.

It was a great pleasure and I would like to thank all who came for participating. The slides are located here.