If you want to deliver a great library to the community, or if you want to bump smoothly your Python's production version, testing on multiple versions of Python is a must have, especially since Python 3.

ShiningPanda plugin for Jenkins proposes an elegant solution to test against multiple versions of Python, without the burden of creating one plan per version.

Let's see how with the pygments example.

Global Configuration

First ensure that the latest version of the ShiningPanda plugin is installed (check on Manage Jenkins > Manage Plugins > Installed page). If not, look for it on Available tab and perform the installation (don't forget to restart Jenkins).

Then declare all the Python installations you want to test on with the Manage Jenkins > Configure System page (one shot configuration):

To add an installation, search the Python section and click on Add Python. Then give the installation a name and enter its home folder (ie. PYTHONHOME).

Create a new job

First, create a New Job, enter its name (here pygments) and select Build multi-configuration project before validating:

Basic setup

As usual, setup:

• The name and description.
• The source repository, here http://dev.pocoo.org/hg/pygments-main.
• The build trigger policy: for instance checking for modifications every five minutes.

Python versions

To be able to select the versions of Python you want to build on, first add a Python axis:

Then check all desired versions:

And if you're on ShiningPanda CI, select versions by expanding the relevant tree:

The builder

A builder is also required:

The Python axis works well with two builders:

• Virtualenv Builder (recommended): creates an isolated environment with virtualenv.
• Python Builder: use it if you do not need to install Python packages to build your project.

In this example a Virtualenv Builder is used to be able to install some useful tools such as nose and coverage:

Here a virtualenv is created in $WORKSPACE/env. As Clear is not checked, this virtual environment is not recreated on each build. Distribute is used as package manager, and every line of Command is expected to return zero as exit code as Ignore exit code is not checked: this is helpful to detect an error as soon as possible in the build process.

As far as Command is concerned, first some tools are installed with pip (nose and coverage), then pygments is installed from source and tests are run with code coverage, and finally code coverage reports are converted in XML.

Post-build actions

Add all the desired post-build actions. Here Jenkins is asked to parse the XML test and coverage reports and to send e-mails on failure.

Result

Finally start a new build by clicking on Build Now: you will find the execution results for each version of Python on the main page of the project.

Tips

Testing on multiple versions of Python can be time and resource consuming. To ensure that a revision is worthwhile to test on all versions, you can execute a touchstone build first:

• If this build is successful, builds on other versions are triggered.
• If this build fails, other versions are not tested.

To enable this option, look for a Execute a touchstone build first checkbox in the Configuration Matrix and check it.

The syntax for filter is PYTHON==<name>, where <name> is the name of the Python installation you want to test on first. Here the touchstone build is executed on CPython 2.7 (PYTHON=="CPython-2.7") but it could also have been any other declared interpreter.