If you want to deliver a great library to the community, or if you want to bump smoothly your Python's production version, testing on multiple versions of Python is a must have, especially since Python 3.
Let's see how with the pygments example.
First ensure that the latest version of the ShiningPanda plugin is installed (check on Manage Jenkins > Manage Plugins > Installed page). If not, look for it on Available tab and perform the installation (don't forget to restart Jenkins).
Then declare all the Python installations you want to test on with the Manage Jenkins > Configure System page (one shot configuration):
To add an installation, search the Python section and click on Add Python. Then give the installation a name and enter its home folder (ie. PYTHONHOME).
First, create a New Job, enter its name (here pygments) and select Build multi-configuration project before validating:
As usual, setup:
To be able to select the versions of Python you want to build on, first add a Python axis:
Then check all desired versions:
And if you're on shiningpanda-ci.com, select versions by expanding the relevant tree:
A builder is also required:
The Python axis works well with two builders:
Here a virtualenv is created in $WORKSPACE/env. As Clear is not checked, this virtual environment is not recreated on each build. Distribute is used as package manager, and every line of Command is expected to return zero as exit code as Ignore exit code is not checked: this is helpful to detect an error as soon as possible in the build process.
As far as Command is concerned, first some tools are installed with pip (nose and coverage), then pygments is installed from source and tests are run with code coverage, and finally code coverage reports are converted in XML.
Add all the desired post-build actions. Here Jenkins is asked to parse the XML test and coverage reports and to send e-mails on failure.
Finally start a new build by clicking on Build Now: you will find the execution results for each version of Python on the main page of the project.
Testing on multiple versions of Python can be time and resource consuming. To ensure that a revision is worthwhile to test on all versions, you can execute a touchstone build first:
To enable this option, look for a Execute a touchstone build first checkbox in the Configuration Matrix and check it.
The syntax for filter is PYTHON==<name>, where <name> is the name of the Python installation you want to test on first. Here the touchstone build is executed on CPython 2.7 (PYTHON=="CPython-2.7") but it could also have been any other declared interpreter.