Python multiple versions testing with Jenkins
If you want to deliver a great library to the community, or if you want to bump smoothly your Python's production version, testing on multiple versions of Python is a must have, especially since Python 3.
ShiningPanda plugin for Jenkins proposes an elegant solution to test against multiple versions of Python, without the burden of creating one plan per version.
Let's see how with the pygments example.
First ensure that the latest version of the ShiningPanda plugin is installed (check on
Manage Jenkins > Manage Plugins > Installed page). If not, look for it on
Available tab and perform the installation (don't forget to restart Jenkins).
Then declare all the Python installations you want to test on with the
Manage Jenkins > Configure System page (one shot configuration):
To add an installation, search the
Python section and click on
Add Python. Then give the installation a name and enter its home folder (ie.
Create a new job
First, create a
New Job, enter its name (here
pygments) and select
Build multi-configuration project before validating:
As usual, setup:
- The name and description.
- The source repository, here
- The build trigger policy: for instance checking for modifications every five minutes.
To be able to select the versions of Python you want to build on, first add a
Then check all desired versions:
And if you're on ShiningPanda CI, select versions by expanding the relevant tree:
A builder is also required:
Python axis works well with two builders:
Virtualenv Builder(recommended): creates an isolated environment with virtualenv.
Python Builder: use it if you do not need to install Python packages to build your project.
In this example a
Virtualenv Builder is used to be able to install some useful tools such as nose and coverage:
Here a virtualenv is created in
Clear is not checked, this virtual environment is not recreated on each build. Distribute is used as package manager, and every line of
Command is expected to return zero as exit code as
Ignore exit code is not checked: this is helpful to detect an error as soon as possible in the build process.
As far as
Command is concerned, first some tools are installed with pip (nose and coverage), then pygments is installed from source and tests are run with code coverage, and finally code coverage reports are converted in XML.
Add all the desired post-build actions. Here Jenkins is asked to parse the XML test and coverage reports and to send e-mails on failure.
Finally start a new build by clicking on
Build Now: you will find the execution results for each version of Python on the main page of the project.
Testing on multiple versions of Python can be time and resource consuming. To ensure that a revision is worthwhile to test on all versions, you can execute a touchstone build first:
- If this build is successful, builds on other versions are triggered.
- If this build fails, other versions are not tested.
To enable this option, look for a
Execute a touchstone build first checkbox in the
Configuration Matrix and check it.
The syntax for filter is
<name> is the name of the Python installation you want to test on first. Here the touchstone build is executed on CPython 2.7 (
PYTHON=="CPython-2.7") but it could also have been any other declared interpreter.