Python & Java: a unified build process (1/4)

Since I graduated, I always worked at companies dealing with more than one language to build their solution:

• an AWT application deployed via Java Web Start, displaying some Python interpreted DSL outputs,
• the use of GWT with a Django project,
• and of course the integration of Jenkins in ShiningPanda's Python backend for our Hosted Continuous Integration Service dedicated to Python (and soon Java).

The target was always to implement a unified build process for the comprehensive solution, building and packaging it by issuing a single command. Most languages have their own build tools, but as soon as you have more than one language at hand, issues arise.

This article describes how we are currently dealing with both Java and Python at ShiningPanda to build our Java and Python based service.

First of all we needed to find a conductor. With Java involved, Maven seemed the smartest choice. So the challenge was to integrate some classical setuptools based Python projects with Maven.

The general idea is to map some of the phases of the Maven lifecycle with the setuptools commands:

• clean with python setup.py clean,
• compile with python setup.py develop (not mapped with Maven install phase to come before the test phase),
• test with python setup.py test,
• deploy with python setup.py register sdist upload.

The following sample project builds a pysample Python project and a jsample Java project. Sources can be found here, and the project is organized as follows:

Let's have a look on the POMs, the project descriptors telling Maven what to do:

• First the root pom.xml: it defines the plugin versions and delegates to the subfolder POMs,
• Then setuptools/pom.xml: this POM defines all the mappings and will be inherited in all Python projects to avoid mapping duplications,
• Finally jsample/pom.xml: a classical Java POM defining the dependences for the jsample project,
• And pysample/pom.xml: a POM that inherits from setuptools/pom.xml.

As you may have guessed, the core of the solutions is setuptools/pom.xml. It uses the exec-maven-plugin to start Python subprocesses calling setup.py:

<project>
  <!-- Project definition [...] -->
  <build>
    <pluginmanagement>
      <plugins>
        <plugin>
          <groupid>org.codehaus.mojo</groupid>
          <artifactid>exec-maven-plugin</artifactid>
          <configuration>
            <executable>python</executable>
            <workingdirectory>${basedir}</workingdirectory>
          </configuration>
          <executions>
            <!-- Mappings [...] -->
          </executions>
        </plugin>
      </plugins>
    </pluginmanagement>
  </build>
</project>

Then we only have to define the mapping in the executions tag:

<executions>
  <execution>
    <id>setuptools clean</id>
    <phase>clean</phase>
    <goals>
      <goal>exec</goal>
    </goals>
    <configuration>
      <arguments>
        <argument>setup.py</argument>
        <argument>clean</argument>
      </arguments>
    </configuration>
  </execution>
  <execution>
    <id>setuptools install</id>
    <phase>compile</phase>
    <goals>
      <goal>exec</goal>
    </goals>
    <configuration>
      <arguments>
        <argument>setup.py</argument>
        <argument>develop</argument>
      </arguments>
    </configuration>
 </execution>
  <execution>
    <id>setuptools test</id>
    <phase>test</phase>
    <goals>
      <goal>exec</goal>
    </goals>
    <configuration>
      <skip>${maven.test.skip}</skip>
      <arguments>
        <argument>setup.py</argument>
        <argument>test</argument>
      </arguments>
    </configuration>
  </execution>
  <execution>
    <id>setuptools deploy</id>
    <phase>deploy</phase>
    <goals>
      <goal>exec</goal>
    </goals>
    <configuration>
      <arguments>
        <argument>setup.py</argument>
        <argument>register</argument>
        <argument>sdist</argument>
        <argument>upload</argument>
      </arguments>
    </configuration>
  </execution>
</executions>

Note that we're skipping the Python tests if the classical -Dmaven.test.skip=true property is set on command line.

Then pysample/pom.xml is quite straightforward (note the parent tag defining the inheritance):

<project>
  <modelversion>4.0.0</modelversion>
  <parent>
    <groupid>com.shiningpanda</groupid>
    <artifactid>setuptools</artifactid>
    <version>0.1-SNAPSHOT</version>
    <relativepath>../setuptools/pom.xml</relativepath>
  </parent>
  <artifactid>pysample</artifactid>
  <packaging>pom</packaging>
  <build>
    <plugins>
      <plugin>
        <groupid>org.codehaus.mojo</groupid>
        <artifactid>exec-maven-plugin</artifactid>
      </plugin>
    </plugins>
  </build>
</project>

And so is the root pom.xml:

<project>
  <!-- Project definition [...] -->
  <build>
    <pluginmanagement>
      <plugins>
        <!-- Define exec-maven-plugin version -->
        <plugin>
          <groupid>org.codehaus.mojo</groupid>
          <artifactid>exec-maven-plugin</artifactid>
          <version>1.2</version>
        </plugin>
      </plugins>
    </pluginmanagement>
  </build>
  <!-- Call subprojects -->
  <modules>
    <module>setuptools</module>
    <module>pysample</module>
    <module>jsample</module>
  </modules>
</project>

That's it. Now issue the following command in the root folder to build your full solution:

mvn clean install
# or to deploy (see our upcoming blog post)
mvn clean deploy

In the next articles we will deal with these topics:

• synchronize project versions,
• automate deploys and releases,
• tips for continuous integration.

So stay tuned! And if you need some help on your build process, contact our service team!