Java Packaging HOWTO

This section includes basic introduction to Java packaging world to people coming from different backgrounds. The goal is to understand language of all groups involved. If you are a Java developer coming into contact with RPM packaging for the first time start reading Java developer section. On the other hand if you are coming from RPM packaging background an introduction to Java world is probably a better starting point.

It should be noted that especially in this section we might sacrifice correctness for simplicity.

Java is a programming language which is usually compiled into bytecode for JVM (Java Virtual Machine). For more details about the JVM and bytecode specification see JVM documentation.

Packaging Java software has specifics which we will try to cover in this section aimed at Java developers who are already familiar with Java language, JVM, classpath handling, Maven, pom.xml file structure and dependencies.

Instead we will focus on basic packaging tools and relationships between Java and RPM world. One of the most important questions is: What is the reason to package software in RPM (or other distribution-specific formats). There are several reasons for it, among others:

Fedora ships with an open-source reference implementation of Java Standard Edition called OpenJDK. OpenJDK provides Java Runtime Environment for Java applications and set of development tools for Java developers.

From users point of view, java command is probably the most interesting. It is a Java application launcher which spawns Java Virtual Machine (JVM), loads specified .class file and executes its main method.

Here is an example how to run sample Java project from section Example Java Project:

OpenJDK provides a lot of interesting tools for Java developers:

Most of the Java application needs to specify classpath in order to work correctly. Fedora contains several tools which make working with classpaths easier.

build-classpath - this tool takes JAR filenames or artifact coordinates as arguments and translates them to classpath-like string. See the following example:

log4j corresponds to log4j.jar stored in %{_javadir}. If the JAR file is stored in subdirectory under %{_javadir}, it is neccessary to pass subdirectory/jarname as an argument to build-classpath. Example:

Another tool is build-jar-repository. It can fill specified directory with symbolic / hard links to specified JAR files. Similarly to build-classpath, JARs can be identified by their names or artifact coordintes.

Similar command rebuild-jar-repository can be used to rebuild JAR repository previously built by build-jar-repository. See man rebuild-jar-repository for more information.

build-classpath-directory is a small tool which can be used to build classpath string from specified directory.

This section describes most of directories used for Java packaging. Each directory is named in RPM macro form, which shows how it should be used in RPM spec files. Symbolic name is followed by usual macro expansion (i.e. physical directory location in the file system) and short description.

Complex Java applications usually consist of multiple components. Each component can have multiple implementations, called artifacts. Artifacts in Java context are usually JAR files, but can also be WAR files or any other kind of file.

There are multiple incompatible ways of identifying (naming) Java artifacts and each build system often encourages usage of specific naming scheme. This means that Linux distributions also need to allow each artifact to be located using several different identifiers, possible using different schemes. On the other hand it is virtually impossible to every naming scheme, so there are some simplifications.

This chapter describes artifact different ways to identify and locate artifacts in system repository.

RPM has multiple types of metadata to describe dependency relationships between packages. The two basic types are Requires and Provides. Requires denote that a package needs something to be present at runtime to work correctly and the package manager is supposed to ensure that requires are met. A single Requires item can specify a package or a virtual provide. RPM Provides are a way to express that a package provides certain capability that other packages might need. In case of Maven packages, the Provides are used to denote that a package contains certain Maven artifact. They add more flexibility to the dependency management as single package can have any number of provides, and they can be moved across different packages without breaking other packages' requires. Provides are usually generated by automatic tools based on the information from the built binaries or package source.

The Java packaging tooling on Fedora provides automatic Requires and Provides generation for packages built using XMvn. The Provides are based on Maven artifact coordinates of artifacts that were installed by the currently being built. They are generated for each subpackage separately. They follow a general format mvn(groupId:artifactId:extension:classifier:version), where the extension is omitted if its jar and classifier is omitted if empty. Version is present only for compat artifacts, but the trailing colon has to be present unless it is a Jar artifact with no classifier.

The generated Requires are based on dependencies specified in Maven POMs in the project. Only dependencies with scope set to either compile, runtime or not set at all are used for Requires generation. Requires do not rely on package names and instead always use virtual provides that were described above, in exactly the same format, in order to be satisfiable by the already existing provides. For packages consisting of multiple subpackages, Requires are generated separately for each subpackage. Additionally, Requires that point to an artifact in a different subpackage of the same source package are generated with exact versions to prevent version mismatches between artifacts belonging to the same project.

The requires generator also always generates Requires on java-headless and javapackages-tools.

If the package is built built using different tool than Apache Maven, but still ships Maven POM(s), the you will still get automatic provides generation if you install the POM using %mvn_artifact and %mvn_install. The requires generation will also be executed but the outcome largely depends on whether the POM contains accurate dependency insformation. If it contains dependency information, you should double-check that it is correct and up-to-date. Otherwise you need to add Requires tags manually as described in the next section.

For packages without POMs it is necessary to specify Requires tags manually. In order to build the package you needed to specify BuildRequires tags. Your Requires tags will therefore likely be a subset of your BuildRequires, excluding build tools and test only dependencies.

The generated Requires and Provides of built packages can be queried using rpm:

While Requires and Provides generation is automated for Maven projects, BuildRequires still remains a manual task. However, there are tools to simplify it to some extent. XMvn ships a script xmvn-builddep that takes a build.log output from mock and prints Maven-style BuildRequires on artifacts that were actually used during the build. It does not help you to figure out what the BuildRequires are before you actually build it, but it may help you to have a minimal set of BuildRequires that are less likely to break, as they do not rely on transitive dependencies.

Javadoc subpackages in Fedora provide automatically generated API documentation for Java libraries and applications. Java Development Kit comes with tool called javadoc. This tool can be used for generating the documentation from specially formated comments in Java source files. Output of this tool, together with license files, usually represents only content of javadoc subpackages. Note javadoc invocation is typically handled by build system and package maintainer does not need to deal with it directly.

Javadoc subpackage shouldn’t depend on its base package and vice versa. The rationale behind this rule is that documentation can usually be used independently from application / library and therefore base package does not need to be always installed. Users are given an option to install application and documentation separately.

As mentioned in section about Java packaging basics, all Java applications need wrapper shell scripts to setup the environment before running JVM and associated Java code.

The javapackages-tools package contains a convenience %jpackage_script macro that can be used to create scripts that work for the majority of packages. See its definition and documentation in /usr/lib/rpm/macros.d/macros.jpackage. One thing to pay attention to is the 6th argument to it - whether to prefer a JRE over a full SDK when looking up a JVM to invoke. Most packages that do not require the full Java SDK will want to set that to true to avoid unexpected results when looking up a JVM when some of the installed JREs do not have the corresponding SDK (*-devel package) installed.

The previous example installs the msv script (5th argument) with main class being msv.textui.Driver (1st argument). No optional flags (2nd argument) or options (3rd argument) are used. This script will add several libraries to classpath before executing main class (4th argument, JAR files separated with :). build-classpath is run on every part of 4th argument to create full classpaths.

Sometimes it may be needed to replace all JAR files in current directory with symlinks to the system JARs located in %{_javadir}. This task can be achieved using tool called xmvn-subst.

The example above shows how easy the symlinking can be. However, there are some limitations. Original JAR files need to carry metadata which tell xmvn-subst for what artifact given file should be substituted. Otherwise xmvn-subst won’t be able to identify the Maven artifact from JAR file.

Apache Ivy provides an automatic dependency management for Ant managed builds. It uses Maven repositories for retrieving artifacts and supports many declarative features of Maven such as handling transitive dependencies.

XMvn supports local resolution of Ivy artifacts, their installation and requires generation.

A subset of macros used to modify Maven POMs also work with ivy.xml files allowing the maintainer to add / remove / change dependencies without the need of making patches and rebasing them with each change. You can use dependency handling macros %pom_add_dep, %pom_remove_dep, %pom_change_dep and generic %pom_xpath_* macros. For more details, see corresponding manpages.

Ivy supports publishing built artifact with ivy:publish task. If your build.xml file already contains a task that calls ivy:publish, you can set the resolver attribute of the ivy:publish element to xmvn. This can be done with simple %pom_xpath_set call. Then when the task is run, XMvn can pick the published artifacts and install them during the run of %mvn_install without needing you to manually specify them with %mvn_artifact.

This step by step guide will show you how to package Maven project. Let’s start with probably the simplest spec file possible.

The spec file above is a real world example how it may look like for simple Maven project. Both %build and %install sections consist only of one line.

Another interesting line:

All Maven projects need to have BuildRequires on maven-local. They also need to have Requires and BuildRequires on jpackages-utils, but build system adds these automatically. The package maintainer does not need to list them explicitly.

By default, resulting JAR files will be installed in %{_javadir}/%{name}, therefore the package needs to own this directory.

The build could fail from many reasons, but one probably most common is build failure due to missing dependencies.

We can try to remove these missing dependencies from pom.xml and make Maven stop complaining about them. However, these removed dependencies may be crucial for building of the project and therefore it may be needed to package them later. Let’s remove the dependencies from pom.xml.

The package maintainer can use a wide variety of “pom_” macros for modifying pom.xml files. See the Macros for POM modification section for more information.

Now try to build the project again. The build will fail with a compilation failure.

Oops, another problem. This time Maven thought it had all the necessary dependencies, but Java compiler found otherwise.

Now it is possible to either patch the source code not to depend on missing libraries or to package them. The second approach is usually correct. It is not necessary to package every dependency right away. The maintainer could package compile time dependencies first and keep the rest for later (test dependencies, etc.). But Maven needs to know that it should not try to run tests now. This can be achieved by passing -f option to %mvn_build macro. Maven will stop complaining about missing test scoped dependencies from now on.

We already have package which provides commons-io:commons-io artifact, let’s add it to the BuildRequires. Also disable tests for now.

Now try to build the project one more time. The build should succeed now. Congrats, you managed to create an RPM from Maven project!

There is plenty of other things maintainer may want to do. For example, they may want to provide symbolic links to the JAR file in %{_javadir}.

This can be easily achieved with %mvn_file macro:

See Alternative JAR File Names section for more information.

Another quite common thing to do is adding aliases to Maven artifact. Try to run rpm -qp --provides on your locally built RPM package:

The output above tells us that the RPM package provides Maven artifact com.example:simplemaven:1.0. Upstream may change the groupId:artifactId with any new release. And it happens. For example org.apache.commons:commons-io changed to commons-io:commons-io some time ago. It is not a big deal for package itself, but it is a huge problem for other packages that depends on that particular package. Some packages may still have dependencies on old groupId:artifactId, which is suddenly unavailable. Luckily, there is an easy way how to solve the problems like these. Package maintainer can add aliases to actually provided Maven artifact.

See Additional Mappings for more information on %mvn_alias.

Rebuild the pacakge and check rpm -qp --provides output again:

Now it does not matter if some other package depends on either of these listed artifact. Both dependencies will always be satisfied with your package.

Maven builds can be configured to produce alternative layout, include additional aliases in package metadata or create separate subpackages for certain artifacts.

Sometimes Maven pom.xml files need to be patched before they are used to build packages. One could use traditional patches to maintain changes, but package maintainers should use %pom_* macros developed specially to ease this task. Using %pom_* macros not only increases readability of the spec file, but also improves maintainability of the package as there are no patches that would need to be rebased with each upstream release.

There are two categories of macros:

The macros are designed to be called from %prep section of spec files. All the macros also have their own manual page. This document provides an overview of how they are used. For the technical details, refer to their respective manpages.

By default, a macro acts on a pom.xml file (or ivy.xml file) in the current directory. Different path can be explicitly specified via an argument (the last one, unless stated otherwise). Multiple paths can be specified as multiple arguments. If a path is a directory, it looks for a pom.xml file in that directory. For example:

Most macros also support recursive mode, where the change is applied to the pom.xml and all its modules recursively. This can be used, for example, to remove a dependency from the whole project. It is activated by -r switch.

Maven wasn’t able to build project com.example:simplemaven because it couldn’t find some dependencies (in this case commons-io:commons-io:jar:2.4 and junit:junit:jar:4.11)

You have multiple options here:

Java compiler couldn’t find given class on classpath or incompatible version was present. This could be caused by following reasons:

Different versions of same library may provide slightly different API. This means that project doesn’t have to be buildable if different version is provided. If the library in local repository is older than the one required by project, then the library could be updated. If the project requires older version, then the project should be ported to latest stable version of the library (this may require cooperation with project’s upstream). If none of these is possible from some reason, it is still possible to introduce new compat package. See compat packages section for more information on this topic.

Sometimes pom.xml doesn’t list all the necessary dependencies, even if it should. Dependencies can also depend on some other and typically all these will be available to the project which is being built. The problem is that local repository may contain different versions of these dependencies. And even if these versions are fully compatible with the project, they may require slightly different set of dependencies. This could lead to build failure if pom.xml doesn’t specify all necessary dependencies and relies on transitive dependencies. Such a missing dependency may be considered a bug in the project. The solution is to explicitly add missing dependency to the pom.xml. This may be easily done by using %pom_add_dep macro. See the section about macros for POM modification for more information.

Most often this error happens when one part of the package depends on an attached artifact which is not being installed. Automatic RPM requires generator then tries to generate requires on artifact which is not being installed. This would most likely result in a broken RPM package so generator halts the build.

There are usually two possible solutions for this problem:

Some Maven artifacts try to depend on exact system paths. Most usually this dependency is either on com.sun:tools or sun.jdk:jconsole. Dependencies with system scope cause issues with our tooling and requires generators so they are not supported.

Easiest way to solve this for above two dependencies is by removing and adding back the dependency without <scope> or <systemPath> nodes:

%add_maven_depmap macro was used to manually install Maven artifacts that were built with Apache Ant or mvn-rpmbuild. It is now deprecated and its invocations should be replaced with %mvn_artifact and %mvn_install.

Artifact files, Maven POM files and their installation directories no longer need to be manually installed, since that is done during run of %mvn_install. The installed files also don’t need to be explicitly enumerated in %files section. Generated file .mfiles should be used instead.

Relevant parts of specfile using %add_maven_depmap:

The same specfile migrated to %mvn_artifact and %mvn_install:

%add_maven_depmap had -a switch to specify artifact aliases and -f switch to support splitting artifacts across multiple subpackages. To achieve the same things with %mvn_* macros, see Additional Mappings and Assignment of the Maven Artifacts to the Subpackages.

Unresolved directive in sections.adoc - include::{EXAMPLE}manpages.adoc[]