Some time ago, in a discussion about incrementing versions in CI builds, I suggested that a release version should be defined by a Product Owner and should not be incremented in CI builds, and that a build number should be used instead to track packages produced by CI pipelines.

One of the participants commented that I must be working in a corporate environment and that most Open Source projects do not have a Product Owner and build numbers are not something that a small team would use. This point of view is incredibly misguided, but it is also, unfortunately, quite widespread.

Product Owners are us

It seems at times that the one thing agile evangelists achieved is that they firmly planted in many heads that agile development is something convoluted, mired in endless meetings, suitable only for corporate teams, and is not the approach a small dynamic Open Source team could ever benefit from.

A Product Owner is not some corporate overlord, but simply a person who defines what the application will be able to do when it is released. Just about anyone who has ever created a source repository and took time to figure out what goes into the next release is, effectively, a Product Owner.

A Product Owner needs a backlog to maintain a prioritized list of desired application capabilities, which may be a fully-functional backlog provided by services like Azure DevOps, or simplified one in a GitHub project, or a stack of sticky notes on the Product Owner's desk.

A release happens when a sufficient number of desired application capabilities from the backlog have been implemented, where "sufficient" is also defined by the Product Owner.

Releases

For the purposes of maintaining release versions, two distinct approaches may be outlined.

In a more traditional approach, the Product Owner defines the scope of a release and plans a set of desired application capabilities to be implemented. Versions of such releases are typically indicative of the level of compatibility between different versions.

For example, a REST API library v1.3.0 may add new API methods, compared to v1.2.3 of the same library, but will not break any existing methods. Similarly, an image editing software v3.17.0 is expected to read a list of image modifications saved by the v3.2.1 of the same application. This, essentially, is what Semantic Versioning promotes.

Time-boxed releases, on the other hand, are done on a fixed schedule and include whatever the development team was able to implement and test within a predefined time frame.

For example, an online shopping application may have a base functionality released at some point as v1.0.0, and each two weeks new functionality may be added, such as introducing new payment methods (v1.1.0), a way to save a shopping cart for a later review (v1.2.0), a way to communicate with a seller (v1.3.0), and so on.

This is where version auto-increments come in and where they may get confused with auto-incremented build numbers.

Release Versions

Regardless of the release approach, release versions serve two distinct purposes.

• A version uniquely identifies a set of features available within a publicly released product.
• A version communicates the level of compatibility between other versions of the same product.

Version components are expected to be incremented monotonically and if somebody sees a version 1.2.0 and then the next one 1.5.0, with no releases in between, this will create confusion for users trying to upgrade from v1.2.0.

Versions in the traditional release approach are generally better understood and in most packages do communicate both of the points above, so package dependency patterns can be set up to pick up important fixes, such as security patches, automatically.

Versions for time-boxed releases will typically be auto-incremented for the given continuous delivery level, such as the minor version in Semantic Versioning notation and may, in theory, be maintained to satisfy both version requirements above, but in practice things get quite fuzzy for users relying on such packages.

For example, @aws-sdk/client-s3 is being released as a part of the aws-sdk-js-v3 framework, which includes multiple packages, all sharing the same version, so some version of a specific client, such as v3.385.0 of @aws-sdk/client-s3, may not even have any client-s3 changes because it was merely bundled with other clients that did.

Time-boxed releases may also be not as concerned with backward compatibility and in case if a bug is found in any of the released versions with a minor version increment, instead of releasing a patch for that bug, the cost of upgrading is often shifted onto the user, forcing them to take not only the bug fix, but also the other changes included into such release.

One important bit about time-boxed release versions that seems to escape many is that even though time-boxed release versions are auto-incremented, they are still defined by the Product Owner, who sets the cadence of time-boxed releases and decides what goes into each one. The auto-increment just reflects the fixed release schedule, not that the Product Owner is removed from the process.

Auto-incremented versions vs. build numbers

So, is there any overlap in auto-incremented time-boxed versions and auto-incremented build numbers? Well, none whatsoever, simply because they serve completely different purposes.

Release versions are intended for application users. Build numbers are intended for development, while a package is being prepared for a release.

Build numbers reflect package maturity towards a given release, whether it is a traditional release or a time-boxed release. There always will be multiple candidate packages for a given release version, regardless of the approach, each assigned a unique build number and tested to determine whether it is suitable for a release.

A version is typically maintained in the source, including possibly CI/CD pipeline source, which can pass the version into the build scripts and into the source during a build.

A build number, or more accurately build metadata, should be maintained outside the source and should never be included anywhere in the source repository. It may, however, be propagated into the source during builds, so a CLI tool, for example, could identify a specific build in which it was generated.

Moreover, there may be multiple build metadata sequences for the same source, some of which may be considered as release candidates and some may never be released, regardless whether they pass all the tests or not (e.g. packages built on a feature branch).

In practical terms, one would increment application version in the source as soon as the new version cycle starts, which means that all packages will be built by CI pipelines with that version and different build metadata from that point on. For a simple project without parallel development around a release in progress, including a possible patch branch created at some later time, it would look like this (release tags are in yellow, version bumps in the source are in light red).

Each package built by a CI pipeline will have the target release version and unique build metadata, and would go through one or more rounds of testing. The package that contains all features intended for that release and also passed all the tests, becomes a candidate package that may be released. When the package is released, the source commit used in the CI pipeline build that generated that package should be tagged with the release version.

Sometimes people rearrange these steps, such as tagging first and then building from a tag, but such changes make the build process more convoluted, such as having to move the release tag if any of the tests fail, etc.

Build metadata challenges

Build metadata is so commonly misunderstood that many tools that are supposed to be aware of build numbers, lack such support, forcing people to resort to various hacks, such as bumping version components instead of build numbers or releasing the same version with different build numbers.

For example, the Python wheel package format does provide a way to track build numbers, such as 27.main in a package name like this:

myapp-2.1.0-27.main-py3-none-any.whl

However, most Python build tools lack support for generating build numbers, so many development teams avoid using build numbers altogether.

npm is even more problematic for tracking build numbers because npm package designers fail to realize the difference between public package repositories, where build numbers should not exist, and development package repositories, where builds numbers are absolutely essential in the release process.

As a result of this npm design, development repositories, such as Azure Artifacts or AWS CodeArtifact will not allow overwriting CI packages with the same version and different build metadata, which is what a development package repository supposed to do, unlike the public ones, which are supposed to drop build metadata.

Sometimes it is possible to work around this deficiency by deleting the previous package using native environment tools, such as using aws codeartifact to delete package abc-1.2.3+35.main.tgz and publish abc-1.2.3+36.main.tgz, but some environments don't allow even that, such as Azure Artifacts, making them unusable for maintaining CI packages.

Muddling through

These challenges make maintaining build numbers more difficult for those who want to follow the most straightforward release approach, which is to keep building and testing CI packages internally, each identified by unique build metadata, until the final candidate package is built that is verified as a suitable one for a release.

Finding a working solution for maintaining build numbers may require some experimentation, based on the package type and DevOps environment. Some of these workarounds are better than others, but all of them are duct-tape solutions needed only because people building those tools don't understand how build numbers work.

For example, Python build module backends, like Setuptools, provide no way to build a binary package with a distinct build number. Fortunately, wheel tags --build can be used to repackage an existing wheel package with a build number, which is accepted in such development repositories as Azure Artifacts and AWS CodeArtifact, making it is possible to keep publishing and testing wheel packages in CI pipelines using standard tools.

Other package formats, such as npm or Nuget, provide no way to add build numbers to packages. Naturally, many development repositories follow this trend and won't accept same version and different build numbers, which makes Azure Artifacts useless in CI pipelines for these package types and requires extra work in AWS CodeArtifact to yank and republish packages.

The solution I ended up using for npm outside of AWS is to have the CI pipeline to rename the package file to include a build number and store it in the project ecosystem storage, such as an S3 bucket or an Azure storage account, and install it locally from a file in a way that can be used across different computers, such as from a common path or a parent directory, like this.

npm install ../mylib-1.2.0+34.tgz

, which is recorded in package.json similar to this:

"dependencies": {
  "mylib": "file:../mylib-1.2.0+34.tgz"
}

It is not as seamless as having a development repository, which would expose the latest package out of a pool of candidates with different build numbers, but it is a usable approach nevertheless that can be easily automated.

Final thoughts

At the heart of agile development is a streamlined path from ideas bouncing around in the backlog to a release with top ideas implemented and thoroughly tested. The power of agile development isn't in agile ceremonies, which often overshadow the real usefulness of the approach, but in that one can plan and manage work in a flexible and less expensive way, compared to the traditional waterfall development.

How versions and build numbers fit into this? They act as a litmus test of whether the team follows basic common-sense principles in developing and delivering their product or not. Somebody who neglects maintaining a meaningful version for their releases will likely be just as sloppy in other development aspects. Even more importantly, without using build metadata one cannot avoid having to repackage artifacts or even to rebuild them from the source, which negates some of the testing for such releases and makes them less robust.

A dynamic development team, even as small as one person, is still expected to produce maintainable releases, with predictable levels of changes between versions. Agile development is supposed to make it easier by bringing more common sense into the development process, so a small team can deliver products with comparable features and quality to those delivered by a 2-3 times larger waterfall team, not to justify cutting corners, just because that is how a small "non-corporate" team rolls.

Diagrams in this post are created with app.diagrams.net.