Release 1.0.0, new repository structure, and new driver versioning scheme

[akudiyar]

Introduction

It has been a very productive 2.5 years for the Tarantool (Cartridge) driver and its adoption and importance for the Tarantool ecosystem have been constantly increasing all this time. Today it is used in several libraries and it is deployed in dozens of production servers. The customers continue launching projects based on the driver and filing bugs and improvement requests. It is also important that the driver is a successful open-source project that attracts users outside of the Tarantool team and allows them to contribute, build their own libraries and solutions, and learn more about Tarantool. Not less important is the fact that open-source ecosystem components for Tarantool help it increase its value and presence in the spectrum of other open-source database tools used by other developers and companies.

This driver is still one of the few that provides full support for both the standalone and cluster Tarantool installations. For historical reasons, it was named after the cluster orchestrator library Cartridge, to not mix it with the oldest driver variant for the single Tarantool instance - tarantool-java, currently not maintained anymore. However, the driver was not designed to comply with only one orchestrator variant - it was intended that any future cluster orchestrators and changes to the Tarantool server protocol should also be reflected in the driver versions. So its current name doesn't look like matching the purposes of further development and support of the other cluster orchestrator libraries.

Until the current driver code gathered a critical mass of production deployments and while its API and internal design were in active development, it was natural to follow the early-semver release enumeration scheme, for making it clear for the users that the API can change between versions. But now the API is closer to its crystallization and also the existing deployments need clearer change notifications and version migration guidelines. So it is obvious now that we need to move forward to establishing a new stable release cycle.

There is also a need of having separate driver packages, one that includes the core open-source functionality and another that includes the closed-source extra functionality that works only with Tarantool Enterprise. The driver structure needs to be changed towards being more pluggable and extensible to support such separation. Also, a huge portion of extensions like built-in discovery and retrying algorithms should not really be a part of the driver core, and separating them from the core API will make it smaller and easier to maintain, fix and evolve.

1. Proposal: new driver name

For the purposes of changing the driver name, we need to change the following things, in this sequence:

1. Repository name in Github - the changes will be handled automatically, and the old repository URL will redirect to the new URL
2. Repository name in pom.xml and documentation (and test resources if necessary)
3. Make the necessary structure refactoring (see the "new repository structure" section below)
4. Create a new branch for the current version (v1)
5. Fix the current version (1.0.0) and make a release
6. Change the package name and spin up a new version (2.0.0) in the master branch
7. Update the dependent libraries, following the same steps.

Creating a new repository for the new driver version with the new name looks like a worse option because it has the following disadvantages:

• automatic redirect from the old repository will be missing
• registration of the new repository in Maven will be required
• automatic upgrades for existing users will be harder to implement
• driver source indexation in the search engines will be lost
• and in general, another source will increase entropy and will mislead the customers, that will continue to take the old driver versions by mistake when they can take the latest ones.

I propose the following naming schema (👉 discussion point):

• io.tarantool.driver and io.tarantool.driver-core for the artifact names (see the "new repository structure" section below)
• tarantool-java-driver for the repository (this naming pattern is very common, see "mongo-java-driver" for example)
• the Java package name (io.tarantool.driver) does not need to be changed

2. Proposal: new repository structure and driver artifacts

I propose the following new artifacts' structure for the driver packages (👉 discussion point):

• the current driver code goes "as is" to the legacy directory - this is needed for maintaining the bugfix releases for version 1.0.0 until all major customers upgrade and we decide to drop this support
• the current driver code is separated into two parts: core and extra, each represented in the top-level directory with a normal Java archetype structure, with the former containing the API and core parts, and the latter containing additional classes that provide some built-in API implementations and helpers
• the enterprise part, also a top-level directory that is a git submodule referencing a private repository
• other top-level directories include at least integration test sources, benchmark script sources, documentation, and bom (bill of materials) - the final directory structure is TBD. Separating integration tests and scripts from the core sources will make them lighter and their dependency trees less error-prone.
• the top-level pom.xml will describe all the above as modules and will provide the dependency and plugin management, see an example in the Cassandra driver.

It will be possible to produce the following artifacts from that structure:

• tarantool-driver-core - from the sources in core directory, includes APIs and the driver core classes
• tarantool-driver - includes classes from both core and extra directories, the outcome will be equivalent to the current driver package (with everything without Enterprise parts)
• tarantool-driver-enterprise - includes everything from tarantool-driver plus the Tarantool Enterprise edition classes. Producing this artifact will be possible only with access to the private repository with Enterprise classes, however, the absence of such access will not prevent testing and building the other (open-source) artifacts.

There will be a separate target in the pom.xml for building each of the artifacts, with the "enterprise" target available only if the "enterprise" private submodule is checked out. The 1.0.0 version bugfix releases will be built from the legacy folder if necessary.

3. Proposal: new versioning scheme

We need to follow the Tarantool releases, but continue supporting the old ones as long as it is possible within the same driver version as we have been doing, clearly making it visible to the users in what versions the incompatible changes are going to be introduced and with what Tarantool version which driver version is compatible.

Goals of the new versioning scheme and the new release cycle:

• to provide a clear understanding for the users of what they can expect for each version change
• to produce a major "release train" version of several artifacts simultaneously with "one button click"
• to be able to release intermediate compatible versions for each artifact meanwhile (e.g. with bugfixes)
• to automate the releases as much as possible

I propose the following new versioning scheme for the driver artifacts (👉 discussion point):

• version contains 3 numbers (major.minor.bugfix) and an optional suffix, that may include words like beta, RC, and RELEASE. Build helper plugin can be used for that. A good example of the suffixes usage is TBD.
• the major part of the version changes every time we make a release with the next major Tarantool version support.
• the minor part of the version changes every time we add new features and support for the new features from the upcoming major Tarantool version. The support of new features must be enabled in the client only when the Tarantool server supports it.
• artifacts that share the same major version should be compatible with each other (can be substituted in the user application POM without any necessary changes).
• if we need to drop some Tarantool version support, we need to first deprecate it in one major version, and only then remove it in the next major version. When we deprecate something, we write warnings in the logs and explicitly state that in the changelogs and documentation.
• minor version changes if we add bugfixes or it is a technical release without any new features.
• each time we publish a new major version release, a new branch is created in the repository. That will allow committing urgent bugfixes to it, while not stopping the development of the next major version release.
• each time we publish a release, we create a tag on the branch for it (as we do it now).
• version must not be changed manually, it should be changed only by the build and release Maven plugins

Example of a release sequence for an artifact with this scheme:

• 2.0.0-RELEASE - Tarantool 3.0 is supported (branch v2 is created)
• 3.0.0 - Added next new features and bugfixes
• 3.1.0 - Added new feature from the upcoming Tarantool 3.1
• 3.1.1 - Added a bugfix and a deprecation warning for Tarantool 1.0
• 2.0.1-RELEASE - Added a bugfix to the branch v2 and a new tag is created on the branch
• 3.1.2-RC - Technical release candidate version
• 3.1.3-RELEASE - Tarantool 3.1 is supported, Tarantool 1.0 support is dropped (branch v3 is created)

[ArtDu]

the current driver code goes "as is" to the legacy directory - this is needed for maintaining the bugfix releases for version 1.0.0 until all major customers upgrade and we decide to drop this support

We just can save old code as is in 1ver branch. But we wouldn't have this in master

Proposal: new repository structure and driver artifacts

Do you suggest something like that? I think I don't get it.

[akudiyar]

We just can save old code as is in 1ver branch. But we wouldn't have this in master

That will not work well, like if we fix something in the code that is actually the same in both versions - it will be cumbersome and not good to fix that in two places. Many other drivers and libraries also follow that path of building several versions from one directory.

I remember how often I was banging my head when I had to backport and merge some new fixes to old branches of software. That approach does not work very well. Meanwhile, the legacy code should exist only until all major clients are updated to 1.0.0.

Do you suggest something like that? I think I don't get it.

That's a good suggestion, and I basically want to discuss what should be present in each part of the driver