Development Model
- mainline: The main tree where the core functionality and core features are being developed.
- subsystem/feature branch: is a branch within the same repository. In our case, we will use the term branch also when referencing branches not in the same repository, which are a copy of a repository sharing the same history.
- upstream: A parent branch the source code is based on. This is the branch you pull from and push to, basically your upstream.
- LTS: Long Term Support
The Zephyr project releases on a time-based cycle, rather than a feature-driven one. Zephyr releases represent an aggregation of the work of many contributors, companies, and individuals. A time-based release process will enable the Zephyr project to provide users with a balance of the latest technologies and features and excellent overall quality. A roughly 3-month release cycle allows us to coordinate development of the features that have actually been implemented, allowing us to maintain the quality of the overall release without delaying everything because of one or two features that are not ready yet. The Zephyr release model is loosely based on the Linux kernel model:
- Release tagging procedure:
- linear mode on master,
- release branches for stabilization
- Each release period will consist of a merge window period followed by one or more release candidates on which only stabilisation changes, bug fixes and documentation can be merged in.
- Merge window mode: all changes are accepted (subject to the different SMEs ACK)
- When the merge window is closed, gatekeeper lays a vN-rc1 tag and we enter the release candidate phase
- CI sees the tag, builds and runs tests; QA analyses the report from the build and test run and gives an ACK/NAK to the build
- Gatekeeper, with QA and any other needed input, determines if the release candidate is a go for release
- If it is a go for release, gatekeeper lays a tag release vN at the same point
- Development on new features continues in feature branches. Once features are ready, they are submitted to mainline during the merge window period.
A relatively straightforward discipline is followed with regard to the merging of patches for each release. At the beginning of each development cycle, the "merge window" is said to be open. At that time, code which is deemed to be sufficiently stable (and which is accepted by the development community) is merged into the mainline tree. The bulk of changes for a new development cycle (and all of the major changes) will be merged during this time.
The merge window lasts for approximately two months. At the end of this time, the gatekeeper will declare that the window is closed and release the first of the "rc". For the code base release which is destined to be 0.4.0, for example, the release which happens at the end of the merge window will be called 0.4.0-rc1. The -rc1 release is the signal that the time to merge new features has passed, and that the time to stabilise the next release of the code base has begun.
Over the next weeks, only patches which fix problems should be submitted to the mainline. On occasion, a more significant change will be allowed, but such occasions are rare; developers who try to merge new features outside of the merge window tend to get an unfriendly reception. As a general rule, if you miss the merge window for a given feature, the best thing to do is to wait for the next development cycle. (An occasional exception is made for drivers for previously unsupported hardware; if they touch no in-tree code, they cannot cause regressions and should be safe to add at any time).
As fixes make their way into the mainline, the patch rate will slow over time. The mainline gatekeeper releases new -rc drops once or twice a week; a normal series will get up to somewhere between -rc4 and -rc6 before the code base is considered to be sufficiently stable and the final 0.4.x release is made.
At that point the whole process starts over again.
Here is the description of the various moderation levels:
- Low:
- Major New Features
- Bug Fixes
- Refactoring
- Structure/Directory Changes
- Medium:
- Bug Fixes, all priorities
- Enhancements
- Minor “self-contained” New Features
- High:
- Bug Fixes: P1 and P2
- Documentation + Test Coverage
The following syntax should be used for releases and tags in Git.
- Release [Major].[Minor].[Patch]
- Release Candidate [Major].[Minor].[Patch]-rc[RC]
- Tagging:
- v[Major].[Minor].[Patch]-rc[RC]
- v[Major].[Minor].[Patch]
- v[Major].[Minor].99 - A tag applied to master branch to signify that work on v[Major].[Minor+1] hs started. For example, v1.7.99 will be tagged at the start of v1.8 process. The tag corresponds to VERSION_MAJOR/VERSION_MINOR/PATCHLEVEL macros as defined for a work-in-progress master version. Presence of this tag allows to receive sensible output for "git describe" on master, as often used for automated builds and CI tools.
Long term support releases are designed to be supported for a longer than normal period and will be the basis for products and certification for various usages. Frequency and maintenance model of LTS released of Zephyr are still TBD.
An LTS release will be branched and maintained independently of the mainline tree. Changes and fixed flow in both directions, however, changes from master to LTS branch will be minimal to fixes that apply to both branches and for existing features only. All fixes for LTS that apply to the mainline tree are pushed to mainline as well.
Gerrit is intended to provide a framework for reviewing every commit before it is accepted into the code base. Changes are uploaded to Gerrit but don’t actually become a part of the project until they’ve been reviewed and accepted. Gerrit is used to support the standard open source practice of submitting patches which are then reviewed by the project members before being applied to the code base. The Zephyr project uses Gerrit for code reviews, and GIT tree management. When submitting a change or an enhancement to any Zephyr component developer should use Gerrit.
All changes submitted to Gerrit are subject to sanity tests that are run on emulated platforms and architectures to identify breakage and regressions that can be immediately identified and additionally performs build tests of a representative number of boards and platforms (subject to hardware availability in the CI infrastructure). Any failures found during the CI test run will result in a negative review. Developers are expected to fix issues and rework their patches and submit again. Any code pushed directly to the GIT trees will not go through this integration testing and might cause regressions. Pushing code directly to Gerrit is currently is disabled for all users and maintainers. The CI infrastructure currently runs the following tests:
- Run checkpatch for code style issues (can vote -1 on errors)
- Run footprint checks on basic kernel code (sends a report if footprint has changed)
- Run sanitycheck script:
- Run kernel tests in QEMU (can vote -1 on errors)
- Build various samples for different boards (can vote -1 on errors)
- Future: run tests on real hardware attached to CI
Feature or enhancement proposals for Zephyr are used to propose a new design or modify an existing feature and get it approved. Currently, the developer mailing list is used to make such proposals. To better integrate with the current tools used in the project and to enable traceability and archiving of such proposals the following is required:
- A well-defined and unified format for enhancement proposals
- Central repository for maintaining such proposals (JIRA)
- A well-defined process for reviews and approval
Maintain such proposals in a document format (similar to the documentation format used in the kernel source tree, using restructured Text) in a Git repository managed using Gerrit. The review and approval process of Gerrit could be used for approving such proposals as well, so there will be no need for a dedicated tool.
- Open an Improvement JIRA ticket with an abstract and the proposal itself
- Send an RFC with the abstract to the mailing list with an abstract and the link to the detailed proposal
- Approve/Reject/Defer based on feedback and comments
To maintain traceability and relation between proposals, changes, features and issues we have established a system that allows cross-referencing a commit with a JIRA entry and vice versa. Any changes that originate from a tracked feature or issue should contain a reference to the feature.
At any time it should be possible to establish the origin of a change and the reason behind it by following the references in the code.
The Zephyr project mailing lists are important tools used as the primary communication tool by project member, contributors, and the community. The mailing list is open for topics related to the project and should be used for collaboration among team members working on the same feature or subsystem or for discussion project direction and daily development of the code base. In general, bug reports and issues should be entered and tracked in the bug tracking system (JIRA) and not broadcasted to the mailing list, the same applies to code reviews. Code should be submitted to Gerrit and using the appropriate tools. To keep the community involved and informed summaries of JIRA and Gerrit traffic are posted to the mailing list on daily basis.
Development in branches before features go to mainline allows teams to work independently on a subsystem or a feature, improves efficiency and turnaround time and encourages collaboration and streamlines communication between developers. Changes submitted to a development branch can evolve and improve incrementally in a branch before they are finally submitted to the mainline tree for final integration. By dedicating an isolated branch to each feature or subsystem, it’s possible to initiate in-depth discussions around new additions before integrating them into the official project.
Individual code changes and fixes can still be submitted to the main tree without going through a development branch, however, such changes can be redirected to a branch owner if the change is deemed to be part of a subsystem that is maintained in a branch and needs review from subject matter experts working in a branch. This might be needed to avoid conflicts when changes to the same files and subsystems happen at the same time.
For example, a change to the networking should be submitted to the networking branch, however, a quick fix to a Makefile or a README that are not part of any particular subsystem can be submitted to the mainline tree directly.
Development branch owners have the following responsibilities:
- Use the infrastructure and tools provided by the project (Gerrit, Git)
- Review changes coming from team members and request review from branch owners when submitting changes.
- Keep the branch in sync with upstream and update on regular basis.
- Push changes frequently to upstream using the following methods:
- Gerrit changes: for example, when reviews have not been done in local branch (one-man branch).
- Merge requests: When a set of changes has been done in a local branch and has been reviewed and tested in a feature branch.
The following main subsystems have been identified. Further breakdown or additional domains or subsystems are possible and TBD.
Subsystem maintainers have privileges to rebase their branch and perform push --force operations, i.e. the ability to edit & overwrite the commit history. This is a powerful tool, and should therefore be used responsibly. The only situation where using push --force is allowed is when rebasing against the master branch (including fixing up of any conflicts related to the rebase). All normal patches must follow the gerrit review process so that they are auditable.
Force push access also comes with a potential race condition if there are multiple people with merge access to the subsystem branch: If one maintainer is doing gerrit reviews and merging patches while another one is doing a rebase and push --force, there is a risk of patches being lost. Because of this, a maintainer doing rebase/force push must communicate with other maintainers so they know to stay clear from the "merge" button.
Maintainers should rebase to master frequently, especially when there are important fixes in master. It is also good practice to perform a rebase as soon as a subsystem -> master merge has happened, so that commit IDs in both branches are in sync. The process of rebasing to master is roughly as follows:
Make sure local tree is in sync with upstream (assumed to be 'origin'):
$ git remote update origin
Set the local HEAD to point at upstream subsystem head:
$ git reset --hard origin/subsystem
Perform the rebase (and fix any ensuing conflicts):
$ git rebase origin/master
Push the resulting tree to the upstream branch:
$ git push --force origin HEAD:subsystem
As mentioned, it's important that no other subsystem maintainer does merges during these steps.
Subsystem maintainers should create merge requests to master periodically while the merge window is open. The length of each period is not fixed, but depends on the amount of activity in the subsystem branch. The period should preferably not be longer than a few weeks, but may be much shorter (a few days) if there are lots of patches going into the subsystem branch, or if there is a risk of conflict with master.
The method of merging to master involves creating a merge commit and pushing it to the refs/for/master branch:
Make sure local tree is in sync with upstream (assumed to be 'origin'):
$ git remote update origin
Set the local HEAD to point at upstream master branch:
$ git reset --hard origin/master
Create the merge commit (--no-ff forces the creation even if fast-forward is possible):
$ git merge --no-ff origin/subsystem
The above step results in opening an editor for the commit message. At this point it doesn't matter what the message is since we need to anyway come back to it later to add a full change-log, Change-Id and Signed-off-by lines.
Prepare a commit history & diffstat for the commit message. Putting this in the commit message is important since the gerrit web interface doesn't provide it:
$ git request-pull origin/master subsystem origin/subsystem
From the output of the above command, copy everything starting with the '---------' line to your paste buffer. After that, go back to editing the commit message, making sure to also add a Signed-off-by line (with the help of -s):
$ git commit -s --amend
While in the commit message editor, paste the copied text from the request-pull output. Above the '-----' line there should be the commit message summary line and a short high-level summary of the changes included with the merge request. The commit message could look something like:
Merge subsystem branch into master
Main changes:
- Add feature X
- Fix for Y
-------------------------------------------------------------
Person A
....
Person B
....
<diffstat>
Change-Id: ...
Signed-off-by: ...
Once the commit message is complete, the merge commit can be pushed to the master branch for review:
$ git push origin HEAD:refs/for/master
If the merge commit is rejected by the master branch maintainer, the above steps will need to be redone (after fixing whatever needed to be fixed in the subsystem branch first). However, it's important to retain the original Change-Id (through e.g. copy-paste) so that it's easy to see/track that the new merge commit is a new revision of the first one.
Once the merge commit has been merged into master it's good practice to perform the rebase procedure on the subsystem branch, so that git commit IDs in both branches are in sync.