Release Management Blog

Thoughts and facts on shipping quality software by the team that ships Firefox every 6 to 8 weeks

Firefox Release Management at FOSDEM 2018

2018-01-11 12:00:00 +0000

Fosdem logo

Like every year, Brussels will host on February 3 & 4 the largest European Open Source event gathering thousands of developers, namely FOSDEM.

Sylvestre, head of the Firefox Release Management team, will explain on Saturday how we ship Firefox to hundreds of millions of users every 6 weeks and how our test suites, pre-release channels, static analyzers, data mining on uplifts, code coverage, fuzzers and community feedback are part of the tools and processes Mozilla uses to ship Firefox every 6 weeks.

Here are the details of the conference:

Firefox: How to ship quality software
6000 new patches, a release every 6 weeks, how Mozilla does it?
Speaker Sylvestre Ledru
Track Mozilla devroom
Room UA2.118 (Henriot)
Day Saturday
Start 14:00
End 14:30

Several members of the Release Management team as well as other Mozilla employees from other departments and Mozilla volunteers will be at Fosdem so don’t hesitate to come and talk to us at the Mozilla Developper Room or at our booth.

In addition to the official schedule for our devroom, our Mozilla Wiki page for Fosdem 2018 contains all the information you may need regarding our participation to this event.

Last but not least, we would like to thank our Belgian Mozilla community, particularly Anthony Maton and Ziggy Maes for organizing our participation at Fosdem!

Janitor project - Newsletter #9

2017-11-29 13:00:00 +0000

Jan Keromnes is a Senior Software Engineer working for the Release Management team on tools and automation and is the lead developer for Janitor.

Janitor offers developer environments as a service for Firefox, Servo and other open source projects. It uses Cloud9 IDE (front-end), Docker servers (back-end), and is 100% web-based so you can jump straight into fresh on-demand environments that are pre-configured and ready for work, without wasting time setting up yet another local checkout or VM.

This newsletter was initially published on the Janitor blog. You can contact the Janitor community on their Discourse forum.

Hi there,

This is your recurrent burst of good news about the Janitor. Thank you ever so much for being part of this community. It really means a lot.

1) Announcing Windows Environments

Janitor is great for quickly fixing platform-specific bugs in your projects, especially if you don’t normally develop on that platform. Today, we only provide Linux containers (Ubuntu 16.04) but many of you asked for native Windows environments on Janitor, so that’s exactly what we plan to give you.

We want to make it easy for you to work on all operating systems, without the hassle of setting up a VM or maintaining a dual boot. In fact, you won’t even need to install anything other than a good web browser (like Firefox Quantum) because our Windows environments will be accessible from the web, with a graphical VNC environment, just like our current Linux containers.

We’re looking into Windows VMs on Azure and TaskCluster workers on AWS. If Mozilla plays along, you should see Windows environments for Firefox on Janitor within just a few months. (If you can help us get there faster, please let us know here, here or here.)

2) Announcing Janitor 0.0.9

So much has happened this year that it was hard to find time to write about our progress. This version bump was long overdue.

Here is a quick rundown of what we did since July:

  • Now serving Cloud9 directly from Janitor ( account no longer required)
  • Made both IDE and VNC load much faster (thanks to better browser caching)
  • Improved Docker proxy allows working in multiple containers at the same time
  • Added the Discourse open source project to Janitor (thanks notriddle)
  • Added janitor.json configuration files to automate your project’s workflows on Janitor (thanks ntim)
  • Added a “Reviews” IDE sidebar with code review comments you need to address (thanks ntim)
  • Added two new Docker servers to our cluster (thanks IRILL for the much needed sponsorship upgrade)
  • Now pulling automated Docker image builds (thanks to Docker Hub and CircleCI)
  • Expanded our API to manage Docker containers (to create / inspect / delete containers and image layers)
  • Created a Docker administration page to efficiently manage our container farm
  • Cleaner UI and more controls in our “Projects” and “Containers” pages (thanks ntim, Coder206 and fbeaufort)
  • Dropped the “The” in “The Janitor” because it’s cleaner (thanks arshad)
  • Refreshed Firefox, Servo and Chromium project logos (thanks Coder206, arshad and ntim)
  • Switched Firefox (hg) from mozilla-central to mozilla-unified (thanks ntim)
  • Upgraded to Git 2.15.0
  • Upgraded to Mercurial 4.4.1
  • Upgraded to Clang 5.0 and replaced Gold with LLD (links Firefox 2x faster)
  • Upgraded to Rust 1.22.1 / 1.23.0-nightly (installed via rustup 1.7.0)
  • Upgraded to Node.js 8.9.1 and npm 5.5.1 (now installed via nvm 0.33.6)
  • Upgraded to Ninja 1.8.2 (now with bash completion)
  • Upgraded to rr 5.0.0
  • Upgraded to the latest Vim 8 and Neovim
  • Installed the latest valgrind (for nbp)
  • Installed the latest tmux (for Paul Rouget)
  • … and many more improvements and bug fixes.

3) Our Cluster Just Got Bigger

Janitor is now used by over 400 developers and our hardware was starting to feel small, so IRILL upgraded their sponsorship, growing our cluster to a total of 6 servers (4 Docker hosts, including 3 at IRILL in Versaille and 1 at Mozilla in California, as well as 2 VPS web app hosts at OVH in Gravelines). This means that Janitor now runs on 42 CPUs, 120 GB RAM and 4 TB disk space.

Here is a picture of EtienneWan and I manually installing the new servers in IRILL’s data center near Paris.

You can really thank IRILL and Sylvestre for keeping us going! In the future we’ll make it much simpler for anyone to join our cluster, in order to accept many more open source projects and developers to Janitor.

4) Janitor Around the World

Here are some events we went to, or plan to attend:

  • Watch how Coder206 presented Janitor to Sudbury’s Google Developer Group, with a cool side-by-side comparison hacking on Servo.
  • Come see two Janitor lightning talks at Mozilla’s All Hands in Austin this December, in the Firefox Lightning Talks and Power tools for open source tracks.
  • Come hack on open source software with Janitor at INSA Lyon or 42 in Paris in just a few months (two hackathons to be announced).

5) Last Stretch to Beta

2017 has been such a wild ride. We significantly lowered the barrier to new contributions for several major open source projects, allowing many people to contribute for the first time to Firefox, Chromium, Servo, Thunderbird (and more), and we proved that it was possible to modernize software development at scale. Now we just need to finish a few more things before we can call our Alpha a resounding success.

In 2018, Janitor Beta will get us to the next level, with Windows environments (and maybe MacOS too); massive Docker scaling improvements; an open build farm that anyone can join; new open source partnerships; and more radical automation to make software development faster and more fun. More on that very soon.

And that’s a wrap for today. How is everything going? We’d love to know! Also our Discourse and IRC channel are great resources to ask questions and learn more about this project.

Stay safe, Team Janitor

P.S. One more thing: Here is a sneak peek at the beautiful new design that ntim, arshad and notriddle are working on for Janitor.

Dawn project - Update on the migration of the aurora populations

2017-05-30 08:30:23 +0000

Project Dawn will change the Firefox release train model by eliminating the Aurora phrase. (See the Hacks blog for more).

We have completed all of the required infrastructure changes and are now ready to migrate the Aurora populations for Firefox Desktop and Firefox for Android. We have different migration paths for these two products as described below.


For Firefox Desktop, the existing Developer Edition (Aurora) users should not notice any change. After the migration their build will still have the Developer Edition branding and the developer specific changes associated with the current Developer Edition build. The technical difference is that the build will be based off our the mozilla-beta branch and, as such, will have the same features and stability as the Beta channel.

We started proposing updates to Developer Edition clients this week. Clients will automatically be updated within a few days or you can force the update by going to the Help menu and opening the About Firefox window.


For Firefox for Android, the current Aurora population on Google Play will be merged with our existing Nightly population. This update will change the branding of the product. In order to make this change, we will now be publishing Nightly to Google Play. Technically, the Android application will keep the same application name (org.mozilla.fennec_aurora) as we do not have a way to change this without requiring manual intervention by the users of this product.

This change will help with the detection of Android specific issues earlier in the cycle so that we can mitigate the impact to Firefox users.

Dawn project or the end of Aurora

2017-04-17 08:30:23 +0000

As described in the post on the Hacks blog, we are changing the release mechanism of Firefox.


In order to address the complexity and cycle length issues, the release management team, in coordination with Firefox product management and engineering, is going to remove the Aurora stabilization phase from the cycle.


On April 18th, Firefox 55 will remain on Nightly. This means Firefox 55 will remain on Nightly for two full cycles. On June 13th, Firefox 55 will migrate directly from Nightly to Beta.


As originally intended, Aurora was to be the first stabilization channel having a user base 10x the size of Nightly so as to provide additional user feedback. This original intent never materialized.

The release cycle time has required that we subvert the model regularly over the years by uplifting new features to meet market requirements.


The stabilization cycle from Nightly to Release will be shortened by 6-8 weeks.

A staged rollout mechanism, similar to what we do today with Release, will be used for the first weeks of Beta.

Our engineering and release workflow will continue to have additional checks and balances rolled out to ensure we ship a high quality release.

We will focus on finding and fixing regressions during the Nightly cycle and alleviate time pressure to ship to reduce the 400-600 patches currently uplifted to Aurora.

A new feature will merge from Nightly to Beta only when it’s deemed ready, based on pre-established criteria determined by engineering, product, and product integrity.

Tooling such as static analysis, linters, and code coverage will be integrated into the development process

Dawn planning


What will happen to the Aurora population on Desktop?

The Aurora population will be migrated to the Beta update channel in April 2017. We plan to keep them on a separate “pre-beta” update channel as compared to the rest of the Beta population. We will use this pre-beta audience to test and improve the stability and quality of initial Beta builds until we are ready to push to 100% of beta population. Because we presented Aurora as a stable product in the past, the beta channel is the closest in terms of stability and quality.

From the next merge (April 18th), users running 54 Aurora will remain on the Aurora channel but updates will be turned off. In case of critical security issues, we might push new updates to these aurora channel users. Aurora channel users will be migrated to Beta channel in April ‘17. For this to happen, we need to make sure that the Developer Edition features are working the same way on the Beta update channel (theme, profile, etc).

What will happen to the Aurora population on Android?

Because Google play doesn’t allow the migration of a population from an application to another, the fennec population on aurora will be migrated to the nightly application. For now, we are planning to reuse the current Google play aurora application and replace it by Nightly to preserve the current population.

Why are we taking different approaches with the Desktop and Android Aurora populations?

Aurora channel on Desktop has been around for a long time and has a substantial end-user base that Beta channel will benefit from.

Fennec Aurora on Google Play is a recent addition and we believe merging this audience with Nightly makes more sense. It also simplifies implementation. !

I am running Developer Edition, what will happen to me?

Developer Edition, currently based off Aurora, will be updated to get builds from the Beta branch. There is nothing Developer Edition users need to do, they will update automatically to the Beta build keeping the Developer Edition themes, tools, and preferences as well as the existing profile.

Will I still be able to test add-ons with Developer Edition?

Yes. You can also test unsigned add-ons on Nightly builds or load WebExtensions temporarily in Beta and Release builds.

We are also continuing to provide unbranded builds of the beta and release branches which are able to run unsigned add-ons - including bootstrapped - for development and experimentation. These versions will not be verified by QE, but will receive updates , which is an improvement to the unbranded builds we currently provide for add-on development.

The majority of Developer Edition users won’t experience any disruption.

How will you mitigate the quality risk from cutting 6-8 weeks of stabilization from the cycle?

Instead of pushing to 100 % of the beta population at once, we will use a staged rollout mechanism to push to a subset of the beta population. For the first phase, we will be pushing to the former aurora population. As a second phase, we will be targeting specific populations (Operating system, graphic card, etc)

In parallel, QE will also do preliminary nightly sign off to detect early new potential issues. Release management will be much more aggressive in term of feature deactivation.

Last but not least, the aurora cycle was used to finalize some features. Instead, feature stabilization will be performed during the nightly cycle.

What are we doing to improve Nightly quality?

To improve the overall quality of nightly, a few initiatives will help.

Nightly merge criteria

New end-user facing features landing in Nightly builds should meet Beta-readiness criteria before they can be pushed to Beta channel.

Static analyzers

In order to detect issues at review phase, static analyzers will be integrated as part of the workflow. They will be able to identify potential defects but also limit the technological debt.

Code coverage

Code coverage results are going to be used to analyze the quality of the testsuite and the risk introduced by the change.

Risk assessment

By correlating various data sources (VCS, Bugzilla, etc), we believe we can identify the potential risks carried by changes before they even land. The idea is to identify the functions where a modification has more chance to induce a regression.

How often will Beta builds be updated?

We will continue to push two Beta builds for Desktop and one Fennec build each week of the Beta cycle.

Will Developer Edition continue to have a separate profile?

Yes. The Developer Edition separate profile feature is a requirement for transition. If for whatever reason this feature cannot be completed by the end of the year we will need to return to creating rebuilds of Developer Edition as previously done to ensure those users are not cast away.

What will happen to the Aurora branch after Firefox 54 moves to Beta?

Updates on aurora channel will be disabled on April 18th. The desktop and Android aurora populations will be migrated as described above.

What criteria will be used to assess feature readiness to move to Beta?

We will be monitoring crash rates, QE’s sign offs, telemetry data and new regressions to determine overall Nightly quality and feature readiness to merge to Beta.

How and who will determine whether a feature is ready to move to Beta?

End-user facing features will be reviewed for beta-readiness before they are pushed to Beta channel. Following is a list of criteria that will be used to evaluate feature readiness to merge to Beta:

  • No significant stability Issues
  • Missing Test Plans
  • Insufficient Testing
  • Feature is not Code Complete
  • Too Many Open Bugs

More detailed criteria defined in this document.

Are there any changes to Release or ESR channel?

No changes are planned for Release or ESR channel users.

Does this change how frequently we push mainline builds to Release channel?

No, but changes added in Nightly can make it into a Release build about 6-8 weeks sooner than they do now.

What will happen for l10n process when we remove Aurora?

Focus for localization will move from mozilla-aurora to mozilla-central. Localization tools (Pootle and Pontoon) will read en-US strings from a special mozilla-central clone: l10n-drivers will review patches with strings landing in the official mozilla-central repository, provide feedback to devs if necessary, and land updates every 2-3 days in this special repository. Localized content will be pushed to l10n-central repositories.

There are no changes for developers working on Firefox: Nightly and mozilla-central remain open to string changes, including the extra six weeks that Firefox 55 will spend in Nightly, while Beta is still considered string frozen, and requests to uplift changes affecting strings are evaluated case by case.

Users interested in helping with localization should download Nightly in their language.

What will happen for l10n process by the end of year?

For Firefox and Firefox for Android we will shift to a model with a single repository for all channels for each locale. This change will be reflected in localization tools, allowing localizers to make a change to a string and see that update applied across all channels at once.

How does Dawn impact engineering planning for landing features?

The biggest shift is that features will have to be completed before merge day. Developers will not be able to finalize feature development during the next branch cycle (as Aurora is used currently). See also “How and who will determine whether a feature is ready to move to Beta?”.

How will bug fixes and features not tracked by project management be impacted by Dawn?

Landing bug fixes in Nightly repository continues as before. Development on features that are not directly end-user visible and not tracked by EPMs, release management continues as before.

If Nightly quality and stability is negatively impacted by these untracked features or bug fixes, we will discuss potential mitigation options such as: back outs, stabilizing quality issues before continuing new feature development work, delaying Merge date, imposing code freeze in Nightly until blocking issues are resolved, etc.

What will happen to the diagnostic assert?

MOZ_DIAGNOSTIC_ASSERT will enabled during the first part of the beta cycle. It will be automatically disabled when EARLY_BETA_OR_EARLIER is no longer defined.