Release Management Blog

Thoughts and facts on shipping quality software by the team that ships Firefox every 6 to 8 weeks

Janitor project - Newsletter 10

2018-02-01 13:00:00 +0000

Jan Keromnes is a Senior Software Engineer working for the Release Management team on tools and automation and is the lead developer for Janitor.

Janitor offers developer environments as a service for Firefox, Servo and other open source projects. It uses Cloud9 IDE (front-end), Docker servers (back-end), and is 100% web-based so you can jump straight into fresh on-demand environments that are pre-configured and ready for work, without wasting time setting up yet another local checkout or VM.

This newsletter was initially published on the Janitor Discourse forum.

Happy 2018 everyone!

We hope you’ve had a smooth start into the year, and wish you all the best in your life and projects. This is your recurrent burst of good news about Janitor.

First Survey

We have big plans for 2018, and about 500 people now use Janitor to contribute to open source software. We’d love to understand what you’re getting out of Janitor, and what we could improve to make your life easier.

2018 Janitor Survey (should take < 3 minutes)

Please help us do our best work this year. In return, we’ll publicly share the stats and insights via our blog.

Towards Windows Support

Last month at Mozilla’s All Hands in Austin, we announced Windows environments in Janitor for mid-2018. You can watch the lightning talk and the slides online.

Since then, we’ve iterated on a prototype Windows image for Firefox (based on a Windows 10 VM in Azure) and we’re now looking into using Azure’s REST API to allow Janitor users to spawn and automatically configure new VMs based on our Firefox Windows image. This is similar to spawning and auto-configuring new Docker containers based on our Linux images today.

It’s still early days, but if you’re excited about Windows support, you can track our progress with the new Janitor Windows roadmap.

Announcing Janitor 0.0.10

We’ve improved, upgraded and extended Janitor in many cool ways. So much that the next release should hopefully take us from Alpha to Beta, which will bring even more exciting features, supported open source projects, users, speed, stability and scalability.

Here is what we did since 0.0.9 was released 4 months ago:

  • Quick preview URLs in the IDE (notriddle)
  • Improved Run scripts for most projects (janx)
  • Enabled collaborative editing in the IDE (janx)
  • New website design for Janitor to be released soon (ntim, arshad, notriddle)
  • New containers page with a cool SSH one-liner (ntim)
  • New blog page populated directly from our Discourse (notriddle)
  • New OVH1 Docker server, our most powerful yet (16 CPU, 64GB RAM, 2TB SSD)
  • Added the PeerTube project (janx, bnjbvr, Chocobozzz)
  • Added the Yuzu Emulator project (etiennewan)
  • Refactored most of our Node.js modules to async/await
  • Tested Janitor on an iPad and it works! (Flaki)
  • Supported UTF-8 in all recent containers
  • Supported multiple email addresses per user, allowing imports from GitHub
  • Supported validation functions and ‘*’ URL parameters in our self-testing API system
  • Latest LLVM toolchain (clang 6.0, lld 6.0, lldb 6.0)
  • Latest Rust toolchain (stable 1.23.0, nightly 1.25.0)
  • Latest Git (2.16.1)
  • Latest Mercurial (4.4.1)
  • Latest Node.js (node 8.9.4, npm 5.6.0, nvm 0.33.8)
  • Latest fd (6.2.0)
  • Latest rg (0.7.1)
  • Latest rr (5.1.0)
  • Latest Vim 8 + latest Neovim
  • Latest Cloud9 SDK and noVNC
  • … plus many more upgrades, bug fixes, stability and performance improvements

And that’s a wrap! As always, please feel free to stop by our IRC channel and Discourse forum to learn more about this project. We’d love to meet you.

Thanks for your time!

Team Janitor

Firefox Release Management at FOSDEM 2018

2018-01-11 12:00:00 +0000

Fosdem logo

Like every year, Brussels will host on February 3 & 4 the largest European Open Source event gathering thousands of developers, namely FOSDEM.

Sylvestre, head of the Firefox Release Management team, will explain on Saturday how we ship Firefox to hundreds of millions of users every 6 weeks and how our test suites, pre-release channels, static analyzers, data mining on uplifts, code coverage, fuzzers and community feedback are part of the tools and processes Mozilla uses to ship Firefox every 6 weeks.

Here are the details of the conference:

Firefox: How to ship quality software
6000 new patches, a release every 6 weeks, how Mozilla does it?
Speaker Sylvestre Ledru
Track Mozilla devroom
Room UA2.118 (Henriot)
Day Saturday
Start 14:00
End 14:30

Several members of the Release Management team as well as other Mozilla employees from other departments and Mozilla volunteers will be at Fosdem so don’t hesitate to come and talk to us at the Mozilla Developper Room or at our booth.

In addition to the official schedule for our devroom, our Mozilla Wiki page for Fosdem 2018 contains all the information you may need regarding our participation to this event.

Last but not least, we would like to thank our Belgian Mozilla community, particularly Anthony Maton and Ziggy Maes for organizing our participation at Fosdem!

Janitor project - Newsletter #9

2017-11-29 13:00:00 +0000

Jan Keromnes is a Senior Software Engineer working for the Release Management team on tools and automation and is the lead developer for Janitor.

Janitor offers developer environments as a service for Firefox, Servo and other open source projects. It uses Cloud9 IDE (front-end), Docker servers (back-end), and is 100% web-based so you can jump straight into fresh on-demand environments that are pre-configured and ready for work, without wasting time setting up yet another local checkout or VM.

This newsletter was initially published on the Janitor blog. You can contact the Janitor community on their Discourse forum.

Hi there,

This is your recurrent burst of good news about the Janitor. Thank you ever so much for being part of this community. It really means a lot.

1) Announcing Windows Environments

Janitor is great for quickly fixing platform-specific bugs in your projects, especially if you don’t normally develop on that platform. Today, we only provide Linux containers (Ubuntu 16.04) but many of you asked for native Windows environments on Janitor, so that’s exactly what we plan to give you.

We want to make it easy for you to work on all operating systems, without the hassle of setting up a VM or maintaining a dual boot. In fact, you won’t even need to install anything other than a good web browser (like Firefox Quantum) because our Windows environments will be accessible from the web, with a graphical VNC environment, just like our current Linux containers.

We’re looking into Windows VMs on Azure and TaskCluster workers on AWS. If Mozilla plays along, you should see Windows environments for Firefox on Janitor within just a few months. (If you can help us get there faster, please let us know here, here or here.)

2) Announcing Janitor 0.0.9

So much has happened this year that it was hard to find time to write about our progress. This version bump was long overdue.

Here is a quick rundown of what we did since July:

  • Now serving Cloud9 directly from Janitor ( account no longer required)
  • Made both IDE and VNC load much faster (thanks to better browser caching)
  • Improved Docker proxy allows working in multiple containers at the same time
  • Added the Discourse open source project to Janitor (thanks notriddle)
  • Added janitor.json configuration files to automate your project’s workflows on Janitor (thanks ntim)
  • Added a “Reviews” IDE sidebar with code review comments you need to address (thanks ntim)
  • Added two new Docker servers to our cluster (thanks IRILL for the much needed sponsorship upgrade)
  • Now pulling automated Docker image builds (thanks to Docker Hub and CircleCI)
  • Expanded our API to manage Docker containers (to create / inspect / delete containers and image layers)
  • Created a Docker administration page to efficiently manage our container farm
  • Cleaner UI and more controls in our “Projects” and “Containers” pages (thanks ntim, Coder206 and fbeaufort)
  • Dropped the “The” in “The Janitor” because it’s cleaner (thanks arshad)
  • Refreshed Firefox, Servo and Chromium project logos (thanks Coder206, arshad and ntim)
  • Switched Firefox (hg) from mozilla-central to mozilla-unified (thanks ntim)
  • Upgraded to Git 2.15.0
  • Upgraded to Mercurial 4.4.1
  • Upgraded to Clang 5.0 and replaced Gold with LLD (links Firefox 2x faster)
  • Upgraded to Rust 1.22.1 / 1.23.0-nightly (installed via rustup 1.7.0)
  • Upgraded to Node.js 8.9.1 and npm 5.5.1 (now installed via nvm 0.33.6)
  • Upgraded to Ninja 1.8.2 (now with bash completion)
  • Upgraded to rr 5.0.0
  • Upgraded to the latest Vim 8 and Neovim
  • Installed the latest valgrind (for nbp)
  • Installed the latest tmux (for Paul Rouget)
  • … and many more improvements and bug fixes.

3) Our Cluster Just Got Bigger

Janitor is now used by over 400 developers and our hardware was starting to feel small, so IRILL upgraded their sponsorship, growing our cluster to a total of 6 servers (4 Docker hosts, including 3 at IRILL in Versaille and 1 at Mozilla in California, as well as 2 VPS web app hosts at OVH in Gravelines). This means that Janitor now runs on 42 CPUs, 120 GB RAM and 4 TB disk space.

Here is a picture of EtienneWan and I manually installing the new servers in IRILL’s data center near Paris.

You can really thank IRILL and Sylvestre for keeping us going! In the future we’ll make it much simpler for anyone to join our cluster, in order to accept many more open source projects and developers to Janitor.

4) Janitor Around the World

Here are some events we went to, or plan to attend:

  • Watch how Coder206 presented Janitor to Sudbury’s Google Developer Group, with a cool side-by-side comparison hacking on Servo.
  • Come see two Janitor lightning talks at Mozilla’s All Hands in Austin this December, in the Firefox Lightning Talks and Power tools for open source tracks.
  • Come hack on open source software with Janitor at INSA Lyon or 42 in Paris in just a few months (two hackathons to be announced).

5) Last Stretch to Beta

2017 has been such a wild ride. We significantly lowered the barrier to new contributions for several major open source projects, allowing many people to contribute for the first time to Firefox, Chromium, Servo, Thunderbird (and more), and we proved that it was possible to modernize software development at scale. Now we just need to finish a few more things before we can call our Alpha a resounding success.

In 2018, Janitor Beta will get us to the next level, with Windows environments (and maybe MacOS too); massive Docker scaling improvements; an open build farm that anyone can join; new open source partnerships; and more radical automation to make software development faster and more fun. More on that very soon.

And that’s a wrap for today. How is everything going? We’d love to know! Also our Discourse and IRC channel are great resources to ask questions and learn more about this project.

Stay safe, Team Janitor

P.S. One more thing: Here is a sneak peek at the beautiful new design that ntim, arshad and notriddle are working on for Janitor.

Dawn project - Update on the migration of the aurora populations

2017-05-30 08:30:23 +0000

Project Dawn will change the Firefox release train model by eliminating the Aurora phrase. (See the Hacks blog for more).

We have completed all of the required infrastructure changes and are now ready to migrate the Aurora populations for Firefox Desktop and Firefox for Android. We have different migration paths for these two products as described below.


For Firefox Desktop, the existing Developer Edition (Aurora) users should not notice any change. After the migration their build will still have the Developer Edition branding and the developer specific changes associated with the current Developer Edition build. The technical difference is that the build will be based off our the mozilla-beta branch and, as such, will have the same features and stability as the Beta channel.

We started proposing updates to Developer Edition clients this week. Clients will automatically be updated within a few days or you can force the update by going to the Help menu and opening the About Firefox window.


For Firefox for Android, the current Aurora population on Google Play will be merged with our existing Nightly population. This update will change the branding of the product. In order to make this change, we will now be publishing Nightly to Google Play. Technically, the Android application will keep the same application name (org.mozilla.fennec_aurora) as we do not have a way to change this without requiring manual intervention by the users of this product.

This change will help with the detection of Android specific issues earlier in the cycle so that we can mitigate the impact to Firefox users.