Release Management Blog Thoughts and facts on shipping quality software <br> by the team that ships Firefox every 4 weeks
https://release.mozilla.org
2023-10-24T14:35:17+00:00Firefox Regional feedback: Let's start with Europe2022-11-18T00:00:00+00:00https://release.mozilla.org/firefox/community/2022/11/18/local-feedback-for-firefox<p>We work hard as an organization to ship the best browser possible every 4 weeks with about 1000 new patches per release.</p>
<p>We ship new features to make our browser useful and easy to use. We also do platform work to be able to render new sites and web applications while remaining compatible with millions of websites created a decade (or more) ago.</p>
<p>This ongoing work also includes updating our translations in more than 100 languages thanks to our impressive community of localizers.</p>
<p>Yes, we want to make sure that Firefox can be used everywhere by everybody.</p>
<p><strong>But could we maybe do better with a tighter feedback loop from our local communities?</strong></p>
<p>Let’s give a few examples from our bug tracker:</p>
<ul>
<li>If <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1729232">spellchecking in French makes my browser crash</a>, this is a major bug for French-speakers, but this is an invisible issue for others.</li>
<li>If the <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1636100">Estonian e-card system stops working in Firefox</a>, this is a blocking issue that may drive our users in this country to switch browsers.</li>
<li>If the <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1661895">keyboard shortcut for Zoom-In stops working on some Japanese keyboards</a>, this is also a bad quality of life regression for some of our users that could have been avoided had we known about it before shipping.</li>
<li>If we build a new feature on Firefox Nightly which <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1798112">breaks a major Italian newspaper website</a>, it’s good to know it early in the development process to make sure this is fixed before the feature goes live.</li>
</ul>
<p>Usually, major issues that may impact users in a specific country are fixed before we ship the final release, but occasionally we discover them after shipping and have to ship a fix in a <a href="https://wiki.mozilla.org/Release_Management/Glossary#Dot_release">dot release</a>.</p>
<h3 id="we-talked-about-significant-breakage-with-a-regional-impact-but-what-about-papercuts">We talked about significant breakage with a regional impact, but what about papercuts?</h3>
<p>Web compatibility, incorrect translations, internationalization issues, PiP subtitles support, certificates… The list of potential problems in a region that may affect our users is long and we may not know about them.</p>
<p>Maybe these issues are discussed in places we don’t know about, in languages we don’t speak. Maybe these issues are already filed in our bug tracker but don’t get prioritized correctly because we don’t know about their regional impact. Maybe a handful of specific regional issues are making Firefox hard to use in a specific country and the information is out there. Maybe all we need is somebody who understands these issues to surface these bugs in Bugzilla to our developers.</p>
<p><em>In a nutshell, we don’t know what we don’t know.</em></p>
<p>That is why I intend to work on studying and setting up basic feedback mechanisms to evaluate the health of Firefox in a few European countries so as to help my team (Release Management) prioritize product fixes for existing bugs which have the highest impact on our users and also to get help identifying regressions on our pre-release channels (Nightly, Beta, Developer Edition).</p>
<p>My very first goal is to make contacts with Mozillians in a handful of European countries (France, Germany, Italy, Poland, Spain) <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> that could help me identify issues that affect them locally, identify their top web compatibility issues, and maybe relay a general message for community feedback on pre-release channels.</p>
<p><strong>To that effect, I created a <a href="https://matrix.to/#/#local-firefox:mozilla.org">Local Firefox</a> room on the Mozilla Matrix instance.</strong>
If you are interested in collaborating with me on this project, you are very welcome to join it and say hello (my nick is <em>Pascal</em>). I can speak with you in French and Spanish as well if you don’t feel comfortable speaking in English.</p>
<p>Thanks!</p>
<p>Pascal</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>I am focusing on a few European countries for timezone and bandwidth reasons since I’ll do that alongside my role as a Firefox Release Manager, but I am open to feedback from other regions as well of course. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Code Coverage on Phabricator2019-01-21T00:00:00+00:00https://release.mozilla.org/tooling/codecoverage/2019/01/21/code-coverage-on-phabricator<p>We have recently implemented a solution to integrate code coverage results into Phabricator.</p>
<aside>This article was initially published on Marco Casteluccio's <a href="https://marco-c.github.io/2019/01/21/code-coverage-phabricator.html">blog</a>.</aside>
<p>Coverage information is uploaded either automatically for revisions after they are landed in mozilla-central (for example for release managers when looking at uplift requests), or on-demand for in-progress revisions.</p>
<p>For revisions under review, in order to upload coverage you just need to trigger a try push containing code coverage builds and tests, e.g. by using:</p>
<pre style="background-color:black;color:white;">
$ mach try fuzzy --full
</pre>
<p>and selecting the relevant ccov builds and test suites. In the future, we will also likely automatically trigger coverage try builds for revisions we deem to be risky, alongside the on-demand option.</p>
<p>Here is an example of a <a href="https://treeherder.mozilla.org/#/jobs?repo=try&revision=38213b49dc00cd108dfa9a246045ed677c34de91">try build</a> which produced the coverage information for <a href="https://phabricator.services.mozilla.com/D14758">my revision</a>:</p>
<figure>
<img src="/images/posts/codecoverage_phabricator/code-coverage-on-phabricator-try-build.png" alt="Try build which produced the coverage information" />
<figcaption><b>Figure 1:</b> Try build which produced the coverage information.</figcaption>
</figure>
<p><br /></p>
<p>Once the try build and linked tests finish, the coverage artifacts get parsed and uploaded to the Phabricator revisions corresponding to the commits in the try push. The analysis works on <em>all</em> commits in the try push that are linked to Phabricator revisions. Stacks of revisions are supported as well.</p>
<p>The coverage information is shown on Phabricator both at a high-level view, in the <em>Revision Contents</em> section, and at a detailed view in the <em>Diff</em> section.</p>
<p>The <em>Revision Contents</em> section contains a list of the files modified by the revision, showing both the coverage percentage of the whole file and the coverage percentage of touched lines. Here’s the screenshot of the section from my revision:</p>
<figure>
<img src="/images/posts/codecoverage_phabricator/code-coverage-on-phabricator-revision-contents.png" alt="Code coverage summary in the 'Revision Contents' section on Phabricator" />
<figcaption><b>Figure 2:</b> Code coverage summary in the 'Revision Contents' section on Phabricator.</figcaption>
</figure>
<p><br /></p>
<p>The <em>Diff</em> section instead shows the coverage line per line, coloring the bar on the right-hand side. <span style="color:#d86"><strong>Orange</strong></span> corresponds to <span style="color:#d86"><strong>uncovered lines</strong></span>, <span style="color:#def"><strong>light blue</strong></span> corresponds to <span style="color:#def"><strong>non-executable lines</strong></span> (e.g. a comment, a definition, and so on), <span style="color:#6bf"><strong>dark blue</strong></span> corresponds to <span style="color:#6bf"><strong>covered lines</strong></span>. When hovering the bar, the corresponding line is highlighted in the same color.
The following screenshots show excerpts of my revision, with covered, uncovered and non-executable lines:</p>
<figure>
<img src="/images/posts/codecoverage_phabricator/code-coverage-on-phabricator-covered-line.png" alt="Example of an added line which was covered by tests" />
<figcaption><b>Figure 3:</b> Example of an added line which was covered by tests.</figcaption>
</figure>
<p><br /></p>
<figure>
<img src="/images/posts/codecoverage_phabricator/code-coverage-on-phabricator-uncovered-line.png" alt="Example of a line which was not covered by tests" />
<figcaption><b>Figure 4:</b> Example of a line which was not covered by tests.</figcaption>
</figure>
Uplift forms get a refresh2018-10-03T08:00:00+00:00https://release.mozilla.org/tooling/bugzilla/2018/10/03/uplift-form<p>Firefox is shipped using a train model. Without going into too much details, this means that we maintain several channel in parallel (Nightly, Beta, Release and ESR). Normal changes happen in Nightly. When a change needs to be cherry-picked from Nightly to another branch, the process is called “Uplift”.</p>
<p>Uplifting is a key tool in the Firefox release management world. When developers want to apply a patch from Nightly to another branch, they will use Bugzilla, answering some questions in a textarea. Then, release managers will make a risk assessment to accept or reject the uplift.
As an example, release managers will see the following comment:</p>
<p><img src="/images/posts/uplift-form/old.png" alt="Uplift form" title="Uplift form" class="center-image" /></p>
<p>The release and quality management team is plugging more and more automation (and Machine Learning in the future) in Bugzilla, and the freeform textarea was making it more difficult (also because developers are free to do anything they want with the prefilled text, even deleting fields).
For this reason, we are moving to a typical form directly in the Bugzilla interface. <a href="https://github.com/mozilla-bteam/bmo/pull/756">The change</a>, developed by <a href="https://mozillians.org/u/kohei.yoshino">Kohei</a> who is volunteering as a <a href="https://twitter.com/BugzillaUX">Bugzilla UX</a> designer, <a href="https://dylan.hardison.net/2018/10/02/happy-bmo-push-day-mojolicious-edition/">has been deployed yesterday</a> (October 2nd)</p>
<p>A screenshot is a better explanation than words:</p>
<p><img src="/images/posts/uplift-form/new.png" alt="The new uplift form" title="The new uplift form" class="center-image" /></p>
<p>Once submitted, the comment will be displayed just like before!</p>
<p>We are planning to move to a similar system for tracking and release notes requests.</p>
<p>As always, don’t hesitate to share feedback to release-mgmt@mozilla.com</p>
Zero coverage report2018-03-23T07:00:00+00:00https://release.mozilla.org/tooling/codecoverage/2018/03/23/code-coverage<p>One of the nice things we can do with code coverage data is looking at which files are completely not covered by any test.</p>
<aside>This article was initially published on Marco Casteluccio's <a href="https://marco-c.github.io/2018/03/27/zero-coverage-reports.html">blog</a>.</aside>
<p>These files might be interesting for two reasons. Either they are:</p>
<ol style="list-style-type:lower-alpha; list-style-position: inside;">
<li>dead code;</li>
<li>code that is not tested at all.</li>
</ol>
<p>Dead code can obviously be removed, bringing a lot of advantages for developers and users alike:</p>
<ul>
<li>Improve maintainability (no need to update the code in case of refactorings in the case of dead code);</li>
<li>Reduce build time for developers and CI;</li>
<li>Reduce the attack surface;</li>
<li>Decrease the size of the resulting binary which can have effects on performance, installation duration, etc.</li>
</ul>
<p>Untested code, instead, can be really problematic. Changes to this code can take more time to be verified, require more QA resources, and so on. In summary, we can’t trust them as we trust code that is properly tested.</p>
<p>A study from <a href="https://www.youtube.com/watch?v=NKEptA3KP08">Google Test Automation Conference 2016</a> showed that an uncovered line (or method) is twice as likely to have a bug fix than a covered line (or method).
On top of that, testing a feature prevents unexpected behavior changes.</p>
<p>Using these reports, we have managed to <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1415819">remove a good amount of code</a> from mozilla-central, so far around 60 files with thousands of lines of code. We are confident that there’s even more code that we could remove or conditionally compile only if needed.</p>
<p>As any modern software, Firefox relies a lot on third party libraries. Currently, most (all?) the content of these libraries is built by default. For example,~400 files are untested in the gfx/skia/ directory).</p>
<p>Reports (updated weekly) can be seen at <a href="https://marco-c.github.io/code-coverage-reports/">https://marco-c.github.io/code-coverage-reports/</a>.
It allows filtering by language (C/C++, JavaScript), filtering out third-party code or header files, showing completely uncovered files only or all files which have uncovered functions (sorted by number of uncovered functions).</p>
<p><img src="/images/posts/codecoverage/uncovered_files.png" alt="uncovered code" title="Uncovered Files" class="center-image" /></p>
<p>Currently there are 2730 uncovered files (2627 C++, 103 JavaScript), 557 if ignoring third party files. As our regular code coverage reports on <a href="https://codecov.io/gh/marco-c/gecko-dev">codecov.io</a>, these reports are restricted to Windows and Linux platforms.</p>
Crash-Stop, an extension to help handle crashes on Bugzilla2018-03-20T07:00:00+00:00https://release.mozilla.org/tooling/crashstop/2018/03/20/crashstop<p><b>Crash-stop</b> is a webextension I wrote for Bugzilla to display crash stats by builds and patch information.</p>
<p>The goal is to have enough information to be able to decide if a patch helped (hence its name) and, if needed, uplift it to the Beta/ESR/Release trains as appropriate.</p>
<p>This project was initially meant to assist release-managers but it’s been useful for developers who fix/monitor crashes or for folks doing bug triage.</p>
<p>A screen snapshot of crash-top from <a href="https://bugzilla.mozilla.org/1432409">bug 1432409</a> (in the “Details” section):</p>
<p><img src="/images/posts/crashstop/bug1432409.png" alt="Crash stop table" title="Crash stop table" class="center-image" /></p>
<h2 id="how-to-read-the-data-in-the-table-above">How to read the data in the table above?</h2>
<ul>
<li>The patches landed in beta on the 2018-02-20 at 23:40</li>
<li>The buildid of b12 is 20180222170353 and of b11 is 20180219114835</li>
<li>The first beta build containing the patches is b12.</li>
<li>The builds which don’t contain the patches are shown in pink</li>
<li>The builds that contain the patch are shown in green.</li>
</ul>
<p>As you can see from the example above, the patches had a very positive effect for the first 2 signatures.</p>
<p>For release channel, the builds are shown in light yellow because no patches were found for that channel (the addon reads all the comments to try to find the push urls). As is obvious in this example, the reassuring data from Beta channel makes for a strong case to request an uplift to release channel.</p>
<p>Recently, I added stuff to show startup crashes, for example in <a href="https://bugzilla.mozilla.org/1435779">bug 1435779</a>:</p>
<p><img src="/images/posts/crashstop/bug1435779.png" alt="Crash stop table" title="Crash stop table" class="center-image" /></p>
<h2 id="recent-updates">Recent updates:</h2>
<ul>
<li>The cells are colored in red when more than 50% of the crashes have the flag startup_crash set to true (on each number in <em>Crashes</em> rows there is a tooltip with the percentage of startup_crash == true).</li>
<li>I added icons for impacted platforms.</li>
<li>Click on signatures or versions to get more information from <a href="https://crash-stats.mozilla.com/">Socorro</a>.</li>
</ul>
<p>All feedback is welcome and appreciated! If you want to request features or more data, or report an error, please feel free to file a bug on GitHub.</p>
<h2 id="source-code-and-extension-download">Source Code and extension download</h2>
<p>The extension can be <a href="https://addons.mozilla.org/firefox/addon/bugzilla-crash-stop/">installed from AMO</a> and the development is done on <a href="https://github.com/mozilla/crashstop">GitHub</a>, pull requests are also welcome!</p>
Janitor project - Newsletter 102018-02-01T13:00:00+00:00https://release.mozilla.org/tooling/janitor/2018/02/01/janitor-newsletter-10<p>Jan Keromnes is a Senior Software Engineer working for the Release Management team on tools and automation and is the lead developer for Janitor.<br /><br /><a href="https://janitor.technology/">Janitor</a> offers developer environments as a service for Firefox, Servo and <a href="https://janitor.technology/projects/">other open source projects</a>. It uses <a href="https://c9.io/">Cloud9 IDE</a> (front-end), <a href="https://www.docker.com/">Docker</a> servers (back-end), and is 100% web-based so you can jump straight into fresh on-demand environments that are pre-configured and ready for work, without wasting time setting up yet another local checkout or VM.<br /><br />This newsletter was initially published on the <a href="https://discourse.janitor.technology/t/janitor-news-10/">Janitor Discourse forum</a>.</p>
<p>Happy 2018 everyone!</p>
<p>We hope you’ve had a smooth start into the year, and wish you all the best in your life and projects.
This is your recurrent burst of good news about <a href="https://janitor.technology">Janitor</a>.</p>
<h2 id="first-survey">First Survey</h2>
<p>We have big plans for 2018, and about 500 people now use Janitor to contribute to open source software. We’d love to understand what you’re getting out of Janitor, and what we could improve to make your life easier.</p>
<p><a href="https://goo.gl/forms/FGjYPybIpfUN3M0m2">2018 Janitor Survey</a> (should take < 3 minutes)</p>
<p>Please help us do our best work this year. In return, we’ll publicly share the stats and insights via our <a href="https://janitor.technology/blog/">blog</a>.</p>
<h2 id="towards-windows-support">Towards Windows Support</h2>
<p>Last month at Mozilla’s All Hands in Austin, we announced Windows environments in Janitor for mid-2018. You can watch the <a href="https://www.youtube.com/watch?v=jHomOSa_WEc">lightning talk</a> and the <a href="https://docs.google.com/presentation/d/1gWu2Q2ejd4eZSuMylHu6W5xuJ89xch3Dve0qek0jlPU/edit?usp=sharing">slides</a> online.</p>
<p>Since then, we’ve iterated on a prototype Windows image for Firefox (based on a <a href="https://azuremarketplace.microsoft.com/en-ca/marketplace/apps/Microsoft.Windows10RS3ProNx64-ARM?tab=Overview">Windows 10 VM</a> in Azure) and we’re now looking into using Azure’s <a href="https://docs.microsoft.com/en-ca/rest/api/compute/virtualmachines">REST API</a> to allow Janitor users to spawn and automatically configure new VMs based on our Firefox Windows image. This is similar to spawning and auto-configuring new Docker containers based on our <a href="https://janitor.technology/projects/">Linux images</a> today.</p>
<p>It’s still early days, but if you’re excited about Windows support, you can track our progress with the new <a href="https://github.com/JanitorTechnology/janitor/projects/4">Janitor Windows roadmap</a>.</p>
<h2 id="announcing-janitor-0010">Announcing Janitor 0.0.10</h2>
<p>We’ve improved, upgraded and extended Janitor in many cool ways. So much that the next release should hopefully take us from Alpha to <a href="https://github.com/JanitorTechnology/janitor/milestone/2">Beta</a>, which will bring even more exciting features, supported open source projects, users, speed, stability and scalability.</p>
<p>Here is what we did since 0.0.9 was released 4 months ago:</p>
<ul>
<li>Quick <a href="https://twitter.com/jankeromnes/status/938446519530901504">preview URLs</a> in the IDE (notriddle)</li>
<li>Improved <a href="https://github.com/JanitorTechnology/dockerfiles/commit/6d3e7835c90fafe76a2e8463b4d5104ac0a39a9d">Run scripts</a> for most projects (janx)</li>
<li><a href="https://github.com/JanitorTechnology/janitor/issues/182">Enabled</a> collaborative editing in the IDE (janx)</li>
<li>New <a href="https://janitor.technology/design/">website design</a> for Janitor to be released soon (ntim, arshad, notriddle)</li>
<li>New <a href="https://janitor.technology/containers-new/">containers page</a> with a cool SSH one-liner (ntim)</li>
<li>New <a href="https://janitor.technology/blog-new/">blog page</a> populated directly from our <a href="https://discourse.janitor.technology/t/newsletters-composed-in-discourse-and-published-in-janitor/100">Discourse</a> (notriddle)</li>
<li>New <a href="https://ovh1.janitor.technology/">OVH1</a> Docker server, our most powerful yet (16 CPU, 64GB RAM, 2TB SSD)</li>
<li><a href="https://twitter.com/jankeromnes/status/956529784569331712">Added</a> the <a href="https://github.com/Chocobozzz/PeerTube">PeerTube</a> project (janx, bnjbvr, Chocobozzz)</li>
<li><a href="https://github.com/yuzu-emu/yuzu/issues/35">Added</a> the <a href="https://github.com/yuzu-emu/yuzu">Yuzu Emulator</a> project (etiennewan)</li>
<li>Refactored most of our Node.js modules to async/await</li>
<li>Tested Janitor on an iPad and <a href="https://twitter.com/mozhacks/status/941802182860406784">it works!</a> (Flaki)</li>
<li>Supported UTF-8 in all recent containers</li>
<li>Supported multiple email addresses per user, allowing imports from GitHub</li>
<li>Supported validation functions and ‘*’ URL parameters in our <a href="https://github.com/JanitorTechnology/selfapi">self-testing API system</a></li>
<li>Latest LLVM toolchain (clang 6.0, lld 6.0, lldb 6.0)</li>
<li>Latest Rust toolchain (stable 1.23.0, nightly 1.25.0)</li>
<li>Latest Git (2.16.1)</li>
<li>Latest Mercurial (4.4.1)</li>
<li>Latest Node.js (node 8.9.4, npm 5.6.0, nvm 0.33.8)</li>
<li>Latest <a href="https://github.com/sharkdp/fd">fd</a> (6.2.0)</li>
<li>Latest <a href="https://github.com/BurntSushi/ripgrep">rg</a> (0.7.1)</li>
<li>Latest <a href="http://rr-project.org/">rr</a> (5.1.0)</li>
<li>Latest Vim 8 + latest Neovim</li>
<li>Latest Cloud9 SDK and noVNC</li>
<li>… plus many more upgrades, bug fixes, stability and performance improvements</li>
</ul>
<p>And that’s a wrap! As always, please feel free to stop by our <a href="https://kiwiirc.com/client/irc.freenode.net/?#janitor">IRC channel</a> and <a href="https://discourse.janitor.technology">Discourse forum</a> to learn more about this project. We’d love to meet you.</p>
<p>Thanks for your time!</p>
<p>Team Janitor</p>
Firefox Release Management at FOSDEM 20182018-01-11T12:00:00+00:00https://release.mozilla.org/events/fosdem/2018/01/11/fosdem2018<p><img src="/images/posts/fosdem/fosdem2018banner.png" alt="Fosdem logo" title="Fosdem logo" class="center-image" /></p>
<p>Like every year, Brussels will host on February 3 & 4 the largest European Open Source event gathering thousands of developers, namely <a href="https://fosdem.org/2018/">FOSDEM</a>.</p>
<p>Sylvestre, head of the Firefox Release Management team, will explain on Saturday how we ship Firefox to hundreds of millions of users every 6 weeks and how our test suites, pre-release channels, static analyzers, data mining on uplifts, code coverage, fuzzers and community feedback are part of the tools and processes Mozilla uses to ship Firefox every 6 weeks.
<!-- more -->
Here are the details of the conference:</p>
<table style="width: 60%; margin: auto; background-repeat: no-repeat; background-position: cover;background-image: linear-gradient(rgba(255, 255, 255, 0.9), rgba(255, 255, 255, 0.9)), url('/images/logos/firefox-logo.svg');">
<tr>
<th colspan="2"><a href="https://fosdem.org/2018/schedule/event/mozilla_how_ship_quality_software/">
Firefox: How to ship quality software</a><br /><small style="font-size: smaller">6000 new patches, a release every 6 weeks, how Mozilla does it?</small>
</th>
</tr>
<tr>
<th>Speaker</th>
<td><a href="https://mozillians.org/fr/u/sylvestre/">Sylvestre Ledru</a></td>
</tr> <tr>
<th>Track</th>
<td><a href="https://fosdem.org/2018/schedule/track/mozilla/">Mozilla devroom</a></td>
</tr>
<tr>
<th>Room</th>
<td><a href="https://fosdem.org/2018/schedule/room/ua2118_henriot/">UA2.118 (Henriot)</a></td>
</tr>
<tr>
<th>Day</th>
<td><a href="https://fosdem.org/2018/schedule/day/saturday/">Saturday</a></td>
</tr>
<tr>
<th>Start</th>
<td><a href="https://fosdem.org/2018/schedule/day/saturday/#1400">14:00</a></td>
</tr>
<tr>
<th>End</th>
<td><a href="https://fosdem.org/2018/schedule/day/saturday/#1430">14:30</a></td>
</tr>
</table>
<p><br />
Several members of the Release Management team as well as other Mozilla employees from other departments and Mozilla volunteers will be at Fosdem so don’t hesitate to come and talk to us at the Mozilla Developper Room or at our booth.</p>
<p>In addition to the <a href="https://fosdem.org/2018/schedule/track/mozilla/">official schedule for our devroom</a>, our <a href="https://wiki.mozilla.org/Fosdem:2018">Mozilla Wiki page for Fosdem 2018</a> contains all the information you may need regarding our participation to this event.</p>
<p>Last but not least, we would like to thank our Belgian Mozilla community, particularly <a href="https://mozillians.org/fr/u/anthony/">Anthony Maton</a> and <a href="https://mozillians.org/fr/u/ZiggyMaes/">Ziggy Maes</a> for organizing our participation at Fosdem!</p>
Janitor project - Newsletter #92017-11-29T13:00:00+00:00https://release.mozilla.org/tooling/janitor/2017/11/29/janitor-newsletter-9<p class="postheader">Jan Keromnes is a Senior Software Engineer working for the Release Management team on tools and automation and is the lead developer for Janitor.<br /><br /><a href="https://janitor.technology/">Janitor</a> offers developer environments as a service for Firefox, Servo and <a href="https://janitor.technology/projects/">other open source projects</a>. It uses <a href="https://c9.io/">Cloud9 IDE</a> (front-end), <a href="https://www.docker.com/">Docker</a> servers (back-end), and is 100% web-based so you can jump straight into fresh on-demand environments that are pre-configured and ready for work, without wasting time setting up yet another local checkout or VM.<br /><br />This newsletter was initially published on the <a href="https://janitor.technology/blog/#janitor-news-9">Janitor blog</a>. You can contact the Janitor community on their <a href="https://discourse.janitor.technology/">Discourse forum</a>.</p>
<p>Hi there,</p>
<p>This is your recurrent burst of good news about the <a href="https://janitor.technology/">Janitor</a>.
Thank you ever so much for being part of this community. It really means a lot.</p>
<h3 id="1-announcing-windows-environments">1) Announcing Windows Environments</h3>
<p>Janitor is great for quickly fixing platform-specific bugs in your projects, especially if you don’t normally develop on that platform. Today, we only provide Linux containers (Ubuntu 16.04) but many of you asked for native Windows environments on Janitor, so that’s exactly what we plan to give you.</p>
<p>We want to make it easy for you to work on all operating systems, without the hassle of setting up a VM or maintaining a dual boot. In fact, you won’t even need to install anything other than a good web browser (like <a href="https://www.mozilla.org/en-US/firefox/new/">Firefox Quantum</a>) because our Windows environments will be accessible from the web, with a graphical VNC environment, just like our current Linux containers.</p>
<p>We’re looking into Windows VMs on Azure and TaskCluster workers on AWS. If Mozilla plays along, you should see Windows environments for Firefox on Janitor within just a few months. (If you can help us get there faster, please let us know <a href="https://github.com/JanitorTechnology/janitor/issues/3">here</a>, <a href="https://discourse.janitor.technology/t/windows-container-support-would-be-really-really-cool/85">here</a> or <a href="https://kiwiirc.com/client/irc.freenode.net/?#janitor">here</a>.)</p>
<h3 id="2-announcing-janitor-009">2) Announcing Janitor 0.0.9</h3>
<p>So much has happened this year that it was hard to find time to write about our progress. This version bump was long overdue.</p>
<p>Here is a quick rundown of what we did since July:</p>
<ul>
<li>Now serving Cloud9 directly from Janitor (<a href="https://c9.io">c9.io</a> account no longer required)</li>
<li>Made both IDE and VNC load much faster (thanks to better browser caching)</li>
<li>Improved Docker proxy allows working in multiple containers at the same time</li>
<li>Added the <a href="https://www.discourse.org/">Discourse</a> open source project to Janitor (thanks notriddle)</li>
<li>Added <a href="https://github.com/JanitorTechnology/dockerfiles/blob/master/firefox/janitor-hg.json">janitor.json</a> configuration files to automate your project’s workflows on Janitor (thanks ntim)</li>
<li>Added a “Reviews” IDE sidebar with code review comments you need to address (thanks ntim)</li>
<li>Added two new Docker servers to our cluster (thanks IRILL for the much needed sponsorship upgrade)</li>
<li>Now pulling automated Docker image builds (thanks to Docker Hub and CircleCI)</li>
<li>Expanded our <a href="https://janitor.technology/reference/api#containers">API</a> to manage Docker containers (to create / inspect / delete containers and image layers)</li>
<li>Created a Docker administration page to efficiently manage our container farm</li>
<li>Cleaner UI and more controls in our “Projects” and “Containers” pages (thanks ntim, Coder206 and fbeaufort)</li>
<li>Dropped the “The” in “The Janitor” because it’s cleaner (thanks arshad)</li>
<li>Refreshed Firefox, Servo and Chromium project logos (thanks Coder206, arshad and ntim)</li>
<li>Switched Firefox (hg) from mozilla-central to mozilla-unified (thanks ntim)</li>
<li>Upgraded to Git 2.15.0</li>
<li>Upgraded to Mercurial 4.4.1</li>
<li>Upgraded to Clang 5.0 and replaced Gold with LLD (links Firefox <a href="https://twitter.com/jankeromnes/status/935804934087421952">2x faster</a>)</li>
<li>Upgraded to Rust 1.22.1 / 1.23.0-nightly (installed via rustup 1.7.0)</li>
<li>Upgraded to Node.js 8.9.1 and npm 5.5.1 (now installed via nvm 0.33.6)</li>
<li>Upgraded to Ninja 1.8.2 (now with bash completion)</li>
<li>Upgraded to rr 5.0.0</li>
<li>Upgraded to the latest Vim 8 <em>and</em> Neovim</li>
<li>Installed the latest valgrind (for nbp)</li>
<li>Installed the latest tmux (for Paul Rouget)</li>
<li>… and many more improvements and bug fixes.</li>
</ul>
<h3 id="3-our-cluster-just-got-bigger">3) Our Cluster Just Got Bigger</h3>
<p>Janitor is now used by over 400 developers and our hardware was starting to feel small, so IRILL upgraded their sponsorship, growing our cluster to a total of 6 servers (4 Docker hosts, including 3 at IRILL in Versaille and 1 at Mozilla in California, as well as 2 VPS web app hosts at OVH in Gravelines). This means that Janitor now runs on 42 CPUs, 120 GB RAM and 4 TB disk space.</p>
<p>Here is a <a href="https://photos.app.goo.gl/QwHhLG56VY5mXy732">picture</a> of EtienneWan and I manually installing the new servers in IRILL’s data center near Paris.</p>
<p>You can really thank IRILL and Sylvestre for keeping us going! In the future we’ll make it much simpler for anyone to join our cluster, in order to accept many more open source projects and developers to Janitor.</p>
<h3 id="4-janitor-around-the-world">4) Janitor Around the World</h3>
<p>Here are some events we went to, or plan to attend:</p>
<ul>
<li>Watch how Coder206 <a href="https://youtu.be/OzbGW9unFdo">presented Janitor</a> to Sudbury’s Google Developer Group, with a cool side-by-side comparison hacking on Servo.</li>
<li>Come see two Janitor lightning talks at Mozilla’s All Hands in Austin this December, in the <a href="https://austinyallhands2017.sched.com/event/Cx7t/firefox-lightning-talks">Firefox Lightning Talks</a> and <a href="https://austinyallhands2017.sched.com/event/CwSj/28-lighting-talks-power-tools-for-open-source">Power tools for open source</a> tracks.</li>
<li>Come hack on open source software with Janitor at INSA Lyon or 42 in Paris in just a few months (two hackathons to be announced).</li>
</ul>
<h3 id="5-last-stretch-to-beta">5) Last Stretch to Beta</h3>
<p>2017 has been such a wild ride. We significantly lowered the barrier to new contributions for several major open source projects, allowing many people to contribute for the first time to Firefox, Chromium, Servo, Thunderbird (and <a href="https://janitor.technology/projects/">more</a>), and we proved that it was possible to modernize software development at scale. Now we just need to finish a <a href="https://github.com/JanitorTechnology/janitor/issues/166">few more things</a> before we can call our Alpha a resounding success.</p>
<p>In 2018, Janitor Beta will get us to the next level, with Windows environments (and maybe MacOS too); massive Docker scaling improvements; an open build farm that anyone can join; new open source partnerships; and more radical automation to make software development faster and more fun. More on that very soon.</p>
<p>And that’s a wrap for today. How is everything going? We’d love to know! Also our <a href="https://discourse.janitor.technology">Discourse</a> and <a href="https://kiwiirc.com/client/irc.freenode.net/?#janitor">IRC channel</a> are great resources to ask questions and learn more about this project.</p>
<p>Stay safe,
Team Janitor</p>
<p>P.S. One more thing: Here is a <a href="https://matrix.org/_matrix/media/v1/download/matrix.org/mTRLzIQmEEdMtlNXIbQiKGIP">sneak peek</a> at the beautiful new design that ntim, arshad and notriddle are working on for Janitor.</p>
Dawn project - Update on the migration of the aurora populations2017-05-30T08:30:23+00:00https://release.mozilla.org/firefox/release/2017/05/30/Dawn-update<p><a href="http://release.mozilla.org/firefox/release/2017/04/17/Dawn-Project-FAQ.html">Project Dawn</a> will change the Firefox release train model by eliminating the Aurora phrase. (See <a href="https://hacks.mozilla.org/2017/04/simplifying-firefox-release-channels/">the Hacks blog</a> for more).</p>
<p>We have completed all of the required infrastructure changes and are now ready to migrate the Aurora populations for Firefox Desktop and Firefox for Android. We have different migration paths for these two products as described below.</p>
<h1 id="desktop">Desktop</h1>
<p>For Firefox Desktop, the existing Developer Edition (Aurora) users should not notice any change. After the migration their build will still have the Developer Edition branding and the developer specific changes associated with the current Developer Edition build. The technical difference is that the build will be based off our the mozilla-beta branch and, as such, will have the same features and stability as the Beta channel.</p>
<p>We started proposing updates to <a href="https://www.mozilla.org/firefox/developer/">Developer Edition clients</a> this week. Clients will automatically be updated within a few days or you can force the update by going to the Help menu and opening the About Firefox window.</p>
<h1 id="android">Android</h1>
<p>For Firefox for Android, the current Aurora population on Google Play will be merged with our existing Nightly population. This update will change the branding of the product. In order to make this change, we will now be publishing Nightly to Google Play. Technically, the Android application will keep the same application name (<a href="https://play.google.com/store/apps/details?id=org.mozilla.fennec_aurora">org.mozilla.fennec_aurora</a>) as we do not have a way to change this without requiring manual intervention by the users of this product.</p>
<p>This change will help with the detection of Android specific issues earlier in the cycle so that we can mitigate the impact to Firefox users.</p>
Dawn project or the end of Aurora2017-04-17T08:30:23+00:00https://release.mozilla.org/firefox/release/2017/04/17/Dawn-Project-FAQ<p>As described in the <a href="https://hacks.mozilla.org/2017/04/simplifying-firefox-release-channels/">post on the Hacks blog</a>, we are changing the release mechanism of Firefox.</p>
<h1 id="what">What</h1>
<p>In order to address the complexity and cycle length issues, the release management team, in coordination with Firefox product management and engineering, is going to remove the Aurora stabilization phase from the cycle.</p>
<h1 id="when">When</h1>
<p>On April 18th, Firefox 55 will remain on Nightly. This means Firefox 55 will remain on Nightly for two full cycles. On June 13th, Firefox 55 will migrate directly from Nightly to Beta.</p>
<h1 id="why">Why</h1>
<p>As originally intended, Aurora was to be the first stabilization channel having a user base 10x the size of Nightly so as to provide additional user feedback. This original intent never materialized.</p>
<p>The release cycle time has required that we subvert the model regularly over the years by uplifting new features to meet market requirements.</p>
<h1 id="how">How</h1>
<p>The stabilization cycle from Nightly to Release will be shortened by 6-8 weeks.</p>
<p>A staged rollout mechanism, similar to what we do today with Release, will be used for the first weeks of Beta.</p>
<p>Our engineering and release workflow will continue to have additional checks and balances rolled out to ensure we ship a high quality release.</p>
<p>We will focus on finding and fixing regressions during the Nightly cycle and alleviate time pressure to ship to reduce the 400-600 patches currently uplifted to Aurora.</p>
<p>A new feature will merge from Nightly to Beta only when it’s deemed ready, based on pre-established criteria determined by engineering, product, and product integrity.</p>
<p>Tooling such as static analysis, linters, and code coverage will be integrated into the development process</p>
<p><img src="/images/posts/dawn-planning.png" alt="Dawn planning" title="Dawn planning" /></p>
<h1 id="faq">FAQ</h1>
<h2 id="what-will-happen-to-the-aurora-population-on-desktop">What will happen to the Aurora population on Desktop?</h2>
<p>The Aurora population will be migrated to the Beta update channel in April 2017. We plan to keep them on a separate “pre-beta” update channel as compared to the rest of the Beta population. We will use this pre-beta audience to test and improve the stability and quality of initial Beta builds until we are ready to push to 100% of beta population. Because we presented Aurora as a stable product in the past, the beta channel is the closest in terms of stability and quality.</p>
<p>From the next merge (April 18th), users running 54 Aurora will remain on the Aurora channel but updates will be turned off. In case of critical security issues, we might push new updates to these aurora channel users. Aurora channel users will be migrated to Beta channel in April ‘17. For this to happen, we need to make sure that the Developer Edition features are working the same way on the Beta update channel (theme, profile, etc).</p>
<h2 id="what-will-happen-to-the-aurora-population-on-android">What will happen to the Aurora population on Android?</h2>
<p>Because Google play doesn’t allow the migration of a population from an application to another, the fennec population on aurora will be migrated to the nightly application. For now, we are planning to reuse the current <a href="https://play.google.com/store/apps/details?id=org.mozilla.fennec_aurora">Google play aurora application</a> and replace it by Nightly to preserve the current population.</p>
<h2 id="why-are-we-taking-different-approaches-with-the-desktop-and-android-aurora-populations">Why are we taking different approaches with the Desktop and Android Aurora populations?</h2>
<p>Aurora channel on Desktop has been around for a long time and has a substantial end-user base that Beta channel will benefit from.</p>
<p>Fennec Aurora on Google Play is a recent addition and we believe merging this audience with Nightly makes more sense. It also simplifies implementation.
!</p>
<h2 id="i-am-running-developer-edition-what-will-happen-to-me">I am running Developer Edition, what will happen to me?</h2>
<p>Developer Edition, currently based off Aurora, will be updated to get builds from the Beta branch. There is nothing Developer Edition users need to do, they will update automatically to the Beta build keeping the Developer Edition themes, tools, and preferences as well as the existing profile.</p>
<h2 id="will-i-still-be-able-to-test-add-ons-with-developer-edition">Will I still be able to test add-ons with Developer Edition?</h2>
<p>Yes. You can also test unsigned add-ons on Nightly builds or <a href="https://blog.mozilla.org/addons/2015/12/23/loading-temporary-add-ons/">load WebExtensions temporarily</a> in Beta and Release builds.</p>
<p>We are also continuing to provide <a href="https://wiki.mozilla.org/Add-ons/Extension_Signing#Unbranded_Builds">unbranded builds</a> of the beta and release branches which are able to run unsigned add-ons - including bootstrapped - for development and experimentation. These versions will not be verified by QE, but will receive updates , which is an improvement to the <a href="https://wiki.mozilla.org/Add-ons/Extension_Signing#Unbranded_Builds">unbranded builds we currently provide</a> for add-on development.</p>
<p>The majority of Developer Edition users won’t experience any disruption.</p>
<h2 id="how-will-you-mitigate-the-quality-risk-from-cutting-6-8-weeks-of-stabilization-from-the-cycle">How will you mitigate the quality risk from cutting 6-8 weeks of stabilization from the cycle?</h2>
<p>Instead of pushing to 100 % of the beta population at once, we will use a staged rollout mechanism to push to a subset of the beta population.
For the first phase, we will be pushing to the former aurora population. As a second phase, we will be targeting specific populations (Operating system, graphic card, etc)</p>
<p>In parallel, QE will also do preliminary nightly sign off to detect early new potential issues. Release management will be much more aggressive in term of feature deactivation.</p>
<p>Last but not least, the aurora cycle was used to finalize some features. Instead, feature stabilization will be performed during the nightly cycle.</p>
<h2 id="what-are-we-doing-to-improve-nightly-quality">What are we doing to improve Nightly quality?</h2>
<p>To improve the overall quality of nightly, a few initiatives will help.</p>
<h3 id="nightly-merge-criteria">Nightly merge criteria</h3>
<p>New end-user facing features landing in Nightly builds should meet Beta-readiness criteria before they can be pushed to Beta channel.</p>
<h3 id="static-analyzers">Static analyzers</h3>
<p>In order to detect issues at review phase, static analyzers will be integrated as part of the workflow. They will be able to identify potential defects but also limit the technological debt.</p>
<h3 id="code-coverage">Code coverage</h3>
<p>Code coverage results are going to be used to analyze the quality of the testsuite and the risk introduced by the change.</p>
<h3 id="risk-assessment">Risk assessment</h3>
<p>By correlating various data sources (VCS, Bugzilla, etc), we believe we can identify the potential risks carried by changes before they even land. The idea is to identify the functions where a modification has more chance to induce a regression.</p>
<h2 id="how-often-will-beta-builds-be-updated">How often will Beta builds be updated?</h2>
<p>We will continue to push two Beta builds for Desktop and one Fennec build each week of the Beta cycle.</p>
<h2 id="will-developer-edition-continue-to-have-a-separate-profile">Will Developer Edition continue to have a separate profile?</h2>
<p>Yes. The Developer Edition separate profile feature is a requirement for transition. If for whatever reason this feature cannot be completed by the end of the year we will need to return to creating rebuilds of Developer Edition as previously done to ensure those users are not cast away.</p>
<h2 id="what-will-happen-to-the-aurora-branch-after-firefox-54-moves-to-beta">What will happen to the Aurora branch after Firefox 54 moves to Beta?</h2>
<p>Updates on aurora channel will be disabled on April 18th. The desktop and Android aurora populations will be migrated as described above.</p>
<h2 id="what-criteria-will-be-used-to-assess-feature-readiness-to-move-to-beta">What criteria will be used to assess feature readiness to move to Beta?</h2>
<p>We will be monitoring crash rates, QE’s sign offs, telemetry data and new regressions to determine overall Nightly quality and feature readiness to merge to Beta.</p>
<h2 id="how-and-who-will-determine-whether-a-feature-is-ready-to-move-to-beta">How and who will determine whether a feature is ready to move to Beta?</h2>
<p>End-user facing features will be reviewed for beta-readiness before they are pushed to Beta channel. Following is a list of criteria that will be used to evaluate feature readiness to merge to Beta:</p>
<ul>
<li>No significant stability Issues</li>
<li>Missing Test Plans</li>
<li>Insufficient Testing</li>
<li>Feature is not Code Complete</li>
<li>Too Many Open Bugs</li>
</ul>
<p>More detailed criteria defined in <a href="https://docs.google.com/document/d/1r_-nEBwRIYexBDdCuLqVwkjlF4TQDDaf8XRAsge2e_g/edit">this document</a>.</p>
<h2 id="are-there-any-changes-to-release-or-esr-channel">Are there any changes to Release or ESR channel?</h2>
<p>No changes are planned for Release or ESR channel users.</p>
<h2 id="does-this-change-how-frequently-we-push-mainline-builds-to-release-channel">Does this change how frequently we push mainline builds to Release channel?</h2>
<p>No, but changes added in Nightly can make it into a Release build about 6-8 weeks sooner than they do now.</p>
<h2 id="what-will-happen-for-l10n-process-when-we-remove-aurora">What will happen for l10n process when we remove Aurora?</h2>
<p>Focus for localization will move from mozilla-aurora to mozilla-central. Localization tools (Pootle and Pontoon) will read en-US strings from a special mozilla-central clone: l10n-drivers will review patches with strings landing in the official mozilla-central repository, provide feedback to devs if necessary, and land updates every 2-3 days in this special repository. Localized content will be pushed to l10n-central repositories.</p>
<p>There are no changes for developers working on Firefox: Nightly and mozilla-central remain open to string changes, including the extra six weeks that Firefox 55 will spend in Nightly, while Beta is still considered string frozen, and requests to uplift changes affecting strings are evaluated case by case.</p>
<p>Users interested in helping with localization should download Nightly in their language.</p>
<h2 id="what-will-happen-for-l10n-process-by-the-end-of-year">What will happen for l10n process by the end of year?</h2>
<p>For Firefox and Firefox for Android we will shift to a model with a single repository for all channels for each locale. This change will be reflected in localization tools, allowing localizers to make a change to a string and see that update applied across all channels at once.</p>
<h2 id="how-does-dawn-impact-engineering-planning-for-landing-features">How does Dawn impact engineering planning for landing features?</h2>
<p>The biggest shift is that features will have to be completed before merge day. Developers will not be able to finalize feature development during the next branch cycle (as Aurora is used currently). See also “How and who will determine whether a feature is ready to move to Beta?”.</p>
<h2 id="how-will-bug-fixes-and-features-not-tracked-by-project-management-be-impacted-by-dawn">How will bug fixes and features not tracked by project management be impacted by Dawn?</h2>
<p>Landing bug fixes in Nightly repository continues as before. Development on features that are not directly end-user visible and not tracked by EPMs, release management continues as before.</p>
<p>If Nightly quality and stability is negatively impacted by these untracked features or bug fixes, we will discuss potential mitigation options such as: back outs, stabilizing quality issues before continuing new feature development work, delaying Merge date, imposing code freeze in Nightly until blocking issues are resolved, etc.</p>
<h2 id="what-will-happen-to-the-diagnostic-assert">What will happen to the diagnostic assert?</h2>
<p>MOZ_DIAGNOSTIC_ASSERT will enabled during the first part of the beta cycle. It will be automatically disabled when EARLY_BETA_OR_EARLIER is no longer defined.</p>