debugging_firefox_with_pernosco.rst

Enable keyboard shortcuts

Debugging Firefox with Pernosco

===============================

This page is intended to help Firefox/Gecko developers get started using `Pernosco <https://pernos.co/>`_

to debug Firefox.

Mozilla employees have a paid subscription to Pernosco, and are encouraged to use it.

Contributors can use Pernosco as well. `A free account <https://pernos.co/pricing/>`_

comes with 5 uploads to see if it is suitable for your workflow.

Prerequisites

-------------

- You need the ability to produce an :code:`rr` trace. See :doc:`debugging_firefox_with_rr`

- Employees, you can also generate traces for Pernosco for some CI job failures via

  Treeherder.

Workflow

--------

At a high level, working with Pernosco consists of recording a trace then uploading

the trace for processing.

Install your Pernosco configuration

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you are using your own Pernoso account, follow the instructions

`here <https://pernos.co/account>`_ to install your credentials.

Mozilla employees, see the `Pernosco install and configuration doc

<https://docs.google.com/document/d/1cb-mkNyJYMudpS0a-R6zxy0kOMIRUSKrxxL-E9zFGKE/edit>`_

for credentials.

In general, you will want:

 * Your email in :code:`~/.config/pernosco/user`

 * Your group in :code:`~/.config/pernosco/group`

 * Your key in :code:`~/.config/pernosco/user_secret_key`

Gather a Local Trace

~~~~~~~~~~~~~~~~~~~~

In general, to gather a trace follow the instruction for recording :code:`rr` traces

in :doc:`debugging_firefox_with_rr`. However, to gather a local rr trace that’s

compatible with Pernosco you need to disable some incompatible CPU features (currently

only AVX-512)

On an Intel machine, you can disable this at the command line. To simplify things a

shell script :code:`pernosco-record` can be created like so,

.. code:: bash

    #!/bin/bash

    rr record --disable-avx-512 "$@"

replacing plain :code:`rr` in command lines (:code:`mach --debugger pernosco-record`)

On AMD Zen Machines, you can disable AVX-512 and 5 level page tables at boot.

On a Debian derived distro this involves editing :code:`/etc/default/grub` and

adding :code:`clearcpuid=304,no5lvl` to :code:`GRUB_CMDLINE_LINUX_DEFAULT`, then

running :code:`sudo update-grub` after.

Upload the trace

~~~~~~~~~~~~~~~~

Find the trace you’re interested in :code:`~/.local/share/rr/<trace>`, and call :code:`pernosco-submit`

.. code:: bash

   pernosco-submit upload ~/.local/share/rr/<trace> PATH_TO_CENTRAL --title "Name Of Trace"

where :code:`PATH TO CENTRAL` is the path to the source of your build, checked out

to the revision matching the build.

The last trace recorded is always symlinked, so you can use :code:`~/.local/share/rr/latest-trace` too.

Wait for email

~~~~~~~~~~~~~~

After the trace is processed, you'll get an email with a link to your trace. Good

hunting!

Treeherder Workflow

-------------------

You can also try to get traces generated via the Treeherder interface. On a failing

test that is eligible, in "Artifacts and Debugging Tools", you'll see a link to

.. image:: ../img/record-this-failure.png

    :width: 35 %

This link brings you to the self-service reproducer:

.. image:: ../img/self-service.png

    :width: 60 %

Select a test and a count, and whether or not chaos-mode should be used, then click

reproduce. At this point there is a **long** wait; the page will notify you when

reproduction has started with an :code:`alert` dialog.