firefox-main/testing/performance/adding-tests/perfdocs/index.rst

Source code

Revision control

Copy as Markdown

Other Tools

========================
Adding Performance Tests
========================
.. contents::
:depth: 3
This guide helps you choose the right framework for a new performance test
and shows how to write one. For general information about performance testing
at Mozilla (handling regressions, running tests, comparisons), see
`Performance Testing in a Nutshell <perftest-in-a-nutshell.html>`_.
Choosing a Framework
--------------------
.. list-table::
:header-rows: 1
:widths: 18 32 8 12 30
* - Framework
- Use when
- Android
- Cross-browser
- Run with
* - `MozPerftest <mozperftest.html>`_ (xpcshell)
- Testing low-level platform code from JS (networking, intl, storage).
No browser chrome needed.
- No
- No
- ``./mach perftest path/to/test.js``
* - `MozPerftest <mozperftest.html>`_ (mochitest)
- Testing browser-level operations that need a content page or chrome
privileges (DOM, accessibility, service workers, ...).
- No
- No
- ``./mach perftest path/to/test.html``
* - `MozPerftest <mozperftest.html>`_ (browsertime)
- Full page-load or navigation scenarios that need a real browser
session driven via Selenium/WebDriver.
- Yes
- Yes (Chrome, Safari)
- ``./mach perftest path/to/perftest_example.js``
* - `MozPerftest <mozperftest.html>`_ (custom script)
- Shell-script-based tests (e.g. startup timing on Android).
- Yes
- Yes, script-dependent
- ``./mach perftest path/to/script.sh``
- Micro-benchmarking C++ or Rust code at the platform level
(parsing, layout, networking internals). Very low overhead.
- No
- No
- ``./mach gtest 'SuiteName.TestName'``
* - `Raptor <raptor.html>`__
- Industry-standard benchmarks (Speedometer, JetStream, MotionMark)
or cross-browser page-load comparisons. See `Getting Help`_ first.
- Yes
- Yes (Chrome, Safari)
- ``./mach raptor``
* - `Talos <talos.html>`__
- Legacy framework. **Do not add new tests here** unless there is a
specific limitation that prevents use of MozPerftest.
- No
- No
- ``./mach talos-test``
- Memory-usage tracking (Are We Slim Yet) across builds.
- No
- No
- ``./mach awsy-test``
For most new tests, **MozPerftest** is the recommended choice.
Pick the flavor (xpcshell, mochitest, browsertime) that matches
the level at which your code operates. For the full reference of
``perfMetadata`` fields, ``options``, and supported flavors, see the
mozperftest `writing guide <writing.html>`_.
MozPerftest: XPCShell Example
-----------------------------
XPCShell tests are the simplest flavor: a plain xpcshell test with a
``perfMetadata`` variable and ``info("perfMetrics", ...)`` calls to report
results.
In-tree examples:
**Test file** (`intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js <https://searchfox.org/mozilla-central/source/intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js>`_):
.. code-block:: javascript
"use strict";
var perfMetadata = {
owner: "Intl team",
name: "Intl.DateTimeFormat",
description: "Test the speed of Intl.DateTimeFormat",
options: {
default: {
perfherder: true,
perfherder_metrics: [
{ name: "DateTimeFormat constructor", unit: "ms" },
{ name: "DateTimeFormat.format", unit: "ms" },
],
},
},
tags: ["intl"],
};
add_task(function test_dateTimeFormat() {
let start = Cu.now();
for (let i = 0; i < 100000; i++) {
new Intl.DateTimeFormat("en-US");
}
let constructorTime = Cu.now() - start;
let fmt = new Intl.DateTimeFormat("en-US");
let date = new Date();
start = Cu.now();
for (let i = 0; i < 100000; i++) {
fmt.format(date);
}
let formatTime = Cu.now() - start;
info(
"perfMetrics",
JSON.stringify({
"DateTimeFormat constructor": constructorTime,
"DateTimeFormat.format": formatTime,
})
);
});
**Manifest** (``perftest.toml``):
.. code-block:: toml
[DEFAULT]
["perftest_dateTimeFormat.js"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"
The ``disabled`` field keeps the test out of the normal unit-test chunks; it
will still run when invoked via ``./mach perftest``.
**Registration** (``moz.build``):
.. code-block:: python
PERFTESTS_MANIFESTS += ["test/xpcshell/perftest.toml"]
Run locally::
./mach perftest intl/benchmarks/test/xpcshell/perftest_dateTimeFormat.js
MozPerftest: Mochitest Example
------------------------------
Mochitest-flavored perftests are standard mochitests with ``perfMetadata`` and
``info("perfMetrics", ...)`` calls. They use the regular mochitest manifest
variable (e.g. ``BROWSER_CHROME_MANIFESTS``) instead of ``PERFTESTS_MANIFESTS``,
and set ``disabled`` in the manifest to prevent the test from running in regular CI.
In-tree examples:
- **mochitest-plain**: `dom/serviceworkers/test/performance/ <https://searchfox.org/mozilla-central/source/dom/serviceworkers/test/performance>`_
- **browser-chrome**: `accessible/tests/browser/performance/ <https://searchfox.org/mozilla-central/source/accessible/tests/browser/performance>`_
- **browser-chrome**: `toolkit/components/ml/tests/browser/ <https://searchfox.org/mozilla-central/source/toolkit/components/ml/tests/browser>`_
**Test file** (`dom/serviceworkers/test/performance/test_caching.html <https://searchfox.org/mozilla-central/source/dom/serviceworkers/test/performance/test_caching.html>`_):
.. code-block:: html
<!DOCTYPE HTML>
<html>
<head>
<title>Service worker caching perftest</title>
</head>
<script src="/tests/SimpleTest/SimpleTest.js"></script>
<script src="perfutils.js"></script>
<script>
"use strict";
var journal = {
"No cache": [],
"Cached": [],
};
const ITERATIONS = 10;
var perfMetadata = {
owner: "DOM LWS",
name: "Service Worker Caching",
description: "Test service worker caching.",
options: {
default: {
perfherder: true,
perfherder_metrics: [
{ name: "No cache", unit: "ms", shouldAlert: true },
{ name: "Cached", unit: "ms", shouldAlert: true },
],
verbose: true,
manifest: "perftest.toml",
manifest_flavor: "plain",
},
},
};
add_task(async () => {
// ... run iterations, collect timing into journal ...
});
add_task(() => {
// Compute medians and report
let metrics = {};
for (const name in journal) {
let sorted = [...journal[name]].sort((a, b) => a - b);
metrics[name] = sorted[Math.floor(sorted.length / 2)];
}
info("perfMetrics", JSON.stringify(metrics));
});
</script>
<body></body>
</html>
**Manifest** (``perftest.toml``):
.. code-block:: toml
[DEFAULT]
["test_caching.html"]
disabled = "Disabled as we want to run this only as perftest, not regular CI"
**Registration** (``moz.build``) -- use the standard mochitest manifest variable
for the flavor you're using, not ``PERFTESTS_MANIFESTS``:
.. code-block:: python
MOCHITEST_MANIFESTS += ["test/performance/perftest.toml"]
# or, for browser-chrome:
# BROWSER_CHROME_MANIFESTS += ["test/browser/performance/perftest.toml"]
Key differences from xpcshell perftests:
- ``perfMetadata`` must include ``manifest`` and ``manifest_flavor`` fields
(``"plain"`` for mochitest-plain, ``"browser-chrome"`` for browser chrome tests).
- ``extra_args`` can specify additional mochitest arguments (e.g. ``["headless"]``).
- The manifest is registered with the standard mochitest variable, not
``PERFTESTS_MANIFESTS``. The ``disabled`` field on each test entry is what
keeps it out of the normal mochitest CI chunks; it will still run when
invoked via ``./mach perftest``.
MozPerftest: Browsertime Example
---------------------------------
Browsertime tests drive a full browser session via Selenium. Use them when
you need real navigation, user interaction, or page-load measurements.
In-tree examples (under `testing/performance/ <https://searchfox.org/mozilla-central/source/testing/performance>`_):
- ``perftest_pageload.js`` -- minimal navigation example.
- ``perftest_facebook.js`` -- form interaction with ``addText``/``click``.
**Test file** (`testing/performance/perftest_pageload.js <https://searchfox.org/mozilla-central/source/testing/performance/perftest_pageload.js>`_):
.. code-block:: javascript
async function setUp(context) {
context.log.info("setUp example!");
}
async function test(context, commands) {
let url = context.options.browsertime.url;
await commands.navigate("https://www.mozilla.org/en-US/");
await commands.wait.byTime(100);
await commands.navigate("about:blank");
await commands.wait.byTime(50);
return commands.measure.start(url);
}
async function tearDown(context) {
context.log.info("tearDown example!");
}
module.exports = {
setUp,
tearDown,
test,
owner: "Performance Team",
name: "pageload",
description: "Measures time to load mozilla page",
};
**Manifest** (`testing/performance/perftest.toml <https://searchfox.org/mozilla-central/source/testing/performance/perftest.toml>`_)
just lists the test files:
.. code-block:: toml
[DEFAULT]
["perftest_pageload.js"]
By convention, browsertime perftest files are prefixed with ``perftest_``.
For full documentation on the browsertime scripting API, see the
MozPerftest: Custom Script Example
-----------------------------------
Custom-script tests are shell scripts (or programs invoked from one) that
write a single ``perfMetrics: ...`` line to stdout. They are useful for
platform-level measurements that don't fit into xpcshell or browsertime --
for example, Android startup timing where the test needs to drive ``adb``.
In-tree examples:
-- Android cold-startup tests (``cvns.sh``, ``cmff.sh``, ...).
-- Android resource-usage tests.
**Test file** (illustrative, based on
.. code-block:: bash
# Name: custom-script-test
# Owner: Perftest Team
# Description: Runs a sample custom script test.
# Options: {"default": {"perfherder": true, "perfherder_metrics": [{ "name": "Registration", "unit": "ms" }]}}
echo Running...
# ${BROWSER_BINARY} is the package name on Android, or the path to the
# browser binary on desktop. Mozperftest verifies that it exists.
echo Binary: ${BROWSER_BINARY}
# The single perfMetrics line is what mozperftest parses for results.
# Curly braces in the JSON must be doubled.
echo 'perfMetrics: [{{"name": "metric1", "shouldAlert": false, "lowerIsBetter": false, "unit": "speed", "values": [1, 2, 3, 4]}}]'
Run locally::
./mach perftest path/to/your-script.sh
GTest: C++ Micro-benchmarks
---------------------------
GTest is the simplest way to add a low-level C++ or Rust performance test.
The ``MOZ_GTEST_BENCH`` macro wraps a test function so that GTest times its
execution and reports the result to Perfherder under the
``platform_microbench`` framework.
In debug/ASAN builds the test runs once without timing (as a correctness check).
In optimized builds it runs 5 times by default (configurable via
``MOZ_GTEST_NUM_ITERATIONS``) and reports the median duration.
In-tree examples:
(currently disabled by default)
.. code-block:: cpp
#include "gtest/MozGTestBench.h"
static void ServoParsingBench() {
// ... setup code ...
for (int i = 0; i < PARSING_REPETITIONS; i++) {
RefPtr<StyleStylesheetContents> stylesheet =
Servo_StyleSheet_FromUTF8Bytes(/* ... */).Consume();
}
}
MOZ_GTEST_BENCH(Stylo, Servo_StyleSheet_FromUTF8Bytes_Bench,
[] { ServoParsingBench(); });
There is also ``MOZ_GTEST_BENCH_F`` for fixture-based tests (wraps ``TEST_F``
instead of ``TEST``).
When the test runs in CI, it outputs::
PERFHERDER_DATA: {"framework": {"name": "platform_microbench"},
"suites": [{"name": "Stylo", "subtests":
[{"name": "Servo_StyleSheet_FromUTF8Bytes_Bench",
"value": 252674, "lowerIsBetter": true}]}]}
.. note::
Micro-benchmark regressions are not treated as strictly as other
performance test regressions. A regression in a micro-benchmark may not
correspond to a user-visible regression, and large changes are expected
when the benchmarked code is intentionally modified.
for more details.
Raptor
------
Raptor is the framework used for industry-standard benchmarks (Speedometer,
JetStream, MotionMark) and cross-browser page-load comparisons.
In-tree examples (under
- ``browsertime_pageload.js`` -- canonical page-load test runner.
- ``speculative-connect.js`` -- privileged-call + custom metric.
**Test file** (excerpt of
.. code-block:: javascript
const { logTest } = require("./utils/profiling");
module.exports = logTest(
"speculative connect pageload",
async function (context, commands) {
await commands.navigate("about:blank");
await commands.wait.byTime(1000);
// Privileged JS to trigger a speculative connection.
const script = `
var URI = Services.io.newURI("${url}");
var principal = Services.scriptSecurityManager
.createContentPrincipal(URI, {});
Services.io.speculativeConnect(URI, principal, callbacks, false);
`;
commands.js.runPrivileged(script);
await commands.wait.byTime(1000);
// Measure the pageload.
await commands.measure.start();
await commands.navigate(url);
await commands.measure.stop();
// Report a custom metric alongside the pageload measurements.
const connect_time = await commands.js.run(
`return (window.performance.timing.connectEnd -
window.performance.timing.navigationStart);`
);
await commands.measure.addObject({
custom_data: { connect_time },
});
return true;
}
);
The wiring (manifest entry, taskcluster kind, alert thresholds, etc.) is
larger than for mozperftest tests -- see the
`contributing guide <contributing.html>`_ for adding new Raptor tests, then
`Raptor documentation <raptor.html>`_ for the full reference.
See `Getting Help`_ before adding a new Raptor test.
Talos
-----
Talos is the legacy framework. **Do not add new tests here** unless there is
a specific limitation that prevents using MozPerftest -- see `Getting Help`_
first.
A Talos test is a Python class that subclasses ``TsBase`` (or another base)
and is registered via ``@register_test()`` in
The class fields configure the runner (URL, cycles, filters, units, etc.).
**Example** (excerpt of ``ts_paint`` from
.. code-block:: python
@register_test()
class ts_paint(TsBase):
"""
Launches tspaint_test.html with the current timestamp in the url,
waits for [MozAfterPaint and onLoad] to fire, then records the end
time and calculates the time to startup.
"""
cycles = 20
timeout = 150
gecko_profile_startup = True
url = "startup_test/tspaint_test.html"
filters = filter.ignore_first.prepare(1) + filter.median.prepare()
tpmozafterpaint = True
unit = "ms"
The page (``startup_test/tspaint_test.html``) lives under
manifests and config wiring are also in that tree. See the
`Talos documentation <talos.html>`_ for the full reference.
AWSY
----
AWSY (Are We Slim Yet) tracks memory usage across builds.
An AWSY test is a Python class that subclasses ``AwsyTestCase``. It declares
the perfherder suites, the memory checkpoints to capture from
``about:memory``, and the URLs to load. AWSY then drives the browser through
those steps and reports the configured metrics.
**Example** (excerpt of
.. code-block:: python
from awsy.awsy_test_case import AwsyTestCase
CHECKPOINTS = [
{
"name": "After tabs open [+30s, forced GC]",
"path": "memory-report-TabsOpenForceGC-4.json.gz",
"name_filter": ["web ", "Web Content"],
"median": True,
},
]
PERF_SUITES = [
{"name": "Base Content Resident Unique Memory", "node": "resident-unique"},
{"name": "Base Content Heap Unclassified", "node": "explicit/heap-unclassified"},
{"name": "Base Content JS", "node": "js-main-runtime/", "alertThreshold": 0.25},
{"name": "Base Content Explicit", "node": "explicit/"},
]
class TestMemoryUsage(AwsyTestCase):
"""Loads about:memory and reports content-process memory usage."""
def urls(self):
return self._urls
def perf_suites(self):
return PERF_SUITES
def perf_checkpoints(self):
return CHECKPOINTS
Run locally::
./mach awsy-test --base
See the
for the full set of fields and configuration options.
Running Tests in CI
-------------------
To run your test in CI you need to add a task definition under the
appropriate taskcluster ``kinds/`` directory:
- mozperftest: `taskcluster/kinds/perftest/ <https://searchfox.org/mozilla-central/source/taskcluster/kinds/perftest>`_
- Raptor and browsertime: `taskcluster/kinds/browsertime/ <https://searchfox.org/mozilla-central/source/taskcluster/kinds/browsertime>`_
This is what determines which platforms and configurations your test runs on.
If you're not sure where your task belongs, see the `Getting Help`_ section
below -- the perf team can help.
For xpcshell and mochitest mozperftest tests, you may also need to update the
``_TRY_MAPPING`` variable in
so CI can locate your test file.
Once your test is registered, ``./mach try perf`` will include it in try
pushes -- but only if the test matches an existing category. If you can't
find your test in the category picker, run with ``--full`` to fall back to
the full ``./mach try fuzzy`` interface, or add a new category for your test.
To reproduce a specific alert::
./mach try perf --alert <ALERT-NUMBER>
Note that ``--alert`` searches across all tasks regardless of category. For
more details, see `Mach Try Perf <mach-try-perf.html>`_.
Getting Help
------------
Reach out to the Performance Testing and Tooling team in the
`#perftest channel on Matrix <https://matrix.to/#/#perftest:mozilla.org>`_
or #perf-help on Slack.