Source code
Revision control
Copy as Markdown
Other Tools
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
---
name: raptor
manifest: testing/raptor/raptor/raptor.toml
static-only: False
metrics:
"First Paint":
aliases:
- fcp
- First Contentful Paint
- First Contentful Composite
description: >
Denotes the first time the browser performs a paint that has content in it (in ms).
"Youtube Playback Metrics":
aliases:
- VP9
- H264
matcher: VP9.*|H264.*
description: >
Metrics starting with VP9/H264 give the number of frames dropped, and painted.
"First Visual Change":
aliases:
- FirstVisualChange
description: The first visual change detected in the test (in ms).
"Last Visual Change":
aliases:
- LastVisualChange
description: The last visual change detected in the test (in ms).
"Load Time":
aliases:
- loadtime
description: The time it took for the page to complete loading (in ms).
"Largest Contentful Paint":
aliases:
- largestContentfulPaint
- lcp
description: >
The time (in ms) at which the largest piece of content on the page was rendered/painted.
"Speed Index":
aliases:
- SpeedIndex
description: >
A metric used to denote the speed at which a page loaded. Lower values indicate faster
pageloads. Units are in (Visually-Complete x Milliseconds). Calculated by using videos
of the pageload which provide a measure of visual completeness. Visual completeness is
calculated by comparing the histogram of a given frame to the final frame of the pageload.
The SpeedIndex is calculated as the area between the curves of a constant line at y=1,
and the graph of the visual completeness from 0ms to when visual completeness reaches
100% (or hits the y=1 line).
"Contentful Speed Index":
aliases:
- ContentfulSpeedIndex
description: >
Similar to SpeedIndex, except that it uses the contentfulness of a frame to determine
visual completeness instead of the histogram differences. The contentfulness is determined
by calculating the number of edges found in the frame. A lower number of edges detected
gives a smaller contentfulness value (and a smaller visually complete value).
"Perceptual Speed Index":
aliases:
- PerceptualSpeedIndex
description: >
Similar to SpeedIndex, except that it uses the structural similarity index measure (ssim)
to determine visual completeness. This technique compares the luminance, contrast, and
structure of the frames (a given frame vs. a final frame) to determine the completeness.
suites:
desktop:
description: "Tests for page-load performance. The links direct to the actual websites that are being tested."
tests:
amazon: ""
bing-search: ""
buzzfeed: ""
cnn: ""
docomo: ""
ebay: ""
espn: ""
expedia: ""
facebook: ""
fandom: ""
google-docs: ""
google-mail: ""
google-search: ""
google-slides: ""
imdb: ""
imgur: ""
instagram: ""
linkedin: ""
microsoft: ""
netflix: ""
nytimes: ""
office: ""
openai: ""
outlook: ""
paypal: ""
pinterest: ""
reddit: ""
samsung: ""
tiktok: ""
tumblr: ""
twitch: ""
twitter: ""
weather: ""
wikia: ""
wikipedia: ""
yahoo-mail: ""
youtube: ""
mobile:
description: "Page-load performance test suite on Android. The links direct to the actual websites that are being tested."
tests:
amazon: ""
youtube: ""
allrecipes: ""
amazon-search: ""
bing: ""
bing-search-restaurants: ""
booking: ""
cnn: ""
cnn-ampstories: ""
dailymail: ""
ebay-kleinanzeigen: ""
ebay-kleinanzeigen-search: ""
espn: ""
facebook: ""
facebook-cristiano: ""
google: ""
google-maps: ""
google-search-restaurants: ""
imdb: ""
instagram: ""
microsoft-support: ""
reddit: ""
stackoverflow: ""
sina: ""
bild-de: ""
wikipedia: ""
youtube-watch: ""
live:
description: "A set of test pages that are run as live sites instead of recorded versions. These tests are available on all browsers, on all platforms."
tests:
booking-sf: ""
discord: ""
fashionbeans: ""
google-accounts: ""
imdb-firefox: ""
medium-article: ""
people-article: ""
reddit-thread: ""
rumble-fox: ""
stackoverflow-question: ""
urbandictionary-define: ""
wikia-marvel: ""
benchmarks:
description: >
Standard benchmarks are third-party tests (i.e. Speedometer) that we have integrated
into Raptor to run per-commit in our production CI. To update any of these benchmarks,
see `Updating Benchmark Tests <browsertime.html#updating-benchmark-tests>`_.
tests:
ares6: ""
assorted-dom: ""
jetstream2: ""
matrix-react-bench: ""
motionmark-1-3: ""
motionmark-htmlsuite-1-3: ""
motionmark-animometer: ""
motionmark-animometer-ramp: ""
motionmark-htmlsuite: ""
motionmark-htmlsuite-ramp: ""
speedometer: ""
speedometer3: ""
stylebench: ""
sunspider: ""
twitch-animation: ""
unity-webgl: ""
wasm-godot-baseline: ""
wasm-godot-optimizing: ""
wasm-godot: ""
wasm-misc-baseline: ""
wasm-misc-optimizing: ""
wasm-misc: ""
webaudio: ""
youtube-playback: ""
youtube-playback-av1-sfr: ""
youtube-playback-h264-1080p30: ""
youtube-playback-h264-1080p60: ""
youtube-playback-h264-full-1080p30: ""
youtube-playback-h264-full-1080p60: ""
youtube-playback-h264-sfr: ""
youtube-playback-hfr: ""
youtube-playback-v9-1080p30: ""
youtube-playback-v9-1080p60: ""
youtube-playback-v9-full-1080p30: ""
youtube-playback-v9-full-1080p60: ""
youtube-playback-vp9-sfr: ""
youtube-playback-widevine-h264-sfr: ""
youtube-playback-widevine-hfr: ""
youtube-playback-widevine-vp9-sfr: ""
scenario:
description: "Tests that perform a specific action (a scenario), i.e. idle application, idle application in background, etc."
tests:
idle: ""
idle-bg: ""
custom:
description: "Browsertime tests that use a custom pageload test script. These use the pageload type, but may have other intentions."
tests:
browsertime: "Used to run vanilla browsertime tests through raptor. For example: `./mach raptor --browsertime -t browsertime --browsertime-arg url=https://www.sitespeed.io --browsertime-arg iterations=3`"
process-switch: "Measures process switch time"
welcome: "Measures pageload metrics for the first-install about:welcome page"
grandprix: "Runs the Grandprix benchmark"
addMab1: "Use add API to send many small ArrayBuffers to IndexedDB in one transaction"
addMabN: "Use add API to send many small ArrayBuffers to IndexedDB independently"
addMar1: "Use add API to send many small Arrays to IndexedDB in one transaction"
addMarN: "Use add API to send many small Arrays to IndexedDB independently"
addMbl1: "Use add API to send many small Blobs to IndexedDB in one transaction"
addMblN: "Use add API to send many small Blobs to IndexedDB independently"
addkAB1: "Use add API to send a couple of large ArrayBuffers to IndexedDB in one transaction"
addkABN: "Use add API to send a couple of large ArrayBuffers to IndexedDB independently"
addkAR1: "Use add API to send a couple of large JS Arrays to IndexedDB in one transaction"
addkARN: "Use add API to send a couple of large JS Arrays to IndexedDB independently"
addkBL1: "Use add API to send a couple of large Blobs to IndexedDB in one transaction"
addkBLN: "Use add API to send a couple of large Blobs to IndexedDB independently"
constant-regression: "Generates a constant value that can be changed to induce a regression."
getkeyrng: "Measures the speed of key-range based reads of IndexedDB"
upload: "Measures http/2 file upload throughput"
upload-h3: "Measures http/3 file upload throughput"
throttled: "Pageload test using a throttled network environment"
sample-python-support: "A sample test that uses a python support file to modify the test command."
connect: "Verifies that connection time on a pageload is zero when a speculative connection has first been made"
tp6-bench: >
A pageload benchmark test that quickly runs through a selection of the
pages available to produce a single overall metric for fcp, lcp, and
loadtime. The metrics from all pages get summarized with the geomean, and mean.
interactive:
description: "Browsertime tests that interact with the webpage. Includes responsiveness tests as they make use of this support for navigation. These form of tests allow the specification of browsertime commands through the test manifest."
tests:
cnn-nav: "Navigates to cnn main page, then to the world sub-page."
facebook-nav: "Navigates to facebook, then the sub-pages friends, marketplace, groups."
reddit-billgates-ama: "Navigates from the Bill Gates AMA to the Reddit members section."
reddit-billgates-post-1: "Navigates the `thisisbillgates` user starting at the main user page, then to the posts, comments, hot, and top sections."
reddit-billgates-post-2: "Navigates the `thisisbillgates` user starting at the main user page, then to the posts, comments, hot, and top sections."
unittests:
description: "These tests aren't used in standard testing, they are only used in the Raptor unit tests (they are similar to raptor-tp6 tests though)."
tests:
test-page-1: ""
test-page-2: ""
test-page-3: ""
test-page-4: ""