Source code
Revision control
Copy as Markdown
Other Tools
Design Specification for ./mach manifest skip-fails
===================================================
The first of the mach manifest subcommands is skip-fails. This command
can be used to automatically edit manifests to skip tests that are
failing as well as file the corresponding bugs for the failures. This
is particularly useful when “greening up” a new platform.
The user documentation for skip-fails is here:
The skip-fails command knows how to manage failures for TOML, WPT
and REFTEST manifests. Additional developer design notes are included here.
Running skip-fails for TOML
---------------------------
The user documentation for TOML manifests is:
Usage
-----
$ ./mach manifest skip-fails -h
usage: mach [global arguments] manifest skip-fails [command arguments]
Global Arguments:
-v, --verbose Print verbose output.
--quiet Don't print as much output.
-l, --log-file FILENAME
Filename to write log data to.
--log-interval Prefix log line with interval from last message rather
than relative time. Note that this is NOT execution
time if there are parallel operations.
--no-interactive Automatically selects the default option on any
interactive prompts. If the output is not a terminal,
then --no-interactive is assumed.
--log-no-times Do not prefix log lines with times. By default, mach
will prefix each output line with the time since
command start.
-h, --help Show this help message.
--debug-command Start a Python debugger when command is dispatched.
--profile-command Capture a Python profile of the mach process as command
is dispatched.
--settings FILENAME Path to settings file.
Sub Command Arguments:
try_url Treeherder URL for try (please use quotes)
-b, --bugzilla BUGZILLA
Bugzilla instance [disable]
-C, --clear-cache [CLEAR_CACHE]
clear cache REVISION (or all)
-c, --carryover Set carryover mode (only skip failures for platform
matches)
-d, --dry-run Determine manifest changes, but do not write them
-F, --use-failures USE_FAILURES
Use failures from file
-f, --save-failures SAVE_FAILURES
Save failures to file
-I, --implicit-vars Use implicit variables in reftest manifests
-i, --task-id TASK_ID
Task id to write a condition for instead of all tasks
from the push
-k, --known-intermittents
Set known intermittents mode (only skip failures known
intermittents)
-M, --max-failures MAX_FAILURES
Maximum number of failures to skip (-1 == no limit)
-m, --meta-bug-id META_BUG_ID
Meta Bug id
-n, --new-version NEW_VERSION
New version to use for annotations
-N, --new-failures Set new failures mode (only add conditions for new
failures)
-r, --failure-ratio FAILURE_RATIO
Ratio of test failures/total to skip [0.4]
-R, --replace-tbd Replace Bug TBD in manifests by filing new bugs
-s, --turbo Skip all secondary failures
-T, --use-tasks USE_TASKS
Use tasks from file
-t, --save-tasks SAVE_TASKS
Save tasks to file
-u, --user-agent USER_AGENT
User-Agent to use for mozci if queries are forbidden
from treeherder
-v, --verbose Verbose mode
Design
------
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the failure ratio of failed runs / total runs is less than 40%
classify "intermittent"
d. Else edit the manifest to skip this failure
2. Manifest variable information
a. Variables used in Python are set:
NOTE: that the task metadata runtimes are turned in to test variants by looking
up the "mozinfo" field that corresponds to the "runtime" key in
taskcluster/test_configs/variants.yml
NOTE: special excpetions are these runtimes are mapped to these variants:
- no-fission -> !fission
- 1proc -> !e10s
NOTE: multiple variants are stored in the field test_variant concatenated by '+',
for example: "socketprocess_networking+!fission"
b. Variables used in JavaScript are set:
c. Use arch instead of processor
aarch64
arm7
x86
x86_64
d. Build types
An an optimized build "opt" implies none of these build types: "debug", "ccov", "asan", "tsan".
A build type "!debug" implies "opt"
e. A "platform match" for the purpose of bug carryover means
- "os" is not specified in the existing condition or "os" is the same, and
- The existing build type is not present and the "os" explictly the same, or
- The existing build type is present:
+ "opt" (or "!debug") in both existing skip-if and proposed skip-if
+ "!opt" or one of ["debug", "asan", "tsan", "ccov" ] in both
(covers the case of "debug" in both)
EXCEPT does not match if one is "debug" and the other is
in ["asan", "tsan", "ccov"]
- Else not a match
f. If the argument `-n, --new-version NEW_VERSION` is used
then the os_version used in any skip-if expression
will be NEW_VERSION -- regardless of the os, arch or
another actual platform information. This is intended to workaround a bug
where task metadata would report "macosx 10.15", yet the worker was actually
"macosx 14.70". By setting `--new-version 14.70` then any skip-if
expression will be created with `os_version == 14.70` which would then match
what mozinfo says on the next try run.
3. Bugzilla interaction
a. The bugzilla instance will be set to (in increasing precedence):
i. Default: "bugzilla.allizom.org"
ii. From the environment variable BUGZILLA
iii. From the command line arg --bugzilla
b. NOTE: if the BUGZILLA env var or command line arg is "disable"
then Bugzilla interaction will be disabled.
c. Setting --dry-run prevents editing manifests, but will not disable reading Bugzilla.
Nevertheless no new bugs will be created nor comments added.
Note existing bugs may be mentioned in an info message as appropriate.
d. If --meta-bug-id is NOT set no bugs will be filed. Any created manifest bugs will
require referencing a meta-bug-id.
e. If there is a skip line for this manifest and the platform matches (see above) then
add an info message about carryover
do NOT edit any existing bug nor create a new bug
If there is an existing manifest comment, then carry it over to the new skip line
Else do NOT add a new comment (as a comment did not exist before)
Else (no skip lines match the platform)
Add a new skip-if line and reference the bug for this manifest + meta-bug-id
(create a new bug if needed)
f. Special case for carryover when a [default] section exists
and contains a skip-if with a platform match
AND our current failure matches that skip-if line for a real manifest
THEN carry over the comment from [default] to the new skip-if line.
4. Modes for skip-fails
a. Normal mode
Tests with failures found based on the above heuristics
are disabled with skip-if conditions in the appropriate manifests
and bugs filed (or updated).
b. Carry-over mode
Only consider adding skip-if conditions which match the
platform (see above) of previous conditions. Does not perform any
bugzilla changes. Requires meta-bug-id ( future actions will be cached in
{topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )
c. Known intermittents mode
Only consider adding skip-if conditions which have known
intermittent bugs (and are not carryover bugs).
form (see above) of previous conditions. Does not perform any
bugzilla changes. Requires meta-bug-id ( future actions will be cached in
{topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )
For each failure, if the job_id can be determined then bug_suggestions will
be retrieved from
each object with "search" field of ^PROCESS-CRASH or ^TEST-UNEXPECTED
will be considered if one of the bugs in
bugs.open_recent.<i>.summary ends with 'single tracking bug$'
then
bugs.open_recent.<i>.id has the bug id
d. New failures mode
Will only edit manifest skip-if conditions for new failures
(i.e. not carryover nor known intermittents). Will record
bug as "TBD". Requires meta-bug-id ( future actions will be cached in
{topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )
e. Replace TBD mode
Will only edit manifest skip-if conditions for new failures
by filing new bugs and replacing TBD with actual bug number.
Requires meta-bug-id ( new bug actions will be read from
{topsrcdir}/.skip_fails_cache/meta-bug-{meta_bug_id}/actions.json )
5. Cache for skip-fails
At the top of the source tree is a cache directory for skip-fails:
".skip_fails_cache". This directory will be created if it is not present.
When running skip-fails if --use-tasks has not been set then
the cache directory will be searched for a tasks file corresponding
the revision (i.e. ".skip_fails_cache/REVISION/tasks.json"). If
that file is present then it is used. Similarly for --use-failures.
If --save-tasks has not been set then, by default, the tasks will
be saved in the cache directory. Similarly for --save-failures.
Any specific cache directory can be cleared with --clear-cache REVISION.
All cache directories can be deleted with --clear-cache (no argument).
Any cache directories which are over 45 days old will be automatically
cleared.
As design changes for skip-fails / TOML, notes will be added here.
Running skip-fails for WPT
---------------------------
The user documentation for WPT manifests is:
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the ratio of failed runs / total runs is less than 40%
classify "intermittent"
d. Else edit the manifest to skip this failure
As design changes for skip-fails / WPT, notes will be added here.
Running skip-fails for REFTEST
------------------------------
The user documentation for REFTEST manifests is:
Current Design notes for skip-fails / REFTEST:
As skip-fails for REFTEST finds a failure it will take the following actions:
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the ratio of failed runs / total runs is less than 40%
classify "low frequency intermittent" (INTERMITTENT)
c. If the ratio of failed runs / total runs is less than 80%
classify "high frequency intermittent" (DISABLE_INTERMITTENT)
d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE)
2. Propose a new range (differences, pixels)
a. For "low frequency intermittent" errors do nothing
b. For "high frequency intermittent" set the range 0-Y (i.e. always
start at zero to avoid UNEXPECTED PASS)
c. For "failure" set the range X-Y (i.e. the range as reported in
reftest_errorsummary.log). If the failure is UNEXPECTED PASS
then set the range to 0-Y.
d. NOTE: The maximum for differences and pixels will be increased
by 5% over the reported values in reftest_errorsummary.log
e. NOTE: If for whatever reason the range is calculated to be 0-0 then
remove the fuzzy-if entirely
3. Merge with an existing fuzzy-if for this OS platform if the other
criteria differ in only one dimension. Current dimensions include:
OS: gtkWidget, winWidget, cocoaWidget, Android
build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer
is64Bit
fission
useDrawSnapshot
swgl
<OTHER>
4. Based on a command line switch choose do use implicit variables (or not).
a. The following variables can implicit values:
is64Bit: true
fission: true
useDrawSnapshot: false
swgl: false
If the value of an implicit variable matches it's default then it may be
elided from the conditional expression.
If the value of an implicit variable is the oppposite of it's default
value then it may be specified in the condition, e.g. !is64Bit
When merging an existing condition with a new failure condition where
the one dimension of difference is an implicit variable then the
condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit)
That is essential so that the reftest sandbox does not add the implicit
variable and cover only one of the two possible values.
b. When implicit variables are not used then the following heuristics apply:
The variable is64Bit will be elided for all platforms except for winWidget.
The variable useDrawSnapshot will be elided for all platforms except gtkWidget.
5. Position of fuzzy-if conditions
In reftest manifest the rightmost applicable condition is used.
a. First sorting criterion is based on OS in this order:
OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"]
This is because appleSilicon is a narrower specification than
cocoaWidget and would be ignored if not to the right of cocoaWidget.
b. The second sorting criterion is the number of dimensions that participate
in the expression (including implicits) from least dimensions to
greatest dimensions.
The last fuzzy-if will be placed on the line immediately to the left of the
"type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present).
Non fuzzy-if statements will appear before the fuzzy-if statements.
In this way the more generic conditions will only be superceded by very specific
conditions.
Design Specification for ./mach manifest clean-skip-fails
=========================================================
The purpose / rationale for this command is to-be-documented.
description="Update manifests to remove skip-if conditions for a specific platform.
Only works for TOML manifests.",
The user documentation is:
Usage
-----
Sub Command Arguments:
manifest_search_path Path to the folder containing the manifests to update,
or the path to a single manifest
-o, --os OS_NAME OS to remove (linux, mac, win)
-s, --os_version OS_VERSION
Version of the OS to remove (eg: 18.04 for linux)
-p, --processor PROCESSOR
Type of processor architecture to remove (eg: x86)
Design Specification for ./mach manifest high-freq-skip-fails
=========================================================
The purpose / rationale for this command is to-be-documented.
description="Update manifests to skip failing tests",
The user documentation is:
Usage
-----
Sub Command Arguments:
-f, --failures FAILURES
Minimum number of failures for the bug to be skipped
-d, --days DAYS Number of days to look for failures since now