Source code
Revision control
Copy as Markdown
Other Tools
Design Specification for ./mach manifest skip-fails
===================================================
The first of the mach manifest subcommands is skip-fails. This command
can be used to automatically edit manifests to skip tests that are
failing as well as file the corresponding bugs for the failures. This
is particularly useful when “greening up” a new platform.
The user documentation for skip-fails is here:
The skip-fails command knows how to manage failures for TOML, WPT
and REFTEST manifests. Additional developer design notes are included here.
Running skip-fails for TOML
---------------------------
The user documentation for TOML manifests is:
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the ratio of failed runs / total runs is less than 40%
classify "intermittent"
d. Else edit the manifest to skip this failure
As design changes for skip-fails / TOML, notes will be added here.
Running skip-fails for WPT
---------------------------
The user documentation for WPT manifests is:
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the ratio of failed runs / total runs is less than 40%
classify "intermittent"
d. Else edit the manifest to skip this failure
As design changes for skip-fails / WPT, notes will be added here.
Running skip-fails for REFTEST
------------------------------
The user documentation for REFTEST manifests is:
Current Design notes for skip-fails / REFTEST:
As skip-fails for REFTEST finds a failure it will take the following actions:
1. Classify the failure based on the following:
a. If less than 3 runs, classify "unknown"
b. If zero failures, classify "success"
c. If the ratio of failed runs / total runs is less than 40%
classify "low frequency intermittent" (INTERMITTENT)
c. If the ratio of failed runs / total runs is less than 80%
classify "high frequency intermittent" (DISABLE_INTERMITTENT)
d. If the ratio is <= 100%, classify "failure" (DISABLE_FAILURE)
2. Propose a new range (differences, pixels)
a. For "low frequency intermittent" errors do nothing
b. For "high frequency intermittent" set the range 0-Y (i.e. always
start at zero to avoid UNEXPECTED PASS)
c. For "failure" set the range X-Y (i.e. the range as reported in
reftest_errorsummary.log). If the failure is UNEXPECTED PASS
then set the range to 0-Y.
d. NOTE: The maximum for differences and pixels will be increased
by 5% over the reported values in reftest_errorsummary.log
e. NOTE: If for whatever reason the range is calculated to be 0-0 then
remove the fuzzy-if entirely
3. Merge with an existing fuzzy-if for this OS platform if the other
criteria differ in only one dimension. Current dimensions include:
OS: gtkWidget, winWidget, cocoaWidget, Android
build_type: optimized, isDebugBuild, isCoverageBuild, AddressSanitizer, ThreadSanitizer
is64Bit
fission
useDrawSnapshot
swgl
<OTHER>
4. Based on a command line switch choose do use implicit variables (or not).
a. The following variables can implicit values:
is64Bit: true
fission: true
useDrawSnapshot: false
swgl: false
If the value of an implicit variable matches it's default then it may be
elided from the conditional expression.
If the value of an implicit variable is the oppposite of it's default
value then it may be specified in the condition, e.g. !is64Bit
When merging an existing condition with a new failure condition where
the one dimension of difference is an implicit variable then the
condition must include the disjunction of the variable, e.g. (is64Bit&&!is64Bit)
That is essential so that the reftest sandbox does not add the implicit
variable and cover only one of the two possible values.
b. When implicit variables are not used then the following heuristics apply:
The variable is64Bit will be elided for all platforms except for winWidget.
The variable useDrawSnapshot will be elided for all platforms except gtkWidget.
5. Position of fuzzy-if conditions
In reftest manifest the rightmost applicable condition is used.
a. First sorting criterion is based on OS in this order:
OSES = ["Android", "cocoaWidget", "appleSilicon", "gtkWidget", "winWidget"]
This is because appleSilicon is a narrower specification than
cocoaWidget and would be ignored if not to the right of cocoaWidget.
b. The second sorting criterion is the number of dimensions that participate
in the expression (including implicits) from least dimensions to
greatest dimensions.
The last fuzzy-if will be placed on the line immediately to the left of the
"type" (e.g. == or !=) (or to the left of an optional "HTTP" expression, if present).
Non fuzzy-if statements will appear before the fuzzy-if statements.
In this way the more generic conditions will only be superceded by very specific
conditions.