Source code

Revision control

Other Tools

1
Expectation Data
2
================
3
4
Introduction
5
------------
6
7
For use in continuous integration systems, and other scenarios where
8
regression tracking is required, wptrunner supports storing and
9
loading the expected result of each test in a test run. Typically
10
these expected results will initially be generated by running the
11
testsuite in a baseline build. They may then be edited by humans as
12
new features are added to the product that change the expected
13
results. The expected results may also vary for a single product
14
depending on the platform on which it is run. Therefore, the raw
15
structured log data is not a suitable format for storing these
16
files. Instead something is required that is:
17
18
* Human readable
19
20
* Human editable
21
22
* Machine readable / writable
23
24
* Capable of storing test id / result pairs
25
26
* Suitable for storing in a version control system (i.e. text-based)
27
28
The need for different results per platform means either having
29
multiple expectation files for each platform, or having a way to
30
express conditional values within a certain file. The former would be
31
rather cumbersome for humans updating the expectation files, so the
32
latter approach has been adopted, leading to the requirement:
33
34
* Capable of storing result values that are conditional on the platform.
35
36
There are few extant formats that meet these requirements, so
37
wptrunner uses a bespoke ``expectation manifest`` format, which is
38
closely based on the standard ``ini`` format.
39
40
Directory Layout
41
----------------
42
43
Expectation manifest files must be stored under the ``metadata``
44
directory passed to the test runner. The directory layout follows that
45
of web-platform-tests with each test path having a corresponding
46
manifest file. Tests that differ only by query string, or reftests
47
with the same test path but different ref paths share the same
48
reference file. The file name is taken from the last /-separated part
49
of the path, suffixed with ``.ini``.
50
51
As an optimisation, files which produce only default results
52
(i.e. ``PASS`` or ``OK``) don't require a corresponding manifest file.
53
54
For example a test with url::
55
56
/spec/section/file.html?query=param
57
58
would have an expectation file ::
59
60
metadata/spec/section/file.html.ini
61
62
63
.. _wptupdate-label:
64
65
Generating Expectation Files
66
----------------------------
67
68
wptrunner provides the tool ``wptupdate`` to generate expectation
69
files from the results of a set of baseline test runs. The basic
70
syntax for this is::
71
72
wptupdate [options] [logfile]...
73
74
Each ``logfile`` is a structured log file from a previous run. These
75
can be generated from wptrunner using the ``--log-raw`` option
76
e.g. ``--log-raw=structured.log``. The default behaviour is to update
77
all the test data for the particular combination of hardware and OS
78
used in the run corresponding to the log data, whilst leaving any
79
other expectations untouched.
80
81
wptupdate takes several useful options:
82
83
``--sync``
84
Pull the latest version of web-platform-tests from the
85
upstream specified in the config file. If this is specified in
86
combination with logfiles, it is assumed that the results in the log
87
files apply to the post-update tests.
88
89
``--no-check-clean``
90
Don't attempt to check if the working directory is clean before
91
doing the update (assuming that the working directory is a git or
92
mercurial tree).
93
94
``--patch``
95
Create a a git commit, or a mq patch, with the changes made by wptupdate.
96
97
``--ignore-existing``
98
Overwrite all the expectation data for any tests that have a result
99
in the passed log files, not just data for the same platform.
100
101
Examples
102
~~~~~~~~
103
104
Update the local copy of web-platform-tests without changing the
105
expectation data and commit (or create a mq patch for) the result::
106
107
wptupdate --patch --sync
108
109
Update all the expectations from a set of cross-platform test runs::
110
111
wptupdate --no-check-clean --patch osx.log linux.log windows.log
112
113
Add expectation data for some new tests that are expected to be
114
platform-independent::
115
116
wptupdate --no-check-clean --patch --ignore-existing tests.log
117
118
Manifest Format
119
---------------
120
The format of the manifest files is based on the ini format. Files are
121
divided into sections, each (apart from the root section) having a
122
heading enclosed in square braces. Within each section are key-value
123
pairs. There are several notable differences from standard .ini files,
124
however:
125
126
* Sections may be hierarchically nested, with significant whitespace
127
indicating nesting depth.
128
129
* Only ``:`` is valid as a key/value separator
130
131
A simple example of a manifest file is::
132
133
root_key: root_value
134
135
[section]
136
section_key: section_value
137
138
[subsection]
139
subsection_key: subsection_value
140
141
[another_section]
142
another_key: another_value
143
144
Conditional Values
145
~~~~~~~~~~~~~~~~~~
146
147
In order to support values that depend on some external data, the
148
right hand side of a key/value pair can take a set of conditionals
149
rather than a plain value. These values are placed on a new line
150
following the key, with significant indentation. Conditional values
151
are prefixed with ``if`` and terminated with a colon, for example::
152
153
key:
154
if cond1: value1
155
if cond2: value2
156
value3
157
158
In this example, the value associated with ``key`` is determined by
159
first evaluating ``cond1`` against external data. If that is true,
160
``key`` is assigned the value ``value1``, otherwise ``cond2`` is
161
evaluated in the same way. If both ``cond1`` and ``cond2`` are false,
162
the unconditional ``value3`` is used.
163
164
Conditions themselves use a Python-like expression syntax. Operands
165
can either be variables, corresponding to data passed in, numbers
166
(integer or floating point; exponential notation is not supported) or
167
quote-delimited strings. Equality is tested using ``==`` and
168
inequality by ``!=``. The operators ``and``, ``or`` and ``not`` are
169
used in the expected way. Parentheses can also be used for
170
grouping. For example::
171
172
key:
173
if (a == 2 or a == 3) and b == "abc": value1
174
if a == 1 or b != "abc": value2
175
value3
176
177
Here ``a`` and ``b`` are variables, the value of which will be
178
supplied when the manifest is used.
179
180
Expectation Manifests
181
---------------------
182
183
When used for expectation data, manifests have the following format:
184
185
* A section per test URL described by the manifest, with the section
186
heading being the part of the test URL following the last ``/`` in
187
the path (this allows multiple tests in a single manifest file with
188
the same path part of the URL, but different query parts).
189
190
* A subsection per subtest, with the heading being the title of the
191
subtest.
192
193
* A key ``expected`` giving the expectation value of each (sub)test.
194
195
* A key ``disabled`` which can be set to any value to indicate that
196
the (sub)test is disabled and should either not be run (for tests)
197
or that its results should be ignored (subtests).
198
199
* A key ``restart-after`` which can be set to any value to indicate that
200
the runner should restart the browser after running this test (e.g. to
201
clear out unwanted state).
202
203
* A key ``fuzzy`` that is used for reftests. This is interpreted as a
204
list containing entries like ``<meta name=fuzzy>`` content value,
205
which consists of an optional reference identifier followed by a
206
colon, then a range indicating the maximum permitted pixel
207
difference per channel, then semicolon, then a range indicating the
208
maximum permitted total number of differing pixels. The reference
209
identifier is either a single relative URL, resolved against the
210
base test URL, in which case the fuzziness applies to any
211
comparison with that URL, or takes the form lhs url, comparison,
212
rhs url, in which case the fuzziness only applies for any
213
comparison involving that specifc pair of URLs. Some illustrative
214
examples are given below.
215
216
* Variables ``debug``, ``os``, ``version``, ``processor`` and
217
``bits`` that describe the configuration of the browser under
218
test. ``debug`` is a boolean indicating whether a build is a debug
219
build. ``os`` is a string indicating the operating system, and
220
``version`` a string indicating the particular version of that
221
operating system. ``processor`` is a string indicating the
222
processor architecture and ``bits`` an integer indicating the
223
number of bits. This information is typically provided by
224
:py:mod:`mozinfo`.
225
226
* Top level keys are taken as defaults for the whole file. So, for
227
example, a top level key with ``expected: FAIL`` would indicate
228
that all tests and subtests in the file are expected to fail,
229
unless they have an ``expected`` key of their own.
230
231
An simple example manifest might look like::
232
233
[test.html?variant=basic]
234
type: testharness
235
236
[Test something unsupported]
237
expected: FAIL
238
239
[test.html?variant=broken]
240
expected: ERROR
241
242
[test.html?variant=unstable]
244
245
A more complex manifest with conditional properties might be::
246
247
[canvas_test.html]
248
expected:
249
if os == "osx": FAIL
250
if os == "windows" and version == "XP": FAIL
251
PASS
252
253
Note that ``PASS`` in the above works, but is unnecessary; ``PASS``
254
(or ``OK``) is always the default expectation for (sub)tests.
255
256
A manifest with fuzzy reftest values might be::
257
258
[reftest.html]
259
fuzzy: [10;200, ref1.html:20;200-300, subtest1.html==ref2.html:10-15;20]
260
261
In this case the default fuzziness for any comparison would be to
262
require a maximum difference per channel of less than or equal to 10
263
and less than or equal to 200 total pixels different. For any
264
comparison involving ref1.html on the right hand side, the limits
265
would instead be a difference per channel not more than 20 and a total
266
difference count of not less than 200 and not more than 300. For the
267
specific comparison subtest1.html == ref2.html (both resolved against
268
the test URL) these limits would instead be 10 to 15 and 0 to 20,
269
respectively.