Source code

Revision control

Other Tools

1
Task Kinds
2
==========
3
4
This section lists and documents the available task kinds.
5
6
build
7
-----
8
9
Builds are tasks that produce an installer or other output that can be run by
10
users or automated tests. This is more restrictive than most definitions of
11
"build" in a Mozilla context: it does not include tasks that run build-like
12
actions for static analysis or to produce instrumented artifacts.
13
14
build-fat-aar
15
-------------
16
17
Build architecture-independent GeckoView AAR (Android ARchive) files. This build-like tasks is an
18
artifact build (ARMv7, but this is arbitrary) that itself depends on arch-specific Android build
19
jobs. It fetches arch-specific AAR files, extracts arch-specific libraries and preference files,
20
and then assembles a multi-architecture "fat AAR". Downstream consumers are expected to use
21
per-ABI feature splits to produce arch-specific APKs.
22
23
If you want to run this task locally, you need to specify these environment variable:
24
- MOZ_ANDROID_FAT_AAR_ARCHITECTURES: must be a comma-separated list of architecture.
25
Eg: "armeabi-v7a,arm64-v8a,x86,x86_64".
26
- each of MOZ_ANDROID_FAT_AAR_ARM64_V8A, MOZ_ANDROID_FAT_AAR_ARMEABI_V7A,
27
MOZ_ANDROID_FAT_AAR_X86, MOZ_ANDROID_FAT_AAR_X86_64 must be a path relative to
28
MOZ_FETCHES_DIR.
29
30
build-signing
31
-------------
32
33
Many builds must be signed. The build-signing task takes the unsigned `build`
34
kind artifacts and passes them through signingscriptworker to a signing server
35
and returns signed results.
36
37
For mac notarization, we download the signed bits that have been notarized by Apple, and we staple the notarization to the app and pkg.
38
39
build-notarization-part-1
40
-------------------------
41
42
We switched to a 3-part mac notarization workflow in bug 1562412. This is the first task, which signs the files and submits them for notarization.
43
44
build-notarization-poller
45
-------------------------
46
47
We switched to a 3-part mac notarization workflow in bug 1562412. This is the second task, which polls Apple for notarization status. Because this is run in a separate, special notarization poller pool, we free up the mac notarization pool for actual signing work.
48
49
artifact-build
50
--------------
51
52
This kind performs an artifact build: one based on precompiled binaries
53
discovered via the TaskCluster index. This task verifies that such builds
54
continue to work correctly.
55
56
hazard
57
------
58
59
Hazard builds are similar to "regular' builds, but use a compiler extension to
60
extract a bunch of data from the build and then analyze that data looking for
61
hazardous behaviors.
62
63
l10n
64
----
65
66
The l10n kind takes the last published nightly build, and generates localized builds
67
from it. You can read more about how to trigger these on the `wiki
69
70
shippable-l10n
71
--------------
72
73
The nightly l10n kind repacks a specific nightly build (from the same source code)
74
in order to provide localized versions of the same source.
75
76
shippable-l10n-signing
77
----------------------
78
79
The shippable l10n signing kind takes artifacts from the shippable-l10n kind and
80
passes them to signing servers to have their contents signed appropriately, based
81
on an appropriate signing format. One signing job is created for each shippable-l10n
82
job (usually chunked).
83
84
For mac notarization, we download the signed bits that have been notarized by Apple, and we staple the notarization to the app and pkg.
85
86
shippable-l10n-notarization-part-1
87
----------------------------------
88
89
We switched to a 3-part mac notarization workflow in bug 1562412. This is the first task, which signs the files and submits them for notarization.
90
91
shippable-l10n-notarization-poller
92
----------------------------------
93
94
We switched to a 3-part mac notarization workflow in bug 1562412. This is the second task, which polls Apple for notarization status. Because this is run in a separate, special notarization poller pool, we free up the mac notarization pool for actual signing work.
95
96
source-test
97
-----------
98
99
Source-tests are tasks that run directly from the Gecko source. This can include linting,
100
unit tests, source-code analysis, or measurement work. While source-test tasks run from
101
a source checkout, it is still possible for them to depend on a build artifact, though
102
often they do not.
103
104
code-review
105
-----------
106
107
Publish issues found by source-test tasks on Phabricator.
108
This is a part of Release Management code review Bot.
109
110
upload-symbols
111
--------------
112
113
Upload-symbols tasks run after builds and upload the symbols files generated by
114
build tasks to Socorro for later use in crash analysis.
115
116
upload-generated-sources
117
------------------------
118
119
Upload-generated-sources tasks run after builds and upload source files that were generated as part of the build process to an s3 bucket for later use in links from crash reports or when debugging shipped builds.
120
121
valgrind
122
--------
123
124
Valgrind tasks produce builds instrumented by valgrind.
125
126
searchfox
127
---------
128
129
Searchfox builds generate C++ index data for Searchfox.
130
131
static-analysis-autotest
132
------------------------
133
134
Static analysis autotest utility in order to be sure that there is no regression
135
when upgrading utilities that impact static-analysis.
136
137
toolchain
138
---------
139
140
Toolchain builds create the compiler toolchains used to build Firefox. These
141
will eventually be dependencies of the builds themselves, but for the moment
142
are run manually via try pushes and the results uploaded to tooltool.
143
144
spidermonkey
145
------------
146
147
Spidermonkey tasks check out the full gecko source tree, then compile only the
148
spidermonkey portion. Each task runs specific tests after the build.
149
150
test
151
----
152
153
The ``desktop-test`` kind defines tests for builds. Its ``tests.yml`` defines
154
the full suite of desktop tests and their particulars, leaving it to the
155
transforms to determine how those particulars apply to the various platforms.
156
157
The process of generating tests goes like this, based on a set of YAML files
158
named in ``kind.yml``:
159
160
* For each build task, determine the related test platforms based on the build
161
platform. For example, a Windows 2010 build might be tested on Windows 7
162
and Windows 10. Each test platform specifies "test sets" indicating which
163
tests to run. This is configured in the file named
164
``test-platforms.yml``.
165
166
* Each test set is expanded to a list of tests to run. This is configured in
167
the file named by ``test-sets.yml``. A platform may specify several test
168
sets, in which case the union of those sets is used.
169
170
* Each named test is looked up in the file named by ``tests.yml`` to find a
171
test description. This test description indicates what the test does, how
172
it is reported to treeherder, and how to perform the test, all in a
173
platform-independent fashion.
174
175
* Each test description is converted into one or more tasks. This is
176
performed by a sequence of transforms defined in the ``transforms`` key in
177
``kind.yml``. See :doc:`transforms`: for more information on these
178
transforms.
179
180
* The resulting tasks become a part of the task graph.
181
182
.. important::
183
184
This process generates *all* test jobs, regardless of tree or try syntax.
185
It is up to a later stages of the task-graph generation (the target set and
186
optimization) to select the tests that will actually be performed.
187
188
docker-image
189
------------
190
191
Tasks of the ``docker-image`` kind build the Docker images in which other
192
Docker tasks run.
193
194
The tasks to generate each docker image have predictable labels:
195
``build-docker-image-<name>``.
196
197
Docker images are built from subdirectories of ``taskcluster/docker``, using
198
``docker build``. There is currently no capability for one Docker image to
199
depend on another in-tree docker image, without uploading the latter to a
200
Docker repository.
201
202
balrog
203
------
204
205
Balrog tasks are responsible for submitting metadata to our update server (Balrog).
206
They are typically downstream of a beetmover job that moves signed MARs somewhere
207
(eg: beetmover and beetmover-l10n for releases, beetmover-repackage for nightlies).
208
209
beetmover
210
---------
211
212
Beetmover, takes specific artifacts, "Beets", and pushes them to a location outside
213
of Taskcluster's task artifacts, (archive.mozilla.org as one place) and in the
214
process determines the final location and a "pretty" name (versioned product name)
215
216
beetmover-l10n
217
--------------
218
219
Beetmover L10n, takes specific artifacts, "Beets", and pushes them to a location outside
220
of Taskcluster's task artifacts, (archive.mozilla.org as one place) and in the
221
process determines the final location and a "pretty" name (versioned product name)
222
This separate kind uses logic specific to localized artifacts, such as including
223
the language in the final artifact names.
224
225
beetmover-repackage
226
-------------------
227
228
Beetmover-repackage is beetmover but for tasks that need an intermediate step
229
between signing and packaging, such as OSX. For more details see the definitions
230
of the Beetmover kind above and the repackage kind below.
231
232
release-beetmover-push-to-release
233
---------------------------------
234
235
release-beetmover-push-to-release publishes promoted releases from the
236
candidates directory to the release directory. This is part of release
237
promotion.
238
239
beetmover-snap
240
--------------
241
Beetmover-source publishes Ubuntu's snap. This is part of release promotion.
242
243
beetmover-source
244
----------------
245
Beetmover-source publishes release source. This is part of release promotion.
246
247
beetmover-geckoview
248
-------------------
249
Beetmover-geckoview publishes the Android library called "geckoview".
250
251
condprof
252
--------
253
condprof creates and updates realistic profiles.
254
255
release-source-checksums-signing
256
--------------------------------
257
release-source-checksums-signing take as input the checksums file generated by
258
source-related beetmover task and sign it via the signing scriptworkers.
259
Returns the same file signed and additional detached signature.
260
261
beetmover-checksums
262
-------------------
263
Beetmover, takes specific artifact checksums and pushes it to a location outside
264
of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
265
process determines the final location and "pretty" names it (version product name)
266
267
release-beetmover-source-checksums
268
----------------------------------
269
Beetmover, takes source specific artifact checksums and pushes it to a location outside
270
of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
271
process determines the final location and "pretty" names it (version product name)
272
273
push-apk
274
--------
275
PushApk publishes Android packages onto Google Play Store. Jobs of this kind take
276
all the signed multi-locales (aka "multi") APKs for a given release and upload them
277
all at once.
278
279
push-apk-checks
280
---------------
281
Runs the checks done in push-apk to ensure APKs are sane before submitting them
282
283
release-balrog-submit-toplevel
284
------------------------------
285
Toplevel tasks are responsible for submitting metadata to Balrog that is not specific to any
286
particular platform+locale. For example: fileUrl templates, versions, and platform aliases.
287
288
Toplevel tasks are also responsible for updating test channel rules to point at the Release
289
being generated.
290
291
release-secondary-balrog-submit-toplevel
292
----------------------------------------
293
Performs the same function as `release-balrog-submit-toplevel`, but against the beta channel
294
during RC builds.
295
296
release-balrog-scheduling
297
-------------------------
298
Schedules a Release for shipping in Balrog. If a `release_eta` was provided when starting the Release,
299
it will be scheduled to go live at that day and time.
300
301
release-secondary-balrog-scheduling
302
-----------------------------------
303
Performs the same function as `release-balrog-scheduling`, except for the beta channel as part of RC
304
Releases.
305
306
release-binary-transparency
307
---------------------------
308
Binary transparency creates a publicly verifiable log of binary shas for downstream
310
311
release-snap-repackage
312
----------------------
313
Generate an installer using Ubuntu's Snap format.
314
315
release-flatpak-repackage
316
-------------------------
317
Generate an installer using Flathub's Flatpak format.
318
319
release-snap-push
320
-----------------
321
Pushes Snap repackage on Snap store.
322
323
release-flatpak-push
324
--------------------
325
Pushes Flatpak repackage on Flathub
326
327
release-secondary-snap-push
328
---------------------------
329
Performs the same function as `release-snap-push`, except for the beta channel as part of RC
330
Releases.
331
332
release-notify-av-announce
333
--------------------------
334
Notify anti-virus vendors when a release is likely shipping.
335
336
release-notify-push
337
-------------------
338
Notify when a release has been pushed to CDNs.
339
340
release-notify-ship
341
-------------------
342
Notify when a release has been shipped.
343
344
release-secondary-notify-ship
345
-----------------------------
346
Notify when an RC release has been shipped to the beta channel.
347
348
release-notify-promote
349
----------------------
350
Notify when a release has been promoted.
351
352
release-notify-started
353
----------------------
354
Notify when a release has been started.
355
356
release-bouncer-sub
357
-------------------
358
Submits bouncer updates for releases.
359
360
release-bouncer-sub-nazgul
361
--------------------------
362
Submits bouncer updates for releases, using new implementation of bouncer-admin (Nazgul).
363
364
release-mark-as-shipped
365
-----------------------
366
Marks releases as shipped in Ship-It v1
367
368
release-bouncer-aliases
369
-----------------------
370
Update Bouncer's (download.mozilla.org) "latest" aliases.
371
372
cron-bouncer-check
373
------------------
374
Checks Bouncer (download.mozilla.org) uptake.
375
376
bouncer-locations
377
-----------------
378
Updates nightly bouncer locations for version bump
379
380
release-bouncer-check
381
---------------------
382
Checks Bouncer (download.mozilla.org) uptake as part of the release tasks.
383
384
release-generate-checksums
385
--------------------------
386
Generate the per-release checksums along with the summaries
387
388
release-generate-checksums-signing
389
----------------------------------
390
Sign the pre-release checksums produced by the above task
391
392
release-generate-checksums-beetmover
393
------------------------------------
394
Submit to S3 the artifacts produced by the release-checksums task and its signing counterpart.
395
396
release-final-verify
397
--------------------
398
Verifies the contents and package of release update MARs.
399
400
release-secondary-final-verify
401
------------------------------
402
Verifies the contents and package of release update MARs for RC releases.
403
404
release-push-langpacks
405
-------------------------------
406
Publishes language packs onto addons.mozilla.org.
407
408
release-beetmover-signed-langpacks
409
----------------------------------
410
Publishes signed langpacks to archive.mozilla.org
411
412
release-beetmover-signed-langpacks-checksums
413
--------------------------------------------
414
Publishes signed langpacks to archive.mozilla.org
415
416
release-update-verify
417
---------------------
418
Verifies the contents and package of release update MARs.
419
release-secondary-update-verify
420
-------------------------------
421
Verifies the contents and package of release update MARs.
422
423
release-update-verify-next
424
--------------------------
425
Verifies the contents and package of release and updare MARs from the previous ESR release.
426
427
release-update-verify-config
428
----------------------------
429
Creates configs for release-update-verify tasks
430
431
release-secondary-update-verify-config
432
--------------------------------------
433
Creates configs for release-secondary-update-verify tasks
434
435
release-update-verify-config-next
436
---------------------------------
437
Creates configs for release-update-verify-next tasks
438
439
release-updates-builder
440
-----------------------
441
Top level Balrog blob submission & patcher/update verify config updates.
442
443
release-version-bump
444
--------------------
445
Bumps to the next version.
446
447
release-source
448
--------------
449
Generates source for the release
450
451
release-source-signing
452
----------------------
453
Signs source for the release
454
455
release-partner-repack
456
----------------------
457
Generates customized versions of releases for partners.
458
459
release-partner-repack-chunking-dummy
460
-------------------------------------
461
Chunks the partner repacks by locale.
462
463
release-partner-repack-signing
464
------------------------------
465
Internal signing of partner repacks.
466
467
For mac notarization, we download the signed bits that have been notarized by Apple, and we staple the notarization to the app and pkg.
468
469
release-partner-repack-notarization-part-1
470
------------------------------------------
471
472
We switched to a 3-part mac notarization workflow in bug 1562412. This is the first task, which signs the files and submits them for notarization.
473
474
release-partner-repack-notarization-poller
475
------------------------------------------
476
477
We switched to a 3-part mac notarization workflow in bug 1562412. This is the second task, which polls Apple for notarization status. Because this is run in a separate, special notarization poller pool, we free up the mac notarization pool for actual signing work.
478
479
release-partner-repack-repackage
480
--------------------------------
481
Repackaging of partner repacks.
482
483
release-partner-repack-repackage-signing
484
----------------------------------------
485
External signing of partner repacks.
486
487
release-partner-repack-beetmover
488
--------------------------------
489
Moves the partner repacks to S3 buckets.
490
491
release-partner-repack-bouncer-sub
492
----------------------------------
493
Sets up bouncer products for partners.
494
495
496
release-early-tagging
497
---------------------
498
Utilises treescript to perform tagging that should happen near the start of a release.
499
500
release-eme-free-repack
501
-----------------------
502
Generates customized versions of releases for eme-free repacks.
503
504
release-eme-free-repack-signing
505
-------------------------------
506
Internal signing of eme-free repacks
507
508
For mac notarization, we download the signed bits that have been notarized by Apple, and we staple the notarization to the app and pkg.
509
510
release-eme-free-repack-notarization-part-1
511
-------------------------------------------
512
513
We switched to a 3-part mac notarization workflow in bug 1562412. This is the first task, which signs the files and submits them for notarization.
514
515
release-eme-free-repack-notarization-poller
516
-------------------------------------------
517
518
We switched to a 3-part mac notarization workflow in bug 1562412. This is the second task, which polls Apple for notarization status. Because this is run in a separate, special notarization poller pool, we free up the mac notarization pool for actual signing work.
519
520
release-eme-free-repack-repackage
521
---------------------------------
522
Repackaging of eme-free repacks.
523
524
release-eme-free-repack-repackage-signing
525
-----------------------------------------
526
External signing of eme-free repacks.
527
528
release-eme-free-repack-beetmover
529
---------------------------------
530
Moves the eme-free repacks to S3 buckets.
531
532
release-eme-free-repack-beetmover-checksums
533
-------------------------------------------
534
Moves the beetmover checksum for eme-free repacks to S3 buckets.
535
536
repackage
537
---------
538
Repackage tasks take a signed output and package them up into something suitable
539
for shipping to our users. For example, on OSX we return a tarball as the signed output
540
and this task would package that up as an Apple Disk Image (.dmg)
541
542
repackage-l10n
543
--------------
544
Repackage-L10n is a ```Repackage``` task split up to be suitable for use after l10n repacks.
545
546
547
repackage-signing
548
-----------------
549
Repackage-signing take the repackaged installers (windows) and signs them.
550
551
repackage-signing-l10n
552
----------------------
553
Repackage-signing-l10n take the repackaged installers (windows) and signs them for localized versions.
554
555
mar-signing
556
-----------
557
Mar-signing takes the complete update MARs and signs them.
558
559
mar-signing-l10n
560
----------------
561
Mar-signing-l10n takes the complete update MARs and signs them for localized versions.
562
563
mar-signing-autograph-stage
564
---------------------------
565
These tasks are only to test autograph-stage, when the autograph team asks for their staging environment to be tested.
566
567
repackage-msi
568
-------------
569
Repackage-msi takes the signed full installer and produces an msi installer (that wraps the full installer)
570
Using the ```./mach repackage``` command
571
572
repackage-signing-msi
573
---------------------
574
Repackage-signing-msi takes the repackaged msi installers and signs them.
575
576
repo-update
577
-----------
578
Repo-Update tasks are tasks that perform some action on the project repo itself,
579
in order to update its state in some way.
580
581
python-dependency-update
582
------------------------
583
Python-dependency-update runs `pip-compile --generate-hashes` against the specified `requirements.in` and
584
submits patches to Phabricator.
585
586
partials
587
--------
588
Partials takes the complete.mar files produced in previous tasks and generates partial
589
updates between previous nightly releases and the new one. Requires a release_history
590
in the parameters. See ``mach release-history`` if doing this manually.
591
592
partials-signing
593
----------------
594
Partials-signing takes the partial updates produced in Partials and signs them.
595
596
post-balrog-dummy
597
-----------------
598
Dummy tasks to consolidate balrog dependencies to avoid taskcluster limits on number of dependencies per task.
599
600
post-beetmover-dummy
601
--------------------
602
Dummy tasks to consolidate beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
603
604
post-beetmover-checksums-dummy
605
------------------------------
606
Dummy tasks to consolidate beetmover-checksums dependencies to avoid taskcluster limits on number of dependencies per task.
607
608
post-langpack-dummy
609
-------------------
610
Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
611
612
fetch
613
-----
614
Tasks that obtain something from a remote service and re-expose it as a
615
task artifact. These tasks are used to effectively cache and re-host
616
remote content so it is reliably and deterministically available.
617
618
packages
619
--------
620
Tasks used to build packages for use in docker images.
621
622
diffoscope
623
----------
624
Tasks used to compare pairs of Firefox builds using https://diffoscope.org/.
625
As of writing, this is mainly meant to be used in try builds, by editing
626
taskcluster/ci/diffoscope/kind.yml for your needs.
627
628
addon
629
-----
630
Tasks used to build/package add-ons.
631
632
openh264-plugin
633
---------------
634
Tasks used to build the openh264 plugin.
635
636
openh264-signing
637
----------------
638
Signing for the openh264 plugin.
639
640
webrender
641
---------
642
Tasks used to do testing of WebRender standalone (without gecko). The
643
WebRender code lives in gfx/wr and has its own testing infrastructure.
644
645
wgpu
646
---------
647
Tasks used to do testing of WebGPU standalone (without gecko). The
648
WebGPU code lives in gfx/wgpu and has its own testing infrastructure.
649
650
github-sync
651
------------
652
Tasks used to do synchronize parts of Gecko that have downstream GitHub
653
repositories.
654
655
instrumented-build
656
------------------
657
Tasks that generate builds with PGO instrumentation enabled. This is an
658
intermediate build that can be used to generate profiling information for a
659
final PGO build. This is the 1st stage of the full 3-step PGO process.
660
661
generate-profile
662
----------------
663
Tasks that take a build configured for PGO and run the binary against a sample
664
set to generate profile data. This is the 2nd stage of the full 3-step PGO
665
process.
666
667
geckodriver-repack
668
------------------
669
Tasks to repackage the geckodriver binary from a build tasks's common
670
test archive into it's own archive.
671
672
geckodriver-signing
673
-------------------
674
Signing for geckodriver binary.
675
676
visual-metrics
677
--------------
678
Tasks that compute visual performance metrics from videos and images captured
679
by other tasks.
680
681
visual-metrics-dep
682
------------------
683
Tasks that compute visual performance metrics from videos and images captured
684
by another task that produces a jobs.json artifact
685
686
iris
687
----
688
Iris testing suite
689
690
maybe-release
691
-------------
692
A shipitscript task that does the following:
693
694
1. Checks if automated releases are disabled
695
2. Checks if the changes between the current revision and the previous releases
696
revision are considered "worthwhile" for a new release.
697
3. Triggers the release via ship-it, which will then create an action task.
698
699
l10n-bump
700
---------
701
Cron-driven tasks that bump l10n-changesets files in-tree, using data from the l10n dashboard.
702
703
merge-automation
704
----------------
705
Hook-driven tasks that automate "Merge Day" tasks during the release cycle.
706
707
system-symbols
708
--------------
709
Generate missing macOS and windows system symbols from crash reports.
710
711
system-symbols-upload
712
---------------------
713
Upload macOS and windows system symbols to tecken.