| bhr_collection.py |
BigQuery reader for the BHR aggregation pipeline.
Streams hang-report pings from BigQuery for a given build-date window,
filtered down to the columns the downstream pipeline actually consumes.
Ported from the original python_mozetl/mozetl/bhr_collection/bhr_collection.py. The
SQL and FARM_FINGERPRINT-based deterministic sampling are unchanged. The
PySpark BigQuery connector has been replaced with the google-cloud-bigquery
Python client; rows are yielded one at a time rather than materialised into
a Spark DataFrame so memory stays bounded at production sample sizes
(~500K rows × ~5-10 KB each would otherwise need 3-5 GB on a single worker).
The google-cloud-bigquery import is deferred to the call site so this
module can be imported in environments where the package isn't installed
(e.g. unit tests that mock the client). Production runs need the package
present in the runtime — that's handled by the TaskCluster Docker image
in a later phase of the migration.
|
27753 |
- |
| bhr_timeseries.py |
Secondary BHR job: build per-signature daily timeseries from the primary
job's historical daily artifacts.
The primary job (bhr_collection.aggregate) writes one columnar profile per
build date. This job consumes a rolling window of those daily artifacts and
produces a single compact artifact giving, for the top-M hang signatures over
the window, their hang milliseconds and count on each day.
A hang signature is identified across days by a content-based key derived from
its reconstructed stack -- the ordered list of (funcName, libName) frames,
leaf->root. funcTable indices are per-file and cannot be used. The frontend
derives the same key from the per-signature `frames` arrays this job emits, so
the two sides join without any shared hashing.
The job is incremental and self-healing. It keeps a state file holding, per day
in the window, that day's top-N signatures (a cushion above the published
top-M). On each run it prunes days that fell out of the window and fills any
in-window day that has an artifact but no stored entry; only newly seen days are
re-read.
A day in a published series is `null`, not 0, when we do not know what happened:
either that day has no artifact, or the day was truncated to its top-N and this
signature fell below the cut. 0 is reserved for days we can positively say the
signature did not hang, which is only when the day kept every signature it saw.
The distinction matters downstream: a zero drawn on a chart asserts the hang did
not occur, when often it just was not big enough that day to be retained.
|
15698 |
- |
| client_metrics.py |
Distinct affected-client counting for BHR via HyperLogLog.
For the "percentage of affected users" metric we need a distinct-client count
per hang signature that is small and, above all, mergeable across days, so a
trailing 7 / 28 / 365 day window can be counted without re-scanning every user.
HyperLogLog gives exactly that: a bounded sketch that unions by a register-wise
maximum.
p=11 gives 2048 registers, measured at about 1.5% mean relative error and 2.6%
at the 95th percentile over the cardinalities we see, which is well inside what
a percentage-of-users figure needs. Precision is deliberately modest because a
sketch is stored per signature per day for a 365-day window, so the per-sketch
size matters far more here than the last fraction of a percent.
Only sketches and counts are ever emitted, never client ids. The HLL is a
minimal pure-Python implementation (no dependency); a production build could
swap in Apache datasketches or BigQuery HLL_COUNT without changing the
interface.
|
5396 |
- |
| heuristics.py |
Hang-signature heuristics for BHR stack aggregation.
Ported from the frontend's getHangFrames in
https://github.com/mozilla/hang-stats/blob/master/bhr.js so that the daily
aggregation job can trim stacks upstream instead of the frontend doing it on
every page render.
History:
- Originally introduced in python_mozetl issue #410 / PR (heuristics
migration) as a Python port of the JS algorithm, with byte-for-byte parity
verified against 2,080 real recorded samples.
- Moved here as part of the bhr_collection migration from python_mozetl into
mozilla-central. The algorithm is unchanged.
The three heuristics:
1. Nested event loop trim — stop walking the stack at the innermost
``nsThread::ProcessNextEvent`` frame. Frames outside that point are
ancestor event loops that don't help identify the hang.
2. Non-Mozilla code collapse — once we cross into Mozilla code walking from
leaf to root, drop all but the immediate entry-point non-Mozilla frame.
We care HOW Firefox code reached system libraries, not the system internals.
3. SpiderMonkey internals strip — between two JS frames, drop frames that
are recognizable as JS engine internals. The JS interpreter machinery
isn't useful signal for hang triage.
Plus an XPConnect-glue special case that strips XPC_WN_* / XPCWrappedNative
/ XPTC__InvokebyIndex chains when they sit between a JS frame and native code.
The function operates on a symbolicated stack — a list of
``(func_name, lib_name)`` tuples in outer-first (root -> leaf) order.
|
5306 |
- |
| leaf_grouping.py |
Group near-duplicate hang signatures by their first *meaningful* frame.
Many hang signatures are really the same underlying hang: they share the same
Firefox code getting stuck and only differ in noise below it (which system
call, allocator path, or JS-interpreter trampoline happened to be sampled; how
deep the nested event loop was). Because a signature is the whole stack, every
such variation becomes its own row, so the true size of a hang is split across
many near-identical entries and the list reads as noise.
This module collapses those signatures server-side. The key idea, following the
hang-triage heuristics, is that the frame that *identifies* a hang is not the
raw leaf (which is usually a lock, a sleep, an allocation, or an event-loop
wait) but the first frame that names actual Firefox work. So before bucketing
we normalize each stack down to its meaningful frames and group by the leaf of
*that*. Concretely, walking from the raw leaf downward we skip:
- external code - system / third-party libraries (ntdll, kernelbase, user32,
vcruntime, ...). We care how Firefox code reached them, not their internals.
- synchronization + allocation primitives - condition-variable / mutex
wrappers and the jemalloc/arena/operator-new machinery. A hang "in a lock"
or "in free()" is identified by *what* is locking or freeing, not the
primitive, so ``SleepConditionVariableSRW`` / ``je_free`` are never a name.
- SpiderMonkey + XPConnect glue - the interpreter/JIT trampolines and
native<->JS call plumbing between script frames. What the engine is doing
internally is rarely the hang; the JS entry point (e.g. ``js::Stringify``)
is kept.
- event-loop + app-run machinery - ``NS_ProcessNextEvent``, the message
pump, ``TaskController``, ``XRE_main`` and friends. Ancestor/nested event
loops don't identify the hang; instead their count is summarized per group
as ``avgEventLoopDepth`` so the nesting isn't lost entirely.
We then bucket by the meaningful leaf, find the common trunk each bucket shares
walking from that leaf downward, and label the group by where its stacks first
branch apart. Only multi-member groups are emitted. The job emits the groups
pre-grouped and pre-named so the frontend does no fuzzy matching of its own;
each group identifies its members so the dashboard can join a displayed
signature to its group.
A member is identified by ``frameKeys``: its stack as funcTable indices,
leaf -> root. The frontend resolves those against the same funcTable and
recomputes the canonical key (see stack_keys), so the two sides still agree
byte-for-byte without any shared hashing. Emitting indices rather than the key
itself keeps the block small: the columnar profile already interns every one of
those strings, and spelling them out again per member made leafGroups roughly
90% of the artifact.
Normalization only chooses how signatures are bucketed and named; it never
changes a signature's identity.
|
16414 |
- |
| moz.build |
|
329 |
- |
| profile_processor.py |
Columnar data structures and the ProfileProcessor aggregator for BHR.
Ported from python_mozetl/mozetl/bhr_collection/bhr_collection.py as part of
the bhr_collection migration. The semantics are unchanged — these classes
take symbolicated, heuristic-trimmed hang samples and build the columnar
output schema (stackTable / funcTable / stringArray / sampleTable /
annotationsTable / dates / libs) the frontend consumes.
The "(root)" sentinel at index 0 of stackTable / pruneStackCache is
intentional: the frontend's stack walker terminates when prefix == 0, so
keeping that slot reserved is load-bearing.
|
18397 |
- |
| stack_keys.py |
Canonical hang-signature keys, shared by the BHR aggregation jobs.
A hang signature is identified by a content-based key derived from its
reconstructed stack: the ordered list of (funcName, libName) frames, leaf->root.
funcTable indices are per-file and cannot be used.
The key has to stay byte-for-byte identical everywhere it is derived, including
the dashboard outside this repo, because that is what joins a displayed hang to
its rolled-up numbers. It lives here so there is a single implementation to keep
in step rather than one per job.
|
2547 |
- |
| symbolication.py |
Symbol-server I/O and breakpad ``.sym`` parsing for BHR aggregation.
Ported from python_mozetl/mozetl/bhr_collection/bhr_collection.py as part of
the bhr_collection migration. Pure-stdlib relocation; semantics are
unchanged.
The Mozilla symbol server returns text in breakpad's ``.sym`` format. Each
file describes one module: ``PUBLIC`` lines map exported names to addresses,
``FUNC`` lines map function symbols to address ranges. ``make_sym_map``
parses one ``.sym`` blob into a ``{address: symbol}`` dict (plus a sorted
key list for bisecting). ``process_module`` is the per-module pipeline:
fetch the ``.sym``, parse it, resolve each requested offset.
``.sym`` files also carry ``INLINE`` / ``INLINE_ORIGIN`` records describing
functions the compiler inlined into a ``FUNC``. An address inside an inlined
region has no return address on the native BHR stack, so without these records
the same source-level call path can appear as two different native stacks
(depending on each build's inlining decisions) and fail to dedup. ``parse_inlines``
resolves the inline ranges so a single address expands to the full inlined chain
``[FUNC, inline@nest0, inline@nest1, ...]`` (outer-first), which lets equivalent
hangs merge. Because of this, ``process_module`` resolves each offset to a
*list* of ``(symbol, module_name)`` frames (length 1 when there are no inlines).
Inlines are parsed in a second pass targeted at only the functions the requested
offsets landed in, since a debug ``.sym`` holds far more ``INLINE`` records than
we ever need. See bug 2052961.
|
18055 |
- |
| tests |
|
|
- |