Source code

Revision control

Copy as Markdown

Other Tools

.. _mozbuild-files:
=============== Files
```` files are the mechanism by which tree metadata (notably
the build configuration) is defined.
Directories in the tree contain ```` files which declare
functionality for their respective part of the tree. This includes
things such as the list of C++ files to compile, where to find tests,
```` files are actually Python scripts. However, their
execution is governed by special rules. This is explained below. Python Sandbox
As mentioned above, ```` files are Python scripts. However,
they are executed in a special Python *sandbox* that significantly
changes and limits the execution environment. The environment is so
different, it's doubtful most ```` files would execute without
error if executed by a vanilla Python interpreter (e.g. ``python``.
The following properties make execution of ```` files special:
1. The execution environment exposes a limited subset of Python.
2. There is a special set of global symbols and an enforced naming
convention of symbols.
3. Some symbols are inherited from previously-executed ````
The limited subset of Python is actually an extremely limited subset.
Only a few symbols from ``__builtin__`` are exposed. These include
``True``, ``False``, ``None``, ``sorted``, ``int``, and ``set``. Global
functions like ``import``, ``print``, and ``open`` aren't available.
Without these, ```` files can do very little. *This is by design*.
The execution sandbox treats all ``UPPERCASE`` variables specially. Any
``UPPERCASE`` variable must be known to the sandbox before the script
executes. Any attempt to read or write to an unknown ``UPPERCASE``
variable will result in an exception being raised. Furthermore, the
types of all ``UPPERCASE`` variables is strictly enforced. Attempts to
assign an incompatible type to an ``UPPERCASE`` variable will result in
an exception being raised.
The strictness of behavior with ``UPPERCASE`` variables is a very
intentional design decision. By ensuring strict behavior, any operation
involving an ``UPPERCASE`` variable is guaranteed to have well-defined
side-effects. Previously, when the build configuration was defined in
``Makefiles``, assignments to variables that did nothing would go
unnoticed. ```` files fix this problem by eliminating the
potential for false promises.
After a ```` file has completed execution, only the
``UPPERCASE`` variables are used to retrieve state.
The set of variables and functions available to the Python sandbox is
defined by the :py:mod:`mozbuild.frontend.context` module. The
data structures in this module are consumed by the
:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct
the sandbox. There are tests to ensure that the set of symbols exposed
to an empty sandbox are all defined in the ``context`` module.
This module also contains documentation for each symbol, so nothing can
sneak into the sandbox without being explicitly defined and documented.
Reading and Traversing Files
The process for reading ```` files roughly consists of:
1. Start at the root ```` (``<topsrcdir>/``).
2. Evaluate the ```` file in a new sandbox.
3. Emit the main *context* and any *sub-contexts* from the executed
4. Extract a set of ```` files to execute next.
5. For each additional ```` file, goto #2 and repeat until all
referenced files have executed.
From the perspective of the consumer, the output of reading is a stream
of :py:class:`mozbuild.frontend.reader.context.Context` instances. Each
``Context`` defines a particular aspect of data. Consumers iterate over
these objects and do something with the data inside. Each object is
essentially a dictionary of all the ``UPPERCASE`` variables populated
during its execution.
.. note::
Historically, there was only one ``context`` per ```` file.
As the number of things tracked by ```` files grew and more
and more complex processing was desired, it was necessary to split these
contexts into multiple logical parts. It is now common to emit
multiple contexts per ```` file.
Build System Reading Mode
The traditional mode of evaluation of ```` files is what's
called *build system traversal mode.* In this mode, the ``CONFIG``
variable in each ```` sandbox is populated from data coming
from ``config.status``, which is produced by ``configure``.
During evaluation, ```` files often make decisions conditional
on the state of the build configuration. e.g. *only compile foo.cpp if
feature X is enabled*.
In this mode, traversal of ```` files is governed by variables
like ``DIRS`` and ``TEST_DIRS``. For example, to execute a child
directory, ``foo``, you would add ``DIRS += ['foo']`` to a ````
file and ``foo/`` would be evaluated.
.. _mozbuild_fs_reading_mode:
Filesystem Reading Mode
There is an alternative reading mode that doesn't involve the build
system and doesn't use ``DIRS`` variables to control traversal into
child directories. This mode is called *filesystem reading mode*.
In this reading mode, the ``CONFIG`` variable is a dummy, mostly empty
object. Accessing all but a few special variables will return an empty
value. This means that nearly all ``if CONFIG['FOO']:`` branches will
not be taken.
Instead of using content from within the evaluated ````
file to drive traversal into subsequent ```` files, the set
of files to evaluate is controlled by the thing doing the reading.
A single ```` file is not guaranteed to be executable in
isolation. Instead, we must evaluate all *parent* ```` files
first. For example, in order to evaluate ``/foo/``, one must
execute ``/`` and have its state influence the execution of
Filesystem reading mode is utilized to power the
:ref:`mozbuild_files_metadata` feature.
Technical Details
The code for reading ```` files lives in
:py:mod:`mozbuild.frontend.reader`. The Python sandboxes evaluation results
(:py:class:`mozbuild.frontend.context.Context`) are passed into
:py:mod:`mozbuild.frontend.emitter`, which converts them to classes defined
in :py:mod:``. Each class in this module defines a
domain-specific component of tree metadata. e.g. there will be separate
classes that represent a JavaScript file vs a compiled C++ file or test
manifests. This means downstream consumers of this data can filter on class
types to only consume what they are interested in.
There is no well-defined mapping between ```` file instances
and the number of :py:mod:`` classes derived from
each. Depending on the content of the ```` file, there may be 1
object derived or 100.
The purpose of the ``emitter`` layer between low-level sandbox execution
and metadata representation is to facilitate a unified normalization and
verification step. There are multiple downstream consumers of the
````-derived data and many will perform the same actions. This
logic can be complicated, so we have a component dedicated to it.
:py:class:`mozbuild.frontend.reader.BuildReader`` and
:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a
stream-based API courtesy of generators. When you hook them up properly,
the :py:mod:`` classes are emitted before all
```` files have been read. This means that downstream errors
are raised soon after sandbox execution.
Lots of the code for evaluating Python sandboxes is applicable to
non-Mozilla systems. In theory, it could be extracted into a standalone
and generic package. However, until there is a need, there will
likely be some tightly coupled bits.