Source code

Revision control

Other Tools

1
.. _localization:
2
3
================
4
Localized Builds
5
================
6
7
Localization repacks
8
====================
9
10
To save on build time, the build system and automation collaborate to allow
11
downloading a packaged en-US Firefox, performing some locale-specific
12
post-processing, and re-packaging a locale-specific Firefox. Such artifacts
13
are termed "single-locale language repacks". There is another concept of a
14
"multi-locale language build", which is more like a regular build and less
15
like a re-packaging post-processing step.
16
17
.. note::
18
19
These builds rely on make targets that don't work for
21
22
Instructions for single-locale repacks for developers
23
-----------------------------------------------------
24
25
This assumes that ``$AB_CD`` is the locale you want to repack with; you
26
find the available localizations on `l10n-central <https://hg.mozilla.org/l10n-central/>`_.
27
28
#. You must have a built and packaged object directory, or a pre-built
29
``en-US`` package.
30
31
.. code-block:: shell
32
33
./mach build
34
./mach package
35
36
#. Repackage using the locale-specific changes.
37
38
.. code-block:: shell
39
40
./mach build installers-$AB_CD
41
42
You should find a re-packaged build at ``OBJDIR/dist/``, and a
43
runnable binary in ``OBJDIR/dist/l10n-stage/``.
44
The ``installers`` target runs quite a few things for you, including getting
45
the repository for the requested locale from
46
https://hg.mozilla.org/l10n-central/. It will clone them into
47
``~/.mozbuild/l10n-central``. If you have an existing repository there, you
48
may want to occasionally update that via ``hg pull -u``. If you prefer
49
to have the l10n repositories at a different location on your disk, you
50
can point to the directory via
51
52
.. code-block:: shell
53
54
ac_add_options --with-l10n-base=/make/this/a/absolute/path
55
56
This build also packages a language pack.
57
58
Instructions for language packs
59
-------------------------------
60
61
Language packs are extensions that contain just the localized resources. Building
62
them doesn't require an actual build, but they're only compatible with the
63
``mozilla-central`` source they're built with.
64
65
66
.. code-block:: shell
67
68
./mach build langpack-$AB_CD
69
70
This target shares much of the logic of the ``installers-$AB_CD`` target above,
71
and does the check-out of the localization repository etc. It doesn't require
72
a package or a build, though. The generated language pack is in
73
``OBJDIR/dist/$(MOZ_PKG_PLATFORM)/xpi/``.
74
75
.. note::
76
77
Despite the platform-dependent location in the build directory, language packs
78
are platform independent, and the content that goes into them needs to be
79
built in a platform-independent way.
80
81
Instructions for multi-locale builds
82
------------------------------------
83
84
If you want to create a single build with multiple locales, you will do
85
86
#. Create a build and package
87
88
.. code-block:: shell
89
90
./mach build
91
./mach package
92
93
#. For each locale you want to include in the build:
94
95
.. code-block:: shell
96
97
export MOZ_CHROME_MULTILOCALE="de it zh-TW"
98
for AB_CD in $MOZ_CHROME_MULTILOCALE; do
99
./mach build chrome-$AB_CD
100
done
101
102
#. Create the multilingual package:
103
104
.. code-block:: shell
105
106
AB_CD=multi ./mach package
107
108
General flow of repacks
109
-----------------------
110
111
The general flow of the locale repacks is controlled by
112
``$MOZ_BUILD_APP/locales/Makefile.in`` and ``toolkit/locales/l10n.mk``, plus
113
the packaging build system. The three main entry points above all trigger
114
related build flows:
115
116
#. Get the localization repository, if needed
117
#. Run l10n-merge with a prior clobber of the merge dir
118
#. Copy l10n files to ``dist``, with minor differences here between ``libs-%`` and ``chrome-%``
119
#. Repackage and package
120
121
Details on l10n-merge are described in its own section below.
122
The copying of files is mainly controlled by ``jar.mn``, in the few source
123
directories that include localizable files. ``libs-%`` is used for repacks,
124
``chrome-%`` for multi-locale packages. The repackaging is dedicated
125
Python code in ``toolkit/mozapps/installer/l10n-repack.py``, using an existing
126
package. It strips existing ``chrome`` l10n resources, and adds localizations
127
and metadata.
128
129
Language packs don't require repackaging. The windows installers are generated
130
by merely packaging an existing repackaged zip into to an installer.
131
132
Exposing strings
133
================
134
135
The localization flow handles a few file formats in well-known locations in the
136
source tree.
137
138
Alongside being built by including the directory in ``$MOZ_BUILD_APP/locales/Makefile.in``
139
and respective entries in a ``jar.mn``, we also have configuration files tailored
140
to localization tools and infrastructure. They're also controlling which
141
files l10n-merge handles, and how.
142
143
These configurations are TOML files. They're part of the bigger
144
localization ecosystem at Mozilla, and `the documentation about the
146
explains how to set them up, and what the entries mean. In short, you find
147
148
.. code-block:: toml
149
150
[[paths]]
151
reference = browser/locales/en-US/**
152
l10n = {l}browser/**
153
154
to add a directory for all localizations. Changes to these files are best
155
submitted for review by :Pike or :flod.
156
157
These configuration files are the future, and right now, we still have
158
support for the previous way to configuring l10n, which is described below.
159
160
The locations are commonly in directories like
161
162
:file:`browser/`\ ``locales/en-US/``\ :file:`subdir/file.ext`
163
164
The first thing to note is that only files beneath :file:`locales/en-US` are
165
exposed to localizers. The second thing to note is that only a few directories
166
are exposed. Which directories are exposed is defined in files called
167
``l10n.ini``, which are at a
169
in the source code.
170
171
An example looks like this
172
173
.. code-block:: ini
174
175
[general]
176
depth = ../..
177
178
[compare]
179
dirs = browser
180
browser/branding/official
181
182
[includes]
183
toolkit = toolkit/locales/l10n.ini
184
185
This tells the l10n infrastructure three things:
186
187
* resolve the paths against the directory two levels up
188
* include files in :file:`browser/locales/en-US` and
189
:file:`browser/branding/official/locales/en-US`
190
* load more data from :file:`toolkit/locales/l10n.ini`
191
192
For projects like Thunderbird and SeaMonkey in ``comm-central``, additional
193
data needs to be provided when including an ``l10n.ini`` from a different
194
repository:
195
196
.. code-block:: ini
197
198
[include_toolkit]
199
type = hg
200
mozilla = mozilla-central
202
l10n.ini = toolkit/locales/l10n.ini
203
204
This tells the l10n infrastructure where to find the repository, and where inside
205
that repository the ``l10n.ini`` file is. This is needed because for local
206
builds, :file:`mail/locales/l10n.ini` references
207
:file:`mozilla/toolkit/locales/l10n.ini`, which is where the comm-central
208
build setup expects toolkit to be.
209
210
Now that the directories exposed to l10n are known, we can talk about the
211
supported file formats.
212
213
File formats
214
------------
215
216
The following file formats are known to the l10n tool chains:
217
218
Fluent
219
Used in Firefox UI, both declarative and programmatically.
220
DTD
221
Deprecated. Used in XUL and XHTML.
222
Properties
223
Used from JavaScript and C++. When used from js, also comes with
225
ini
226
Used by the crashreporter and updater, avoid if possible.
227
inc
228
Used during builds, for example to create metadata for
229
language packs or bookmarks.
230
231
Adding new formats involves changing various different tools, and is strongly
232
discouraged.
233
234
Exceptions
235
----------
236
Generally, anything that exists in ``en-US`` needs a one-to-one mapping in
237
all localizations. There are a few cases where that's not wanted, notably
238
around locale configuration and locale-dependent metadata.
239
240
For optional strings and files, l10n-merge won't add ``en-US`` content if
241
the localization doesn't have that content.
242
243
For the TOML files, the
245
is a good reference. In short, filters match the localized source code, optionally
246
a ``key``, and an action. An example like
247
248
.. code-block:: toml
249
250
[[filters]]
251
path = "{l}browser/defines.inc"
252
key = "MOZ_LANGPACK_CONTRIBUTORS"
253
action = "ignore"
254
255
indicates that the ``MOZ_LANGPACK_CONTRIBUTORS`` in ``browser/defines.inc``
256
is optional.
257
258
For the legacy ini configuration files, there's a Python module
259
``filter.py`` next to the main ``l10n.ini``, implementing :py:func:`test`, with the following
260
signature
261
262
.. code-block:: python
263
264
def test(mod, path, entity = None):
265
if does_not_matter:
266
return "ignore"
267
if show_but_do_not_merge:
268
return "report"
269
# default behavior, localizer or build need to do something
270
return "error"
271
272
For any missing file, this function is called with ``mod`` being
273
the *module*, and ``path`` being the relative path inside
274
:file:`locales/en-US`. The module is the top-level dir as referenced in
275
:file:`l10n.ini`.
276
277
For missing strings, the :py:data:`entity` parameter is the key of the string
278
in the en-US file.
279
280
l10n-merge
281
==========
282
283
The chrome registry in Gecko doesn't support fallback from a localization to ``en-US`` at runtime.
284
Thus, the build needs to ensure that the localization as it's built into
285
the package has all required strings, and that the strings don't contain
286
errors. To ensure that, we're *merging* the localization and ``en-US``
287
at build time, nick-named l10n-merge.
288
289
For Fluent, we're also removing erroneous messages. For many errors in Fluent,
290
that's cosmetic, but when a localization has different values or attributes
291
on a message, that's actually important so that the DOM bindings of Fluent
292
can apply the translation without having to load the ``en-US`` source to
293
compare against.
294
295
The process can be manually triggered via
296
297
.. code-block:: bash
298
299
$> ./mach build merge-$AB_CD
300
301
It creates another directory in the object dir, :file:`browser/locales/merge-dir/$AB_CD`, in
302
which the modified files are stored. The actual repackaging process looks for
303
the localized files in the merge dir first, then the localized file, and then
304
in ``en-US``. Thus, for the ``de`` localization of
305
:file:`browser/locales/en-US/chrome/browser/browser.dtd`, it checks
306
307
1. :file:`$objdir/browser/locales/merge-de/browser/chrome/browser/browser.dtd`
308
2. :file:`$(LOCALE_BASEDIR)/de/browser/chrome/browser/browser.dtd`
309
3. :file:`browser/locales/en-US/chrome/browser/browser.dtd`
310
311
and will include the first of those files it finds.
312
313
l10n-merge modifies a file if it supports the particular file type, and there
314
are missing strings which are not filtered out, or if an existing string
315
shows an error. See the Checks section below for details. If the files are
316
not modified, l10n-merge copies them over to the respective location in the
317
merge dir.
318
319
Checks
320
------
321
322
As part of the build and other localization tool chains, we run a variety
323
of source-based checks. Think of them as linters.
324
325
The suite of checks is usually determined by file type, i.e., there's a
326
suite of checks for DTD files and one for properties files, etc.
327
328
Localizations
329
-------------
330
331
Now that we talked in-depth about how to expose content to localizers,
332
where are the localizations?
333
334
We host a mercurial repository per locale. All of our
335
localizations can be found on https://hg.mozilla.org/l10n-central/.
336
337
You can search inside our localized files on
338
`Transvision <https://transvision.mozfr.org/>`_ and