| ABIArgGenerator.h |
jit_ABIArgGenerator_h |
2633 |
| ABIFunctionList-inl.h |
|
12724 |
| ABIFunctions.h |
jit_VMFunctions_h |
2888 |
| AliasAnalysis.cpp |
|
10300 |
| AliasAnalysis.h |
jit_AliasAnalysis_h |
1453 |
| AlignmentMaskAnalysis.cpp |
|
3205 |
| AlignmentMaskAnalysis.h |
namespace jit |
694 |
| arm |
|
|
| arm64 |
|
|
| Assembler.h |
jit_Assembler_h |
1174 |
| AtomicOp.h |
jit_AtomicOp_h |
2983 |
| AtomicOperations.h |
[SMDOC] Atomic Operations
The atomic operations layer defines types and functions for
JIT-compatible atomic operation.
The fundamental constraints on the functions are:
- That their realization here MUST be compatible with code the JIT
generates for its Atomics operations, so that an atomic access
from the interpreter or runtime - from any C++ code - really is
atomic relative to a concurrent, compatible atomic access from
jitted code. That is, these primitives expose JIT-compatible
atomicity functionality to C++.
- That accesses may race without creating C++ undefined behavior:
atomic accesses (marked "SeqCst") may race with non-atomic
accesses (marked "SafeWhenRacy"); overlapping but non-matching,
and hence incompatible, atomic accesses may race; and non-atomic
accesses may race. The effects of races need not be predictable,
so garbage can be produced by a read or written by a write, but
the effects must be benign: the program must continue to run, and
only the memory in the union of addresses named in the racing
accesses may be affected.
The compatibility constraint means that if the JIT makes dynamic
decisions about how to implement atomic operations then
corresponding dynamic decisions MUST be made in the implementations
of the functions below.
The safe-for-races constraint means that by and large, it is hard
to implement these primitives in C++. See "Implementation notes"
below.
The "SeqCst" suffix on operations means "sequentially consistent"
and means such a function's operation must have "sequentially
consistent" memory ordering. See mfbt/Atomics.h for an explanation
of this memory ordering.
Note that a "SafeWhenRacy" access does not provide the atomicity of
a "relaxed atomic" access: it can read or write garbage if there's
a race.
Implementation notes.
It's not a requirement that these functions be inlined; performance
is not a great concern. On some platforms these functions may call
functions that use inline assembly. See GenerateAtomicOperations.py.
In principle these functions will not be written in C++, thus
making races defined behavior if all racy accesses from C++ go via
these functions. (Jitted code will always be safe for races and
provides the same guarantees as these functions.)
The appropriate implementations will be platform-specific and
there are some obvious implementation strategies to choose
from, sometimes a combination is appropriate:
- generating the code at run-time with the JIT;
- hand-written assembler (maybe inline); or
- using special compiler intrinsics or directives.
Trusting the compiler not to generate code that blows up on a
race definitely won't work in the presence of TSan, or even of
optimizing compilers in seemingly-"innocuous" conditions. (See
https://www.usenix.org/legacy/event/hotpar11/tech/final_files/Boehm.pdf
for details.)
|
13065 |
| AutoWritableJitCode.h |
jit_AutoWritableJitCode_h |
3424 |
| BacktrackingAllocator.cpp |
|
169753 |
| BacktrackingAllocator.h |
|
30204 |
| Bailouts.cpp |
|
13221 |
| Bailouts.h |
|
8248 |
| BaselineBailouts.cpp |
BaselineStackBuilder helps abstract the process of rebuilding the C stack on
the heap. It takes a bailout iterator and keeps track of the point on the C
stack from which the reconstructed frames will be written.
It exposes methods to write data into the heap memory storing the
reconstructed stack. It also exposes method to easily calculate addresses.
This includes both the virtual address that a particular value will be at
when it's eventually copied onto the stack, as well as the current actual
address of that value (whether on the heap allocated portion being
constructed or the existing stack).
The abstraction handles transparent re-allocation of the heap memory when it
needs to be enlarged to accommodate new data. Similarly to the C stack, the
data that's written to the reconstructed stack grows from high to low in
memory.
The lowest region of the allocated memory contains a BaselineBailoutInfo
structure that points to the start and end of the written data.
|
74979 |
| BaselineCacheIRCompiler.cpp |
|
146845 |
| BaselineCacheIRCompiler.h |
|
6816 |
| BaselineCodeGen.cpp |
HandlerArgs = |
198938 |
| BaselineCodeGen.h |
|
17977 |
| BaselineDebugModeOSR.cpp |
|
19200 |
| BaselineDebugModeOSR.h |
|
962 |
| BaselineFrame-inl.h |
jit_BaselineFrame_inl_h |
3982 |
| BaselineFrame.cpp |
|
5415 |
| BaselineFrame.h |
|
13003 |
| BaselineFrameInfo-inl.h |
jit_BaselineFrameInfo_inl_h |
1372 |
| BaselineFrameInfo.cpp |
|
6453 |
| BaselineFrameInfo.h |
|
13008 |
| BaselineIC.cpp |
|
79308 |
| BaselineIC.h |
|
16850 |
| BaselineICList.h |
jit_BaselineICList_h |
2037 |
| BaselineJIT.cpp |
|
32271 |
| BaselineJIT.h |
|
20297 |
| BitSet.cpp |
|
2627 |
| BitSet.h |
jit_BitSet_h |
4250 |
| BytecodeAnalysis.cpp |
stackDepth= |
10982 |
| BytecodeAnalysis.h |
jit_BytecodeAnalysis_h |
2276 |
| CacheIR.cpp |
|
441656 |
| CacheIR.h |
|
16423 |
| CacheIRCloner.h |
jit_CacheIRCloner_h |
2181 |
| CacheIRCompiler.cpp |
|
323824 |
| CacheIRCompiler.h |
|
47964 |
| CacheIRGenerator.h |
|
36240 |
| CacheIRHealth.cpp |
|
13581 |
| CacheIRHealth.h |
JS_CACHEIR_SPEW |
4484 |
| CacheIROps.yaml |
|
53444 |
| CacheIRReader.h |
isSpread = |
4947 |
| CacheIRSpewer.cpp |
|
12381 |
| CacheIRSpewer.h |
JS_CACHEIR_SPEW |
3217 |
| CacheIRWriter.h |
|
22948 |
| CalleeToken.h |
namespace js::jit |
2242 |
| CodeGenerator.cpp |
|
651079 |
| CodeGenerator.h |
|
17861 |
| CompactBuffer.h |
|
7020 |
| CompileInfo.h |
env chain and argument obj |
14294 |
| CompileWrappers.cpp |
static |
6454 |
| CompileWrappers.h |
|
4167 |
| Disassemble.cpp |
|
3139 |
| Disassemble.h |
jit_Disassemble_h |
680 |
| EdgeCaseAnalysis.cpp |
|
1461 |
| EdgeCaseAnalysis.h |
jit_EdgeCaseAnalysis_h |
700 |
| EffectiveAddressAnalysis.cpp |
|
7124 |
| EffectiveAddressAnalysis.h |
namespace jit |
874 |
| ExecutableAllocator.cpp |
willDestroy = |
10594 |
| ExecutableAllocator.h |
|
6356 |
| FixedList.h |
jit_FixedList_h |
2249 |
| FlushICache.cpp |
|
4642 |
| FlushICache.h |
Flush the instruction cache of instructions in an address range. |
3355 |
| FoldLinearArithConstants.cpp |
namespace jit |
3672 |
| FoldLinearArithConstants.h |
namespace jit |
639 |
| GenerateAtomicOperations.py |
INLINE_ATTR void %(fun_name)s() {
asm volatile ("mfence\n\t" ::: "memory");
} |
34831 |
| GenerateCacheIRFiles.py |
\
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#ifndef %(includeguard)s
#define %(includeguard)s
/* This file is generated by jit/GenerateCacheIRFiles.py. Do not edit! */
%(contents)s
#endif // %(includeguard)s
|
20258 |
| GenerateLIRFiles.py |
\
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#ifndef %(includeguard)s
#define %(includeguard)s
/* This file is generated by jit/GenerateLIRFiles.py. Do not edit! */
%(contents)s
#endif // %(includeguard)s
|
9678 |
| GenerateMIRFiles.py |
\
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#ifndef %(includeguard)s
#define %(includeguard)s
/* This file is generated by jit/GenerateMIRFiles.py. Do not edit! */
%(contents)s
#endif // %(includeguard)s
|
12977 |
| ICState.h |
|
7378 |
| ICStubSpace.h |
jit_ICStubSpace_h |
2013 |
| InlinableNatives.cpp |
|
12607 |
| InlinableNatives.h |
|
11685 |
| InlineList.h |
|
15541 |
| InlineScriptTree-inl.h |
jit_InlineScriptTree_inl_h |
2269 |
| InlineScriptTree.h |
jit_InlineScriptTree_h |
4163 |
| InstructionReordering.cpp |
|
8278 |
| InstructionReordering.h |
|
593 |
| InterpreterEntryTrampoline.cpp |
countIncludesThis = |
8639 |
| InterpreterEntryTrampoline.h |
The EntryTrampolineMap is used to cache the trampoline code for
each script as they are created. These trampolines are created
only under --emit-interpreter-entry and are used to identify which
script is being interpeted when profiling with external profilers
such as perf.
The map owns the JitCode objects that are created for each script,
and keeps them alive at least as long as the script associated
with it in case we need to re-enter the trampoline again.
As each script is finalized, the entry is manually removed from
the table in BaseScript::finalize which will also release the
trampoline code associated with it.
During a moving GC, the table is rekeyed in case any scripts
have relocated.
|
2382 |
| Invalidation.h |
jit_Invalidation_h |
1891 |
| Ion.cpp |
|
82687 |
| Ion.h |
this |
4371 |
| IonAnalysis.cpp |
|
162240 |
| IonAnalysis.h |
|
6561 |
| IonCacheIRCompiler.cpp |
|
79406 |
| IonCacheIRCompiler.h |
jit_IonCacheIRCompiler_h |
3071 |
| IonCompileTask.cpp |
|
6311 |
| IonCompileTask.h |
jit_IonCompileTask_h |
3097 |
| IonGenericCallStub.h |
jit_IonGenericCallStub_h |
1206 |
| IonIC.cpp |
static |
21142 |
| IonIC.h |
|
20358 |
| IonOptimizationLevels.cpp |
|
5400 |
| IonOptimizationLevels.h |
|
5803 |
| IonScript.h |
|
19117 |
| IonTypes.h |
[SMDOC] Avoiding repeated bailouts / invalidations
To avoid getting trapped in a "compilation -> bailout -> invalidation ->
recompilation -> bailout -> invalidation -> ..." loop, every snapshot in
Warp code is assigned a BailoutKind. If we bail out at that snapshot,
FinishBailoutToBaseline will examine the BailoutKind and take appropriate
action. In general:
1. If the bailing instruction comes from transpiled CacheIR, then when we
bail out and continue execution in the baseline interpreter, the
corresponding stub should fail a guard. As a result, we will either
increment the enteredCount for a subsequent stub or attach a new stub,
either of which will prevent WarpOracle from transpiling the failing stub
when we recompile.
Note: this means that every CacheIR op that can bail out in Warp must
have an equivalent guard in the baseline CacheIR implementation.
FirstExecution works according to the same principles: we have never hit
this IC before, but after we bail to baseline we will attach a stub and
recompile with better CacheIR information.
2. If the bailout occurs because an assumption we made in WarpBuilder was
invalidated, then FinishBailoutToBaseline will set a flag on the script
to avoid that assumption in the future: for example, UninitializedLexical.
3. Similarly, if the bailing instruction is generated or modified by a MIR
optimization, then FinishBailoutToBaseline will set a flag on the script
to make that optimization more conservative in the future. Examples
include LICM, EagerTruncation, and HoistBoundsCheck.
4. Some bailouts can't be handled in Warp, even after a recompile. For
example, Warp does not support catching exceptions. If this happens
too often, then the cost of bailing out repeatedly outweighs the
benefit of Warp compilation, so we invalidate the script and disable
Warp compilation.
5. Some bailouts don't happen in performance-sensitive code: for example,
the |debugger| statement. We just ignore those.
|
40056 |
| Jit.cpp |
osrFrame = |
6769 |
| Jit.h |
jit_Jit_h |
1149 |
| JitAllocPolicy.h |
|
5279 |
| JitCode.h |
|
5902 |
| JitcodeMap.cpp |
|
38876 |
| JitcodeMap.h |
The jitcode map implements tables to allow mapping from addresses in jitcode
to the list of (JSScript*, jsbytecode*) pairs that are implicitly active in
the frame at that point in the native code.
To represent this information efficiently, a multi-level table is used.
At the top level, a global AVL-tree of JitcodeGlobalEntry describing the
mapping for each individual JitCode generated by compiles. The entries are
ordered by their nativeStartAddr.
Every entry in the table is of fixed size, but there are different entry
types, distinguished by the kind field.
|
26790 |
| JitCommon.h |
|
2292 |
| JitContext.cpp |
|
3757 |
| JitContext.h |
jit_JitContext_h |
4375 |
| Jitdump.h |
This file provides the necessary data structures to meet the JitDump
specification as of
https://github.com/torvalds/linux/blob/f2906aa863381afb0015a9eb7fefad885d4e5a56/tools/perf/Documentation/jitdump-specification.txt
|
1592 |
| JitFrames-inl.h |
jit_JitFrames_inl_h |
862 |
| JitFrames.cpp |
|
87956 |
| JitFrames.h |
HASCACHEDSAVEDFRAME_BIT |
24107 |
| JitHints-inl.h |
jit_JitHints_inl_h |
1652 |
| JitHints.cpp |
|
2223 |
| JitHints.h |
[SMDOC] JitHintsMap
The Jit hints map is an in process cache used to collect Baseline and Ion
JIT hints to try and skip as much of the warmup as possible and jump
straight into those tiers. Whenever a script enters one of these tiers
a hint is recorded in this cache using the script's filename+sourceStart
value, and if we ever encounter this script again later, e.g. during a
navigation, then we try to eagerly compile it into baseline and ion
based on its previous execution history.
|
5491 |
| JitOptions.cpp |
|
15785 |
| JitOptions.h |
|
5277 |
| JitRuntime.h |
|
16613 |
| JitScript-inl.h |
jit_JitScript_inl_h |
1144 |
| JitScript.cpp |
depth= |
23257 |
| JitScript.h |
[SMDOC] ICScript Lifetimes
An ICScript owns an array of ICEntries, each of which owns a linked
list of ICStubs.
A JitScript contains an embedded ICScript. If it has done any trial
inlining, it also owns an InliningRoot. The InliningRoot owns all
of the ICScripts that have been created for inlining into the
corresponding JitScript. This ties the lifetime of the inlined
ICScripts to the lifetime of the JitScript itself.
We store pointers to ICScripts in two other places: on the stack in
BaselineFrame, and in IC stubs for CallInlinedFunction.
The ICScript pointer in a BaselineFrame either points to the
ICScript embedded in the JitScript for that frame, or to an inlined
ICScript owned by a caller. In each case, there must be a frame on
the stack corresponding to the JitScript that owns the current
ICScript, which will keep the ICScript alive.
Each ICStub is owned by an ICScript and, indirectly, a
JitScript. An ICStub that uses CallInlinedFunction contains an
ICScript for use by the callee. The ICStub and the callee ICScript
are always owned by the same JitScript, so the callee ICScript will
not be freed while the ICStub is alive.
The lifetime of an ICScript is independent of the lifetimes of the
BaselineScript and IonScript/WarpScript to which it
corresponds. They can be destroyed and recreated, and the ICScript
will remain valid.
|
18850 |
| JitSpewer.cpp |
|
19029 |
| JitSpewer.h |
Information during sinking |
10100 |
| JitZone.h |
|
13434 |
| JSJitFrameIter-inl.h |
jit_JSJitFrameIter_inl_h |
1806 |
| JSJitFrameIter.cpp |
|
24648 |
| JSJitFrameIter.h |
|
26221 |
| JSONSpewer.cpp |
|
6788 |
| JSONSpewer.h |
JS_JITSPEW |
1192 |
| KnownClass.cpp |
|
3217 |
| KnownClass.h |
|
879 |
| Label.cpp |
|
883 |
| Label.h |
|
3254 |
| LICM.cpp |
OUT |
13204 |
| LICM.h |
jit_LICM_h |
629 |
| Linker.cpp |
|
2354 |
| Linker.h |
jit_Linker_h |
1248 |
| LIR.cpp |
|
21418 |
| LIR.h |
|
64269 |
| LIROps.yaml |
|
76977 |
| loong64 |
|
|
| Lowering.cpp |
useAtStart = |
242395 |
| Lowering.h |
useAtStart = |
2863 |
| MachineState.h |
jit_MachineState_h |
3636 |
| MacroAssembler-inl.h |
|
38654 |
| MacroAssembler.cpp |
|
279152 |
| MacroAssembler.h |
|
256270 |
| mips-shared |
|
|
| mips32 |
|
|
| mips64 |
|
|
| MIR.cpp |
|
210117 |
| MIR.h |
Everything needed to build actual MIR instructions: the actual opcodes and
instructions, the instruction interface, and use chains.
|
376940 |
| MIRGenerator.h |
|
5480 |
| MIRGraph.cpp |
|
41960 |
| MIRGraph.h |
|
30472 |
| MIROps.yaml |
|
65202 |
| MoveEmitter.h |
jit_MoveEmitter_h |
1172 |
| MoveResolver.cpp |
|
14054 |
| MoveResolver.h |
|
9924 |
| moz.build |
|
9802 |
| none |
|
|
| PcScriptCache.h |
jit_PcScriptCache_h |
2458 |
| PerfSpewer.cpp |
|
38984 |
| PerfSpewer.h |
|
7234 |
| ProcessExecutableMemory.cpp |
Inspiration is V8's OS::Allocate in platform-win32.cc.
VirtualAlloc takes 64K chunks out of the virtual address space, so we
keep 16b alignment.
x86: V8 comments say that keeping addresses in the [64MiB, 1GiB) range
tries to avoid system default DLL mapping space. In the end, we get 13
bits of randomness in our selection.
x64: [2GiB, 4TiB), with 25 bits of randomness.
|
31221 |
| ProcessExecutableMemory.h |
|
5307 |
| RangeAnalysis.cpp |
|
119199 |
| RangeAnalysis.h |
|
25935 |
| ReciprocalMulConstants.cpp |
|
4472 |
| ReciprocalMulConstants.h |
jit_ReciprocalMulConstants_h |
990 |
| Recover.cpp |
|
60749 |
| Recover.h |
|
28079 |
| RegExpStubConstants.h |
jit_RegExpStubConstants_h |
1202 |
| RegisterAllocator.cpp |
|
22457 |
| RegisterAllocator.h |
|
10958 |
| Registers.h |
|
9357 |
| RegisterSets.h |
|
40257 |
| RematerializedFrame-inl.h |
|
746 |
| RematerializedFrame.cpp |
static |
6430 |
| RematerializedFrame.h |
|
7432 |
| riscv64 |
|
|
| SafepointIndex-inl.h |
jit_SafepointIndex_inl_h |
694 |
| SafepointIndex.cpp |
|
732 |
| SafepointIndex.h |
namespace jit |
2376 |
| Safepoints.cpp |
|
17714 |
| Safepoints.h |
jit_Safepoints_h |
4253 |
| ScalarReplacement.cpp |
|
96362 |
| ScalarReplacement.h |
jit_ScalarReplacement_h |
675 |
| ScalarTypeUtils.h |
jit_ScalarTypeUtils_h |
1328 |
| ScriptFromCalleeToken.h |
namespace js::jit |
1023 |
| shared |
|
|
| SharedICHelpers-inl.h |
jit_SharedICHelpers_inl_h |
1328 |
| SharedICHelpers.h |
jit_SharedICHelpers_h |
1280 |
| SharedICRegisters.h |
jit_SharedICRegisters_h |
1348 |
| ShuffleAnalysis.cpp |
|
25176 |
| ShuffleAnalysis.h |
|
4568 |
| Simulator.h |
jit_Simulator_h |
1037 |
| Sink.cpp |
|
9457 |
| Sink.h |
jit_Sink_h |
598 |
| Snapshots.cpp |
|
20031 |
| Snapshots.h |
|
16031 |
| StackSlotAllocator.h |
jit_StackSlotAllocator_h |
3500 |
| TemplateObject-inl.h |
jit_TemplateObject_inl_h |
3392 |
| TemplateObject.h |
jit_TemplateObject_h |
2583 |
| Trampoline.cpp |
|
9349 |
| TrialInlining.cpp |
|
31022 |
| TrialInlining.h |
[SMDOC] Trial Inlining
WarpBuilder relies on transpiling CacheIR. When inlining scripted
functions in WarpBuilder, we want our ICs to be as monomorphic as
possible. Functions with multiple callers complicate this. An IC in
such a function might be monomorphic for any given caller, but
polymorphic overall. This make the input to WarpBuilder less precise.
To solve this problem, we do trial inlining. During baseline
execution, we identify call sites for which it would be useful to
have more precise inlining data. For each such call site, we
allocate a fresh ICScript and replace the existing call IC with a
new specialized IC that invokes the callee using the new
ICScript. Other callers of the callee will continue using the
default ICScript. When we eventually Warp-compile the script, we
can generate code for the callee using the IC information in our
private ICScript, which is specialized for its caller.
The same approach can be used to inline recursively.
|
6015 |
| TypeData.h |
jit_TypeData_h |
1314 |
| TypePolicy.cpp |
|
43906 |
| TypePolicy.h |
|
19548 |
| ValueNumbering.cpp |
[SMDOC] IonMonkey Value Numbering
Some notes on the main algorithm here:
- The SSA identifier id() is the value number. We do replaceAllUsesWith as
we go, so there's always at most one visible value with a given number.
- Consequently, the GVN algorithm is effectively pessimistic. This means it
is not as powerful as an optimistic GVN would be, but it is simpler and
faster.
- We iterate in RPO, so that when visiting a block, we've already optimized
and hashed all values in dominating blocks. With occasional exceptions,
this allows us to do everything in a single pass.
- When we do use multiple passes, we just re-run the algorithm on the whole
graph instead of doing sparse propagation. This is a tradeoff to keep the
algorithm simpler and lighter on inputs that don't have a lot of
interesting unreachable blocks or degenerate loop induction variables, at
the expense of being slower on inputs that do. The loop for this always
terminates, because it only iterates when code is or will be removed, so
eventually it must stop iterating.
- Values are not immediately removed from the hash set when they go out of
scope. Instead, we check for dominance after a lookup. If the dominance
check fails, the value is removed.
|
46781 |
| ValueNumbering.h |
jit_ValueNumbering_h |
4635 |
| VMFunctionList-inl.h |
|
25578 |
| VMFunctions.cpp |
Unexpected return type for a VMFunction. |
91932 |
| VMFunctions.h |
|
27191 |
| WarpBuilder.cpp |
maybePred = |
108436 |
| WarpBuilder.h |
Intentionally not implemented |
12782 |
| WarpBuilderShared.cpp |
|
4153 |
| WarpBuilderShared.h |
|
12609 |
| WarpCacheIRTranspiler.cpp |
|
186947 |
| WarpCacheIRTranspiler.h |
jit_WarpCacheIRTranspiler_h |
905 |
| WarpOracle.cpp |
|
44153 |
| WarpOracle.h |
jit_WarpOracle_h |
2508 |
| WarpSnapshot.cpp |
|
13636 |
| WarpSnapshot.h |
|
18778 |
| wasm32 |
|
|
| WasmBCE.cpp |
|
4949 |
| WasmBCE.h |
jit_wasmbce_h |
964 |
| x64 |
|
|
| x86 |
|
|
| x86-shared |
|
|
| XrayJitInfo.cpp |
|
622 |