Source code
Revision control
Copy as Markdown
Other Tools
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*-
* vim: set ts=8 sts=2 et sw=2 tw=80:
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
/*
* Streaming access to the raw tokens of JavaScript source.
*
* Because JS tokenization is context-sensitive -- a '/' could be either a
* regular expression *or* a division operator depending on context -- the
* various token stream classes are mostly not useful outside of the Parser
* where they reside. We should probably eventually merge the two concepts.
*/
#ifndef frontend_TokenStream_h
#define frontend_TokenStream_h
/*
* [SMDOC] Parser Token Stream
*
* A token stream exposes the raw tokens -- operators, names, numbers,
* keywords, and so on -- of JavaScript source code.
*
* These are the components of the overall token stream concept:
* TokenStreamShared, TokenStreamAnyChars, TokenStreamCharsBase<Unit>,
* TokenStreamChars<Unit>, and TokenStreamSpecific<Unit, AnyCharsAccess>.
*
* == TokenStreamShared → ∅ ==
*
* Certain aspects of tokenizing are used everywhere:
*
* * modifiers (used to select which context-sensitive interpretation of a
* character should be used to decide what token it is) and modifier
* assertion handling;
* * flags on the overall stream (have we encountered any characters on this
* line? have we hit a syntax error? and so on);
* * and certain token-count constants.
*
* These are all defined in TokenStreamShared. (They could be namespace-
* scoped, but it seems tentatively better not to clutter the namespace.)
*
* == TokenStreamAnyChars → TokenStreamShared ==
*
* Certain aspects of tokenizing have meaning independent of the character type
* of the source text being tokenized: line/column number information, tokens
* in lookahead from determining the meaning of a prior token, compilation
* options, the filename, flags, source map URL, access to details of the
* current and next tokens (is the token of the given type? what name or
* number is contained in the token? and other queries), and others.
*
* All this data/functionality *could* be duplicated for both single-byte and
* double-byte tokenizing, but there are two problems. First, it's potentially
* wasteful if the compiler doesnt recognize it can unify the concepts. (And
* if any-character concepts are intermixed with character-specific concepts,
* potentially the compiler *can't* unify them because offsets into the
* hypothetical TokenStream<Unit>s would differ.) Second, some of this stuff
* needs to be accessible in ParserBase, the aspects of JS language parsing
* that have meaning independent of the character type of the source text being
* parsed. So we need a separate data structure that ParserBase can hold on to
* for it. (ParserBase isn't the only instance of this, but it's certainly the
* biggest case of it.) Ergo, TokenStreamAnyChars.
*
* == TokenStreamCharsShared → ∅ ==
*
* Some functionality has meaning independent of character type, yet has no use
* *unless* you know the character type in actual use. It *could* live in
* TokenStreamAnyChars, but it makes more sense to live in a separate class
* that character-aware token information can simply inherit.
*
* This class currently exists only to contain a char16_t buffer, transiently
* used to accumulate strings in tricky cases that can't just be read directly
* from source text. It's not used outside character-aware tokenizing, so it
* doesn't make sense in TokenStreamAnyChars.
*
* == TokenStreamCharsBase<Unit> → TokenStreamCharsShared ==
*
* Certain data structures in tokenizing are character-type-specific: namely,
* the various pointers identifying the source text (including current offset
* and end).
*
* Additionally, some functions operating on this data are defined the same way
* no matter what character type you have (e.g. current offset in code units
* into the source text) or share a common interface regardless of character
* type (e.g. consume the next code unit if it has a given value).
*
* All such functionality lives in TokenStreamCharsBase<Unit>.
*
* == SpecializedTokenStreamCharsBase<Unit> → TokenStreamCharsBase<Unit> ==
*
* Certain tokenizing functionality is specific to a single character type.
* For example, JS's UTF-16 encoding recognizes no coding errors, because lone
* surrogates are not an error; but a UTF-8 encoding must recognize a variety
* of validation errors. Such functionality is defined only in the appropriate
* SpecializedTokenStreamCharsBase specialization.
*
* == GeneralTokenStreamChars<Unit, AnyCharsAccess> →
* SpecializedTokenStreamCharsBase<Unit> ==
*
* Some functionality operates differently on different character types, just
* as for TokenStreamCharsBase, but additionally requires access to character-
* type-agnostic information in TokenStreamAnyChars. For example, getting the
* next character performs different steps for different character types and
* must access TokenStreamAnyChars to update line break information.
*
* Such functionality, if it can be defined using the same algorithm for all
* character types, lives in GeneralTokenStreamChars<Unit, AnyCharsAccess>.
* The AnyCharsAccess parameter provides a way for a GeneralTokenStreamChars
* instance to access its corresponding TokenStreamAnyChars, without inheriting
* from it.
*
* GeneralTokenStreamChars<Unit, AnyCharsAccess> is just functionality, no
* actual member data.
*
* Such functionality all lives in TokenStreamChars<Unit, AnyCharsAccess>, a
* declared-but-not-defined template class whose specializations have a common
* public interface (plus whatever private helper functions are desirable).
*
* == TokenStreamChars<Unit, AnyCharsAccess> →
* GeneralTokenStreamChars<Unit, AnyCharsAccess> ==
*
* Some functionality is like that in GeneralTokenStreamChars, *but* it's
* defined entirely differently for different character types.
*
* For example, consider "match a multi-code unit code point" (hypothetically:
* we've only implemented two-byte tokenizing right now):
*
* * For two-byte text, there must be two code units to get, the leading code
* unit must be a UTF-16 lead surrogate, and the trailing code unit must be
* a UTF-16 trailing surrogate. (If any of these fail to hold, a next code
* unit encodes that code point and is not multi-code unit.)
* * For single-byte Latin-1 text, there are no multi-code unit code points.
* * For single-byte UTF-8 text, the first code unit must have N > 1 of its
* highest bits set (and the next unset), and |N - 1| successive code units
* must have their high bit set and next-highest bit unset, *and*
* concatenating all unconstrained bits together must not produce a code
* point value that could have been encoded in fewer code units.
*
* This functionality can't be implemented as member functions in
* GeneralTokenStreamChars because we'd need to *partially specialize* those
* functions -- hold Unit constant while letting AnyCharsAccess vary. But
* C++ forbids function template partial specialization like this: either you
* fix *all* parameters or you fix none of them.
*
* Fortunately, C++ *does* allow *class* template partial specialization. So
* TokenStreamChars is a template class with one specialization per Unit.
* Functions can be defined differently in the different specializations,
* because AnyCharsAccess as the only template parameter on member functions
* *can* vary.
*
* All TokenStreamChars<Unit, AnyCharsAccess> specializations, one per Unit,
* are just functionality, no actual member data.
*
* == TokenStreamSpecific<Unit, AnyCharsAccess> →
* TokenStreamChars<Unit, AnyCharsAccess>, TokenStreamShared,
* ErrorReporter ==
*
* TokenStreamSpecific is operations that are parametrized on character type
* but implement the *general* idea of tokenizing, without being intrinsically
* tied to character type. Notably, this includes all operations that can
* report warnings or errors at particular offsets, because we include a line
* of context with such errors -- and that necessarily accesses the raw
* characters of their specific type.
*
* Much TokenStreamSpecific operation depends on functionality in
* TokenStreamAnyChars. The obvious solution is to inherit it -- but this
* doesn't work in Parser: its ParserBase base class needs some
* TokenStreamAnyChars functionality without knowing character type.
*
* The AnyCharsAccess type parameter is a class that statically converts from a
* TokenStreamSpecific* to its corresponding TokenStreamAnyChars. The
* TokenStreamSpecific in Parser<ParseHandler, Unit> can then specify a class
* that properly converts from TokenStreamSpecific Parser::tokenStream to
* TokenStreamAnyChars ParserBase::anyChars.
*
* Could we hardcode one set of offset calculations for this and eliminate
* AnyCharsAccess? No. Offset calculations possibly could be hardcoded if
* TokenStreamSpecific were present in Parser before Parser::handler, assuring
* the same offsets in all Parser-related cases. But there's still a separate
* TokenStream class, that requires different offset calculations. So even if
* we wanted to hardcode this (it's not clear we would, because forcing the
* TokenStreamSpecific declarer to specify this is more explicit), we couldn't.
*/
#include "mozilla/ArrayUtils.h"
#include "mozilla/Assertions.h"
#include "mozilla/Attributes.h"
#include "mozilla/Casting.h"
#include "mozilla/Maybe.h"
#include "mozilla/MemoryChecking.h"
#include "mozilla/Span.h"
#include "mozilla/TextUtils.h"
#include "mozilla/Utf8.h"
#include <algorithm>
#include <stdarg.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <type_traits>
#include "jspubtd.h"
#include "frontend/ErrorReporter.h"
#include "frontend/ParserAtom.h" // ParserAtom, ParserAtomsTable, TaggedParserAtomIndex
#include "frontend/Token.h"
#include "frontend/TokenKind.h"
#include "js/CharacterEncoding.h" // JS::ConstUTF8CharsZ
#include "js/ColumnNumber.h" // JS::LimitedColumnNumberOneOrigin, JS::ColumnNumberOneOrigin, JS::ColumnNumberUnsignedOffset
#include "js/CompileOptions.h"
#include "js/friend/ErrorMessages.h" // JSMSG_*
#include "js/HashTable.h" // js::HashMap
#include "js/RegExpFlags.h" // JS::RegExpFlags
#include "js/UniquePtr.h"
#include "js/Vector.h"
#include "util/Unicode.h"
#include "vm/ErrorReporting.h"
struct KeywordInfo;
namespace js {
class FrontendContext;
namespace frontend {
// True if str is a keyword.
bool IsKeyword(TaggedParserAtomIndex atom);
// If `name` is reserved word, returns the TokenKind of it.
// TokenKind::Limit otherwise.
extern TokenKind ReservedWordTokenKind(TaggedParserAtomIndex name);
// If `name` is reserved word, returns string representation of it.
// nullptr otherwise.
extern const char* ReservedWordToCharZ(TaggedParserAtomIndex name);
// If `tt` is reserved word, returns string representation of it.
// nullptr otherwise.
extern const char* ReservedWordToCharZ(TokenKind tt);
enum class DeprecatedContent : uint8_t {
// No deprecated content was present.
None = 0,
// Octal literal not prefixed by "0o" but rather by just "0", e.g. 0755.
OctalLiteral,
// Octal character escape, e.g. "hell\157 world".
OctalEscape,
// NonOctalDecimalEscape, i.e. "\8" or "\9".
EightOrNineEscape,
};
struct TokenStreamFlags {
// Hit end of file.
bool isEOF : 1;
// Non-whitespace since start of line.
bool isDirtyLine : 1;
// Hit a syntax error, at start or during a token.
bool hadError : 1;
// The nature of any deprecated content seen since last reset.
// We have to uint8_t instead DeprecatedContent to work around a GCC 7 bug.
uint8_t sawDeprecatedContent : 2;
TokenStreamFlags()
: isEOF(false),
isDirtyLine(false),
hadError(false),
sawDeprecatedContent(uint8_t(DeprecatedContent::None)) {}
};
template <typename Unit>
class TokenStreamPosition;
/**
* TokenStream types and constants that are used in both TokenStreamAnyChars
* and TokenStreamSpecific. Do not add any non-static data members to this
* class!
*/
class TokenStreamShared {
protected:
// 1 current + (3 lookahead if EXPLICIT_RESOURCE_MANAGEMENT is enabled
// else 2 lookahead and rounded up to ^2)
// NOTE: This must be power of 2, in order to make `ntokensMask` work.
static constexpr size_t ntokens = 4;
static constexpr unsigned ntokensMask = ntokens - 1;
template <typename Unit>
friend class TokenStreamPosition;
public:
#ifdef ENABLE_EXPLICIT_RESOURCE_MANAGEMENT
// We need a lookahead buffer of atleast 3 for the AwaitUsing syntax.
static constexpr unsigned maxLookahead = 3;
#else
static constexpr unsigned maxLookahead = 2;
#endif
using Modifier = Token::Modifier;
static constexpr Modifier SlashIsDiv = Token::SlashIsDiv;
static constexpr Modifier SlashIsRegExp = Token::SlashIsRegExp;
static constexpr Modifier SlashIsInvalid = Token::SlashIsInvalid;
static void verifyConsistentModifier(Modifier modifier,
const Token& nextToken) {
MOZ_ASSERT(
modifier == nextToken.modifier || modifier == SlashIsInvalid,
"This token was scanned with both SlashIsRegExp and SlashIsDiv, "
"indicating the parser is confused about how to handle a slash here. "
"See comment at Token::Modifier.");
}
};
static_assert(std::is_empty_v<TokenStreamShared>,
"TokenStreamShared shouldn't bloat classes that inherit from it");
template <typename Unit, class AnyCharsAccess>
class TokenStreamSpecific;
template <typename Unit>
class MOZ_STACK_CLASS TokenStreamPosition final {
public:
template <class AnyCharsAccess>
inline explicit TokenStreamPosition(
TokenStreamSpecific<Unit, AnyCharsAccess>& tokenStream);
private:
TokenStreamPosition(const TokenStreamPosition&) = delete;
// Technically only TokenStreamSpecific<Unit, AnyCharsAccess>::seek with
// Unit constant and AnyCharsAccess varying must be friended, but 1) it's
// hard to friend one function in template classes, and 2) C++ doesn't
// allow partial friend specialization to target just that single class.
template <typename Char, class AnyCharsAccess>
friend class TokenStreamSpecific;
const Unit* buf;
TokenStreamFlags flags;
unsigned lineno;
size_t linebase;
size_t prevLinebase;
Token currentToken;
unsigned lookahead;
Token lookaheadTokens[TokenStreamShared::maxLookahead];
};
template <typename Unit>
class SourceUnits;
/**
* This class maps:
*
* * a sourceUnits offset (a 0-indexed count of code units)
*
* to
*
* * a (1-indexed) line number and
* * a (0-indexed) offset in code *units* (not code points, not bytes) into
* that line,
*
* for either |Unit = Utf8Unit| or |Unit = char16_t|.
*
* Note that, if |Unit = Utf8Unit|, the latter quantity is *not* the same as a
* column number, which is a count of UTF-16 code units. Computing a column
* number requires the offset within the line and the source units of that line
* (including what type |Unit| is, to know how to decode them). If you need a
* column number, functions in |GeneralTokenStreamChars<Unit>| will consult
* this and source units to compute it.
*/
class SourceCoords {
// For a given buffer holding source code, |lineStartOffsets_| has one
// element per line of source code, plus one sentinel element. Each
// non-sentinel element holds the buffer offset for the start of the
// corresponding line of source code. For this example script,
// assuming an initialLineOffset of 0:
//
// 1 // xyz [line starts at offset 0]
// 2 var x; [line starts at offset 7]
// 3 [line starts at offset 14]
// 4 var y; [line starts at offset 15]
//
// |lineStartOffsets_| is:
//
// [0, 7, 14, 15, MAX_PTR]
//
// To convert a "line number" to an "index" into |lineStartOffsets_|,
// subtract |initialLineNum_|. E.g. line 3's index is
// (3 - initialLineNum_), which is 2. Therefore lineStartOffsets_[2]
// holds the buffer offset for the start of line 3, which is 14. (Note
// that |initialLineNum_| is often 1, but not always.
//
// The first element is always initialLineOffset, passed to the
// constructor, and the last element is always the MAX_PTR sentinel.
//
// Offset-to-{line,offset-into-line} lookups are O(log n) in the worst
// case (binary search), but in practice they're heavily clustered and
// we do better than that by using the previous lookup's result
// (lastIndex_) as a starting point.
//
// Checking if an offset lies within a particular line number
// (isOnThisLine()) is O(1).
//
Vector<uint32_t, 128> lineStartOffsets_;
/** The line number on which the source text begins. */
uint32_t initialLineNum_;
/**
* The index corresponding to the last offset lookup -- used so that if
* offset lookups proceed in increasing order, and and the offset appears
* in the next couple lines from the last offset, we can avoid a full
* binary-search.
*
* This is mutable because it's modified on every search, but that fact
* isn't visible outside this class.
*/
mutable uint32_t lastIndex_;
uint32_t indexFromOffset(uint32_t offset) const;
static const uint32_t MAX_PTR = UINT32_MAX;
uint32_t lineNumberFromIndex(uint32_t index) const {
return index + initialLineNum_;
}
uint32_t indexFromLineNumber(uint32_t lineNum) const {
return lineNum - initialLineNum_;
}
public:
SourceCoords(FrontendContext* fc, uint32_t initialLineNumber,
uint32_t initialOffset);
[[nodiscard]] bool add(uint32_t lineNum, uint32_t lineStartOffset);
[[nodiscard]] bool fill(const SourceCoords& other);
std::optional<bool> isOnThisLine(uint32_t offset, uint32_t lineNum) const {
uint32_t index = indexFromLineNumber(lineNum);
if (index + 1 >= lineStartOffsets_.length()) { // +1 due to sentinel
return std::nullopt;
}
return (lineStartOffsets_[index] <= offset &&
offset < lineStartOffsets_[index + 1]);
}
/**
* A token, computed for an offset in source text, that can be used to
* access line number and line-offset information for that offset.
*
* LineToken *alone* exposes whether the corresponding offset is in the
* the first line of source (which may not be 1, depending on
* |initialLineNumber|), and whether it's in the same line as
* another LineToken.
*/
class LineToken {
uint32_t index;
#ifdef DEBUG
uint32_t offset_; // stored for consistency-of-use assertions
#endif
friend class SourceCoords;
public:
LineToken(uint32_t index, uint32_t offset)
: index(index)
#ifdef DEBUG
,
offset_(offset)
#endif
{
}
bool isFirstLine() const { return index == 0; }
bool isSameLine(LineToken other) const { return index == other.index; }
void assertConsistentOffset(uint32_t offset) const {
MOZ_ASSERT(offset_ == offset);
}
};
/**
* Compute a token usable to access information about the line at the
* given offset.
*
* The only information directly accessible in a token is whether it
* corresponds to the first line of source text (which may not be line
* 1, depending on the |initialLineNumber| value used to construct
* this). Use |lineNumber(LineToken)| to compute the actual line
* number (incorporating the contribution of |initialLineNumber|).
*/
LineToken lineToken(uint32_t offset) const;
/** Compute the line number for the given token. */
uint32_t lineNumber(LineToken lineToken) const {
return lineNumberFromIndex(lineToken.index);
}
/** Return the offset of the start of the line for |lineToken|. */
uint32_t lineStart(LineToken lineToken) const {
MOZ_ASSERT(lineToken.index + 1 < lineStartOffsets_.length(),
"recorded line-start information must be available");
return lineStartOffsets_[lineToken.index];
}
};
enum class UnitsType : unsigned char {
PossiblyMultiUnit = 0,
GuaranteedSingleUnit = 1,
};
class ChunkInfo {
private:
// Column number offset in UTF-16 code units.
// Store everything in |unsigned char|s so everything packs.
unsigned char columnOffset_[sizeof(uint32_t)];
unsigned char unitsType_;
public:
ChunkInfo(JS::ColumnNumberUnsignedOffset offset, UnitsType type)
: unitsType_(static_cast<unsigned char>(type)) {
memcpy(columnOffset_, offset.addressOfValueForTranscode(), sizeof(offset));
}
JS::ColumnNumberUnsignedOffset columnOffset() const {
JS::ColumnNumberUnsignedOffset offset;
memcpy(offset.addressOfValueForTranscode(), columnOffset_,
sizeof(uint32_t));
return offset;
}
UnitsType unitsType() const {
MOZ_ASSERT(unitsType_ <= 1, "unitsType_ must be 0 or 1");
return static_cast<UnitsType>(unitsType_);
}
void guaranteeSingleUnits() {
MOZ_ASSERT(unitsType() == UnitsType::PossiblyMultiUnit,
"should only be setting to possibly optimize from the "
"pessimistic case");
unitsType_ = static_cast<unsigned char>(UnitsType::GuaranteedSingleUnit);
}
};
enum class InvalidEscapeType {
// No invalid character escapes.
None,
// A malformed \x escape.
Hexadecimal,
// A malformed \u escape.
Unicode,
// An otherwise well-formed \u escape which represents a
// codepoint > 10FFFF.
UnicodeOverflow,
// An octal escape in a template token.
Octal,
// NonOctalDecimalEscape - \8 or \9.
EightOrNine
};
class TokenStreamAnyChars : public TokenStreamShared {
private:
// Constant-at-construction fields.
FrontendContext* const fc;
/** Options used for parsing/tokenizing. */
const JS::ReadOnlyCompileOptions& options_;
/**
* Pointer used internally to test whether in strict mode. Use |strictMode()|
* instead of this field.
*/
StrictModeGetter* const strictModeGetter_;
/** Input filename or null. */
JS::ConstUTF8CharsZ filename_;
// Column number computation fields.
// Used only for UTF-8 case.
/**
* A map of (line number => sequence of the column numbers at
* |ColumnChunkLength|-unit boundaries rewound [if needed] to the nearest code
* point boundary). (|TokenStreamAnyChars::computeColumnOffset| is the sole
* user of |ColumnChunkLength| and therefore contains its definition.)
*
* Entries appear in this map only when a column computation of sufficient
* distance is performed on a line -- and only when the column is beyond the
* first |ColumnChunkLength| units. Each line's vector is lazily filled as
* greater offsets require column computations.
*/
mutable HashMap<uint32_t, Vector<ChunkInfo>> longLineColumnInfo_;
// Computing accurate column numbers requires at *some* point linearly
// iterating through prior source units in the line, to properly account for
// multi-unit code points. This is quadratic if counting happens repeatedly.
//
// But usually we need columns for advancing offsets through scripts. By
// caching the last ((line number, offset) => relative column) mapping (in
// similar manner to how |SourceCoords::lastIndex_| is used to cache
// (offset => line number) mappings) we can usually avoid re-iterating through
// the common line prefix.
//
// Additionally, we avoid hash table lookup costs by caching the
// |Vector<ChunkInfo>*| for the line of the last lookup. (|nullptr| means we
// must look it up -- or it hasn't been created yet.) This pointer is nulled
// when a lookup on a new line occurs, but as it's not a pointer at literal,
// reallocatable element data, it's *not* invalidated when new entries are
// added to such a vector.
/**
* The line in which the last column computation occurred, or UINT32_MAX if
* no prior computation has yet happened.
*/
mutable uint32_t lineOfLastColumnComputation_ = UINT32_MAX;
/**
* The chunk vector of the line for that last column computation. This is
* null if the chunk vector needs to be recalculated or initially created.
*/
mutable Vector<ChunkInfo>* lastChunkVectorForLine_ = nullptr;
/**
* The offset (in code units) of the last column computation performed,
* relative to source start.
*/
mutable uint32_t lastOffsetOfComputedColumn_ = UINT32_MAX;
/**
* The column number offset from the 1st column for the offset (in code units)
* of the last column computation performed, relative to source start.
*/
mutable JS::ColumnNumberUnsignedOffset lastComputedColumnOffset_;
// Intra-token fields.
/**
* The offset of the first invalid escape in a template literal. (If there is
* one -- if not, the value of this field is meaningless.)
*
* See also |invalidTemplateEscapeType|.
*/
uint32_t invalidTemplateEscapeOffset = 0;
/**
* The type of the first invalid escape in a template literal. (If there
* isn't one, this will be |None|.)
*
* See also |invalidTemplateEscapeOffset|.
*/
InvalidEscapeType invalidTemplateEscapeType = InvalidEscapeType::None;
// Fields with values relevant across tokens (and therefore potentially across
// function boundaries, such that lazy function parsing and stream-seeking
// must take care in saving and restoring them).
/** Line number and offset-to-line mapping information. */
SourceCoords srcCoords;
/** Circular token buffer of gotten tokens that have been ungotten. */
Token tokens[ntokens] = {};
/** The index in |tokens| of the last parsed token. */
unsigned cursor_ = 0;
/** The number of tokens in |tokens| available to be gotten. */
unsigned lookahead = 0;
/** The current line number. */
unsigned lineno;
/** Various flag bits (see above). */
TokenStreamFlags flags = {};
/** The offset of the start of the current line. */
size_t linebase = 0;
/** The start of the previous line, or |size_t(-1)| on the first line. */
size_t prevLinebase = size_t(-1);
/** The user's requested source URL. Null if none has been set. */
UniqueTwoByteChars displayURL_ = nullptr;
/** The URL of the source map for this script. Null if none has been set. */
UniqueTwoByteChars sourceMapURL_ = nullptr;
// Assorted boolean fields, none of which require maintenance across tokens,
// stored at class end to minimize padding.
/**
* Whether syntax errors should or should not contain details about the
* precise nature of the error. (This is intended for use in suppressing
* content-revealing details about syntax errors in cross-origin scripts on
* the web.)
*/
const bool mutedErrors;
/**
* An array storing whether a TokenKind observed while attempting to extend
* a valid AssignmentExpression into an even longer AssignmentExpression
* (e.g., extending '3' to '3 + 5') will terminate it without error.
*
* For example, ';' always ends an AssignmentExpression because it ends a
* Statement or declaration. '}' always ends an AssignmentExpression
* because it terminates BlockStatement, FunctionBody, and embedded
* expressions in TemplateLiterals. Therefore both entries are set to true
* in TokenStreamAnyChars construction.
*
* But e.g. '+' *could* extend an AssignmentExpression, so its entry here
* is false. Meanwhile 'this' can't extend an AssignmentExpression, but
* it's only valid after a line break, so its entry here must be false.
*
* NOTE: This array could be static, but without C99's designated
* initializers it's easier zeroing here and setting the true entries
* in the constructor body. (Having this per-instance might also aid
* locality.) Don't worry! Initialization time for each TokenStream
*/
bool isExprEnding[size_t(TokenKind::Limit)] = {}; // all-false initially
// End of fields.
public:
TokenStreamAnyChars(FrontendContext* fc,
const JS::ReadOnlyCompileOptions& options,
StrictModeGetter* smg);
template <typename Unit, class AnyCharsAccess>
friend class GeneralTokenStreamChars;
template <typename Unit, class AnyCharsAccess>
friend class TokenStreamChars;
template <typename Unit, class AnyCharsAccess>
friend class TokenStreamSpecific;
template <typename Unit>
friend class TokenStreamPosition;
// Accessors.
unsigned cursor() const { return cursor_; }
unsigned nextCursor() const { return (cursor_ + 1) & ntokensMask; }
unsigned aheadCursor(unsigned steps) const {
return (cursor_ + steps) & ntokensMask;
}
const Token& currentToken() const { return tokens[cursor()]; }
bool isCurrentTokenType(TokenKind type) const {
return currentToken().type == type;
}
[[nodiscard]] bool checkOptions();
private:
TaggedParserAtomIndex reservedWordToPropertyName(TokenKind tt) const;
public:
TaggedParserAtomIndex currentName() const {
if (isCurrentTokenType(TokenKind::Name) ||
isCurrentTokenType(TokenKind::PrivateName)) {
return currentToken().name();
}
MOZ_ASSERT(TokenKindIsPossibleIdentifierName(currentToken().type));
return reservedWordToPropertyName(currentToken().type);
}
bool currentNameHasEscapes(ParserAtomsTable& parserAtoms) const {
if (isCurrentTokenType(TokenKind::Name) ||
isCurrentTokenType(TokenKind::PrivateName)) {
TokenPos pos = currentToken().pos;
return (pos.end - pos.begin) != parserAtoms.length(currentToken().name());
}
MOZ_ASSERT(TokenKindIsPossibleIdentifierName(currentToken().type));
return false;
}
bool isCurrentTokenAssignment() const {
return TokenKindIsAssignment(currentToken().type);
}
// Flag methods.
bool isEOF() const { return flags.isEOF; }
bool hadError() const { return flags.hadError; }
DeprecatedContent sawDeprecatedContent() const {
return static_cast<DeprecatedContent>(flags.sawDeprecatedContent);
}
private:
// Workaround GCC 7 sadness.
void setSawDeprecatedContent(DeprecatedContent content) {
flags.sawDeprecatedContent = static_cast<uint8_t>(content);
}
public:
void clearSawDeprecatedContent() {
setSawDeprecatedContent(DeprecatedContent::None);
}
void setSawDeprecatedOctalLiteral() {
setSawDeprecatedContent(DeprecatedContent::OctalLiteral);
}
void setSawDeprecatedOctalEscape() {
setSawDeprecatedContent(DeprecatedContent::OctalEscape);
}
void setSawDeprecatedEightOrNineEscape() {
setSawDeprecatedContent(DeprecatedContent::EightOrNineEscape);
}
bool hasInvalidTemplateEscape() const {
return invalidTemplateEscapeType != InvalidEscapeType::None;
}
void clearInvalidTemplateEscape() {
invalidTemplateEscapeType = InvalidEscapeType::None;
}
private:
// This is private because it should only be called by the tokenizer while
// tokenizing not by, for example, BytecodeEmitter.
bool strictMode() const {
return strictModeGetter_ && strictModeGetter_->strictMode();
}
void setInvalidTemplateEscape(uint32_t offset, InvalidEscapeType type) {
MOZ_ASSERT(type != InvalidEscapeType::None);
if (invalidTemplateEscapeType != InvalidEscapeType::None) {
return;
}
invalidTemplateEscapeOffset = offset;
invalidTemplateEscapeType = type;
}
public:
// Call this immediately after parsing an OrExpression to allow scanning the
// next token with SlashIsRegExp without asserting (even though we just
// peeked at it in SlashIsDiv mode).
//
// It's OK to disable the assertion because the places where this is called
// have peeked at the next token in SlashIsDiv mode, and checked that it is
// *not* a Div token.
//
// To see why it is necessary to disable the assertion, consider these two
// programs:
//
// x = arg => q // per spec, this is all one statement, and the
// /a/g; // slashes are division operators
//
// x = arg => {} // per spec, ASI at the end of this line
// /a/g; // and that's a regexp literal
//
// The first program shows why orExpr() has use SlashIsDiv mode when peeking
// ahead for the next operator after parsing `q`. The second program shows
// why matchOrInsertSemicolon() must use SlashIsRegExp mode when scanning
// ahead for a semicolon.
void allowGettingNextTokenWithSlashIsRegExp() {
#ifdef DEBUG
// Check the precondition: Caller already peeked ahead at the next token,
// in SlashIsDiv mode, and it is *not* a Div token.
MOZ_ASSERT(hasLookahead());
const Token& next = nextToken();
MOZ_ASSERT(next.modifier == SlashIsDiv);
MOZ_ASSERT(next.type != TokenKind::Div);
tokens[nextCursor()].modifier = SlashIsRegExp;
#endif
}
#ifdef DEBUG
inline bool debugHasNoLookahead() const { return lookahead == 0; }
#endif
bool hasDisplayURL() const { return displayURL_ != nullptr; }
char16_t* displayURL() { return displayURL_.get(); }
bool hasSourceMapURL() const { return sourceMapURL_ != nullptr; }
char16_t* sourceMapURL() { return sourceMapURL_.get(); }
FrontendContext* context() const { return fc; }
using LineToken = SourceCoords::LineToken;
LineToken lineToken(uint32_t offset) const {
return srcCoords.lineToken(offset);
}
uint32_t lineNumber(LineToken lineToken) const {
return srcCoords.lineNumber(lineToken);
}
uint32_t lineStart(LineToken lineToken) const {
return srcCoords.lineStart(lineToken);
}
/**
* Fill in |err|.
*
* If the token stream doesn't have location info for this error, use the
* caller's location (including line/column number) and return false. (No
* line of context is set.)
*
* Otherwise fill in everything in |err| except 1) line/column numbers and
* 2) line-of-context-related fields and return true. The caller *must*
* fill in the line/column number; filling the line of context is optional.
*/
bool fillExceptingContext(ErrorMetadata* err, uint32_t offset) const;
MOZ_ALWAYS_INLINE void updateFlagsForEOL() { flags.isDirtyLine = false; }
private:
/**
* Compute the column number offset from the 1st code unit in the line in
* UTF-16 code units, for given absolute |offset| within source text on the
* line of |lineToken| (which must have been computed from |offset|).
*
* A column number offset on a line that isn't the first line is just
* the actual column number in 0-origin. But a column number offset
* on the first line is the column number offset from the initial
* line/column of the script. For example, consider this HTML with
* line/column number keys:
*
* Column number in 1-origin
* 1 2 3
* 123456789012345678901234 567890
*
* Column number in 0-origin, and the offset from 1st column
* 1 2 3
* 0123456789012345678901234 567890
* ------------------------------------
* 1 | <html>
* 2 | <head>
* 3 | <script>var x = 3; x < 4;
* 4 | const y = 7;</script>
* 5 | </head>
* 6 | <body></body>
* 7 | </html>
*
* The script would be compiled specifying initial (line, column) of (3, 10)
* using |JS::ReadOnlyCompileOptions::{lineno,column}|, which is 0-origin.
* And the column reported by |computeColumn| for the "v" of |var| would be
* 11 (in 1-origin). But the column number offset of the "v" in |var|, that
* this function returns, would be 0. On the other hand, the column reported
* by |computeColumn| would be 1 (in 1-origin) and the column number offset
* returned by this function for the "c" in |const| would be 0, because it's
* not in the first line of source text.
*
* The column number offset is with respect *only* to the JavaScript source
* text as SpiderMonkey sees it. In the example, the "<" is converted to
* "<" by the browser before SpiderMonkey would see it. So the column number
* offset of the "4" in the inequality would be 16, not 19.
*
* UTF-16 code units are not all equal length in UTF-8 source, so counting
* requires *some* kind of linear-time counting from the start of the line.
* This function attempts various tricks to reduce this cost. If these
* optimizations succeed, repeated calls to this function on a line will pay
* a one-time cost linear in the length of the line, then each call pays a
* separate constant-time cost. If the optimizations do not succeed, this
* function works in time linear in the length of the line.
*
* It's unusual for a function in *this* class to be |Unit|-templated, but
* while this operation manages |Unit|-agnostic fields in this class and in
* |srcCoords|, it must *perform* |Unit|-sensitive computations to fill them.
* And this is the best place to do that.
*/
template <typename Unit>
JS::ColumnNumberUnsignedOffset computeColumnOffset(
const LineToken lineToken, const uint32_t offset,
const SourceUnits<Unit>& sourceUnits) const;
template <typename Unit>
JS::ColumnNumberUnsignedOffset computeColumnOffsetForUTF8(
const LineToken lineToken, const uint32_t offset, const uint32_t start,
const uint32_t offsetInLine, const SourceUnits<Unit>& sourceUnits) const;
/**
* Update line/column information for the start of a new line at
* |lineStartOffset|.
*/
[[nodiscard]] MOZ_ALWAYS_INLINE bool internalUpdateLineInfoForEOL(
uint32_t lineStartOffset);
public:
const Token& nextToken() const {
MOZ_ASSERT(hasLookahead());
return tokens[nextCursor()];
}
bool hasLookahead() const { return lookahead > 0; }
void advanceCursor() { cursor_ = (cursor_ + 1) & ntokensMask; }
void retractCursor() { cursor_ = (cursor_ - 1) & ntokensMask; }
Token* allocateToken() {
advanceCursor();
Token* tp = &tokens[cursor()];
MOZ_MAKE_MEM_UNDEFINED(tp, sizeof(*tp));
return tp;
}
// Push the last scanned token back into the stream.
void ungetToken() {
MOZ_ASSERT(lookahead < maxLookahead);
lookahead++;
retractCursor();
}
public:
void adoptState(TokenStreamAnyChars& other) {
// If |other| has fresh information from directives, overwrite any
// previously recorded directives. (There is no specification directing
// that last-in-source-order directive controls, sadly. We behave this way
// in the ordinary case, so we ought do so here too.)
if (auto& url = other.displayURL_) {
displayURL_ = std::move(url);
}
if (auto& url = other.sourceMapURL_) {
sourceMapURL_ = std::move(url);
}
}
// Compute error metadata for an error at no offset.
void computeErrorMetadataNoOffset(ErrorMetadata* err) const;
// ErrorReporter API Helpers
// Provide minimal set of error reporting API given we cannot use
// ErrorReportMixin here. "report" prefix is added to avoid conflict with
// ErrorReportMixin methods in TokenStream class.
void reportErrorNoOffset(unsigned errorNumber, ...) const;
void reportErrorNoOffsetVA(unsigned errorNumber, va_list* args) const;
const JS::ReadOnlyCompileOptions& options() const { return options_; }
JS::ConstUTF8CharsZ getFilename() const { return filename_; }
};
constexpr char16_t CodeUnitValue(char16_t unit) { return unit; }
constexpr uint8_t CodeUnitValue(mozilla::Utf8Unit unit) {
return unit.toUint8();
}
template <typename Unit>
class TokenStreamCharsBase;
template <typename T>
inline bool IsLineTerminator(T) = delete;
inline bool IsLineTerminator(char32_t codePoint) {
return codePoint == '\n' || codePoint == '\r' ||
codePoint == unicode::LINE_SEPARATOR ||
codePoint == unicode::PARA_SEPARATOR;
}
inline bool IsLineTerminator(char16_t unit) {
// Every LineTerminator fits in char16_t, so this is exact.
return IsLineTerminator(static_cast<char32_t>(unit));
}
template <typename Unit>
struct SourceUnitTraits;
template <>
struct SourceUnitTraits<char16_t> {
public:
static constexpr uint8_t maxUnitsLength = 2;
static constexpr size_t lengthInUnits(char32_t codePoint) {
return codePoint < unicode::NonBMPMin ? 1 : 2;
}
};
template <>
struct SourceUnitTraits<mozilla::Utf8Unit> {
public:
static constexpr uint8_t maxUnitsLength = 4;
static constexpr size_t lengthInUnits(char32_t codePoint) {
return codePoint < 0x80 ? 1
: codePoint < 0x800 ? 2
: codePoint < 0x10000 ? 3
: 4;
}
};
/**
* PeekedCodePoint represents the result of peeking ahead in some source text
* to determine the next validly-encoded code point.
*
* If there isn't a valid code point, then |isNone()|.
*
* But if there *is* a valid code point, then |!isNone()|, the code point has
* value |codePoint()| and its length in code units is |lengthInUnits()|.
*
* Conceptually, this class is |Maybe<struct { char32_t v; uint8_t len; }>|.
*/
template <typename Unit>
class PeekedCodePoint final {
char32_t codePoint_ = 0;
uint8_t lengthInUnits_ = 0;
private:
using SourceUnitTraits = frontend::SourceUnitTraits<Unit>;
PeekedCodePoint() = default;
public:
/**
* Create a peeked code point with the given value and length in code
* units.
*
* While the latter value is computable from the former for both UTF-8 and
* JS's version of UTF-16, the caller likely computed a length in units in
* the course of determining the peeked value. Passing both here avoids
* recomputation and lets us do a consistency-checking assertion.
*/
PeekedCodePoint(char32_t codePoint, uint8_t lengthInUnits)
: codePoint_(codePoint), lengthInUnits_(lengthInUnits) {
MOZ_ASSERT(codePoint <= unicode::NonBMPMax);
MOZ_ASSERT(lengthInUnits != 0, "bad code point length");
MOZ_ASSERT(lengthInUnits == SourceUnitTraits::lengthInUnits(codePoint));
}
/** Create a PeekedCodeUnit that represents no valid code point. */
static PeekedCodePoint none() { return PeekedCodePoint(); }
/** True if no code point was found, false otherwise. */
bool isNone() const { return lengthInUnits_ == 0; }
/** If a code point was found, its value. */
char32_t codePoint() const {
MOZ_ASSERT(!isNone());
return codePoint_;
}
/** If a code point was found, its length in code units. */
uint8_t lengthInUnits() const {
MOZ_ASSERT(!isNone());
return lengthInUnits_;
}
};
inline PeekedCodePoint<char16_t> PeekCodePoint(const char16_t* const ptr,
const char16_t* const end) {
if (MOZ_UNLIKELY(ptr >= end)) {
return PeekedCodePoint<char16_t>::none();
}
char16_t lead = ptr[0];
char32_t c;
uint8_t len;
if (MOZ_LIKELY(!unicode::IsLeadSurrogate(lead)) ||
MOZ_UNLIKELY(ptr + 1 >= end || !unicode::IsTrailSurrogate(ptr[1]))) {
c = lead;
len = 1;
} else {
c = unicode::UTF16Decode(lead, ptr[1]);
len = 2;
}
return PeekedCodePoint<char16_t>(c, len);
}
inline PeekedCodePoint<mozilla::Utf8Unit> PeekCodePoint(
const mozilla::Utf8Unit* const ptr, const mozilla::Utf8Unit* const end) {
if (MOZ_UNLIKELY(ptr >= end)) {
return PeekedCodePoint<mozilla::Utf8Unit>::none();
}
const mozilla::Utf8Unit lead = ptr[0];
if (mozilla::IsAscii(lead)) {
return PeekedCodePoint<mozilla::Utf8Unit>(lead.toUint8(), 1);
}
const mozilla::Utf8Unit* afterLead = ptr + 1;
mozilla::Maybe<char32_t> codePoint =
mozilla::DecodeOneUtf8CodePoint(lead, &afterLead, end);
if (codePoint.isNothing()) {
return PeekedCodePoint<mozilla::Utf8Unit>::none();
}
auto len =
mozilla::AssertedCast<uint8_t>(mozilla::PointerRangeSize(ptr, afterLead));
MOZ_ASSERT(len <= 4);
return PeekedCodePoint<mozilla::Utf8Unit>(codePoint.value(), len);
}
inline bool IsSingleUnitLineTerminator(mozilla::Utf8Unit unit) {
// BEWARE: The Unicode line/paragraph separators don't fit in a single
// UTF-8 code unit, so this test is exact for Utf8Unit but inexact
// for UTF-8 as a whole. Users must handle |unit| as start of a
// Unicode LineTerminator themselves!
return unit == mozilla::Utf8Unit('\n') || unit == mozilla::Utf8Unit('\r');
}
// This is the low-level interface to the JS source code buffer. It just gets
// raw Unicode code units -- 16-bit char16_t units of source text that are not
// (always) full code points, and 8-bit units of UTF-8 source text soon.
// TokenStreams functions are layered on top and do some extra stuff like
// converting all EOL sequences to '\n', tracking the line number, and setting
// |flags.isEOF|. (The "raw" in "raw Unicode code units" refers to the lack of
// EOL sequence normalization.)
//
// buf[0..length-1] often represents a substring of some larger source,
// where we have only the substring in memory. The |startOffset| argument
// indicates the offset within this larger string at which our string
// begins, the offset of |buf[0]|.
template <typename Unit>
class SourceUnits {
private:
/** Base of buffer. */
const Unit* base_;
/** Offset of base_[0]. */
uint32_t startOffset_;
/** Limit for quick bounds check. */
const Unit* limit_;
/** Next char to get. */
const Unit* ptr;
public:
SourceUnits(const Unit* units, size_t length, size_t startOffset)
: base_(units),
startOffset_(startOffset),
limit_(units + length),
ptr(units) {}
bool atStart() const {
MOZ_ASSERT(!isPoisoned(), "shouldn't be using if poisoned");
return ptr == base_;
}
bool atEnd() const {
MOZ_ASSERT(!isPoisoned(), "shouldn't be using if poisoned");
MOZ_ASSERT(ptr <= limit_, "shouldn't have overrun");
return ptr >= limit_;
}
size_t remaining() const {
MOZ_ASSERT(!isPoisoned(),
"can't get a count of remaining code units if poisoned");
return mozilla::PointerRangeSize(ptr, limit_);
}
size_t startOffset() const { return startOffset_; }
size_t offset() const {
return startOffset_ + mozilla::PointerRangeSize(base_, ptr);
}
const Unit* codeUnitPtrAt(size_t offset) const {
MOZ_ASSERT(!isPoisoned(), "shouldn't be using if poisoned");
MOZ_ASSERT(startOffset_ <= offset);
MOZ_ASSERT(offset - startOffset_ <=
mozilla::PointerRangeSize(base_, limit_));
return base_ + (offset - startOffset_);
}
const Unit* current() const { return ptr; }
const Unit* limit() const { return limit_; }
Unit previousCodeUnit() {
MOZ_ASSERT(!isPoisoned(), "can't get previous code unit if poisoned");
MOZ_ASSERT(!atStart(), "must have a previous code unit to get");
return *(ptr - 1);
}
MOZ_ALWAYS_INLINE Unit getCodeUnit() {
return *ptr++; // this will nullptr-crash if poisoned
}
Unit peekCodeUnit() const {
return *ptr; // this will nullptr-crash if poisoned
}
/**
* Determine the next code point in source text. The code point is not
* normalized: '\r', '\n', '\u2028', and '\u2029' are returned literally.
* If there is no next code point because |atEnd()|, or if an encoding
* error is encountered, return a |PeekedCodePoint| that |isNone()|.
*
* This function does not report errors: code that attempts to get the next
* code point must report any error.
*
* If a next code point is found, it may be consumed by passing it to
* |consumeKnownCodePoint|.
*/
PeekedCodePoint<Unit> peekCodePoint() const {
return PeekCodePoint(ptr, limit_);
}
private:
#ifdef DEBUG
void assertNextCodePoint(const PeekedCodePoint<Unit>& peeked);
#endif
public:
/**
* Consume a peeked code point that |!isNone()|.
*
* This call DOES NOT UPDATE LINE-STATUS. You may need to call
* |updateLineInfoForEOL()| and |updateFlagsForEOL()| if this consumes a
* LineTerminator. Note that if this consumes '\r', you also must consume
* an optional '\n' (i.e. a full LineTerminatorSequence) before doing so.
*/
void consumeKnownCodePoint(const PeekedCodePoint<Unit>& peeked) {
MOZ_ASSERT(!peeked.isNone());
MOZ_ASSERT(peeked.lengthInUnits() <= remaining());
#ifdef DEBUG
assertNextCodePoint(peeked);
#endif
ptr += peeked.lengthInUnits();
}
/** Match |n| hexadecimal digits and store their value in |*out|. */
bool matchHexDigits(uint8_t n, char16_t* out) {
MOZ_ASSERT(!isPoisoned(), "shouldn't peek into poisoned SourceUnits");