Lexical structure

An identifier starts with _, a Unicode letter, or a non-ASCII scalar with the Unicode Emoji property. Later scalars may also be Unicode numbers and combining marks.

The character classes use Unicode 16.0.0. Letters use Lu, Ll, Lt, Lm, Lo, numbers use Nd, Nl, No, and combining marks use Mn, Mc, Me. A combining mark cannot start a name.

Exclude every scalar with Default_Ignorable_Code_Point from both positions, even when it is a letter or combining mark. This includes variation selectors, join controls, combining grapheme joiner, Hangul fillers, and emoji tag characters. No script or emoji sequence receives an exception. Each scalar in a multi-scalar emoji name must qualify independently. String data follows the string rules.

When no string delimiter immediately follows, the lexer classifies the complete identifier spelling. A spelling in ReservedWord becomes a keyword token. The complete spelling _ becomes an underscore token. Other spellings become identifier tokens.

' followed by IdentifierSpelling forms a label token. Its body is not reclassified as a keyword.

The complete name must already be in Unicode 16.0.0 NFC form in the source. This applies to declarations, references, members, fields, label bodies, and template tag names. Check the whole sequence before keyword or tag classification.

Bindings and lookup compare the admitted scalar sequence exactly. They do not normalize a spelling silently or create a second alias. NFC uses canonical equivalence. It does not apply compatibility normalization or case folding.

A non-NFC name produces TPZ0007 during lex. Mark the complete name span, excluding a label's apostrophe and a tag's string delimiter, and suggest its NFC spelling. Review the replacement for collisions before changing the source.

A name position that requires an identifier rejects keyword tokens.

A field-name position accepts an identifier token or a keyword token.

For identifier tokens with the spellings below, the parser assigns a contextual role when the surrounding form matches.

The punctuation scanner chooses the longest matching punctuation spelling. Comments, string delimiters, and label prefixes have separate lexical forms. The adjacent >> token supplies two type-closing delimiters. It has no expression-operator meaning.

Space (U+0020) and horizontal tab (U+0009) separate tokens. LF (U+000A), CRLF, and bare CR (U+000D) each supply one line break to layout processing. Each spelling preserves the same statement boundary. Indentation does not delimit blocks. Braces and the surrounding syntax determine block and list boundaries.

A line comment begins with // and ends before the next LF, CRLF, or bare CR, or at the end of the source. Its terminating line break remains available to layout processing.

Write every comment line with //. The /* ... */ spelling is rejected with TPZ2001 during parsing. Its opener is tokenized as ordinary / and * punctuation. Comment markers inside a string remain text.

LF, CRLF, and bare CR separate completed items at program level and in statement blocks when no continuation condition applies, including the break that ends a line comment. A semicolon is rejected and never discards a value. Blank lines do not introduce an extra item. No implicit separator is inserted immediately before a closing delimiter or the end of the source. The final expression remains the block value across trailing line breaks and comments. To discard that value explicitly, write let _ = expression.

A line break continues the surrounding form while its active context is parentheses, square brackets, interpolation, record fields, or import selections. An explicit ; in those contexts produces TPZ1001. Use , between list elements.

A newline continues the form at these token positions.

A line break on either side of | continues a binding, case, or for pattern while that pattern is being read.

An unrecognized source character produces TPZ0001 at that character's location.

CLI diagnostic labels use 1-based line, col, endLine, and endCol. Columns count Unicode scalar values from the start of the line. The lo and hi fields use 0-based UTF-8 byte offsets. Both end positions are exclusive.