artichoke-lang/docs/Reference-Tokens.md

Token Reference

Token definitions used by the tokenizer.

Literals

• tkInteger - decimal integers (10, 42).
• tkDecimal - floating-point literals (3.14159265358979).
• tkString - double-quoted strings.
• tkCharacter - character literals.
• tkIdentifier - identifiers and names.
• tkEOF - end-of-file sentinel.

Operators

Includes all operator tokens such as ::, ->, .[, .#, .*, .@, and the compound assignments (+=, etc.).

Keywords

import, using, struct, enum, fn, return, let, def, if, else, for, while, do, loop, match, switch, defer, errdefer, true, false, null, typename, this, _ (default pattern), and logical aliases or, and.

Notes

• Keyword recognition uses a trie to ensure single-pass tokenization.
• Multi-character operators (::, .[, .#, .*, .@, >>=) rely on lookahead logic.
• Generic parsing may split >> into > + > to disambiguate template closers.