Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit lays the foundational groundwork for the artichoke language
parser by introducing the formal language grammar specification.
The tokenizer was updated to include new operators and keywords, also
added the posibility to handle comments.
Key Additions:
- Implemented support for C-style block comments (`/* ... */`),
including error handling for unclosed comments.
- Added all necessary tokens for missing keywords (e.g., `module`,
`export`, `using`, `match`, `loop`) and operators (e.g., `+=`, `:=`,
`.#`, `.*`, `.@`).
- The `Token` enum has been expanded to reflect the full language
feature set.
Documentation:
- Added `docs/grammar.ebnf` which contains the official, well-structured
EBNF grammar for the language.
- Added `docs/readme.md` providing a detailed technical overview of the
language's features, syntax, and semantics.
BREAKING CHANGE: The `kwVariant` and `kwMut` tokens have been removed to
align with the updated language design defined in the new grammar.
Fixed some minor mistakes (wrong messages/errors) due to copy/pasting
code.
Fixed that digits weren't allowed in identifiers before.
Also minor improvements in some functions/code parts.
Initial version of Tokenizer and Token
Generator template for coroutines (used in tokenizer)
Utilities like string related functions, TrieMap, and error handling
TODO: Add tests for Tokenizer
TODO: Add tests for Generator
