Signed-off-by: erick-alcachofa <erick@artichoke.dev>
Major refactoring of the Parser and Tokenizer components to improve code
maintainability, strengthen error messaging, and streamline AST
generation.
This version intentionally focuses on top-level declarations, with
statement parsing stubbed for the next development phase.
- **Path Sanitization**: Added `sanitizePath` to extract filenames from
input paths, ensuring consistent `unitName` identification regardless
of directory depth.
- **Improved Output**: Wrapped AST string output in Markdown code blocks
and added a commented-out entry for the new DOT graph visualization.
- **Unified Consumption**: Replaced manual token checks with a more
robust `consume()` method that leverages `peekExpect()` for
centralized error handling.
- **New Predicates**: Introduced `match()` and `matchAndConsume()`
helpers to handle optional tokens and branching logic without
redundant peek/consume calls.
- **Exception Handling**: Standardized the use of `langException` across
all parsing functions, providing more descriptive "Expected X, found
Y" messages.
- **Declarations**: Refactored `parseTopLevelDeclaration` and
sub-parsers (Module, Struct, Enum, Fn) to use the new matching
patterns.
- **Looping Logic**: Replaced recursive-style parsing loops with
`while(keepParsing)` iterative blocks to prevent stack depth issues
and clarify termination conditions (e.g., finding a closing brace or
failing to find a comma).
- **Namespaced Identifiers**: Rewrote `parseNamespacedIdentifier` to
correctly handle multi-part paths (`A::B::C`) and edge cases.
- **Generic Support**: Improved handling of generic parameter and
argument lists, ensuring strict enforcement of delimiters like `<` and
`>`.
- **Contextual Errors**: Updated `peekExpect` to accept a custom
`message` string, allowing the parser to describe *what* it was
looking for (e.g., "Expected ';'").
- **Token Lookahead**: Enhanced `peek` and `peekExpect` reliability with
better bounds checking and buffer management.
- **Removed `lib/src/Parser/AST/AST.cpp`**: Deleted the monolithic AST
stringification file in favor of the previously introduced modular
implementations.
- **Build System**: Updated `.gitignore` to ignore
`cpm-package-lock.cmake`.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
Introduced `peekExpect(std::size_t, TokenV)` to the Tokenizer class, enabling
token lookahead with explicit token type checks. This method returns an
`Unexpected` error with diagnostic info if the expected token type does not
match the peeked token.
Includes a special case handling (workaround) for distinguishing between
`>` and `>>` tokens when parsing the token stream.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit introduces a comprehensive test suite for the tokenizer
using the Catch2 framework. To support this and improve the project
structure, the build system and the tokenizer's API have been
significantly updated.
- Removed `cmake/testing.cmake` as it's no longer needed.
- A new `TokenizerRange` class provides a C++20-style range interface,
allowing for simple `for-each` loop iteration over tokens. This is
used extensively in the new tests.
- The CMake build system has been refactored:
- An `ENABLE_TESTING` option (OFF by default) now controls whether
the test suite is built.
- The core library is now compiled into an object library, which is
then used to produce both a shared (`.so`/`.dll`) and a static
(`.a`/`.lib`) library. This improves build efficiency and provides
more flexible linkage options.
- The frontend executable now links against the static version of
the library.
- Implemented tests for tokenizer using Catch2 framework, covering
various cases like identifiers, keywords, numbers, etc. that already
catched some issues in current implementation.
- Several parsing bugs and edge cases in the tokenizer were fixed,
including the handling of unterminated strings and invalid numeric
literals. The README has been updated with instructions for building
and running tests.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit lays the foundational groundwork for the artichoke language
parser by introducing the formal language grammar specification.
The tokenizer was updated to include new operators and keywords, also
added the posibility to handle comments.
Key Additions:
- Implemented support for C-style block comments (`/* ... */`),
including error handling for unclosed comments.
- Added all necessary tokens for missing keywords (e.g., `module`,
`export`, `using`, `match`, `loop`) and operators (e.g., `+=`, `:=`,
`.#`, `.*`, `.@`).
- The `Token` enum has been expanded to reflect the full language
feature set.
Documentation:
- Added `docs/grammar.ebnf` which contains the official, well-structured
EBNF grammar for the language.
- Added `docs/readme.md` providing a detailed technical overview of the
language's features, syntax, and semantics.
BREAKING CHANGE: The `kwVariant` and `kwMut` tokens have been removed to
align with the updated language design defined in the new grammar.
Initial version of Tokenizer and Token
Generator template for coroutines (used in tokenizer)
Utilities like string related functions, TrieMap, and error handling
TODO: Add tests for Tokenizer
TODO: Add tests for Generator
