Signed-off-by: erick-alcachofa <erick@artichoke.dev>
- Updated `main.cpp` to read source files and invoke the parser using
`MatcherRule<rules::Program>`, providing clear error output or success
indication.
- Replaced `peek` with `peekExpect` in `Parser` to ensure expected token
types.
- Replaced parentheses with curly braces for struct and enum definitions
as specified on grammar.
- Fixed `StructMembers` and `EnumMembers` to allow zero or more entries
as specified on grammar.
- Replaced `MatcherNoneOrOnce` with `MatcherNoneOrMore` in expression
rules to support chained binary expressions as specified on grammar.
- Reorganized `PrimaryTypeExpression` to include all valid expression
types, and reordered rules for better match accuracy.
- Fixed `TypesListR` to correctly refer to `Type` instead of
`TypesListR` in its recursive rule.
These changes fixes language grammar coverage, fix structural parsing
bugs, and enable full file parsing from the frontend.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
Introduced `peekExpect(std::size_t, TokenV)` to the Tokenizer class, enabling
token lookahead with explicit token type checks. This method returns an
`Unexpected` error with diagnostic info if the expected token type does not
match the peeked token.
Includes a special case handling (workaround) for distinguishing between
`>` and `>>` tokens when parsing the token stream.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
The EBNF grammar definition contained several redundancies,
inconsistencies, and minor omissions. This commit refactors the grammar
to make it more concise, readable, and robust for parsing.
Key changes include:
- **Rule Simplification**: Redundant intermediate rules (`fn_params`,
`statements`, `assign_expression`) have been removed. Rules like
`code_block` and `import_target` are now more concisely expressed
using standard EBNF operators (`?`, `*`).
- **EOF Enforcement**: The top-level `program` rule now requires an
`<eof>` token. This is a crucial fix to ensure the parser consumes the
entire file and fails on trailing invalid tokens.
- **Optional Generics**: Generic parameters (`<... >`) are now correctly
marked as optional on `function`, `struct`, and `enum` declarations,
which was the original intent.
- **Flexible For-Loops**: The update/increment expression (the third
part) in a C-style `for` loop is now optional, aligning with behavior
in languages like C and C++.
- **Primary Expressions**: Primary type expressions failed to parse
correctly namespaced elements and types, now it's fixed and improved.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit introduces the initial implementation of the language
parser. It is designed as a compile-time, template-based recursive
descent parser using a system of parser combinators.
The new `Parser.hpp` header contains the core parsing infrastructure:
- A set of parser combinator structs (`MatcherSequence`, `MatcherAnyOf`,
`MatcherNoneOrMore`, etc.) to build complex parsing logic.
- C++20 concepts (`ParserRule`, `ParserRuleDefinition`) to define and
constrain grammar rules.
- A `GenericParser` to recursively dispatch parsing tasks.
- A `rules` namespace that defines the entire language grammar, with
each rule implemented as a struct using the combinator framework.
- An initial `MatcherEOF` to ensure the entire input is consumed.
The parser currently validates the token stream against the defined
grammar but does not yet build a meaningful Abstract Syntax Tree (AST).
Placeholder structures are in place for future AST construction.
WARN: This is an experimental implementation just for fun and may not be
the implementation that might end up being used.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit introduces a comprehensive test suite for the tokenizer
using the Catch2 framework. To support this and improve the project
structure, the build system and the tokenizer's API have been
significantly updated.
- Removed `cmake/testing.cmake` as it's no longer needed.
- A new `TokenizerRange` class provides a C++20-style range interface,
allowing for simple `for-each` loop iteration over tokens. This is
used extensively in the new tests.
- The CMake build system has been refactored:
- An `ENABLE_TESTING` option (OFF by default) now controls whether
the test suite is built.
- The core library is now compiled into an object library, which is
then used to produce both a shared (`.so`/`.dll`) and a static
(`.a`/`.lib`) library. This improves build efficiency and provides
more flexible linkage options.
- The frontend executable now links against the static version of
the library.
- Implemented tests for tokenizer using Catch2 framework, covering
various cases like identifiers, keywords, numbers, etc. that already
catched some issues in current implementation.
- Several parsing bugs and edge cases in the tokenizer were fixed,
including the handling of unterminated strings and invalid numeric
literals. The README has been updated with instructions for building
and running tests.
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
This commit lays the foundational groundwork for the artichoke language
parser by introducing the formal language grammar specification.
The tokenizer was updated to include new operators and keywords, also
added the posibility to handle comments.
Key Additions:
- Implemented support for C-style block comments (`/* ... */`),
including error handling for unclosed comments.
- Added all necessary tokens for missing keywords (e.g., `module`,
`export`, `using`, `match`, `loop`) and operators (e.g., `+=`, `:=`,
`.#`, `.*`, `.@`).
- The `Token` enum has been expanded to reflect the full language
feature set.
Documentation:
- Added `docs/grammar.ebnf` which contains the official, well-structured
EBNF grammar for the language.
- Added `docs/readme.md` providing a detailed technical overview of the
language's features, syntax, and semantics.
BREAKING CHANGE: The `kwVariant` and `kwMut` tokens have been removed to
align with the updated language design defined in the new grammar.
Fixed some minor mistakes (wrong messages/errors) due to copy/pasting
code.
Fixed that digits weren't allowed in identifiers before.
Also minor improvements in some functions/code parts.
Initial version of Tokenizer and Token
Generator template for coroutines (used in tokenizer)
Utilities like string related functions, TrieMap, and error handling
TODO: Add tests for Tokenizer
TODO: Add tests for Generator
