2 Commits

Author SHA1 Message Date
b99f3586dc
chore(license): Added NOTICE header to all source files
Signed-off-by: erick-alcachofa <erick@artichoke.dev>
2025-12-25 13:12:41 -06:00
85a34bdd65
feat: Added Token, Tokenizer, Generator, and some utilities
Initial version of Tokenizer and Token
Generator template for coroutines (used in tokenizer)
Utilities like string related functions, TrieMap, and error handling

TODO: Add tests for Tokenizer
TODO: Add tests for Generator
2025-03-10 01:20:23 -06:00