1 Commits

Author SHA1 Message Date
85a34bdd65
feat: Added Token, Tokenizer, Generator, and some utilities
Initial version of Tokenizer and Token
Generator template for coroutines (used in tokenizer)
Utilities like string related functions, TrieMap, and error handling

TODO: Add tests for Tokenizer
TODO: Add tests for Generator
2025-03-10 01:20:23 -06:00