(sigil format tokenize)
(sigil format tokenize) - Positioned Sigil tokenizer (io-free)
The positioned tokenizer extracted from (sigil format) so downstream consumers (e.g. Slate's Tier-B syntax highlighting, compiled to WASM) can import JUST the tokenizer without pulling in the formatter or its dependencies (sigil-json, io/ports/fs).
This library is deliberately io-free: it depends only on (sigil core). tokenize takes a source string + filename and returns a (cons chars tokens) result; a token is a raw 6-slot vector.
Example: (let* ((r (tokenize "(+ 1 2)" "test.sgl")) (chars (tokenize-result-chars r)) (toks (tokenize-result-tokens r))) (token-type (car toks)) ; => lparen (token-value (car toks) chars)) ; => "("
Exports
token?procedureCheck if value is a token
token-typeprocedureToken type accessor
token-startprocedureToken start position accessor
token-endprocedureToken end position accessor
token-lineprocedureToken line accessor
token-columnprocedureToken column accessor
token-indentprocedureToken indent accessor
token-valueprocedureExtract the text value of a token from the source chars vector
tokenize-result-charsprocedureGet the chars vector from a tokenize result
tokenize-result-tokensprocedureGet the token list from a tokenize result
char-digit?procedureCheck if character is a digit
delimiter?procedureCheck if character is a delimiter
looks-like-number?procedureCheck if string looks like a number
tokenizeprocedureTokenize entire source into a list of tokens Returns (cons chars tokens) where chars is the source as a vector