Strict Sigil value reader

Status: approved contract.

Purpose and boundary

A .sgv file contains one Sigil value, read as data and never evaluated. The strict reader accepts a bounded subset of the ordinary reader's grammar. It returns plain runtime values without invoking the compiler, expander, evaluator, constructors looked up by name, or user procedures.

The convention is "if it ends in .sgv, it cannot run." The extension is a consumer contract, not an operating system access control. Consumers must use this API and must never pass its result to evaluation or treat a value as a command. They must validate their own schema, allowed keys, types, ranges, paths, and authorization rules before applying data.

Ordinary read, read-expr, and syntax readers keep their existing behavior. In particular, ordinary #{...} and #[...] currently produce constructor call lists. Strict reads produce actual dictionaries and arrays directly.

Public Sigil API

The standard library module (sigil sgv) exports:

(read-sgv-string text
  max-bytes: 1048576 max-depth: 64 max-token-bytes: 65536
  max-elements: 65536 max-integer-magnitude: 140737488355327
  max-decimal-bytes: 32)

(read-sgv-bytevector bytes
  max-bytes: 1048576 max-depth: 64 max-token-bytes: 65536
  max-elements: 65536 max-integer-magnitude: 140737488355327
  max-decimal-bytes: 32)

(read-sgv port
  max-bytes: 1048576 max-depth: 64 max-token-bytes: 65536
  max-elements: 65536 max-integer-magnitude: 140737488355327
  max-decimal-bytes: 32)

The required positional argument is respectively a string, bytevector, or open input port; the result is any?. All keywords accept exact integers. Each omitted keyword uses its default independently. Invalid arguments raise type or range errors before input is consumed. No argument is evaluated by the reader itself.

The string and bytevector entries use their explicit byte lengths. They do not copy or NUL-terminate the whole input before checking its byte limit. Both reject invalid UTF-8, including malformed strings supplied by native code. The port entry consumes from the current position through EOF and does not close the caller's port. It supports text and binary input ports, including ports backed by an already-open file descriptor. It respects already-buffered bytes and does not reopen a pathname.

read-sgv is a whole-document operation, not a stream of successive values. It buffers at most max-bytes + 1 bytes, stops on the first excess byte, and does not first call an unbounded read-all helper. It may wait for EOF on a pipe; callers control I/O deadlines and should use finite input sources. Syntax failure may leave a port consumed through EOF; byte-limit failure consumes at most the allowed bytes plus one. There is no rollback promise.

The module is a thin wrapper over runtime primitives. Its transitive runtime closure must contain neither compiler nor expander. An input list such as (delete-file "example") is simply a list containing a symbol and a string. The reader has no route from that list to procedure application.

Native API

Declarations live in packages/sigil-lib/include/sigil/sigil.h:

typedef struct {
    size_t max_bytes;
    size_t max_depth;
    size_t max_token_bytes;
    size_t max_elements;
    int64_t max_integer_magnitude;
    size_t max_decimal_bytes;
} SigilSgvLimits;

typedef struct {
    SigilSgvErrorCode code;
    size_t byte_offset;
    size_t line;
    size_t column;
    char message[192];
} SigilSgvError;

SigilSgvLimits sigil_sgv_default_limits(void);
Value sigil_read_sgv_len(SigilVM *vm, const char *source, size_t length,
                        const SigilSgvLimits *limits, SigilSgvError *error);
Value sigil_read_sgv_port(SigilVM *vm, Value port,
                         const SigilSgvLimits *limits, SigilSgvError *error);

A non-NULL VM is required. NULL limits select defaults; otherwise every field is explicit. NULL error discards diagnostics. Success sets the error code to SIGIL_SGV_OK. Failure returns SIGIL_UNDEFINED, a value absent from the accepted grammar, and fills caller-owned diagnostics. No error string allocation or ownership transfer is exposed. A NULL source is valid only with length zero. Input remains owned by the caller.

Accepted grammar

The entire input must be valid UTF-8 without U+0000 or Unicode format characters (category Cf). A leading U+FEFF BOM has a distinct BOM refusal; any other raw or escaped format character has a FORMAT refusal. Its structure is:

document = trivia datum trivia EOF
datum    = list | vector | dictionary | array | string
         | symbol | keyword | boolean | integer | decimal
list     = "(" (trivia datum)* trivia ")"
         | "[" (trivia datum)* trivia "]"
vector   = "#(" (trivia datum)* trivia ")"
array    = "#[" (trivia datum)* trivia "]"
dictionary = "#{" (trivia key trivia datum)* trivia "}"
integer  = ("+" | "-")? ASCII-DIGIT+
decimal  = ("+" | "-")? ASCII-DIGIT+ "." ASCII-DIGIT+
boolean  = "#t" | "#f" | "#true" | "#false"

Boolean spellings are ASCII case-insensitive, as in the ordinary reader, and must end at a token delimiter. Partial spellings and suffixes such as #tr, #truex, and #f0 are errors even inside a container. All container delimiters must match. Lists are proper; dotted pairs are refused.

Unescaped symbols are nonempty UTF-8 tokens excluding ASCII whitespace, ASCII control characters, DEL, parentheses, brackets, braces, double quote, semicolon, apostrophe, backtick, comma, hash, vertical bar, and backslash. An unescaped token ending in : with at least one preceding character is a keyword, stored without that colon. Non-ASCII Unicode White_Space characters also require bars in symbols; they are not token separators. A bare : is a symbol. A bare . is refused. Tokens beginning with a digit, a sign followed by a digit or dot, or a dot followed by a digit are numeric candidates and must match the integer or decimal grammar in full. They never fall back to symbols. The spellings inf and nan, optionally suffixed with .0 and optionally signed, are also reserved numeric candidates and refused in every ASCII case variant. Case matching is independent of the caller's locale. Other tokens such as +, -, ..., and version-2 are symbols.

Vertical-bar escaped symbols and double-quoted strings use the existing reader's escapes: \a, \b, \t, \n, \r, \", \\, \|, and \xHEX;. Hex escapes require at least one digit and a Unicode scalar other than U+0000; surrogate code points and values above U+10FFFF are refused. Raw C0 controls in strings require an explicit escape; this includes raw ESC, tabs, and newlines. Escaped C0 controls remain allowed except U+0000. Strings also support the ordinary backslash-newline continuation. An escaped symbol ends at its closing bar and must be followed by a delimiter or EOF. Its contents are always a symbol, so |123| and |name:| do not become an integer or keyword. There is no escaped-keyword notation.

Dictionary keys are strings, symbols, keywords, booleans, or integers. Compound keys and duplicate keys are refused, avoiding ambiguous policy entries and recursive hashing of attacker-selected structures. Duplicate identity follows the runtime's dictionary key equality. Dictionary values can be any accepted datum. Empty lists, vectors, arrays, dictionaries, strings, and escaped symbols are valid.

Bounded base-ten integers and plain decimals are accepted. Exponents, rationals, complex numbers, infinities, NaNs, exactness prefixes, and every radix prefix are refused. Integer accumulation checks the configured magnitude before multiplication or addition, never creates a bignum, and never invokes general numeric conversion. Leading zeros are allowed but still consume token and byte budgets. Decimals require digits on both sides of the point: .5, 1., and -.5 are errors. Their complete spelling, including sign and point, has an independent small byte limit as well as the token limit. Conversion uses fixed 256-bit integer numerator and denominator storage, extracts 53 significant binary bits with an exact remainder, and rounds ties to even before assembling the binary64 bits. It allocates no locale, uses no floating arithmetic or floating-environment API, and neither depends on nor changes caller locale/rounding state. The result is a finite IEEE binary64 inexact value. For example, 0.85 denotes the nearest representable value (approximately 0.84999999999999998), and 1.4 approximately 1.3999999999999999. Negative zero is preserved. Decimals do not use the integer magnitude limit; their spelling cap (hard maximum 64 bytes) bounds both conversion work and finite magnitude. Exact fractional quantities should use schema-defined strings or scaled integers. Tests cover positive/negative values, rounding, negative zero, the byte boundary and one past it, and missing point digits.

Comments

ASCII space, tab, LF, and CRLF are trivia. A lone CR, form feed, or vertical tab is refused anywhere in the raw input. Explicit string/symbol escapes can still denote those controls. Policy character properties are pinned to Unicode 17.0.0 General_Category and White_Space, independent of the host C locale. Semicolon line comments, nested #| ... |# block comments, and #; datum comments are accepted before, inside, and after the one retained datum. An unterminated block comment or missing discarded datum is an error. ;;; is an ordinary line comment, without docstring accumulation.

Datum comments discard exactly one datum parsed under the same grammar and limits. Forbidden syntax cannot be hidden inside one. Discarded values count toward the total element budget. Each nested datum-comment payload and each nested block comment consumes depth budget; comments cannot bypass the recursion limit. A commented-out second datum is permitted; a second retained datum is not. A comment-only document has no datum and is refused.

Limits and accounting

Limits are per call and inclusive. They cannot disable checks.

Keyword / C fieldDefaultValid range
max-bytes: / max_bytes1,048,5761 through 16,777,216
max-depth: / max_depth641 through 128
max-token-bytes: / max_token_bytes65,5361 through 1,048,576
max-elements: / max_elements65,5361 through 1,048,576
max-integer-magnitude: / max_integer_magnitude140,737,488,355,3270 through 140,737,488,355,327
max-decimal-bytes: / max_decimal_bytes321 through 64

The byte budget includes all input, including comments, whitespace, and trailing bytes. UTF-8 is validated before allocating parsed values. For known-length input above the limit, size refusal precedes scanning.

The root datum has depth one. A child datum has its parent's depth plus one, including an atom inside a container. Thus () has depth one and (0) has depth two. Comment parsing shares the active recursion budget; it does not reset it. The hard cap is well below MAX_READER_DEPTH and the ordinary reader's parenthesis-location capacity.

Each datum, including a container or an empty list, counts as one element. The root counts. Each dictionary key and value counts separately. Discarded datums and their descendants count identically. Trivia does not count as an element. Counting happens before parsing or allocating the next datum.

The token budget bounds both the source spelling of each integer, decimal, string, symbol, and keyword, excluding surrounding quotes/bars and a keyword's final colon, and its decoded UTF-8 content. Escape spelling counts even when it decodes to fewer bytes. Checks precede token-buffer growth and interning. The integer magnitude bound is symmetric: -M <= n <= M.

Strict containers use this total element budget rather than the ordinary reader's fixed per-container limits. Dictionary and array creation uses the runtime's direct allocation APIs, with all intermediate values rooted. All temporary storage and roots are released on both success and failure. Strict aggregate builders keep a bounded number of roots per nesting level, using rooted managed storage or a rooted list while collecting elements. Finalization writes list, vector, and still-private array elements in place with write barriers. Persistent array updates must not be used here: they copy the entire buffer per element. Dictionary rebuilding uses bounded HAMT paths rather than copying the entire dictionary per entry. They must not push one VM-stack root per element: the configured element cap can exceed the VM stack's capacity. Exercise large containers and reads with existing VM stack occupancy under GC stress. Rejected documents intern no names. During parsing, symbol and keyword spellings are stored as rooted strings inside private tagged placeholders. These placeholders are distinguishable from every accepted datum and are never returned to the caller. Temporary dictionaries use their structural equality to reject duplicate names without interning. After the complete document, including discarded datums and trailing trivia, passes validation, a bounded-depth traversal replaces retained placeholders with interned symbols/keywords and rebuilds dictionary keys. Discarded names are never interned. No parsing or schema rejection remains after interning begins; allocation failures during this final construction follow runtime allocation failure behavior. Repeated distinct rejected documents must leave both the symbol and keyword table counts unchanged, including under GC stress.

Error contract

The first detected failure wins. Offsets are zero-based bytes from the start of this read; line and column are one-based and columns count Unicode scalars. Invalid UTF-8 points at its first offending sequence. Unexpected EOF points one byte past the final byte. Limit diagnostics include the limit name, attempted count or magnitude, and configured maximum.

Scheme entry points raise an ordinary catchable syntax error for invalid documents, with a bounded message containing reason code, offset, line, and column. Reasons use the code names below, not enum ordinals. The native sigil_sgv_error_name function returns the same uppercase names. Invalid limits use range errors; port failures use I/O errors. No input token, source excerpt, or caller-controlled filename is included. The C enum uses the following suffixes after SIGIL_SGV_:

CodeRefused input or failure
ARGUMENTInvalid limits, VM, buffer, or port arguments
BYTES, DEPTH, TOKEN, ELEMENTS, INTEGER, DECIMALCorresponding configured limit exceeded
UTF8, NULMalformed UTF-8 or raw/escaped U+0000
BOM, FORMATLeading BOM or any other raw/escaped Unicode format character
WHITESPACELone CR, raw VT/FF, or non-ASCII whitespace in an unquoted symbol
CONTROLRaw C0 control in a string; use an explicit escape
EMPTY, TRAILINGMissing datum or bytes after the retained datum and trivia
LABELAny #N= or #N#, including shared and cyclic forms
QUOTEQuote, quasiquote, unquote, splicing, and their hash syntax variants
NUMBERLeading/trailing decimal points, exponents, nonfinite values, rational/complex syntax, radix/exactness prefixes, malformed numbers
CHARACTER, BYTEVECTOR#\... and #u8(...) literals
DOTTEDDotted lists
KEY, DUPLICATECompound dictionary key or repeated key
COMMENTUnterminated block comment or missing datum-comment payload
ESCAPEInvalid string or symbol escape, including surrogate escapes
SYNTAXMismatched delimiters, malformed booleans, incomplete containers, invalid tokens, all other unsupported hash syntax, directives, and shebangs
IO, MEMORYInput failure or allocation failure

Malformed syntax after a valid retained datum is TRAILING, without parsing or allocating another retained value. Invalid UTF-8 or NUL anywhere is still rejected during preflight. Ordinary VM-wide fatal resource failures retain their existing behavior; configured document limits are recoverable errors.

Implementation and verification

Add strict state to the existing Reader in reader.c and initialize it only in the new entry point. Reuse character I/O, token buffers, string and symbol parsing, container parsers, and error positioning. Strict checks run before forbidden dispatch, numeric expansion, buffer growth, or allocation. Do not implement an independent parser or evaluate constructor expressions. The port adapter belongs alongside the existing runtime port operations.

Native and Sigil tests cover each accepted type, each refusal code, exact boundaries and one past every limit, discarded inputs, dictionary ambiguity, boolean boundaries, every UTF-8 failure class, raw and escaped NUL, and deep mixed containers. Existing ordinary reader tests establish compatibility. A bundled program importing (sigil sgv) exercises strings, bytes, and ports while its closure is checked for compiler and expander absence.

A fixed-seed Sigil fuzz test mutates a checked-in corpus of valid and invalid documents, caps input and parser budgets, and exercises byte and port reads. Structural mutations cover nested containers, large collections, duplicate keys, and malformed delimiters. Distinct rejected names must leave both intern-table sizes unchanged, measured with gc-stats's interned-symbols and interned-keywords counters. The default is 3,000 iterations; SIGIL_SGV_FUZZ_ITERATIONS=100000 enables a longer opt-in run (valid range 1 through 1,000,000). A process timeout and memory limit bound the fuzz run itself. Run native reader coverage with address/undefined-behavior sanitizers when available. Run the complete suite before finalizing implementation commits.