Language Reader Protocol v1

Status: frozen implementation contract. The compiler implementation is intentionally deferred to the language-reader L1–L4 phases.

A language reader is an ordinary installed Sigil module that turns a source file's notation into ordinary located Sigil syntax. The normal expander, compiler, module system, dependency graph, artifact store, cache, and backends remain authoritative. There is no reader registry and no parallel compilation or cache path.

Source header

The input is UTF-8 bytes. An optional UTF-8 BOM may precede the header. An optional #! shebang may be physical line 1, in which case the header must be the next physical line. Otherwise the header must be physical line 1. Its grammar is:

#lang HSPACE+ LANGUAGE-ID HSPACE* (NEWLINE | EOF)

LANGUAGE-ID is one or more slash-separated ASCII segments. Each segment matches [a-z0-9]+(-[a-z0-9]+)*: lowercase letters and decimal digits, with single internal hyphens. Empty segments, leading/trailing/repeated hyphens, uppercase or non-ASCII letters, ., .., absolute paths, backslashes, NUL, comments, and trailing tokens are invalid. Header arguments do not exist in v1. There is no special #lang sigil; ordinary Sigil omits the header.

The candidate header line is scanned with a hard 4096-byte bound, including its newline. The scanner reads no more than 4096 bytes from the candidate line's start. A longer line fails rather than being truncated or interpreted as ordinary Sigil. Without a #lang candidate, the existing standard whole-buffer reader is used unchanged, including existing shebang behavior.

The scanner examines only the candidate line and, solely to diagnose displacement, one following physical line, all within the same 4096-byte prefix from the candidate line's start. If the candidate does not start #lang but that immediately following line does, it reports #lang must be first line after optional shebang. It never searches the rest of the file. If neither line starts #lang, it returns “no language header” and passes the original complete buffer unchanged to the standard reader. Thus ordinary no-header files do not acquire alternate-reader parsing or preprocessing.

#lang sigil/doc maps mechanically to (sigil doc reader) and #lang acme/config to (acme config reader). The compiler has no alias table. The canonical extension for Sigil documents is .sgd by convention, but extensions are semantically irrelevant: the validated #lang header alone selects a reader.

Invocation

The reader module is written using standard Sigil syntax in v1 and must export the required entry point:

(read-language-source source context) -> proper list of syntax objects

The protocol recognizes exactly this one required entry point. The module may export additional ordinary bindings; they have no reader-protocol meaning in v1.

The compiler resolves and loads the module through the same package and explicit load paths used by ordinary imports. It never installs a package, consults a registry, or uses the network. The exported binding—not an unexported implementation binding—is called under the compilation yield barrier.

source is the complete immutable source string, including BOM, shebang, and header. context is an immutable alist with exactly these initial fields:

• source-name:: the compiler's source filename;
• language-id:: the validated header identifier string;
• header-end-offset:: the byte offset immediately after the header newline, or the source byte length when the header ends at EOF.

The result must be a finite proper list. Every element must be a syntax object. EOF, undefined, bare datums, improper lists, and cyclic lists are protocol errors. The list then enters the ordinary define-library discovery, expansion, CPS conversion, optimization, and emission pipeline.

Source ownership

The original alternate-language file is authoritative. Returned syntax carries its filename, byte offset, 1-based line and column, and byte span. Offsets and spans must be within the immutable source. Displayed columns count Unicode scalar values, while byte offsets and spans count UTF-8 bytes.

Generated syntax and syntax claiming another source are valid only when they carry an explicit, bounded, acyclic generated-origin chain. The chain's outer location is the authoritative manuscript command or expression; diagnostics report it first and may show inner generated locations as context. V1 bounds the chain at 32 origin entries. A longer chain or a repeated syntax/location identity is a source violation. Generated temporary Sigil text must never become the primary author location.

L1 owns public construction and access operations for byte spans and generated-origin chains, including safe traversal. L2 validates every returned syntax tree, its spans, the 32-entry bound, acyclicity, and outer manuscript ownership before compilation.

Bootstrap, authority, and state

Reader modules are normal compiled modules executing with the same compile-time authority as macros and build scripts. V1 does not claim to sandbox them. A reader module may not itself require an alternate reader while it is being bootstrapped. The compiler reports that cycle; it does not recurse or fall back to standard syntax.

Loading and invoking a reader must restore the VM's current module, error state, GC roots, yield barrier, and compile-only/phantom-module state on success and every failure. Two files using different readers in one VM must not contaminate one another. Editors must not execute a reader merely to open a file; pure header inspection and explicit trust are a later tooling capability.

Tooling discovery and activation

(sigil language) exposes pure inspect-language-source and inspect-language-file operations. They return #f for an ordinary Sigil input or an immutable protocol-1 header value containing the validated language id, its conventional reader module, and the byte offset immediately after the header. The availability query performs ordinary module lookup without loading the reader, running its top level, invoking an auto-compile hook, installing a package, or using the network. A bundled reader may be reported as available-unidentified; because no independent artifact identity can be proven, that state is never persistently trusted.

activate-language-capabilities is the explicit authority-bearing operation. It loads the already-resolved reader and recognizes an optional exported language-capabilities procedure. The procedure returns a proper version-1 alist; the optional formatter and document-symbols values must be procedures. Additional ordinary exports and capability keys are allowed. There is no capability export by default, so a reader remains valid without editor integration.

An LSP approval lasts only for its current server session. Slate may persist approval only under the exact workspace root, validated language id, and independently resolved reader artifact identity. A reader identity change revokes approval. Neither project files nor manuscript content can grant trust. Until explicit activation, alternate-language buffers are inspected passively and never enter standard Sigil analysis or formatting as a fallback.

Compilation dependencies and caching

An alternate-language file uses the one normal Sigil compilation pipeline and cache. The conventional reader module is an ordinary compiler dependency. The protocol version, resolved reader artifact identity, and reader's normal transitive dependencies therefore participate in dependency discovery and artifact identity just like other compiler-relevant inputs.

Changing reader source or bytecode rebuilds every source it reads even if those source bytes and mtimes are unchanged. With no changes, the normal cache must hit. This must hold independently for CLI auto-compilation, bytecode package/store builds, native builds, and WASM builds. A generated-source workaround or document-specific cache does not satisfy the contract.

Diagnostics

Diagnostics have a stable class and required leading message. Implementations may append paths, identifiers, nested causes, and source excerpts deterministically.

After a syntactically valid #lang header is recognized, every resolution, loading, invocation, validation, or compilation failure is terminal. Standard-reader fallback is forbidden.

Implementation ownership

• L1 exposes only the standard-reader and located-syntax primitives required by package readers.
• L2 owns bounded header dispatch, module loading and invocation, result/location validation, state restoration, and fixture compilation.
• L3 records the reader and its transitives in the existing dependency graph and proves normal cache correctness in every backend.
• L4 inventories all whole-file entry points and closes ordinary-Sigil regressions.
• L5 adds pure, non-executing header inspection and optional trusted tooling capabilities.

The fixture contract in test/fixtures/language-readers-v1/ is the first conformance example. It is deliberately inert until L1/L2 implement the public primitives and dispatch seam.