Scripting survey
This is a measured assessment of the scripting surface on November 2, 2026. The startup and output profile gives the newer, timestamped breakdown and first-byte measurements. Sigil can already express the six paired examples below. It loses badly on short-process startup and bulk regular-expression scanning in the measured release. The measurements do not justify a general claim that it is faster or that every existing script can be converted unchanged.
Population and method
The inventory driver is scripting-survey.sgl. It reads source files without executing them. It excludes .git, bare Git storage, .sigil dependency caches, _archive, task-*, build, and node_modules directories. It includes hidden source files, vendored sources, extensionless commands, JavaScript support modules, and direct Sigil test programs. Declarative package and project manifests and library declarations are excluded from the Sigil count. File counts are not command counts.
Across the inspected sibling checkouts and the separate site checkout:
| Language | Files |
|---|---|
| Shell | 1,212 |
| Python | 217 |
| JavaScript | 1,306 |
| Sigil direct programs | 3,108 |
| Total | 5,843 |
All 5,843 files have a primary category, additional matching categories, and an explicit exception field. There were zero read failures. The complete local inventory can be regenerated with the driver; the public repository subset is scripting-survey-data.json. The aggregate counts are in scripting-survey-summary.json. Alternate checkouts with names outside the exclusions are counted separately. Third-party code is retained in the census and identified separately.
There are 2,735 non-Sigil files. Of these, 2,534 are outside identified vendor and downloaded-tool trees; 2,198 remain after automatic and hash-verified exception classifications. These are candidates for assessment, not proven conversions. Browser assets, imported support modules, and independent commands still require different treatment before setting a conversion target.
The monorepo subset contains 237 shell, 49 Python, 602 JavaScript, and 836 direct Sigil programs. Loader signatures identify 162 monorepo JavaScript files, including 130 of its 505 top-level JavaScript test files. Across all checkouts there are 317 direct matches. This explicitly checks WebAssembly construction, compilation, and named loader calls; a direct-wasm filename alone grants no exception. Comment-only matches are excluded. Indirect loader use needs call-site review. The source sample found three missed runtime adapters or indirect execution fixtures. Hash-verified corrections across duplicate checkouts add five files: the refined census contains 322 WebAssembly exceptions, 164 in the monorepo, including 131 top-level tests. This remains a lower-bound classification rather than a complete static call graph. JavaScript that loads or executes WebAssembly is counted separately and is not rewrite work. Pure text/JSON validation stays eligible.
Bootstrap paths identify six provisional exception-1 files; host setup paths identify eleven provisional exception-2 files. No harness is exempted merely because it tests a compiler: a pinned prior interpreter can often run it. No temporary-gap exception is invented by the classifier.
Category assessment
Three non-Sigil programs per primary category were initially sampled: 42 in total. Each excerpt includes up to 110 source lines; longer files require follow-up before a whole-program conversion claim. The sampling driver retains the selected paths and excerpts alongside the full local inventory. Additional monorepo examples were read in full for the paired conversions. Categories are ordered signatures, not a parser: a signal mentioned in a comment can affect the primary category. Additional categories remain available per file.
A second, independently selected sample draws three eligible first-party files per category with seed 20261102. All 42 were inspected, including the remaining source of the four programs longer than the bounded excerpts. Source judgments and file hashes are retained locally; the public aggregate is scripting-coverage-summary.json.
| Primary category | Non-Sigil files | Sample | Current surface and limits |
|---|---|---|---|
| Process control and pipelines | 69 | 3 | Structured argv and per-stage outcomes; pipeline deadlines are unsupported. |
| Captures and exit status | 68 | 3 | $?, causes, stdout, stderr, timeout evidence; $ buffers output. |
| Text, lines, regex | 90 | 3 | Line ports and RE2-style regex; measured regex throughput is poor. |
| JSON and structured data | 278 | 3 | Native encoding and decoding; arrays and keyword object keys require learning. |
| File trees, modes, links | 174 | 3 | Traversal, metadata, link operations; symlink policy must be explicit. |
| Hashes and checksums | 362 | 3 | Checksum tools work through argv; native file hashing exists in current source. |
| HTTP | 15 | 3 | Curl outcomes and HTTP libraries; authentication and service fixtures stay explicit. |
| Polling and timeouts | 498 | 3 | wait-until and command deadlines exist; retry policy repeats across scripts. |
| Parallelism and background jobs | 96 | 3 | spawn, process handles, and (sigil threads) bounded parallel-map exist. |
| Arguments and help | 137 | 3 | arg, script-args, and (sigil args) exist. |
| Temporary files and cleanup | 45 | 3 | Scoped temporary trees and dynamic-wind; immediate exit cannot unwind. |
| Signals | 762 | 3 | (sigil signals) has callbacks and explicit dispatch; synchronous blocking needs care. |
| TTY and PTY | 26 | 3 | Terminal and PTY primitives exist; child input is not automatically terminal input. |
| Other | 115 | 3 | Includes tool configuration and imported modules; not all are executable commands. |
Gaps and existing alternatives
The measured priority is startup, then printing, as detailed in the separate profile. For functional blockers, the reviewed 42-file sample finds zero proven absolute blockers for each family below, so their observed blocker counts are tied at zero. Three boundary cases remain unknown. This does not establish that the unreviewed population has no blockers.
The following search order ranks signature matches among the 2,198 eligible first-party non-Sigil files, not confirmed blockers. Matches overlap. Binary matches often mean ordinary UTF-8 file reads; live-tee matches include inherited stdio and a completed one-line report. Neither proves the stronger missing API.
| Search order | Need | Candidate files | Evidence and next action |
|---|---|---|---|
| 1 | Binary capture that preserves bytes by default | 722 | Executed: three non-UTF-8 bytes become seven with default capture. ansi: #t preserves all three; native spawn file sinks also work. Prefer an explicit byte API and clearer redirect contract. |
| 2 | Live tee with retained stdout/stderr and bounded termination | 238 | Source: checked commands buffer; pipeline pumping retains output and lacks deadlines. A streaming capture API needs process-layer design. Ordinary tee on already-produced text is not blocked. |
| 3 | A deadline for an entire pipeline | 64 | Source: pipeline option validation rejects timeout:. Bound every stage and terminate owned descendants without losing stage outcomes. |
| 4 | Retry with capped backoff | 52 | Added (sigil shell patterns) with bounded attempts, capped delay, final outcome identity, and exception propagation. Twelve focused tests and CLI/prelude integration pass. This removes repeated loops rather than unlocking an otherwise impossible program. |
Two tempting gap claims were rejected after API inspection: 91 signal-cleanup matches do not establish missing signal support, and 15 parallel-map matches do not establish missing bounded parallelism. Both already have library APIs. wait-until, run-with-timeout preserving both streams, temporary cleanup, and command-specific environments also already exist. A checksum sidecar is a useful recipe around hashing, rather than evidence that checksums are missing.
Startup and regex throughput are measured performance gaps. The counts above cannot predict how many scripts their improvement would unblock: that depends on each script's latency and throughput requirements.
Scorecard
The driver scripting-scorecard.sgl uses current-jiffy to time synchronous invocations. Linux 6.17.12, nice 10, CPUs 10 and 11; released Sigil 0.24.1 (9e8f48cd). The source checkout is based on 391234d347dd3c287aa1deec7cc9c88856236eaf. These are different identities: the released executable interprets the measurement tools independently of the checkout. A shared development executable was inspected but not used as a substitute for a verified build of that checkout.
| Empty-file interpreter | First observed launch, ms | Warm median, ms (15 launches) |
|---|---|---|
| Sigil | 600.941 | 294.887 |
| Bash | 3.900 | 3.863 |
| Python | 26.758 | 17.235 |
| Node | 30.772 | 33.274 |
All launches include the same driver capture path. First launch is not an OS-cold measurement: page caches were not evicted. Physical cold startup remains unmeasured; fresh interpreter processes and fresh language caches do not establish a cold filesystem. No cache-dropping operation was performed. The workstation was shared, and the inventory overlapped these initial measurements. The samples are observations at the stated resource limit, not isolated CPU microbenchmarks. The newer profile interleaves interpreters and supplies measured startup phases, printing body time, and first-byte latency.
The executable-cache probe adds a narrower cold measurement without host-wide cache eviction. It copies each executable without reflinks, syncs only that file, applies per-file fadvise, and checks zero resident executable pages with fincore before every cold run. Sigil's embedded library bundle is part of that executable. Seven paired launches per interpreter gave:
| Interpreter | Executable-cache cold, ms | Same-copy warm, ms | Original-path warm control, ms |
|---|---|---|---|
| Sigil | 588.498 | 389.664 | 408.917 |
| Bash | 22.583 | 5.760 | 6.478 |
| Node | 266.515 | 33.700 | 35.276 |
| Python | 22.515 | 19.779 | 21.877 |
Shared libraries, the source file, and other OS caches remain uncontrolled. This is executable-cache cold, not a fully cold operating system. Host load was about 4.9 during this later experiment; the original-path control also slowed. Do not subtract its timings from the earlier quiet phase table or attribute the entire difference to cache state.
A batch of 100 true spawn/capture operations took 110.881 ms median over seven batches: 1.109 ms per operation including driver overhead. This measures Sigil's capture path, not an isolated kernel spawn cost or a cross-language spawn ranking.
For text throughput, the driver repeats the actual release notes four times: 1,284,532 UTF-8 bytes and 320 release headings. Seven scans with a compiled (?m)^## pattern took 5.821 s median; grep -c '^## ' over the same file took 6.391 ms including process launch. Both counts were checked. Throughput is about 0.221 MB/s versus 201 MB/s. Sigil can invoke grep with structured argv, but its native regex loop loses by about 911 times on this particular scan.
Paired rewrites
Six candidates live in scripts/rewrites. Existing commands remain authoritative. The comparison driver runs each original and candidate against independently created identical fixture trees. It compares stdout, stderr, exit status, file hashes, modes, directory entries, and symlink targets. Only the temporary root and the invoked script name are normalized. It excludes the identical script copies themselves from filesystem output. No network access or publication is involved.
All 18 cases passed, including failures. Four separate negative controls also reject a change confined to stdout, stderr, exit status, or a written file. The recorded results retain each comparison.
| Original | Cases | Physical lines, original / Sigil | Code lines, original / Sigil |
|---|---|---|---|
extract-release-body.sh | Present, absent, usage failure | 21 / 22 | 15 / 20 |
generate-release-metadata.sh | Two platforms, no binaries, invalid allocator | 125 / 49 | 65 / 47 |
check-zig-version-pin | Consistent, build drift, empty pin | 59 / 31 | 40 / 29 |
ci-relink-workspace-deps.sh | Fresh links, existing file, blocked parent | 26 / 16 | 15 / 14 |
check-package-boundaries.sh | Complete, missing classification, invalid row | 70 / 50 | 56 / 48 |
test/integration/test-package-boundary-classification.sh | Real checker, failing checker, false success | 35 / 35 | 28 / 33 |
Code-line counting excludes blank lines, full-line comments, and shebangs; it does not pretend that a line of awk and a Sigil form have equal complexity. The smallest text filter gets longer. The metadata conversion gets shorter while using an encoder rather than assembling unescaped JSON.
These fixtures are bounded evidence, not exhaustive equivalence. System-tool diagnostics, alternate platforms, malformed files, symlink cycles, and signal delivery still need targeted cases before replacing a production command.
Correctness and migration traps
scripting-correctness.sgl runs five contrasts: an unset variable, a set -e hole in a function used as an OR-list condition, a failing pipeline producer, unquoted word splitting, and the real metadata script's unescaped JSON. All five demonstrate the expected difference. Bash's set -u, pipefail, and correct quoting handle three of these cases; the comparison is about defaults and maintenance traps, not impossibility.
The driver also reproduces two Sigil traps. Bare numeric -0 becomes 0, while quoted "-0" survives; no signals are sent by that probe. Default ANSI removal is unsafe for arbitrary binary stdout, while ansi: #t preserves raw bytes. Other migration rules include comma interpolation, whole-argument quoting, different in: meanings for synchronous commands and spawn, and the need to inspect an ignored $? outcome. The paired examples have one confirmed original JSON-construction defect and zero shell word-splitting sites in the Sigil candidates; the other five originals have zero demonstrated quoting or error-handling defects on these fixtures. That gives one demonstrated defect in the original set and zero in the candidates, rather than a guessed count of every possible trap. The five independent correctness contrasts above are separate demonstrations, not five defects in each rewritten script.
Coverage limits
Verified paired coverage is six scripts out of 2,198 eligible first-party foreign files, or 0.273%, on the recorded fixtures. This is a lower-bound verification share, separate from source feasibility.
The seed-20261102 sample uses the original 2,203-file frame, before the five review corrections. Of 42 files, 36 appear feasible with existing APIs and external tools, three are WebAssembly runtime exceptions, and three remain unknown. Feasible means the required operations have identified equivalents; it permits converting data-helper dependencies and retaining an allowed Node WebAssembly driver. It does not promise unchanged diagnostics or performance.
Weighting each sampled file by its category population divided by three gives 94.4% source-feasible, 3.5% unknown, and 2.1% missed exceptions in that frame. The conservative 95% weighted Hoeffding margin is 32.3 percentage points, so the feasible-share interval is roughly 62.1–100%. This wide bound does not account for mistaken source judgments, duplicate checkout bias, or excluded files. The sample is too small to advertise a precise population percentage.
After retry, streaming, and pipeline helper improvements, this sample's source feasibility estimate remains 94.4%: no sampled conversion was proved to depend on those missing helpers. Their benefit is a cleaner and more robust implementation, rather than an invented increase from overlapping signatures. An exact whole-population blocker ranking remains unmeasured.
Reproduce
Use a pinned released interpreter, an ordinary disposable working directory, and explicit source paths. Each driver prints or writes its evidence:
sigil scripts/scripting-survey.sgl SOURCE-ROOT SITE-ROOT OUTPUT
sigil scripts/review-scripting-survey.sgl OUTPUT SOURCE-ROOT SITE-ROOT OUTPUT
sigil scripts/sample-scripting-coverage.sgl OUTPUT SOURCE-ROOT SITE-ROOT REVIEW
sigil scripts/assess-scripting-coverage.sgl REVIEW docs/internals
sigil scripts/apply-scripting-reviews.sgl OUTPUT SOURCE-ROOT SITE-ROOT
sigil scripts/summarize-scripting-survey.sgl OUTPUT REPOSITORY docs/internals
sigil scripts/scripting-scorecard.sgl OUTPUT PINNED-SIGIL RELEASES.md
sigil scripts/compare-scripting-rewrites.sgl REPOSITORY PINNED-SIGIL OUTPUT
sigil scripts/scripting-correctness.sgl REPOSITORY PINNED-SIGIL OUTPUT
sigil scripts/profile-executable-cache.sgl PINNED-SIGIL FRESH-OUTPUTThe complete inventory is local because sibling paths and source excerpts can describe unrelated projects. Only aggregate counts and this repository's per-file records are included here. No unrelated source is copied into this repository.
The assessment refuses a changed sample or changed source hashes; new source needs a new review. The public aggregate alone is not a substitute for those judgments. The retry module passed the full development test suite with effect proofs and the documented registry/signing fixtures: 4,196 passed, two skipped.