sigildocs

Test execution evidence

The experimental evidence mode qualifies one static, in-process sigil test --sgl invocation. It combines the runtime's acquired-input journal with the runner's selection and completion records. Ordinary tests and existing report fields remain available without enabling evidence.

Use the external observer with an instrumented executable:

python3 scripts/observe-test-evidence.py \
  --evidence-dir /tmp/my-new-test-observation \
  --expect-file /absolute/project/test/test-subject.sgl \
  --expect-input /absolute/lib/example/subject.sgb EXPECTED_SHA256 \
  -- /absolute/sigil --no-auto-compile -L /absolute/lib \
  test --sgl --report /tmp/test-report.json \
  /absolute/project/test/test-subject.sgl

The evidence directory must be new. The observer records the invoked executable hash, arguments, working directory, process status, signal and timeout, stdout, stderr, journal hash, and reconciliation result in observed.json. A timeout kills the process group created for this invocation. Exit codes are 0 for a complete passing run, 1 for a complete failing run or discovery refusal, and 2 for incomplete evidence. Missing records, truncated JSON, inconsistent counts, unsupported execution modes, signals, and timeouts cannot qualify as a pass.

Repeat --expect-file to supply the independently expected selected population. Repeat --expect-input to require particular consumed module bytes in the runner's VM, with acquisition completed before its run terminal. Without these expectations, the observer checks the producer's internal accounting; it does not establish that discovery or module resolution chose the intended test suite or implementation. Compute expected digests before execution. Path labels identify requests, while the runtime hashes the buffer it actually acquired; a module can replace its pathname during initialization. A matching input record alone does not establish every later binding or retained closure's identity. See runtime evidence for module generations, unsupported identities, process ownership, and precise runtime claims.

The observer inherits the caller's module and cache environment. Use owned fixtures and explicit library paths when validating changes. It does not infer that an installed library matches nearby source files, nor does it certify children or arbitrary native/custom harnesses. Watch mode, implicit mixed native runs, and custom runtime harnesses are explicitly unsupported. Start a new process for each qualified run.

The runner writes one run-plan before discovery refusals, then file-start and file-end around each selected file, and finally run-terminal. Their payloads are JSON strings in the runtime envelope's payload_json field. Every payload includes invocation_id; the envelope supplies run, process, VM and sequence identity.

  • The plan records the available discovered, selected and backend-excluded paths, selection scope, filter, fail-fast, strictness and containment support. Ignored files are explicitly not-enumerated.
  • File terminals record the outcome, registered and selected test names, completed test results, and passed, failed, skipped and total counts. Names retain registration order and may repeat. Load failures include a synthetic failing result.
  • The run terminal records selected, started, terminal and unstarted files. requested-stop preserves selected work omitted after fail-fast; discovery-rejected completes an early refusal; abandoned cannot qualify. A file with no selected tests can complete with zero tests.

When evidence is enabled, --report adds an evidence reference containing schema, run ID, journal path, VM ID, invocation ID and completion claim. Existing summary, failures, stdout fields and the Report: output line are retained. A report or child terminal alone cannot prove process completion; the external observer supplies that check. Early discovery refusals need not create a report.

Quiet and report modes now apply --filter before execution and honor --fail-fast between tests and files, including tests registered before a partial load failure in verbose mode. Completed skipped and pending tests count in skipped and total, with the same report field names as before.

Run the real CLI regression controls explicitly against the executable being qualified:

python3 test/integration/test-runner-evidence.py /absolute/instrumented/sigil

The controls retain their owned raw evidence directory and include consumed source/bytecode self-overwrite, early refusals, filtering, fail-fast, a trapped exit, an externally observed hard kill, and altered-journal rejection.