dsh-lit-filecheck

A DSH plugin that brings the lit + FileCheck test loop into a session. Three tools:

Tool	What it does
lit_run	Runs llvm-lit in a suite root and reports total/passed/failed/skipped, splitting each failure block into per-CHECK diagnostics: which line holds the CHECK, which line of the actual IR broke it
lit_rerun_failed	Re-runs only the tests that failed in this plugin instance's previous lit_run (no full-suite rescan)
filecheck	Runs FileCheck once on an explicit pair of "test file with CHECK lines + actual IR file" and reports match/mismatch with the offending lines

Plugin id lit-filecheck, npm package dsh-lit-filecheck@0.1.0, ESM, no background jobs and no persisted state (the rerun target lives in plugin-instance memory only).

Install

As a bundle, cordis.patch.yml inserts the plugin entry:

- insert:
    - id: lit-filecheck
      name: 'dsh-lit-filecheck'
      config:
        timeoutMs: 120000
        maxOutputBytes: 1048576
        maxChecks: 50
        logDir: .lit-logs
        requireFileCheck: true

Configuration

Key	Default	Meaning
toolchainPath	—	Path to toolchain.json. Lookup order: this key > MLIR_TOOLCHAIN_JSON > <session cwd>/mlir-lang/toolchain.json
litCommand	—	Full argv array for lit, highest priority, e.g. ["D:/Code/python/python.exe","D:/Code/llvm-build/bin/llvm-lit.py"]. No probing when set
litPath	—	Path to llvm-lit.py or an executable. A .py file gets pythonPath prepended; .cmd / .bat are rejected (cmd forwarding mangles arguments on Windows)
pythonPath	—	Interpreter for the .py lit; otherwise D:/Code/python/python.exe and PATH are tried
filecheckPath	—	FileCheck.exe; otherwise searched in llvmRoot/bin, D:/Code/llvm-build/bin, D:/Code/llvm/bin, PATH
timeoutMs	120000	Total budget for one lit run. Recommend >= 1000; the floor is 1 only so the timeout path stays acceptance-testable
maxOutputBytes	1048576	Per-stream retention cap for stdout and stderr (truncated + warned when exceeded)
maxChecks	50	Maximum CHECK diagnostics parsed per run
logDir	.lit-logs	Log directory, relative to the session cwd
requireFileCheck	true	true = tool-missing when FileCheck is absent; false = warn and keep running lit (failure blocks then usually carry no CHECK diagnostics)

Out-of-range numbers never throw: one warning plus fallback to the default.

Tool reference

lit_run

Parameters: suiteDir (required), filter, jobs (1–64), timeoutMs.

Command (cwd = suiteDir):

<litCommand...> -v --no-progress-bar --show-unsupported [--filter=<filter>] [-j <jobs>] .

Result fields:

ok, total, passed, failed, skipped,
failures: [{ test, status, reason, runLine,
             checks: [{ checkFile, line, column, prefix, pattern,
                        expected, matched, actualLine, actualText, message }],
             checksTruncated }],
logPath, truncated, timedOut, durationMs,
preflight: { lit: { ok, command }, filecheck: { ok, path } },
warnings, error

ok is true only when preflight passed, exit code is 0, failed === 0, and the run did not time out. A failing run still returns complete failures / checks / logPath — a red run is the informative one.

reason ∈ filecheck-mismatch (CHECK diagnostics present), command-failed, timeout, no-filecheck-diagnostic, unknown.

lit_rerun_failed

No parameters. The command carries only the previously failed paths and no --filter:

<litCommand...> -v --no-progress-bar <failed test paths...>

• Last run had no failures → ok:true, noop:true, total:0, lit is not started;
• Never ran lit_run → no-previous-run;
• A failed test file was deleted → listed in missingTests with a warning; all gone → path-not-found;
• Same result shape as lit_run plus onlyFailed / noop / source{suiteDir,startedAt} / missingTests.

filecheck

Parameters: testFile, irFile (both required, existing regular files), extraArgs, timeoutMs (default 30000).

<FileCheck.exe> --color=false <extraArgs...> <testFile> --input-file=<irFile>

extraArgs containing --input-file or - is rejected (invalid-extra-args) because it would override irFile.

Returns ok, mismatches[], stdout, stderr, truncated, logPath, durationMs, error, hint; only exit code 0 gives ok:true. No CHECK directives at all is no-check-lines, never a guessed "pass".

Shape of a failure diagnostic

<check-file>:<line>:<column>: error: CHECK: expected string not found in input

The plugin reads line <line> of the test file as pattern and the text after CHECK: as expected; actualLine / actualText come from the first available note: note: possible intended match located ... first, then note: scanning from here. When neither note exists they stay null — nothing is invented.

Crash dumps (PLEASE submit a bug report / Stack dump) never masquerade as diagnostics; they go to the log and warnings only.

Error codes

invalid-input, path-not-found, not-a-directory, tool-missing, spawn-failed, timeout, lit-internal-error, no-previous-run, invalid-extra-args, no-check-lines, log-write-failed, parse-failed.

Failures are returned as ok:false + error{code,message} + an actionable Chinese hint; nothing throws.

Logs

The full output is written to <logDir>/lit-<yyyyMMdd-HHmmss>-<rand>.log with argv, cwd, start/end time, exit code, counts, failure details, and complete stdout/stderr. The stdout/stderr in the tool result are clipped to maxOutputBytes (head and tail kept). If the log cannot be written, logPath:null plus a log-write-failed warning — the result itself is still returned.

Tests

npm test        # 68 cases: 66 pass, 2 real-lit cases skipped by default
npm run smoke   # smoke: really runs the plain suite and prints all three results

Real lit integration tests are skipped by default; run them with:

$env:LIT_FILECHECK_REAL='1'; node --test --test-isolation=none test/integration.test.mjs

Verification status (this machine, 2026-11-05, LLVM 23.1.4):

• lit side verified for real: test/fixtures/plain/ has 5 tests, 3 pass and 2 fail; lit_run reports total=5/passed=3/failed=2 and lit_rerun_failed runs exactly those 2. Failure-block parsing is calibrated against captured real output in test/golden/plain-lit-output.txt.
• FileCheck side not verified against a real binary: this machine's Windows LLVM 23.1.4 package ships no FileCheck.exe, and there is no MSVC toolchain to build one. CHECK-level parsing (line → the CHECK line, actualLine → the actual IR line) is covered by hand-written test/golden/filecheck-stderr.txt and test/golden/lit-filecheck-output.txt following spec §9.3. Re-run a real CHECK-level acceptance once FileCheck is available.

Known limitations

• lit.cfg.py must be pure ASCII: on Windows lit reads config files with the platform locale (GBK) and dies on non-ASCII comments with UnicodeDecodeError.
• A .cmd / .bat llvm-lit is rejected on purpose; use llvm-lit.py + pythonPath.
• Under the DSH restricted sandbox, piped Node spawns are denied (spawn EPERM) and lit's multiprocessing pool hits PermissionError; the tools then return spawn-failed. Host-side runs work normally.