Test Troubleshooting
What a test run says when the machine, the checkout or a test itself is at fault, and how to tell which and fix it. Each symptom below has come up before, so a session can find its cause here rather than again.
A required capability is missing
A test panicked with LNK_REQUIRE_... is set, but it lacks .... The
machine was declared to provide a capability it doesn't. Either provide
it, or leave the variable unset so its tests skip, saying why
(Skipping Tests).
lacks bubblewrap
The sandbox's tests need unprivileged user namespaces. Install
bubblewrap, and on Ubuntu let AppArmor allow the namespaces with
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0. Check
with bwrap --unshare-net --ro-bind / / true, the gate's own probe.
lacks Landlock
The kernel withholds Landlock, which no package installs: a container
gets its host kernel's. Leave LNK_REQUIRE_NATIVE unset on that
machine, or use one whose kernel has it, Linux 5.13 or newer.
If an agent's wrapped sandbox reports a Landlock version-probe failure,
its test needs gate::sandbox_with_landlock, even when bubblewrap works.
A test expecting a sandbox refusal must stop at that gate. Running with
--no-sandbox and then asserting a refusal tests a different contract.
lacks a test folder out of the temp folders
A fixture the sandbox reads is under /tmp, which the sandbox refuses
a spec to read, since anyone can write there. Run from a checkout
outside /tmp. A socket fixture uses TempDir::short_out_of_tmp, which
needs a cache folder short enough, or a short LNK_TEST_STATE_DIR
(LNK_TEST_STATE_DIR).
A program a test runs fails
A test ran a host program and it failed, or the test skipped saying the program is missing. Probe the program the way the gate does, from a folder outside the checkout, before installing anything.
An agent refuses to start from a user namespace
A command runner may put the test itself in a user namespace. Link
correctly refuses unsandboxed agents there, even when the underlying
host supports bubblewrap. Compare /proc/self/uid_map and the
bubblewrap probe in the runner's supported execution modes. Run the
tests with the required host capabilities rather than weakening Link's
machine identity checks.
A test tries to write to the real home
A failure naming the real user's Link folder, especially on a read-only
home, means a child inherited that user's settings. Start it through
env::command, or apply env::clean, with the fixture's own home.
The test must work whether the real home is writable or read-only.
attempting to make an HTTP request, but --offline was specified
A test read cargo metadata --offline, which resolves every
platform's dependencies, and a build had fetched only this machine's.
Run cargo fetch --locked once in the workspace.
Unknown binary 'rust-analyzer' in official toolchain
The code tool's test ran rust-analyzer in a sandbox, outside the
checkout and without the RUSTUP_TOOLCHAIN Cargo sets, so rustup chose
its default toolchain, which lacks it. Make the pinned toolchain the
default and give it the component, as CI does:
rustup default <pinned> and rustup component add rust-analyzer.
Rustup loses its default inside a language-server sandbox
A rustup proxy can run on the host and fail inside a sandbox that
drops RUSTUP_HOME. Select the toolchain before confinement and put
its sysroot's bin directory first on the fixture's PATH, so the
server and its compiler run directly. Grant that toolchain's folders
without forwarding the host's environment or weakening confinement.
A test skips with a browser or display installed
The browser test finds Chromium on the PATH, in CHROMIUM or under
PLAYWRIGHT_BROWSERS_PATH, and the desktop tests need Xvfb on the
PATH. Set CHROMIUM to the browser's program when it is installed
somewhere else, and install xvfb for the desktop tests.
panic in a destructor during cleanup, then every test fails
A helper panicked while it held a lock other tests share, such as a port block registry. Every later test then panics on the poisoned lock, one of them inside a drop as it unwinds, and the test program aborts. Its agents keep running and hold their ports for the next run. A shared helper lets its lock go before it panics, and a drop never panics.
Text file busy
A test ran a program it had just written with std::fs::write, and a
child another test forked held the file open. Write it with the kit's
write_program or copy_program
(Programs and processes).
A test passes alone and fails in the full run
A test is green alone and red in cargo test --workspace, or only
while something else builds. The cause is a race or an elapsed-time
assertion, never chance. Run its whole test program several times on one
build, under load, to count it, then fix the cause
(Load and timing).
waited ... for ...; the last check saw
A wait ran past its deadline. Read what the last check saw: if the
awaited event never came, the code under test lost it, which is the bug.
A machine slower than most sets LNK_TEST_TIME_SCALE once. A longer
deadline alone never fixes a lost event.
An assertion that something took less than some time
A test proved an order by elapsed time, such as an answer told within twice a pause. A loaded machine runs any step late, so the test proves the order by events the code reports instead.
Streamed output is empty after cancellation
A fixed pause can expire before the client displays the first fragment,
even when the producer has yielded it. Capture output with
io::ObservedOutput and wait for its written-byte count before sending
the interrupt. Keep the assertions for the partial output and later
answer, so synchronization preserves the behavior being checked.
Extra typing notifications in a connection test
A bridge refreshes its typing indicator while a reply is pending, and load can give its child time to refresh again. Check the active, stopped and single-delivery transitions while allowing repeated active indicators. Duplicate stops or replies and typing restarted after a stop remain failures.
Address already in use, or a port taken by another program
Two tests took the same port, one probed free and bound later, or one
the code under test hands out itself. A test hands its server a
listener it already holds (Held::loopback). A port a program the test
starts takes itself is held across test programs with HeldNumber
before it's probed, from a range the code's own lookup never gives out.
A port is the proxy's own, or a test server can't bind its port
A port probed free from the system's own range, bound to 0 and let
go, can be any port there: one another listener bound to 0 is handed
next, or one Link fixes for itself in that range, such as the sandbox
proxy's own. A test's ports come from below that range, held with
HeldNumber. A port reused from such a fixed range can still be
in TIME_WAIT from a run before, so a server the test starts sets
SO_REUSEADDR, as Rust's listeners and Python's http.server do.
A port held for a test is taken anyway
HeldNumber::port() holds one port. A program that binds the ports
after its own too, such as a harness that serves its tools on the next
one, takes a port another test holds. Such a program is given a block,
held whole by its base, as the harness tests' free_block does. An
agent's settings name its block's size too (block), or its start
checks the default 200 ports above its base, finds another test's agent
there and moves to ports no test holds.
An agent's first answer times out on a busy machine
An agent's start can take most of a minute on a loaded machine before
its harness answers, so a test whose answer deadline started at the
spawn spends that minute on the start. The test waits for the agent's
health check first (the harness conformance kit's answering), then
times the answer.
a user ... is already on this machine in a root test
A root test that makes system users removes them when it ends, but a
test killed mid-run never ends, and its user stays. The host then
refuses to take over a user it didn't make. Remove the user a killed
run left, with userdel -r lnk-<name>, and never kill a run of the root
tests partway.
Broken pipe asking a fake
A fake answered without reading the question it was sent and exited, so the asker's write met a closed pipe whenever the fake ended first. A fake reads its whole input before it answers, as the program it stands in for does.
A test fails only with LNK_TEST_STATE_DIR set
A test passes with its folders in the temp folder and fails with them in
the state folder, or leaves files there. The test or the code under it
assumed something only /tmp gives.
A sandboxed program can't open a folder it was given
A program a sandbox runs may read and write the folders it was given,
and never the folders that hold them. Under /tmp every parent is
readable, so code that opens a folder through its parent passes there
and fails with Permission denied elsewhere. It opens the folder by its
own path. A test sees the difference with LNK_TEST_STATE_DIR set.
Folders left in LNK_TEST_STATE_DIR after a run
A turn that ends after its test can write a file again once the test's
TempDir is gone, such as measures.jsonl. Folders in the state folder
use a compact process ID and counter, but the ordinary lnk-t... stale
folder sweep does not remove those suite folders. Stop the late writer
before its guard drops, and remove abandoned suite folders only after
verifying their owning processes have ended.
A check fails on files the change didn't touch
A check of the repository's files names files outside the change, or fails on a machine whose tools differ.
A check names files under .claude/worktrees
An agent session keeps worktrees inside the checkout, under
.agents/worktrees, .claude/worktrees or .codex. A check that walks
the folders reads each worktree as more of this checkout. It lists files
from Git, or skips those folders.
ShellCheck disagrees between machines
ShellCheck 0.9, Ubuntu 24.04's, reports a function run only by a trap as
SC2317, which 0.10 renamed SC2329. A disable comment names both,
# shellcheck disable=SC2317,SC2329, so it passes either version.
The first build is slow
A session's first cargo installs the pinned toolchain, then builds
the workspace from nothing. An environment's setup script installs the
toolchain ahead, so the session's first check spends only the build.