Test Troubleshooting

What a test run says when the machine, the checkout or a test itself is at fault, and how to tell which and fix it. Each symptom below has come up before, so a session can find its cause here rather than again.

A required capability is missing

A test panicked with LNK_REQUIRE_... is set, but it lacks .... The machine was declared to provide a capability it doesn't. Either provide it, or leave the variable unset so its tests skip, saying why (Skipping Tests).

lacks bubblewrap

The sandbox's tests need unprivileged user namespaces. Install bubblewrap, and on Ubuntu let AppArmor allow the namespaces with sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0. Check with bwrap --unshare-net --ro-bind / / true, the gate's own probe.

lacks Landlock

The kernel withholds Landlock, which no package installs: a container gets its host kernel's. Leave LNK_REQUIRE_NATIVE unset on that machine, or use one whose kernel has it, Linux 5.13 or newer. If an agent's wrapped sandbox reports a Landlock version-probe failure, its test needs gate::sandbox_with_landlock, even when bubblewrap works. A test expecting a sandbox refusal must stop at that gate. Running with --no-sandbox and then asserting a refusal tests a different contract.

lacks a test folder out of the temp folders

A fixture the sandbox reads is under /tmp, which the sandbox refuses a spec to read, since anyone can write there. Run from a checkout outside /tmp. A socket fixture uses TempDir::short_out_of_tmp, which needs a cache folder short enough, or a short LNK_TEST_STATE_DIR (LNK_TEST_STATE_DIR).

A program a test runs fails

A test ran a host program and it failed, or the test skipped saying the program is missing. Probe the program the way the gate does, from a folder outside the checkout, before installing anything.

An agent refuses to start from a user namespace

A command runner may put the test itself in a user namespace. Link correctly refuses unsandboxed agents there, even when the underlying host supports bubblewrap. Compare /proc/self/uid_map and the bubblewrap probe in the runner's supported execution modes. Run the tests with the required host capabilities rather than weakening Link's machine identity checks.

A test tries to write to the real home

A failure naming the real user's Link folder, especially on a read-only home, means a child inherited that user's settings. Start it through env::command, or apply env::clean, with the fixture's own home. The test must work whether the real home is writable or read-only.

attempting to make an HTTP request, but --offline was specified

A test read cargo metadata --offline, which resolves every platform's dependencies, and a build had fetched only this machine's. Run cargo fetch --locked once in the workspace.

Unknown binary 'rust-analyzer' in official toolchain

The code tool's test ran rust-analyzer in a sandbox, outside the checkout and without the RUSTUP_TOOLCHAIN Cargo sets, so rustup chose its default toolchain, which lacks it. Make the pinned toolchain the default and give it the component, as CI does: rustup default <pinned> and rustup component add rust-analyzer.

Rustup loses its default inside a language-server sandbox

A rustup proxy can run on the host and fail inside a sandbox that drops RUSTUP_HOME. Select the toolchain before confinement and put its sysroot's bin directory first on the fixture's PATH, so the server and its compiler run directly. Grant that toolchain's folders without forwarding the host's environment or weakening confinement.

A test skips with a browser or display installed

The browser test finds Chromium on the PATH, in CHROMIUM or under PLAYWRIGHT_BROWSERS_PATH, and the desktop tests need Xvfb on the PATH. Set CHROMIUM to the browser's program when it is installed somewhere else, and install xvfb for the desktop tests.

panic in a destructor during cleanup, then every test fails

A helper panicked while it held a lock other tests share, such as a port block registry. Every later test then panics on the poisoned lock, one of them inside a drop as it unwinds, and the test program aborts. Its agents keep running and hold their ports for the next run. A shared helper lets its lock go before it panics, and a drop never panics.

Text file busy

A test ran a program it had just written with std::fs::write, and a child another test forked held the file open. Write it with the kit's write_program or copy_program (Programs and processes).

A test passes alone and fails in the full run

A test is green alone and red in cargo test --workspace, or only while something else builds. The cause is a race or an elapsed-time assertion, never chance. Run its whole test program several times on one build, under load, to count it, then fix the cause (Load and timing).

waited ... for ...; the last check saw

A wait ran past its deadline. Read what the last check saw: if the awaited event never came, the code under test lost it, which is the bug. A machine slower than most sets LNK_TEST_TIME_SCALE once. A longer deadline alone never fixes a lost event.

An assertion that something took less than some time

A test proved an order by elapsed time, such as an answer told within twice a pause. A loaded machine runs any step late, so the test proves the order by events the code reports instead.

Streamed output is empty after cancellation

A fixed pause can expire before the client displays the first fragment, even when the producer has yielded it. Capture output with io::ObservedOutput and wait for its written-byte count before sending the interrupt. Keep the assertions for the partial output and later answer, so synchronization preserves the behavior being checked.

Extra typing notifications in a connection test

A bridge refreshes its typing indicator while a reply is pending, and load can give its child time to refresh again. Check the active, stopped and single-delivery transitions while allowing repeated active indicators. Duplicate stops or replies and typing restarted after a stop remain failures.

Address already in use, or a port taken by another program

Two tests took the same port, one probed free and bound later, or one the code under test hands out itself. A test hands its server a listener it already holds (Held::loopback). A port a program the test starts takes itself is held across test programs with HeldNumber before it's probed, from a range the code's own lookup never gives out.

A port is the proxy's own, or a test server can't bind its port

A port probed free from the system's own range, bound to 0 and let go, can be any port there: one another listener bound to 0 is handed next, or one Link fixes for itself in that range, such as the sandbox proxy's own. A test's ports come from below that range, held with HeldNumber. A port reused from such a fixed range can still be in TIME_WAIT from a run before, so a server the test starts sets SO_REUSEADDR, as Rust's listeners and Python's http.server do.

A port held for a test is taken anyway

HeldNumber::port() holds one port. A program that binds the ports after its own too, such as a harness that serves its tools on the next one, takes a port another test holds. Such a program is given a block, held whole by its base, as the harness tests' free_block does. An agent's settings name its block's size too (block), or its start checks the default 200 ports above its base, finds another test's agent there and moves to ports no test holds.

An agent's first answer times out on a busy machine

An agent's start can take most of a minute on a loaded machine before its harness answers, so a test whose answer deadline started at the spawn spends that minute on the start. The test waits for the agent's health check first (the harness conformance kit's answering), then times the answer.

a user ... is already on this machine in a root test

A root test that makes system users removes them when it ends, but a test killed mid-run never ends, and its user stays. The host then refuses to take over a user it didn't make. Remove the user a killed run left, with userdel -r lnk-<name>, and never kill a run of the root tests partway.

Broken pipe asking a fake

A fake answered without reading the question it was sent and exited, so the asker's write met a closed pipe whenever the fake ended first. A fake reads its whole input before it answers, as the program it stands in for does.

A test fails only with LNK_TEST_STATE_DIR set

A test passes with its folders in the temp folder and fails with them in the state folder, or leaves files there. The test or the code under it assumed something only /tmp gives.

A sandboxed program can't open a folder it was given

A program a sandbox runs may read and write the folders it was given, and never the folders that hold them. Under /tmp every parent is readable, so code that opens a folder through its parent passes there and fails with Permission denied elsewhere. It opens the folder by its own path. A test sees the difference with LNK_TEST_STATE_DIR set.

Folders left in LNK_TEST_STATE_DIR after a run

A turn that ends after its test can write a file again once the test's TempDir is gone, such as measures.jsonl. Folders in the state folder use a compact process ID and counter, but the ordinary lnk-t... stale folder sweep does not remove those suite folders. Stop the late writer before its guard drops, and remove abandoned suite folders only after verifying their owning processes have ended.

A check fails on files the change didn't touch

A check of the repository's files names files outside the change, or fails on a machine whose tools differ.

A check names files under .claude/worktrees

An agent session keeps worktrees inside the checkout, under .agents/worktrees, .claude/worktrees or .codex. A check that walks the folders reads each worktree as more of this checkout. It lists files from Git, or skips those folders.

ShellCheck disagrees between machines

ShellCheck 0.9, Ubuntu 24.04's, reports a function run only by a trap as SC2317, which 0.10 renamed SC2329. A disable comment names both, # shellcheck disable=SC2317,SC2329, so it passes either version.

The first build is slow

A session's first cargo installs the pinned toolchain, then builds the workspace from nothing. An environment's setup script installs the toolchain ahead, so the session's first check spends only the build.