Testing Kit

link-test-kit holds what keeps a test free of races and of the machine it runs on, written once. A test uses the kit's helper rather than the raw call it replaces, and rather than a helper of its own: a helper written twice is fixed in one place and not the other. A suite's own fixtures, such as a fake model or a spec it builds, stay with the suite.

Use it

The kit is for tests only. A package names it as a development dependency:

[dev-dependencies]
link-test-kit.workspace = true

A home for each test program

Link finds its folders, the machine folder ~/.lnk and the Link folder ~/Link, from HOME. Code a test calls in-process, and a program it starts without env::command, would find the account's real ones there: a stale approvals store left by one run would change the next, and a test would change the Link the account uses. So every test program holds home! at its root, a crate's unit tests under #[cfg(test)] as the root's last item, and each integration test program anywhere at its root:

#[cfg(test)]
link_test_kit::home!();

What it sets

Before the program's main, while no thread can read the environment, home! sets HOME to a folder the kit makes for it, whenever HOME is unset, relative or the account's own (the home the system keeps for the account, home::account()). The folder is short, out of /tmp (TempDir::short_out_of_tmp), empty when the program starts and removed when it exits.

Sockets and the toolchain

XDG_RUNTIME_DIR becomes the made home's .run, so the program's sockets are its own and never in the machine-wide runtime folder. CARGO_HOME and RUSTUP_HOME, when unset, name the account's, so a cargo the test starts finds the toolchain. A test program a test starts with env::command keeps the home it was given, and so does one run as a user a test made, whose own home is in a test's folder. The repository checks refuse a test program whose root lacks it (the check).

Why a constructor

The macro is the kit's only code that runs without being called. It is a constructor, as a C program's are, which the system's loader runs as the program starts, so cargo run and every program in a Link release, which never hold it, are untouched.

In a crate only for tests

A crate that is itself only for tests, such as the harness conformance suite or a fixture, names it as an ordinary dependency. Code in a release never does. The workspace reaches it through its workspace.dependencies, by packages/test-kit.

Inject Failures

fault::FileFault removes or truncates one existing regular file below a test's TempDir, returning its observed before and after sizes. It refuses paths outside that folder, symbolic links, shared hard links and unchanged truncations. Fixture ancestors must remain fixed during injection. It keeps no original contents and restores nothing, so the case observes the product's own recovery or an explicit repair.

Lost Replies

fault::Once injects one error around a caller-selected operation. With fault::Moment::Before, the operation never runs. With After, its first successful result is discarded after its effects complete. Later calls run normally. An underlying error does not count as an injected lost reply. Call assert_injected before accepting the experiment, then assert the product's state, retry behavior and cleanup independently.

IDs in Output

Names remain labels under name, even when a hostname resembles an ID. Identity fields still require their kind and axis.

Check JSON

ids::assert_named(&json) instead of a test's own search for IDs in a command's --json: it fails on every ID written outside its form, a bare one or <kind>:<id> under a field that names no axis, and names where each is. ids::assert_named_output(stdout) does the same for a reply of one JSON value or one per line. Stable IDs explains the form.

What it holds

One module per concern. Each item is also at the crate's root, such as link_test_kit::TempDir.

temp

TempDir::new(name), instead of std::env::temp_dir(), tempfile, or a folder removed by hand: it is removed when the test ends, failed or not, and a killed test's folder goes on the next run. TempDir::under(parent, name) instead of a folder made in parent by hand, such as CARGO_TARGET_TMPDIR. TempDir::short(name) instead of a socket under CARGO_TARGET_TMPDIR or a Mac's TMPDIR, past the socket path limit: its path is at most SHORT (44) bytes, and a longer name fails the test.

Smaller socket budgets

TempDir::short_with_limit(name, bytes) gives a smaller budget to an owner such as SSH, whose control socket adds directories, a hash and a temporary suffix. A suite folder that exceeds it falls back to /tmp.

Suite Folders

With LNK_TEST_STATE_DIR, compact process IDs and counters keep suite folders distinct across test programs, including after a folder is removed.

Permission Boundaries

On Unix, TempDir::set_mode(mode) changes the guard's owned folder for a test of an unsafe or restricted ancestor. Keep it under another private guard when testing a writable ancestor. Program files use write_program or copy_program.

Real paths

TempDir::new uses the temp folder's real path, because a Mac's goes through the /var link, which SQLite refuses and which a resolved path never matches. A short folder stays under /tmp, also a link on a Mac, to keep its path short, so a test that opens SQLite in one resolves the folder first.

Short folders out of /tmp

TempDir::short_out_of_tmp(name) is for a fixture a sandbox reads that holds sockets, such as an agent's home. It uses a short LNK_TEST_STATE_DIR when set, else this user's cache folder (XDG_CACHE_HOME, else ~/.cache) when a short folder fits there. A checkout's CARGO_TARGET_TMPDIR can be too long for a socket under it. Otherwise it falls back to TempDir::short, and gate::out_of_tmp reports that the folder is in /tmp.

programs

write_program(path, text) instead of std::fs::write and a mode change before running the file ("Text file busy"), and copy_program(from, to) instead of std::fs::copy before running the copy. link_program(from, to) instead of copy_program for a large built program a test only runs, as a Mac checks each new copy on its first run. cargo() instead of Command::new("cargo"), which carries the test's Cargo variables and rebuilds the workspace. For a Unix permission boundary, write_program_with_mode(path, text, mode) prepares the executable with the explicit access mode under test.

A stand-in lnk

A stand-in lnk is written with write_core(path, script), a #!/bin/sh script. Before the script runs, the stand-in answers the core's history query, a hidden protocol flag that asks for the release's history of renamed and removed modules and commands, with an empty one in the protocol version its asker states. A program reading a spec whose names follow that history asks its core for it, and refuses the spec when the core doesn't answer.

programs() and programs_in(root)

Instead of building lnk or its modules from a test: programs() is the target/<profile> the test was built in, where cargo build --workspace --bins put them, and programs_in(root) the same for another workspace, the nearest target/<profile> above root. require(program, by) fails with one shared sentence naming by.

process

ChildGroup::spawn(&mut command) owns a background process in a group of its own, after env::command or env::clean sets its environment. Its drop stops the group and reaps the direct child, including on panic. take_stdin releases a piped fixture handshake. take_stdout and take_stderr give access to piped readiness signals and diagnostics; stop ends the fixture early. kill_group(child) stops an existing group; interrupt(child) sends SIGINT to one child.

Observe Interrupted Exit

group.interrupt() sends SIGINT to the owned leader without reaping it, and group.terminate() sends it SIGTERM, as a service manager's stop does. Use try_status to observe that exit, then let group cleanup end any descendants.

Fixtures of a test's own program

process::rerun(test, home, log) runs the test's own program again as a dedicated fixture of test, with env::command's environment over home. The fixture's output, its libtest summary among it, goes to log and never to the run's own output, so a reader of that output, such as QA or the timing report, finds one summary, the test's own. process::rerun_said(log) reads it back for a failure message, each line marked fixture| .

Detached sessions

ChildGroup::spawn_session owns a new session and its process group when a fixture must not inherit the runner's controlling terminal. It also accepts a command already configured to create its own session.

Group ownership

The guard's interrupt method sends Ctrl-C to its owned leader while retaining ownership until cleanup. The guard keeps the leader unreaped until cleanup so its group number cannot be reused. On Linux and macOS, try_status observes a natural exit without reaping it. The guard stops descendants that remain in the group; detached descendants need another owner. An orphaned grandchild is reaped by its adopter, so cleanup proves it stopped rather than claiming to reap another process's children. Use a domain fixture for its protocol, with the guard owning the process beneath it.

Detached descendants

On Linux, left_under(folder) lists every running process that names a test's own folder in its command line or environment, and end_under(folder) ends each of them. A fixture whose programs start daemons in sessions of their own, beyond any group it owns, calls end_under on its folder in its teardown, so nothing of it outlives the test.

Dedicated supervisors

On Linux, process::Subreaper::for_fixture lets a dedicated fixture process adopt orphaned descendants when the container's main process does not reap them. Its has_children method observes live or unreaped children without consuming their exit status. After every ChildGroup drops, its reaped method waits without blocking and reports whether any children remain. It never signals them. Use it only in a dedicated supervisor, since ordinary test processes can have other tests' owned children. Drop reaps exited adoptees and restores the previous setting.

net

Held::loopback() instead of a port found free (bound to 0 and let go) and bound later by the server, which another program can take in between: the test holds the listener and gives it to the server, .listener() in its own process, .hand_to(command, variable) to a child. nodelay_of(group, port, peer) instead of reading a connection's TCP_NODELAY another process holds by hand.

HeldNumber

HeldNumber::first(kind, candidates, free) instead of a number probed free and taken later by a program the test starts, such as an X display or an agent's ports. It holds the first candidate whose lock no other test holds, across every test program on the machine, and that free says nothing else has taken, until it's dropped. HeldNumber::hold(kind, number) holds one number for a caller with a search of its own, such as the conformance kit's port blocks.

HeldNumber::port

HeldNumber::port() instead of a port bound to 0 and let go for a program the test starts that binds its port itself, such as an sshd or a harness given LNK_PORT: a loopback port from 12000 to 15999, below the system's ports and Link's own, read with .port_number(). The availability probe binds without listening, so a concurrent fork retaining its descriptor cannot delay the program's listener.

release and gate

release::serve(name, versions) instead of a release server of a test's own, with the tests' signer. gate::bubblewrap(), landlock(), sshd(), root() and out_of_tmp(dir) instead of a check of a test's own and a silent return (Gates); gate::program(name, args) instead of a have(program) of a test's own, for a host program such as python3 or git; and gate::sandbox(lnk_sandbox, home) for whether the workspace's built lnk-sandbox runs a sandbox here. For wrapped agents, gate::sandbox_with_landlock() checks namespaces and Landlock.

env and users

env::command(program, home) instead of Command::new with env_clear and a HOME of the test's, or the test's whole environment inherited (A child's environment), and env::clean(command, home) the same for a command made another way, such as cargo(). users::root(), me() and my_group() instead of geteuid() == 0, id -un, id -gn, or reading USER, and users::command_as(uid, gid, groups, program, home) instead of setpriv written by hand, to run a program as another user.

keychain

On a Mac, Link always keeps secrets in the login Keychain through Apple's security tool, and LNK_SECURITY names a program to run in its place. env::clean points it at keychain::stand_in(dir), a script keeping each item as a file under the child's HOME, so no test writes your Keychain. Elsewhere there is no Keychain and it sets nothing. A command built without the kit's environment adds .envs(keychain::isolated(home)) instead of setting the variable by hand. A test of the Keychain path on any Unix sets LNK_SECURITY to keychain::stand_in(dir) itself.

A Keychain that refuses

keychain::refusing(dir) is a security that fails every command, as a locked Keychain does, for a test that a caller's command fails with it and leaves the value in no file.

wait

eventually(what, deadline, check) instead of a loop of sleep and an Instant::now() deadline, or a wait_for, until or wait_until of a test's own, and within(what, deadline, check) the same loop where the deadline passing is the caller's to handle. eventually_async and within_async are the same loop with tokio::time::sleep, in an async test, and eventually_await and within_await the same for a check that is itself awaited, such as a request to a server, or a loop around one.

A whole test's deadline

Deadline::new(what, total) bounds a whole test, set as it starts and cloned into its threads, so a test that hangs anywhere ends by then and says what it was waiting for. Its eventually and within end by the deadline if it comes first, and output(command) runs a program as output() does but stops it with its process group past the deadline.

A permit before the waits

permit::confined(n) before a test starts n confined programs, held to its end: a test program's tests share half the CPUs' worth, and a test waits for its permit before any deadline of its own begins.

wait's polls

wait::exited(what, child, deadline) instead of polling try_wait, wait::accepting(what, address, deadline) and wait::accepting_unix instead of polling connect until a server listens, and wait::appears(what, path, deadline) instead of polling exists.

Observed Output

io::ObservedOutput captures bytes through std::io::Write and publishes their total through written(). Wait on that receiver's current value with eventually_async before interrupting a streamed response, then assert its captured into_bytes(). The observation follows the actual write, including when the observer subscribes after it happened.

Where write_program comes from

write_program and copy_program are lnk's own (link_files::write_program), so a program the code under test writes and one a test writes are written alike (why).

Waiting

A test waits for the condition it needs, never for a fixed time. Its check returns a bool, an Option or a Result: done with true, Some or Ok, and otherwise what it saw. The kit polls it, from 5 ms doubling to 200 ms between checks, until the deadline. Past it, eventually fails with what was awaited, how long, and the last value the check saw:

waited 10.0s for the provider's port file; the last check saw: absent

within

within returns the same as an error instead: for a test that says more on failure, such as a server's log, or that expects the wait to run out.

LNK_TEST_TIME_SCALE

Every deadline is multiplied by LNK_TEST_TIME_SCALE, a positive number, 1 when unset. A loaded or slow machine sets it once, such as LNK_TEST_TIME_SCALE=3, and every wait gets more time, rather than each test adding slack of its own.

Gates

A gate answers whether a test can run on this machine, and the test returns where it can't. It is the only way a test skips: a test that prints its own skip and returns can't be required by any variable (Skipping Tests).

if !link_test_kit::gate::bubblewrap() {
    return;
}

What it prints

Where the capability is missing, the gate prints the test, what it lacked and why:

proxy::the_listening_proxy_refuses_this_machine: skipping, as it lacks bubblewrap: bwrap: No such file or directory

Under a variable that requires it

Under the variable that requires the capability, it fails the test instead, so a machine meant to run every test never passes by skipping one. LNK_REQUIRE_SANDBOX requires bubblewrap(), sshd(), sandbox(lnk_sandbox, home) and out_of_tmp(dir), LNK_REQUIRE_NATIVE requires landlock(), and LNK_REQUIRE_PROGRAMS requires program(name, args). A workspace program that isn't built is never a skip: sandbox() fails the test, as the run is missing a step.

root() and a suite's own gate

root() has none: CI runs as an ordinary user, and code only root may run is run as root before it merges (Test environments). A suite's own gate, for a capability only its code can probe, says so with gate::lacks(what, why, variable).

A child's environment

env::command gives a program the test runs nothing of the test's environment but what it needs to run here, so it passes or fails alike whoever runs it, from whatever shell:

  • HOME, the folder the test gives, and TMPDIR, that folder's .tmp, made for it.
  • XDG_RUNTIME_DIR, that folder's .run, made private to the user (env::made_runtime). A test that sets a child's environment whole with env_clear() gives it the same, or the child's sockets are in the machine-wide runtime folder every test shares.
  • PATH, where the host programs a gate found are.
  • The proxy and CA variables (HTTPS_PROXY, NO_PROXY, SSL_CERT_FILE and the like: env::PASSED), through which this machine reaches the network.
  • The test's own LNK_*, such as LNK_REQUIRE_SANDBOX or LNK_TEST_TIME_SCALE.

What it never passes

None that holds a secret: what link_secretive::process::variables leaves out never reaches the child. The runtime folder is beside the home's .tmp, so a grant of a folder in that home never holds its sockets. It inherits none of the user’s USER, LOGNAME or XDG_* values. A program that needs its user asks the system. A variable the test sets on the command afterwards is added to these.

Command Recording

A controller can observe test commands through the kit's shared command recording contract. It covers commands started by env::command and fixture executables handed to product code.

Clippy keeps tests on the kit

Clippy and a repository check refuse the raw calls and patterns the kit replaces, and a call that stays says why where it is (Clippy Keeps Tests on the Kit).

Custom test programs

A test requiring harness = false uses harness::Counts::summary for its final libtest summary, with actual outcomes and elapsed time. harness::Selection handles include filters, --exact, --skip, --list, --ignored and --show-output, rejecting unsupported flags. It executes nothing. The custom main preserves its child modes and failure behavior, and explicitly reports cases not run after a failure as ignored, with their reason.

Platforms

Each helper has a Unix implementation. Where one has no Windows implementation yet, the kit fails to compile on Windows with an error naming it: process::kill_group, process::interrupt, net::Held::hand_to, users, and the check that a test process is gone, which TempDir uses to remove a killed test's folders. nodelay_of reads the flag on Linux and macOS and returns none elsewhere. gate::bubblewrap and gate::landlock say elsewhere that they are Linux's, and users::command_as is Linux's.