Testing Kit
link-test-kit holds what keeps a test free of races and of the
machine it runs on, written once. A test uses the kit's helper rather
than the raw call it replaces, and rather than a helper of its own: a
helper written twice is fixed in one place and not the other. A suite's
own fixtures, such as a fake model or a spec it builds, stay with the
suite.
Use it
The kit is for tests only. A package names it as a development dependency:
[dev-dependencies]
link-test-kit.workspace = trueA home for each test program
Link finds its folders, the machine folder ~/.lnk and the Link folder
~/Link, from HOME. Code a test calls in-process, and a program it
starts without env::command, would find the account's real ones there:
a stale approvals store left by one run would change the next, and a
test would change the Link the account uses. So every test program
holds home! at its root, a crate's unit tests under #[cfg(test)] as
the root's last item, and each integration test program anywhere at its
root:
#[cfg(test)]
link_test_kit::home!();What it sets
Before the program's main, while no thread can read the environment,
home! sets HOME to a folder the kit makes for it, whenever HOME is
unset, relative or the account's own (the home the system keeps for the
account, home::account()). The folder is short, out of /tmp
(TempDir::short_out_of_tmp), empty when the program starts and
removed when it exits.
Sockets and the toolchain
XDG_RUNTIME_DIR becomes the made home's .run, so the program's
sockets are its own and never in the machine-wide runtime folder.
CARGO_HOME and RUSTUP_HOME, when unset, name the account's, so a
cargo the test starts finds the toolchain. A test program a test
starts with env::command keeps the home it was given, and so does one
run as a user a test made, whose own home is in a test's folder. The repository
checks refuse a test program whose root lacks it
(the check).
Why a constructor
The macro is the kit's only code that runs without being called. It is
a constructor, as a C program's are, which the system's loader runs as
the program starts, so cargo run and every program in a Link release, which
never hold it, are untouched.
In a crate only for tests
A crate that is itself only for tests, such as the harness conformance
suite or a fixture, names it as an ordinary dependency. Code in a release
never does. The workspace reaches it through its
workspace.dependencies, by packages/test-kit.
Inject Failures
fault::FileFault removes or truncates one existing regular file below a
test's TempDir, returning its observed before and after sizes. It refuses
paths outside that folder, symbolic links, shared hard links and unchanged
truncations. Fixture ancestors must remain fixed during injection. It keeps
no original contents and restores nothing, so the case observes the product's
own recovery or an explicit repair.
Lost Replies
fault::Once injects one error around a caller-selected operation. With
fault::Moment::Before, the operation never runs. With After, its first
successful result is discarded after its effects complete. Later calls run
normally. An underlying error does not count as an injected lost reply.
Call assert_injected before accepting the experiment, then assert the
product's state, retry behavior and cleanup independently.
IDs in Output
Names remain labels under name, even when a hostname resembles an ID.
Identity fields still require their kind and axis.
Check JSON
ids::assert_named(&json) instead of a test's own search for IDs in a
command's --json: it fails on every ID written outside its form, a
bare one or <kind>:<id> under a field that names no axis, and names
where each is. ids::assert_named_output(stdout) does the same for a
reply of one JSON value or one per line. Stable IDs
explains the form.
What it holds
One module per concern. Each item is also at the crate's root, such as
link_test_kit::TempDir.
temp
TempDir::new(name), instead of std::env::temp_dir(), tempfile, or
a folder removed by hand: it is removed when the test ends, failed or
not, and a killed test's folder goes on the next run.
TempDir::under(parent, name) instead of a folder made in parent by
hand, such as CARGO_TARGET_TMPDIR. TempDir::short(name) instead of a socket under CARGO_TARGET_TMPDIR or a Mac's TMPDIR,
past the socket path limit: its path is at most SHORT (44) bytes, and
a longer name fails the test.
Smaller socket budgets
TempDir::short_with_limit(name, bytes) gives a smaller budget to an
owner such as SSH, whose control socket adds directories, a hash and a
temporary suffix. A suite folder that exceeds it falls back to /tmp.
Suite Folders
With LNK_TEST_STATE_DIR, compact process IDs and counters keep suite
folders distinct across test programs, including after a folder is removed.
Permission Boundaries
On Unix, TempDir::set_mode(mode) changes the guard's owned folder for a
test of an unsafe or restricted ancestor. Keep it under another private
guard when testing a writable ancestor. Program files use write_program
or copy_program.
Real paths
TempDir::new uses the temp folder's real path, because a Mac's goes through
the /var link, which SQLite refuses and which a resolved path never matches.
A short folder stays under /tmp, also a link on a Mac, to keep its path short,
so a test that opens SQLite in one resolves the folder first.
Short folders out of /tmp
TempDir::short_out_of_tmp(name) is for a fixture a sandbox reads that
holds sockets, such as an agent's home. It uses a short
LNK_TEST_STATE_DIR when set, else this user's cache folder
(XDG_CACHE_HOME, else ~/.cache) when a short folder fits there. A
checkout's CARGO_TARGET_TMPDIR can be too long for a socket under it.
Otherwise it falls back to TempDir::short, and gate::out_of_tmp
reports that the folder is in /tmp.
programs
write_program(path, text) instead of std::fs::write and a mode
change before running the file ("Text file busy"), and
copy_program(from, to) instead of std::fs::copy before running the
copy. link_program(from, to) instead of copy_program for a large built
program a test only runs, as a Mac checks each new copy on its first run.
cargo() instead of Command::new("cargo"), which carries the
test's Cargo variables and rebuilds the workspace.
For a Unix permission boundary, write_program_with_mode(path, text, mode)
prepares the executable with the explicit access mode under test.
A stand-in lnk
A stand-in lnk is written with write_core(path, script), a
#!/bin/sh script. Before the script runs, the stand-in answers the
core's history query, a hidden protocol flag that asks for the release's
history of renamed and removed modules and commands, with an empty one
in the protocol version its asker states. A program reading a spec whose names follow that history
asks its core for it, and refuses the spec when the core doesn't
answer.
programs() and programs_in(root)
Instead of building lnk or its modules from a test: programs() is
the target/<profile> the test was built in, where cargo build --workspace --bins put them, and programs_in(root) the same for
another workspace, the nearest target/<profile> above root.
require(program, by) fails with one shared sentence naming by.
process
ChildGroup::spawn(&mut command) owns a background process in a group
of its own, after env::command or env::clean sets its environment.
Its drop stops the group and reaps the direct child, including on panic.
take_stdin releases a piped fixture handshake. take_stdout and take_stderr give access to piped readiness signals
and diagnostics; stop ends the fixture early. kill_group(child)
stops an existing group; interrupt(child) sends SIGINT to one child.
Observe Interrupted Exit
group.interrupt() sends SIGINT to the owned leader without reaping it, and
group.terminate() sends it SIGTERM, as a service manager's stop does. Use
try_status to observe that exit, then let group cleanup end any descendants.
Fixtures of a test's own program
process::rerun(test, home, log) runs the test's own program again as a
dedicated fixture of test, with env::command's environment over home.
The fixture's output, its libtest summary among it, goes to log and never
to the run's own output, so a reader of that output, such as QA or the
timing report, finds one summary, the test's own. process::rerun_said(log)
reads it back for a failure message, each line marked fixture| .
Detached sessions
ChildGroup::spawn_session owns a new session and its process group when a
fixture must not inherit the runner's controlling terminal. It also accepts a
command already configured to create its own session.
Group ownership
The guard's interrupt method sends Ctrl-C to its owned leader while
retaining ownership until cleanup. The guard keeps the leader unreaped until cleanup so its group number
cannot be reused. On Linux and macOS, try_status observes a natural
exit without reaping it. The guard stops descendants that remain in
the group; detached descendants need another owner. An orphaned
grandchild is reaped by its adopter, so cleanup proves it stopped
rather than claiming to reap another process's children. Use a domain
fixture for its protocol, with the guard owning the process beneath it.
Detached descendants
On Linux, left_under(folder) lists every running process that names a
test's own folder in its command line or environment, and
end_under(folder) ends each of them. A fixture whose programs start
daemons in sessions of their own, beyond any group it owns, calls
end_under on its folder in its teardown, so nothing of it outlives the
test.
Dedicated supervisors
On Linux, process::Subreaper::for_fixture lets a dedicated fixture process
adopt orphaned descendants when the container's main process does not reap
them. Its has_children method observes live or unreaped children without
consuming their exit status. After every ChildGroup drops, its reaped method waits without blocking
and reports whether any children remain. It never signals them. Use it only
in a dedicated supervisor, since ordinary test processes can have other tests'
owned children. Drop reaps exited adoptees and restores the previous setting.
net
Held::loopback() instead of a port found free (bound to 0 and let go)
and bound later by the server, which another program can take in
between: the test holds the listener and gives it to the server,
.listener() in its own process, .hand_to(command, variable) to a
child. nodelay_of(group, port, peer) instead of reading a
connection's TCP_NODELAY another process holds by hand.
HeldNumber
HeldNumber::first(kind, candidates, free) instead of a number probed
free and taken later by a program the test starts, such as an X display
or an agent's ports. It holds the first candidate whose lock no other
test holds, across every test program on the machine, and that free
says nothing else has taken, until it's dropped. HeldNumber::hold(kind, number) holds one number for a caller with a search of its own, such as
the conformance kit's port blocks.
HeldNumber::port
HeldNumber::port() instead of a
port bound to 0 and let go for a program the test starts that binds its
port itself, such as an sshd or a harness given LNK_PORT: a loopback
port from 12000 to 15999, below the system's ports and Link's own, read
with .port_number().
The availability probe binds without listening, so a concurrent fork
retaining its descriptor cannot delay the program's listener.
release and gate
release::serve(name, versions) instead of a release server of a
test's own, with the tests' signer. gate::bubblewrap(),
landlock(), sshd(), root() and out_of_tmp(dir) instead of a
check of a test's own and a silent return (Gates);
gate::program(name, args) instead of a have(program) of a test's
own, for a host program such as python3 or git; and
gate::sandbox(lnk_sandbox, home) for whether the workspace's built
lnk-sandbox runs a sandbox here.
For wrapped agents, gate::sandbox_with_landlock() checks namespaces and Landlock.
env and users
env::command(program, home) instead of Command::new with
env_clear and a HOME of the test's, or the test's whole environment
inherited (A child's environment), and
env::clean(command, home) the same for a command made another way,
such as cargo(). users::root(), me() and my_group() instead of
geteuid() == 0, id -un, id -gn, or reading USER, and
users::command_as(uid, gid, groups, program, home) instead of
setpriv written by hand, to run a program as another user.
keychain
On a Mac, Link always keeps secrets in the login Keychain through
Apple's security tool, and LNK_SECURITY names a program to run in
its place. env::clean points it at keychain::stand_in(dir), a script
keeping each item as a file under the child's HOME, so no test writes
your Keychain. Elsewhere there is no Keychain and it sets nothing. A
command built without the kit's environment adds
.envs(keychain::isolated(home)) instead of setting the variable by
hand. A test of the Keychain path on any Unix sets LNK_SECURITY to
keychain::stand_in(dir) itself.
A Keychain that refuses
keychain::refusing(dir) is a security that fails every command, as
a locked Keychain does, for a test that a caller's command fails with it
and leaves the value in no file.
wait
eventually(what, deadline, check) instead of a loop of sleep and an
Instant::now() deadline, or a wait_for, until or wait_until of
a test's own, and within(what, deadline, check) the same loop where
the deadline passing is the caller's to handle. eventually_async and
within_async are the same loop with tokio::time::sleep, in an async
test, and eventually_await and within_await the same for a check
that is itself awaited, such as a request to a server, or a loop around
one.
A whole test's deadline
Deadline::new(what, total) bounds a whole test, set as it starts and
cloned into its threads, so a test that hangs anywhere ends by then and
says what it was waiting for. Its eventually and within end by the
deadline if it comes first, and output(command) runs a program as
output() does but stops it with its process group past the deadline.
A permit before the waits
permit::confined(n) before a test starts n confined programs, held to
its end: a test program's tests share half the CPUs' worth, and a test
waits for its permit before any deadline of its own begins.
wait's polls
wait::exited(what, child, deadline) instead of polling try_wait,
wait::accepting(what, address, deadline) and wait::accepting_unix
instead of polling connect until a server listens, and
wait::appears(what, path, deadline) instead of polling exists.
Observed Output
io::ObservedOutput captures bytes through std::io::Write and publishes
their total through written(). Wait on that receiver's current value
with eventually_async before interrupting a streamed response, then
assert its captured into_bytes(). The observation follows the actual
write, including when the observer subscribes after it happened.
Where write_program comes from
write_program and copy_program are lnk's own
(link_files::write_program), so a program the code under test writes
and one a test writes are written alike
(why).
Waiting
A test waits for the condition it needs, never for a fixed time. Its
check returns a bool, an Option or a Result: done with true,
Some or Ok, and otherwise what it saw. The kit polls it, from 5 ms
doubling to 200 ms between checks, until the deadline. Past it,
eventually fails with what was awaited, how long, and the last value
the check saw:
waited 10.0s for the provider's port file; the last check saw: absentwithin
within returns the same as an error instead: for a test that says
more on failure, such as a server's log, or that expects the wait to
run out.
LNK_TEST_TIME_SCALE
Every deadline is multiplied by LNK_TEST_TIME_SCALE, a positive
number, 1 when unset. A loaded or slow machine sets it once, such as
LNK_TEST_TIME_SCALE=3, and every wait gets more time, rather than
each test adding slack of its own.
Gates
A gate answers whether a test can run on this machine, and the test returns where it can't. It is the only way a test skips: a test that prints its own skip and returns can't be required by any variable (Skipping Tests).
if !link_test_kit::gate::bubblewrap() {
return;
}What it prints
Where the capability is missing, the gate prints the test, what it lacked and why:
proxy::the_listening_proxy_refuses_this_machine: skipping, as it lacks bubblewrap: bwrap: No such file or directoryUnder a variable that requires it
Under the variable that requires the capability, it fails the test
instead, so a machine meant to run every test never passes by skipping
one. LNK_REQUIRE_SANDBOX requires bubblewrap(), sshd(),
sandbox(lnk_sandbox, home) and out_of_tmp(dir),
LNK_REQUIRE_NATIVE requires landlock(), and LNK_REQUIRE_PROGRAMS
requires program(name, args). A workspace program that isn't built is
never a skip: sandbox() fails the test, as the run is missing a step.
root() and a suite's own gate
root() has none: CI runs as an ordinary user, and code only root may
run is run as root before it merges
(Test environments). A suite's own gate, for a
capability only its code can probe, says so with gate::lacks(what, why, variable).
A child's environment
env::command gives a program the test runs nothing of the test's
environment but what it needs to run here, so it passes or fails alike
whoever runs it, from whatever shell:
HOME, the folder the test gives, andTMPDIR, that folder's.tmp, made for it.XDG_RUNTIME_DIR, that folder's.run, made private to the user (env::made_runtime). A test that sets a child's environment whole withenv_clear()gives it the same, or the child's sockets are in the machine-wide runtime folder every test shares.PATH, where the host programs a gate found are.- The proxy and CA variables (
HTTPS_PROXY,NO_PROXY,SSL_CERT_FILEand the like:env::PASSED), through which this machine reaches the network. - The test's own
LNK_*, such asLNK_REQUIRE_SANDBOXorLNK_TEST_TIME_SCALE.
What it never passes
None that holds a secret: what link_secretive::process::variables
leaves out never reaches the child. The runtime folder is beside the
home's .tmp, so a grant of a folder in that home never holds its
sockets. It inherits none of the user’s USER, LOGNAME or XDG_* values. A program that needs its user
asks the system. A variable the test sets on the command afterwards is
added to these.
Command Recording
A controller can observe test commands through the kit's shared
command recording contract. It covers commands
started by env::command and fixture executables handed to product code.
Clippy keeps tests on the kit
Clippy and a repository check refuse the raw calls and patterns the kit replaces, and a call that stays says why where it is (Clippy Keeps Tests on the Kit).
Custom test programs
A test requiring harness = false uses harness::Counts::summary for
its final libtest summary, with actual outcomes and elapsed time.
harness::Selection handles include filters, --exact, --skip,
--list, --ignored and --show-output, rejecting unsupported flags.
It executes nothing. The custom main preserves its child modes and
failure behavior, and explicitly reports cases not run after a failure
as ignored, with their reason.
Platforms
Each helper has a Unix implementation. Where one has no Windows
implementation yet, the kit fails to compile on Windows with an error
naming it: process::kill_group, process::interrupt,
net::Held::hand_to, users, and the check
that a test process is gone, which TempDir uses to remove a killed
test's folders. nodelay_of reads the flag on Linux and macOS and
returns none elsewhere. gate::bubblewrap and gate::landlock say
elsewhere that they are Linux's, and users::command_as is Linux's.