Skipping Tests

A test skips only when the machine can't give it what it needs, and it says what and why. This page holds when a skip is right, how a test makes one and every variable that decides whether it skips, fails or runs.

When a test skips

A skip is the machine's, never the code's. A test skips when the host lacks a capability, such as bubblewrap's namespaces, Landlock or root, or a program it runs, such as sshd, Xvfb or Chromium. It never skips because it is slow, flaky, broken or waiting on a fix: those are failures, fixed at their cause or the test deleted.

What is a failure instead

Some missing things are the run's mistake rather than the machine's. A workspace program the test runs that wasn't built is a failure, since cargo build --workspace --bins comes first. So is a request the session's network refuses, since a test answers every name it reaches itself through LNK_TEST_RESOLVE. A gate that passes but whose capability then breaks under the test is a bug in the gate.

Skips are not coverage

libtest counts a test that skipped through its gate as passed, so a green run says nothing about what skipped. A test's gate output is captured with the rest of its output, and shows only with cargo test -- --show-output. A run's coverage is the tests that ran, read from that output, never its pass count (Results and coverage).

How a test skips

A test asks the testing kit's gate in gate.rs before it needs the capability. The gate probes the machine and returns true, or prints <test>: skipping, as it lacks <what>: <why> and returns false, and the test returns. Under the capability's LNK_REQUIRE_* variable the gate panics with the same words instead, so the skip fails the run. The line is a print macro, not link-format's aside: the harness captures it per test, which is how --show-output shows which test skipped. The reasons carry no backticks.

One gate a capability

Each capability has one gate, and every test that needs it asks that gate: bubblewrap, landlock, sshd, root, program and out_of_tmp. A capability only a suite can probe goes through gate::lacks, with the variable that requires it. A test never prints its own skip and returns, which no variable can turn into a failure.

Probe what the test will run

A gate runs the capability as the test will run it, never a stand-in that passes where the real one fails. Rustup picks a toolchain by the folder and by RUSTUP_TOOLCHAIN, which Cargo sets for every test. A program the test runs in a sandbox, outside the checkout and without that variable, can get the machine's default toolchain, so its probe runs in the test's folder with the variable removed.

Look everywhere a program is installed

A gate that finds a program by its place names every place it is installed. Playwright puts Chromium in chromium-<build>/chrome-linux/ in older builds and in chrome-linux64/ in newer ones, and a probe that knew one skipped on a machine that had the other.

Variables that require a capability

Each variable turns its capability's skips into failures. Set one only on a machine that provides everything it names, as CI does: there, a skip is a regression. A machine missing any of it leaves the variable unset, and the tests that need it skip, saying why.

LNK_REQUIRE_SANDBOX

Requires bubblewrap with unprivileged user namespaces and its own network, sshd for the sandbox's guests, and test folders out of the temp folders. That means a checkout outside /tmp, and for a fixture that must stay out of /tmp with a short path, a cache folder short enough or a short LNK_TEST_STATE_DIR. Ubuntu's AppArmor withholds the namespaces until kernel.apparmor_restrict_unprivileged_userns=0.

LNK_REQUIRE_NATIVE

Requires the kernel's Landlock, which the sandbox's executable allowlists enforce. The kernel provides it, Linux 5.13 or newer with it enabled, and a container can't add it. Probe it with the kit's gate::landlock, which asks the kernel for Landlock's version.

LNK_REQUIRE_PROGRAMS

Requires the host programs tests run, such as python3, git, zsh, ssh, Xvfb, rust-analyzer in the default toolchain and a Chromium. Each is probed by running it, or by finding it where it is installed, so a program that exists but doesn't run counts as missing.

Variables a test reads from the machine

These variables give a test what only the machine knows. None skips a test by itself, and each is set only where its value is true.

LNK_TEST_STATE_DIR

A short folder outside /tmp, such as ~/.lnk/tests, under which every TempDir goes instead of the temp folder. A socket fixture that must stay out of the sandbox's /tmp uses it, else this user's cache folder, since a checkout's own path can pass a socket's length limit. Nothing needs it set, and a test passes with it set or not.

LNK_TEST_TIME_SCALE

A number, such as 3, that every deadline of the kit's waits is multiplied by, on a machine slower than most. It is set once for the machine, never for one test, and never to hide a race (Load and timing).

CHROMIUM and PLAYWRIGHT_BROWSERS_PATH

CHROMIUM names the Chromium the browser tool's test runs. Without it, the test finds one on the PATH, then under PLAYWRIGHT_BROWSERS_PATH, where Playwright installs its browsers.

Capabilities nothing requires

A few capabilities stay unrequired, because CI can't or shouldn't have them. Root is one: GitHub's runners run as an ordinary user, and code only root runs is run as root before it merges. Another OS is another: a Linux-only test skips on macOS. Each says so through its gate, like any other skip.

Choose an environment's variables

To set a machine up, probe each capability, install only the programs the components under test need and set each LNK_REQUIRE_* variable the machine fully provides. Then run the suite once with -- --show-output, read every skipping, as it lacks line, and either provide what it names or leave its variable unset. A machine's failure that only it has gets a gate and a rule in Test Environments.