Test Timings

Turn a saved Cargo test run into a timing breakdown by plugin and package. The link-test-timing report separates validation phases from ranked execution durations, test counts and an optional pie chart, with structured data for later comparison.

Capture a run

Run these commands from the workspace being measured, after building its programs. The Bash block preserves Cargo's status while saving both output streams and compiler artifacts, which identify test programs even when several packages use the same test name.

mkdir -p target/test-timing
cargo metadata --no-deps --format-version 1 > target/test-timing/metadata.json
set -o pipefail
set +e
TIMEFORMAT='%R'
{ time cargo test --locked --workspace --no-fail-fast --color never --message-format=json-render-diagnostics 2>&1; } 2> target/test-timing/wall-seconds | tee target/test-timing/cargo.log
capture_status=("${PIPESTATUS[@]}")
test_status=${capture_status[0]}
printf '%s\n' "$test_status" > target/test-timing/status
if [ "${capture_status[1]}" -ne 0 ]; then
    printf 'Could not save complete test output\n' >&2
    exit "${capture_status[1]}"
fi

Keep measurements comparable

Use the same machine, thread count, test selection and required capabilities for comparisons. Tests retain their normal concurrency. Avoid competing builds during a timing run, and build the workspace's programs before testing, as for an ordinary test run.

Preserve failure

The saved status belongs to the Cargo command, not the reporter. Automation exits with that status after rendering and storing its artifacts. A reporter that successfully describes a failed test run does not turn that run into a passing check.

Capture a validation cycle

Test execution is only one part of feedback time. Begin a full-cycle measurement before setup and checks, and record formatting, Clippy, binary builds and the test command separately. Add setup, web or other phases only when their durations were measured. Never infer these costs by subtracting test execution from elapsed time.

Record command wall times

Use Bash's builtin time around each existing command, preserving its status before another command runs. This captures one formatting phase without introducing a test runner. Repeat for the other checks, retaining the raw outputs and using the test-log capture above for execution.

TIMEFORMAT='%R'
set +e
{ time cargo fmt --all --check 2>&1; } 2> target/test-timing/fmt.seconds
fmt_status=$?
printf '%s\n' "$fmt_status" > target/test-timing/fmt.status

Save phase evidence

Pass --phases with a JSON file in this shape, replacing the illustrative values with recorded evidence. A phase's status is its actual command exit code. Missing or null durations and statuses remain unknown. Names are unique and may describe setup, web checks or any other measured work, without changing the reporter.

{
  "schema": 1,
  "elapsed_seconds": 420.5,
  "cache_state": null,
  "phases": [
    {"name": "fmt", "wall_seconds": 1.0, "status": 0},
    {"name": "clippy", "wall_seconds": 120.0, "status": 0},
    {"name": "bin-build", "wall_seconds": 160.0, "status": 0},
    {"name": "test-command", "wall_seconds": 126.0, "status": 101},
    {"name": "web", "wall_seconds": null, "status": null}
  ]
}

Keep elapsed separate from work

Measure overall elapsed independently across stated start and end boundaries. Overlapping phases can make summed command wall times larger than elapsed time, so the reporter never substitutes their sum. The test-command phase includes preparation and execution already. Adding the program breakdown to it would count execution twice.

Describe cache and scope

Record known cache and build context in cache_state, or leave it null for unknown. A fast run, a cache action or a warm registry alone does not establish a warm build. Record the revision, environment and command selection too, and do not call a run cold unless its relevant cache and build state were established.

CI capture boundaries

The Rust validation jobs retain raw logs, metadata, phase files and reports outside the build cache. Their overall elapsed runs from after checkout through validation, before reporting and upload, at one-second precision. Individual commands use Bash timing. Cache restoration has unknown phase duration, and separate web or architecture jobs are outside these per-job measurements.

Render the report

The reporter reads saved files and runs no tests. Run it from this workspace, passing the captured files and an output directory. The optional revision, machine and thread labels record the comparison's context without discovering environment variables or credentials.

cargo run --locked -p link-test-timing -- --log target/test-timing/cargo.log --metadata target/test-timing/metadata.json --status "$(cat target/test-timing/status)" --output target/test-timing/report --revision "$(git rev-parse HEAD)" --machine linux-4cpu --threads 16

Include validation phases

Add --phases target/test-timing/phases.json when rendering saved validation evidence. The phase table precedes the execution breakdown. Its independent elapsed time and summed measured work stay separate from the test command's --wall-seconds and libtest program durations. Failed phases remain visible even when the test command passes.

Open or save the result

Open target/test-timing/report/index.html in a browser. It needs no server, scripts or network access. The sibling timings.json stores the versioned measurement, program outcomes and warnings. Keep generated reports as local files or CI artifacts rather than tracked source.

Compare two runs

Pass --baseline with an earlier timings.json when rendering another run. The report shows both group durations and their difference. Missing groups are marked as a changed selection rather than silently assigned zero. Groups with missing summaries show no duration change, and incomplete runs remain visibly incomplete.

Read the measurements

Each duration comes from a test program's libtest summary, which already includes the effects of concurrent tests. The chart shows its share of summed program wall times, never a sum of individual tests. Cargo's printed precision can round very short programs to zero.

Ranked table

The table shows each area's total test time, share of measured time, passed, skipped and failed tests, in that order. Counts come from libtest summaries. Skipped means explicitly ignored by libtest, while capability-gated early returns can appear as passes. Missing summaries leave their counts unknown, and each row notes missing coverage. Wholly missing summaries show unknown counts, and mixed coverage marks the observed counts as partial.

Environment exclusions

The optional --filtered-skip-reason records why every filtered test in the captured run was individually verified as an environment exclusion. Only with this option does the skipped count include filtered tests alongside explicit ignores. Use it only when all filters represent such exclusions, never for an arbitrary test selection. Passed and failed counts stay unchanged, and the saved programs retain their raw ignored and filtered counts.

Ownership

Compiler artifacts map executable paths to package IDs in the matching Cargo metadata. A plugin's nested packages join its group, using the nearest containing package with a binary. Shared packages, core and repository tests retain separate groups, and calls into a dependency remain part of the calling test's time.

Missing attribution

An ordinary saved text log can identify programs with unique target names. Ambiguous names remain unassigned, with a warning. Capture compiler artifacts and matching metadata for reliable attribution. Doc tests are matched by their library target name.

Command and preparation time

The report separately shows Cargo's printed preparation time and any whole-command duration supplied with --wall-seconds. It never infers the latter from test summaries. Preparation includes Cargo's work before execution, which can include cache checks and compilation. Validation-phase durations are recorded separately and do not imply that a test's preparation was the only compilation in the cycle.

Failed and incomplete runs

A missing summary has unknown duration, not zero. A nonzero Cargo status, missing status, unassigned package or unsuccessful program prevents a complete result. Ignored tests remain visible. Capability-gated early returns can appear as passes, so timing does not prove the gated work ran.

Custom harnesses

Tests use Rust's standard harness by default. A required custom main with harness = false uses the shared selection and summary contract through the testing kit. A missing summary is a reporting gap that leaves timing and outcome coverage incomplete. Crashes, truncated logs and nonconforming harnesses can omit summaries, so missing timings alone do not identify their cause.

Other executors can adapt saved case results to the library's Program measurements and call measured with MeasurementKind::Cases. Cargo's parser uses the same aggregation, report types and HTML renderer. Unknown durations stay separate from unknown outcomes, so a skipped case can retain its known count without inventing execution time.

Read compatible saved evidence

Use Report::parse for saved reports and baselines. It validates the schema and durations for either execution model. Earlier reports default to test programs, and baseline comparisons stay within one model. Case execution success establishes timing evidence, with its caller responsible for interpreting product assertions and output approval.