Relay servers you set up in one command

Set up, run and recover your own relay, the server that gives each tunnel its public address, https://<app>.<user>.<domain>. One command sets a server up, and one command replaces it without losing users, certificates or its IP. The examples use relay.example.com for its domain, and the scripts are in plugins/networking/relay/deploy.

# from anywhere: ok VERSION when the relay is up
curl https://relay.example.com/_link/health
# on the server: relay and update logs
journalctl -u link-relay -u link-relay-update -f
# update now, not in 5 minutes
systemctl start link-relay-update
# who is connected
link-relay admin tunnel list
# cut someone off now
link-relay admin user block alice
# back up now
systemctl start link-backup

On your Mac, one command replaces the server:

plugins/networking/relay/deploy/rebuild.sh --old <old droplet ip> --reserved-ip <reserved ip> --domain relay.example.com

How it's laid out

Caddy handles HTTPS and gets a Let's Encrypt certificate for each hostname on first use, once the relay approves it. link-relay, on 127.0.0.1:7080, routes each hostname to its user's lnk, which reaches it over a WebSocket from lnk tunnel open. The server is disposable, so you replace it rather than repair it.

browser ─https─▶ Caddy (:443, TLS) ─http─▶ link-relay (127.0.0.1:7080) ◀─wss─ lnk tunnel open

Day to day

A relay updates itself, so most of this is checking on it and managing its users. Setting one up is on Set Up a Relay.

Check it's healthy

On the server, the health check prints ok VERSION, and a failed update or backup shows in systemctl --failed.

curl localhost:7080/_link/health
systemctl --failed
systemctl list-timers link-relay-update.timer link-backup.timer
# the last updates
journalctl -u link-relay-update -n 20 --no-pager
# who is banned from SSH
fail2ban-client status sshd

Deploy a change

No release publishes the relay, so a server's updater, which checks the relay-latest release every 5 minutes, finds nothing new. Build the relay from your checkout of main and install it with bootstrap's --from, which installs the binary and the deploy files in that folder as they are, with no signature check. Bootstrap keeps users, certificates and settings, and re-applies the firewall and hardening.

# a static relay, which runs on any x86_64 Linux
rustup target add x86_64-unknown-linux-musl
cargo build --release --locked -p link-relay-server --target x86_64-unknown-linux-musl
cp -r plugins/networking/relay/deploy/files link-files
cp target/x86_64-unknown-linux-musl/release/link-relay link-files/link-relay-x86_64-linux
scp -r link-files root@relay.example.com:/root/link-files
ssh root@relay.example.com 'bash -s -- --from /root/link-files' < plugins/networking/relay/deploy/bootstrap.sh

From a signed release

Where a signed release is published, bootstrap without --from installs it, and the updater follows it. Run bootstrap from link-public/ in your checkout of main, never from a download, because it carries the release key that everything it installs is checked against.

# the server never installs an older relay
V=$(grep -m1 '^version' Cargo.toml | cut -d'"' -f2)
ssh root@relay.example.com "bash -s -- --domain relay.example.com --email you@example.com --min-version $V" < plugins/networking/relay/deploy/bootstrap.sh

What bootstrap installs

Bootstrap installs a release's deploy files only if they match its SHA256SUMS and that file is signed by a key in the script. It refuses a release older than the highest of --min-version, the floor it kept before and the relay the server already runs, and keeps that version in /etc/link/min-relay-version as a floor the updater holds to, until a higher one is given.

The version floor

Re-run bootstrap with --min-version too, as above: only on a fresh server with no floor does it take any signed release. create-droplet.sh and rebuild.sh pass their checkout's version themselves, so run them from a checkout of main, and one whose version isn't released yet is refused.

Roll back

Install the older build the same way, with --from. From a signed release, run LINK_ALLOW_DOWNGRADE=1 link-relay-update on the server: the timer's own runs refuse a build older than the installed relay, or than the floor in /etc/link/min-relay-version. A build that failed its health check is rolled back by itself and skipped from then on.

Manage users

users lists the users file's users (ok), everyone else with a tunnel open (account), and everyone blocked. Machines log in and sign out at the account service, not here. tunnels lists the connections serving, each with its user, uptime, requests, lnk version and IP; a tunnel's standby, waiting to take it over (Protocol), shows once it serves. kick and block disconnect a tunnel's standby too.

# every user: ok / account / blocked
link-relay admin user list
link-relay admin tunnel list
# disconnect one tunnel: lnk stops
link-relay admin tunnel disconnect my-app.alice
# disconnect alice and refuse her tokens
link-relay admin user block alice
link-relay admin user unblock alice

Blocking

block needs no restart, lasts across restarts and works by username, for accounts and the users file's users alike. It's kept in /var/lib/private/link/blocked, which is in the snapshot. link-relay admin talks to the relay on the same machine, at http://127.0.0.1:7080 unless you pass --relay, and the relay answers admin requests only from loopback.

Add a user of your own

This appends the token's hash to the users file and prints the token once, and you give it to them. To remove one, delete their line from /etc/link/users and restart. A name in the users file is yours: an account by the same name is refused.

t=$(openssl rand -hex 24); echo "alice sha256:$(printf %s "$t" | sha256sum | cut -d' ' -f1)" >> /etc/link/users && systemctl restart link-relay && echo "$t"

Change a setting

Settings live in /etc/link/relay.env, which only root can read and the systemd unit loads. Configuration lists every one. Edit the file and restart, or re-run bootstrap with its flag.

echo 'LINK_NOTICE=maintenance at 22:00 UTC' >> /etc/link/relay.env && systemctl restart link-relay

The website and the limits

To send browsers on the bare domain to the website, re-run bootstrap with --website https://www.example.com. /_link/* and /install.sh stay on the relay. Limits such as requests, tunnels, body sizes and timeouts are compiled into the relay, and the tunnel's security lists them.

Recover from a loss

Replacing a server keeps its users, certificates and IP, and a restore brings them back from an off-site backup.

Replace the server

Use this for a broken server, a new region or a new size. It exports the old server's state, keeps a copy in the current folder, creates and bootstraps a new droplet with that state, and moves the reserved IP to it. Agents reconnect by themselves and keep their URLs. Any other flag goes to create-droplet.sh (--region, --size, ...).

plugins/networking/relay/deploy/rebuild.sh --old <old droplet ip> --reserved-ip <reserved ip> --domain relay.example.com

The old droplet

The old droplet keeps running, so you can switch back by reassigning the IP. Delete it once the new one works.

doctl compute droplet list --tag-name link-relay
doctl compute droplet delete <old id>

Restore from a backup

When the old server is gone, restore its last off-site backup. You need age (brew install age) and the backup settings on your Mac, which Set up off-site backups saves. fetch-backup.sh writes ./link-state-latest.tar.gz.

plugins/networking/relay/deploy/fetch-backup.sh --env ~/.config/link/backup.env
plugins/networking/relay/deploy/create-droplet.sh --state link-state-latest.tar.gz --reserved-ip <reserved ip> --domain relay.example.com

Its key and an older backup

fetch-backup.sh decrypts with ~/.ssh/id_ed25519. Pass --identity <file> for another key and --name state-<time>.tar.gz.age for an older backup. Without an off-site backup the users file is lost, so its tokens have to be reissued. Accounts are the account service's, and log in as before.

Trusting a snapshot

A snapshot is only as trustworthy as the bucket it came from, because whoever can write there can upload one of their own, with its own users and settings. A restore never takes the release key or the version floor from it, and bootstrap writes both from your checkout after the import.

Local snapshots

To copy the server's local snapshots to your Mac, run scp 'root@relay.example.com:/var/backups/link/*' ~/relay-backups/; they're kept in /var/backups/link on the server.

More about it

The rest of the relay's pages, for setting one up and for what it does and why.

Set Up a Relay: a new relay, its roles, its measures, logins, backups and the release key
Security: what bootstrap and the updater install, the checks before they do, and how the server is hardened
Configuration: every deploy script, and the relay's settings
Decisions: why it works this way

Docs

Configuration: The scripts that set up, rebuild and back up a relay server, in plugins/networking/relay/deploy, and the flags each takes
Public Suffix List: A relay serves every tunnel under one domain, its own setting (LINK_DOMAIN, Change a setting): <app>.<user>.<domain>
Security: How a relay server is hardened, and what its operator should keep doing
Decisions: Why the relay is hosted the way the runbook describes
Set Up a Relay: How to set up your own relay once, give it a role, send its measures, take logins from an account service, back it up off-site and rotate its release key (Your own relay)