Set Up a Relay

How to set up your own relay once, give it a role, send its measures, take logins from an account service, back it up off-site and rotate its release key (Your own relay). You do the first sections once, in order, and the rest when you need them.

Set up a new relay

Do this once, before the first server: the releases repo, the signing key, a reserved IP, then the server. Then take logins from an account service and set up off-site backups.

The public releases repo

Create so-local/link-releases on GitHub, public, with a README. In so-local/link, Settings → Environments → New environment, make release: Deployment branches and tags → Selected branches → main, and Required reviewers → yourself. The jobs that sign and publish run only there.

Its token

Make a fine-grained token with access to that repo only and Contents: Read and write, and save it as that environment's secret RELEASES_TOKEN, which release.yml publishes lnk with.

The relay it runs

No release publishes the relay, so a server runs one you build from your checkout and install with bootstrap's --from, as in Deploy a change. create-droplet.sh and rebuild.sh run bootstrap without it, which finds no signed release and stops, so run bootstrap with --from on the server they made.

The release signing key

Make it as in Rotate the release signing key, save the private key as the release environment's secret RELEASE_SIGNING_KEY, and put the public line in scripts/release/release-signers, core/cli/install.sh and plugins/networking/relay/deploy/bootstrap.sh. Losing the private key means you have to rotate.

doctl and a reserved IP

The reserved IP stays when servers are replaced, so DNS never changes. Make an API token in DigitalOcean (API → Generate), and have at least one SSH key on the account: the scripts use all of them.

brew install doctl && doctl auth init
doctl compute ssh-key list
# prints the IP
doctl compute reserved-ip create --droplet-id <droplet id>

Its DNS records

Point relay.example.com and *.relay.example.com (A records) at the reserved IP. Add AAAA records only once curl -6 https://relay.example.com/ works from outside: a broken one stalls clients that try IPv6 first.

The server

create-droplet.sh prints a token for dana. On your Mac, export LNK_RELAY=https://relay.example.com LNK_TOKEN=<token>, then lnk tunnel open 3000 --public.

plugins/networking/relay/deploy/create-droplet.sh --domain relay.example.com --email you@example.com --user dana --reserved-ip <reserved ip>

Without DigitalOcean

bootstrap.sh works on any Ubuntu 24.04 server with a public IP. Run it as root from your checkout with --domain, --email and --user.

ssh root@<server ip> 'bash -s -- --domain relay.example.com --email you@example.com --user dana' \
  < plugins/networking/relay/deploy/bootstrap.sh

Roles

A relay does one job, its role (LINK_ROLE), so a flood of tunnels never takes down the machines' connections or the webhooks beside them. Each role runs on a domain of its own, which is also the audience its access tokens must name, so a token for one relay is refused by every other. They share the protocol, the account service's tokens, flow control and limits.

tunnels

The default. It serves <app>.<user>.<domain>, users' own domains and share links, and takes connections from any account the account service signs for, and from the users file.

remote

No public host: /_link/machines/<id>/<path> reaches <path> on the machine with that ID, for a person of the account it's signed in to. It takes connections from machines signed in to an account, each as itself, its tunnel named by its ID, with no --name, --auth, --account or --domain. It needs LINK_ACCOUNTS_URL.

hooks

No public host: /h/<path> on its domain goes, as it is, to its one tunnel. It takes only the tunnel LINK_HOOKS_TUNNEL names, <owner>/<tunnel>, from that owner, with no --auth, --account or --domain.

# a hooks relay serving one account and its tunnel named hooks
echo 'LINK_ROLE=hooks' >> /etc/link/relay.env
echo 'LINK_HOOKS_TUNNEL=acct-0123456789abcdef0123456789abcdef/hooks' >> /etc/link/relay.env
systemctl restart link-relay

A hooks relay's front

A hooks relay is a service's way in for webhooks, behind a front that sends it /h/* (the webhooks), and /_link/connect, /_link/accounts and /_link/health (the service's own connection), on a domain it shares with the service. It takes no LINK_WEBSITE, since its paths, a sender's GET included, are its tunnel's; and only plain requests, never a WebSocket, a dot segment or an encoded slash.

A front on another machine

A relay knows only its own connections, so a front sends a hooks relay's paths to the one its service is connected to. With the front not on the relay's machine, the relay listens where the front reaches it (LINK_LISTEN) and believes the front's X-Forwarded-For (LINK_TRUSTED_PROXIES, its address ranges), so its limits are kept by each sender, not by the front. Whoever sends to /h/ is checked by the service, by its secret or signature, as a tunnel's visitors are by its app.

A remote relay's tokens

A request to a remote relay carries a person's access token for it, Authorization: Bearer, granting machine.remote on machine:<id> and naming the account that machine is signed in to. The relay keeps the token to itself: the machine never sees it. A machine that isn't connected, or isn't the caller's, is answered 502 alike; a token that isn't one is answered 401. An account holds at most LINK_MACHINES_PER_USER machines on one remote relay, 100 unless set.

Browsers on a remote relay

Browsers reach a remote relay from the account service's origin (LINK_ACCOUNTS_URL, where its pages are) and no other. The relay answers their preflight itself, letting that origin send a token and a body, and on every answer it says that origin alone may read it, dropping whatever Access-Control-* headers the machine sent.

Its measures

A relay counts what only it sees, in Link's measures, and sends them as OTLP JSON to the endpoint LINK_MEASURES names, every LINK_MEASURES_EVERY (60s unless set), the first as it starts: https://, or http:// to this machine, such as an OpenTelemetry collector beside it. Without it, it sends none.

# a collector on the same machine
echo 'LINK_MEASURES=http://localhost:4318' >> /etc/link/relay.env
systemctl restart link-relay

Its credentials

Set LINK_MEASURES_HEADER_FILE to a private file of name: value headers when the endpoint needs credentials. Values are read as secrets and sent only to that endpoint. Without it, a send has no credential.

link.relay.connections.sum, requests.sum and bytes.sum

Agent connections held, including standbys, sampled as a gauge; public requests received, excluding internal relay paths; and the encoded payload bytes sent to and received from agents.

link.hooks.undelivered.sum

Webhooks a hooks relay couldn't hand on through its tunnel, reported by a hooks relay alone: the relay answered a 5xx itself, because the tunnel wasn't connected, was lost mid-request or didn't answer in time, or the relay was too busy. A 4xx it answers is the sender's, and an answer the tunnel's service gives, a 5xx too, was handed on.

How they're counted

Requests, bytes and undelivered hooks are running totals since the relay started (OTLP's cumulative sum, its start the relay's), so a send that's lost is made up by the next. A send that fails is logged, never retried.

Its unit

Its unit is the relay: kind relay, its ID the relay's domain (LINK_DOMAIN), its name its role. LINK_MEASURES_MACHINE names the hosting machine by permanent ID when several deployments share a domain. The relay then has ID <machine>/relay below that machine, so their counts remain separate.

Take logins from an account service

People then log in with lnk auth login, instead of you editing /etc/link/users. Local Link's relay uses Local Link's account service. An account service is any that answers Link's login API, here https://accounts.example.com.

KEY=$(curl -s https://accounts.example.com/api/v1/keys | jq -r '[.keys[] | "\(.kid):\(.x)"] | join(",")')
ssh root@relay.example.com "bash -s -- --accounts-url https://accounts.example.com --accounts-key $KEY" < plugins/networking/relay/deploy/bootstrap.sh

What it saves

This re-runs bootstrap, which keeps everything else, and saves LINK_ACCOUNTS_URL and LINK_ACCOUNTS_KEY in /etc/link/relay.env. The keys are the account service's public keys, each <kid>:<x>, which its access tokens are signed with; check them with the service's operator, not only over the network. When the service adds a key, run it again with both.

Logging in

On any machine, lnk auth login --relay https://relay.example.com then prints a link and a code, and you approve the machine in the browser, signed in to your account.

What the relay keeps

The relay keeps no logins and never calls the account service: each machine signs for access tokens there with its key, made for this relay and 5 minutes, which the relay checks with the public keys alone. A machine signed out there is refused within 5 minutes.

Usernames and visitors

An account that changes its username there takes the new name with its next access token; the old name is free for another account, whose first tunnel under it closes the old holder's. Anyone with an account there can open tunnels, and their tunnels show the visitor warning. What it checks is in the tunnel's security.

Tunnels that let in some people

Visitors to a tunnel that lets in only some people (lnk tunnel open --account) sign in through the same account service, which must list this relay among those it signs for. Without an account service, the relay refuses such tunnels.

Set up off-site backups

Every night the server snapshots its state (users, tokens, settings, certificates) into /var/backups/link and keeps 7. By default that's the only copy, so losing the server loses the users file. Set this up once, and each snapshot is also encrypted with age to your public key and uploaded to a bucket. The server can't read old backups, and neither can anyone with the bucket.

Any S3 bucket

Any bucket that speaks S3's API works: Google Cloud Storage, AWS S3, or another S3-compatible store. The snapshots are a few KB, so the cost is close to nothing.

A bucket and a key on Google Cloud

Make a bucket and a key that can write only to it, with gcloud (<bucket>, <project> and <region> are yours). The last command prints an HMAC key: its access id and secret are the key id and secret below, and the URL is https://storage.googleapis.com/<bucket>.

gcloud storage buckets create gs://<bucket> --project <project> --location <region> --uniform-bucket-level-access
gcloud iam service-accounts create link-backup --project <project>
gcloud storage buckets add-iam-policy-binding gs://<bucket> --member serviceAccount:link-backup@<project>.iam.gserviceaccount.com --role roles/storage.objectUser
gcloud storage hmac create link-backup@<project>.iam.gserviceaccount.com

A bucket and a key on AWS

Make the bucket and an IAM user allowed s3:PutObject and s3:GetObject on it alone. The URL is https://s3.<region>.amazonaws.com/<bucket>.

Expire old backups

Optionally, expire old backups after 90 days, with a lifecycle rule on the bucket. state-latest is rewritten nightly, so it never expires.

Save the settings on your Mac

Save them for restores. On AWS, LINK_BACKUP_REGION is the bucket's region, such as us-east-1.

mkdir -p ~/.config/link && cat > ~/.config/link/backup.env <<'EOF'
LINK_BACKUP_URL=https://storage.googleapis.com/<bucket>
LINK_BACKUP_REGION=auto
LINK_BACKUP_KEY_ID=<access id>
LINK_BACKUP_SECRET=<secret>
EOF
chmod 600 ~/.config/link/backup.env

Send them to the server, and test

Send the settings to the server with your public key (id_rsa.pub if that's your key). It should print uploaded state-….tar.gz.age.

scp ~/.config/link/backup.env root@relay.example.com:/etc/link/backup.env
scp ~/.ssh/id_ed25519.pub root@relay.example.com:/etc/link/backup-recipients
ssh root@relay.example.com "chmod 600 /etc/link/backup.env && systemctl start link-backup && journalctl -u link-backup -n 3 --no-pager"

Check you can restore

Fetch the latest backup, list it, and remove the copy. A failed upload fails the link-backup unit (systemctl --failed) and keeps the local snapshot.

plugins/networking/relay/deploy/fetch-backup.sh --env ~/.config/link/backup.env && tar -tzf link-state-latest.tar.gz && rm link-state-latest.tar.gz

Keep a second decryption key

Only the private halves of the keys in /etc/link/backup-recipients open the backups. So as not to depend on your Mac alone, run age-keygen -o link-backup.key, store that file in your password manager, and append its age1… public key as another line of /etc/link/backup-recipients. Restore with fetch-backup.sh --identity link-backup.key.

Rotate the release signing key

Every release's SHA256SUMS is signed. Servers, lnk upgrade and install.sh install nothing without a valid signature, so someone who gets into the releases repo or its token still can't push code.

Make the new key

Make it on your Mac and keep the private half in a password manager.

ssh-keygen -t ed25519 -N "" -C link-release -f ~/.ssh/link-release

Add its public key

Add it as a second line in scripts/release/release-signers and in RELEASE_SIGNERS= in both core/cli/install.sh and plugins/networking/relay/deploy/bootstrap.sh, in the form link-release namespaces="link-release" ssh-ed25519 AAAA… link-release. A test fails if they differ.

Release, then re-run bootstrap

Release, then re-run bootstrap from that checkout on each server (Deploy a change). It installs its keys in /etc/link/release-signers.

Sign with it

In so-local/link, Settings → Environments → release, set the environment secret RELEASE_SIGNING_KEY to the whole new private key file (cat ~/.ssh/link-release | pbcopy). Remove the old line in a later release.

If a key leaks

Remove its line at once, set a new key as above, release, then re-run bootstrap from that checkout on each server and have users reinstall. Bootstrap refuses a signed release until one signed by a key it still trusts replaces it.

On the server

What runs a relay on a server, and what it keeps there.

On a server, systemd runs link-relay.service with the settings in /etc/link/relay.env. With no flags it runs in dev mode, on 127.0.0.1:7080 with the single user dev; for development, see running it locally.

link-relay
link-relay --base-domain relay.example.com --users-file /etc/link/users --state-dir /var/lib/link
link-relay --version

What's on a server

link-state exports and imports the state, which is /etc/link (users, settings), /var/lib/private/link (blocks, share links, the hosts approved for certificates, the custom domains proven) and Caddy's certificates. link-backup runs link-state backup nightly. If you lose your SSH key, use the droplet's Recovery Console in DigitalOcean.