Decisions

Why buckets work the way they do.

The engine

Why rclone moves the files, and how it signs in and encrypts.

Why is rclone the engine?

rclone already reaches many backends, and its check and cryptcheck compare checksums, through encryption too. The plugin pins rclone 1.75.1, downloads it on first use, and installs it only if its SHA-256 matches the one in the plugin's source. It runs with its own config file and no RCLONE_* variables.

How does a bucket sign in to AWS or Google Cloud?

As its lnk cloud account, the one your boxes there use: the backend's adapter hands rclone the account's sign-in for each command, and rclone.conf keeps none of it. Buckets had signed in their own way, which meant a second set of bugs and a service account that worked for boxes but not buckets. Why, and how a Google Cloud account signs in: clouds and boxes' decisions.

Why is everything else rclone does passed through, not wrapped?

Link's commands stay the safe path, and rclone's power is one command away. backend create takes any rclone storage type and lets rclone ask its own questions, lnk bucket rclone runs rclone itself on the connected backends, and push and pull pass flags after --.

Why rclone's crypt for encryption?

It encrypts file names as well as contents, and cryptcheck keeps clean possible through the encryption. Encrypting with age before upload would do neither. It's set per backend at backend create, and always on for an agent's files.

Pushing and cleaning

Why push only adds, and what clean checks before it deletes.

Why does push never sync deletes or renames?

A sync that deletes is how backups get lost. push only adds, and replaces only with --replace; remove deletes from backends on purpose.

How are good copies protected from bad local files?

push refuses to replace a copy that differs from the file here. Link doesn't turn on bucket versioning instead: rclone can't set it up the same way on every backend, and it keeps paying for old versions.

When is a copy "checked"?

When its checksum matches the file here. Where the backend has no checksum, Link reads it byte for byte; through encryption, it uses cryptcheck. The manifest records these checks as a cache for tree: clean checks again every time, but for an S3 copy it read back or proved before, unchanged here and there since.

Why does clean read some S3 copies back?

On S3, rclone takes the MD5 of an object uploaded in parts from metadata its uploader set, which S3 never checks, so anyone who can write to the bucket could forge it. Reading every copy back would cost a full download on every clean, and would never work on archived copies.

Only those with the uploader's MD5

Link instead asks for each object's metadata and reads back only those carrying an MD5 there; every other S3 MD5 is the ETag, which S3 computed. rclone hides the ETag and drops that metadata from its own listing, so Link lists with rclone's --s3-directory-bucket, under which an object's MD5 shows only when it comes from the metadata. push keeps comparing checksums: it is not what deletes files.

Proved once

What a clean proved is kept, so a dry run and then a clean, or a clean tried again, don't download every large copy each time. The copy is known by the time S3 wrote it, from a listing that costs one request per thousand objects: S3 alone sets that time, so a copy replaced since shows another, and is checked again. The same listing, taken again after the check, catches a copy replaced while it ran; a clean that found every copy proved checks nothing, so it lists once.

Where it lists

It lists only where the files are: for each path given to clean, the folder its files share, so clean photos/2019 docs lists those two and not the whole bucket. Past eight such folders it lists the one they all share instead. Each backend is listed, checked and listed again in a thread of its own, as push sends.

Why does push skip files unchanged since their check?

Reading every file to hash it, once to send and again to check, on every push and for every backend, makes a push of a large folder take as long as the first one. push skips a file whose size and modification time match the manifest and whose copy there was checked, and checks only what it sent, like rsync without --checksum. push --recheck reads everything, and clean checks every file before deleting any, taking an S3 copy as proved only by the time S3 wrote it.

Why does push send to every backend at once?

A push to three backends shouldn't take three times as long. Each backend is sent and checked in a thread of its own, with rclone's progress as a line every few seconds after the backend's name, since its redrawn progress would mix. What only happens once, like a sign-in or the Keychain, is asked first, one at a time. The manifest is written after, one backend after another, so no backend's record overwrites another's.

Why is there no copy of the manifest in each backend?

The bucket itself is the truth. A new machine lists it (connect --bucket, tree --refresh) and loses only the "checked" times, which clean redoes regardless.

Buckets

Why each folder has a bucket of its own, and how buckets are named, made and removed.

Why a bucket per folder, not one bucket with a folder per part?

Every backend can limit a key to one bucket simply, but not to a prefix. A bucket per folder (~/Link/Personal, and each agent's files folder) also lets each have its own settings, so an agent's is always encrypted, and yours holds exactly your folder. The path picks the bucket. Your files' bucket is made at backend create; an agent's on its first push. Before each agent had a folder of its own, a single agent folder had a bucket; that bucket is now main's.

Why are bucket names random?

Bucket names are global and anyone can probe whether one exists. A new bucket is local-link- plus 12 random letters and digits, with no username or account name. On storage with folders, it's a local-link folder holding personal and one folder per agent ID.

Why are backend create and backend connect two commands?

A command named connect that also makes a bucket surprises people. And because backend connect stops when the bucket isn't there, a typo in --bucket never makes a new one. lnk cloud connect makes no bucket either; it prints the backend create command.

Why does backend delete delete everything in the buckets?

So that what Link made can be removed with Link. It still refuses, without --everywhere, when a file's only copy is there. backend disconnect stays the way to forget a backend and keep its files.

Why are a backend's buckets backend delete, and a file's copies remove?

A bucket Link made goes out of existence with delete, and a file's copy in a backend is a connection between two things, ended with remove (the grammar). Each refuses the other's argument, before anything is deleted: backend delete refuses a path, and remove a connected backend's name that is no file of yours, and each says which command you meant.

Agents' files

How an agent's bucket is reached from a box, found from another computer, and restored.

How does a box back up an agent's files?

With a key that reaches only its agent's bucket: an IAM user or a service account per bucket, made through your account when the agent arrives, and replaced or revoked when it leaves. A box holds no account of yours, so a box broken into reaches that one bucket, still encrypted with the agent's password.

Why doesn't your own computer get a key per bucket?

It holds your cloud accounts themselves (lnk cloud), next to the buckets' keys, so a key that reaches only one bucket wouldn't narrow what someone who took those files could reach. A box holds no account, so there it does.

How does another computer find an agent's bucket?

By labels on the bucket: the agent's ID and name, and your account's owner ID, found through lnk cloud. No list is kept anywhere; the bucket is its own record, as a box is by its tags.

How does an agent's password move with it?

It's carried in the move's stream, as rclone keeps it, and removed from the machine the agent left. There's one password per agent, whichever backend or machine, and it never goes on a command line.

Why does the bucket plugin restore an agent's buckets?

Finding an agent's buckets by their labels, asking their password, making a key and putting them in place is bucket knowledge, so lnk bucket restore does all of it and pulls the files, and lnk agent restore runs that one command before unpacking the harness's state. The harness never builds what lnk bucket adopt reads, and a change to how buckets are found or adopted stays in the bucket plugin.

Where a share link's files come from, where it lives, and what ends it.

Straight from your backend, through URLs it signs on your machine (S3 presigned URLs, through rclone's link). The relay holds only the URLs, so files never pass through Local Link and backend keys never leave the machine. That's also why a link lasts at most 7 days, S3's longest signature.

Why can only S3 storage share?

Share links need signed URLs that expire. rclone can't sign Google Cloud Storage URLs, which need a service account's private key, and other storage's links, such as Dropbox or Drive, don't expire. So only aws and s3 backends share. Encrypted buckets are refused, an agent's always, because a link would hand out ciphertext.

On your own host, https://<user>.<domain>/b/<id>, with an id of 24 random letters and digits. Who shared it is in the address, and one host per user holds their links; it gets a certificate only while they have one.

Why do some share pages show the visitor warning?

A share page is the relay's, under its name, and links to whatever URLs the sharer sent. The relay takes only URLs shaped like S3's presigned ones, but S3-compatible storage can be any host, so it can't tell storage from a site dressed as it. So an account's share page shows the visitor warning, as its tunnels do.

Through lnk auth token, a short token for the share API: the relay takes one for at most 15 minutes from its making. It doesn't read the tunnel plugin's saved login token: plugins never read each other's files, and the bucket plugin never holds a token that runs tunnels.

As for tunnels: --auth, checked by the relay with HTTP Basic, which browsers and lnk bucket pull both speak. The relay keeps only its hash.

The link on the relay. URLs already handed out can't be recalled, since a presigned URL is valid until it expires. So the page says how long it lasts, and --expires can make a share short.