Decisions

Why buckets work the way they do.

The engine

Why rclone moves the files, and how it signs in and encrypts.

Why is rclone the engine?

rclone already reaches many clouds, and its check and cryptcheck compare checksums, through encryption too. The module pins rclone 1.75.1, downloads it on first use, and installs it only if its SHA-256 matches the one in the module's source. It runs with its own config file and no RCLONE_* variables.

How does a bucket sign in to AWS or Google Cloud?

As its lnk cloud account, the one your boxes there use: the cloud's adapter hands rclone the account's sign-in for each command, and rclone.conf keeps none of it. Buckets had signed in their own way, which meant a second set of bugs and a service account that worked for boxes but not buckets. Why, and how a Google Cloud account signs in: clouds and boxes' decisions.

Why is everything else rclone does passed through, not wrapped?

Link's commands stay the safe path, and rclone's power is one command away. lnk storage create takes any rclone storage type and lets rclone ask its own questions, lnk storage rclone runs rclone itself on the connected clouds, and push and pull pass flags after --.

Why rclone's crypt for encryption?

It encrypts file names as well as contents, and cryptcheck keeps clean possible through the encryption. Encrypting with age before upload would do neither. It's set per cloud at lnk storage create, and always on for an owner's folder.

Pushing and cleaning

Why push only adds, and what clean checks before it deletes.

Why does push never sync deletes or renames?

A sync that deletes is how backups get lost. push only adds, and replaces only with --replace; remove deletes from clouds on purpose.

How are good copies protected from bad local files?

push refuses to replace a copy that differs from the file here. Link doesn't turn on bucket versioning instead: rclone can't set it up the same way on every cloud, and it keeps paying for old versions.

When is a copy "checked"?

When its checksum matches the file here. Where the cloud has no checksum, Link reads it byte for byte; through encryption, it uses cryptcheck. The manifest records these checks as a cache for tree: clean checks again every time, but for an S3 copy it read back or proved before, unchanged here and there since.

Why does clean read some S3 copies back?

On S3, rclone takes the MD5 of an object uploaded in parts from metadata its uploader set, which S3 never checks, so anyone who can write to the bucket could forge it. Reading every copy back would cost a full download on every clean, and would never work on archived copies.

Only those with the uploader's MD5

Link instead asks for each object's metadata and reads back only those carrying an MD5 there; every other S3 MD5 is the ETag, which S3 computed. rclone hides the ETag and drops that metadata from its own listing, so Link lists with rclone's --s3-directory-bucket, under which an object's MD5 shows only when it comes from the metadata. push keeps comparing checksums: it is not what deletes files.

Proved once

What a clean proved is kept, so a dry run and then a clean, or a clean tried again, don't download every large copy each time. The copy is known by the time S3 wrote it, from a listing that costs one request per thousand objects: S3 alone sets that time, so a copy replaced since shows another, and is checked again. The same listing, taken again after the check, catches a copy replaced while it ran; a clean that found every copy proved checks nothing, so it lists once.

Where it lists

It lists only where the files are: for each path given to clean, the folder its files share, so clean photos/2019 docs lists those two and not the whole bucket. Past eight such folders it lists the one they all share instead. Each cloud is listed, checked and listed again in a thread of its own, as push sends.

Why does push skip files unchanged since their check?

Reading every file to hash it, once to send and again to check, on every push and for every cloud, makes a push of a large folder take as long as the first one. push skips a file whose size and modification time match the manifest and whose copy there was checked, and checks only what it sent, like rsync without --checksum. push --recheck reads everything, and clean checks every file before deleting any, taking an S3 copy as proved only by the time S3 wrote it.

Why does push send to every cloud at once?

A push to three clouds shouldn't take three times as long. Each cloud is sent and checked in a thread of its own, with rclone's progress as a line every few seconds after the cloud's name, since its redrawn progress would mix. What only happens once, like a sign-in or the Keychain, is asked first, one at a time. The manifest is written after, one cloud after another, so no cloud's record overwrites another's.

Why is there no copy of the manifest in each cloud?

The bucket itself is the truth. A new machine lists it (connect --bucket, tree --refresh) and loses only the "checked" times, which clean redoes regardless.

Buckets

Why each folder has a bucket of its own, and how buckets are named, made and removed.

Why a bucket per folder, not one bucket with a folder per part?

Every cloud can limit a key to one bucket simply, but not to a prefix. A bucket per folder (~/Link/Files, and each owner's folder) also lets each have its own settings, so an owner's is always encrypted, and yours holds exactly your folder. The path picks the bucket. Your files' bucket is made at lnk storage create; an owner's on its first push. Either is made only after a yes, since the cloud bills it.

Why are bucket names random?

Bucket names are global and anyone can probe whether one exists. A new bucket is lnk-bucket- plus 12 random letters and digits, with no username or account name. On storage with folders, it's a local-link folder holding personal and one folder per owner, by its ID's name-safe form, <kind>-<id>.

Why are lnk storage create and lnk storage connect two commands?

A command named connect that also makes a bucket surprises people. And because lnk storage connect stops when the bucket isn't there, a typo in --bucket never makes a new one. lnk cloud connect makes no bucket either; it prints the lnk storage create command.

Why does lnk storage delete delete everything in the buckets?

So that what Link made can be removed with Link. It still refuses, without --everywhere, when a file's only copy is there. lnk storage disconnect stays the way to forget a cloud and keep its files.

Why are a cloud's buckets lnk storage delete, and a file's copies remove?

A bucket Link made goes out of existence with delete, and a file's copy in a cloud is a connection between two things, ended with remove (the grammar). Each refuses the other's argument, before anything is deleted: lnk storage delete refuses a path, and remove a connected cloud's name that is no file of yours, and each says which command you meant.

Owners' folders

Why lnk storage knows owners only by their IDs, and how an owner's bucket is reached from a box, found from another computer, and restored.

Why owners, not the modules that keep them?

The storage module is built on by others, never the reverse: what another module keeps reaches it only as an answer to its own questions, in its own types (Storage Contract). An owner is a stable ID, <kind>:<id>, on the one generic every kind of ID rides, so any module can name what it keeps, and a new kind needs no change here. The kind is data: lnk storage never branches on it.

Why is a bucket by its owner's ID, not its name?

A name can change, and two owners can share one over time; an ID never does. A person still types a name, --owner <kind>:<name>, which the module that keeps the kind turns into the ID (storage.owner), and only the ID is kept. A program that holds the ID passes it directly.

Why don't settings commands ask which folders are owners'?

A module answers storage.folders so a push never sweeps an owner's folder into your bucket, and clean never deletes files an owner uses. That protection matters only where files are touched. list, show, create, connect, disconnect and delete change only clouds.toml, which keys each bucket by its folder's key. Asking would couple them to every other module, so one broken module stopped them all, even for someone with no owners. An owner is named there by its ID, never from a second store of names.

Why do file commands go on without an answer?

A module that fails is warned of and its folders left out, rather than stopping a push, pull, tree or clean of your own files. An owner's folder still needs its own module's answer to be reached at all.

Why one name-safe form of an owner's ID?

An ID holds a :, which an rclone section, a file's name or a bucket's label can't. One function writes <kind>-<id> for all of them, and reads it back, so the same owner has the same name in each, and two owners never share one: lnk storage takes only IDs whose id is lowercase letters and digits, so the last - splits the form one way.

How does a box back up an owner's folder?

With a key that reaches only its owner's bucket: an IAM user or a service account per bucket, made through your account when the folder arrives, and replaced or revoked when it leaves. A box holds no account of yours, so a box broken into reaches that one bucket, still encrypted with the owner's password.

Why doesn't your own computer get a key per bucket?

It holds your cloud accounts themselves (lnk cloud), next to the buckets' keys, so a key that reaches only one bucket wouldn't narrow what someone who took those files could reach. A box holds no account, so there it does.

How does another computer find an owner's bucket?

By labels on the bucket: the owner's ID and name, and your account's owner ID, found through lnk cloud. No list is kept anywhere; the bucket is its own record, as a box is by its tags.

How does an owner's password move with its folder?

The module that keeps the owner carries it, as rclone keeps it, from export to adoption, and the machine the folder left releases its local record. There's one password per owner, whichever cloud or machine, and it never goes on a command line.

Why does the storage module restore an owner's buckets?

Finding an owner's buckets by their labels, asking their password, making a key and putting them in place is bucket knowledge. The storage restore interface does that work and pulls the files before the owning module restores its part. The caller does not build storage's adoption payload, and a change to how buckets are found or adopted stays in the storage module.