Buckets keep your folders in your backends

Your folders, in your backends. Back up your files to every backend you connect, check every copy, and free the space here once they're safe.

lnk up buckets                          # install the bucket plugin
lnk bucket backend create aws                   # a bucket in your aws account, connecting it first if needed
lnk bucket push photos/2019             # ~/Link/Personal/photos/2019 to every backend, then checked
lnk bucket tree --level 2               # what is where, per backend, even offline
lnk bucket clean photos/2019            # delete here, only if every backend has a checked copy
lnk bucket pull 'photos/2019/**/*.jpg'  # back from the cheapest backend, checked

What it needs

Nothing runs in the background, and your files never pass through Local Link's servers. You need a Mac or Linux and an account with a backend: AWS, Google Cloud, any S3 storage, or an external drive. lnk up buckets adds the plugin. Your backend bills storage and downloads, and share links also need a Local Link account (lnk auth login).

Its paths

Paths are in ~/Link/Personal or an agent's files folder (~/Link/Agents/<name>/Home). A relative path starts from the current folder when you're in one of them, and from ~/Link/Personal otherwise. pull, remove and link create take globs, so quote them and use ** to span folders.

Connect a backend

Connect as many backends as you like, and push copies to every one.

lnk bucket backend create aws --profile work       # connects that ~/.aws login as the account aws, then makes the bucket
lnk bucket backend create gcp                      # signs you in with gcloud, in the browser, needs gcloud
lnk bucket backend create gcp --cloud gcp-ci       # as the account gcp-ci, a service account for one
lnk bucket backend create s3 --endpoint https://0123abcd.r2.cloudflarestorage.com --key-id AKIAEXAMPLE
lnk bucket backend create folder --path /Volumes/MyDrive  # an external drive
lnk bucket backend create b2                       # any rclone storage, and rclone asks what it needs
lnk bucket backend list                          # what is connected, and what each holds

What backend create makes

backend create makes a private bucket for your files, once per backend, and tests the keys. If a test fails after the bucket is made, it deletes the bucket again, or says which one it couldn't.

AWS and Google Cloud

On AWS and Google Cloud a bucket signs in as a cloud account from lnk cloud connect, the same one your boxes there use. backend create connects it first when there's none, asking what lnk cloud connect asks. A service account connects there, with its key: lnk cloud connect gcp --name gcp-ci --credential-file key.json.

--encrypt

Add --encrypt to encrypt your files and their names before upload. Link makes a password and shows it once, and if you lose it, nobody can read the files.

Back up

push sends files to every backend at once and checks every copy it sent. It skips files unchanged since their copy was checked, never deletes anything in a backend, and never replaces a copy that differs from the file here unless you pass --replace.

lnk bucket push photos/2019
lnk bucket push photos --backend aws -- --bwlimit 10M --transfers 8

Back up each agent's files

Each agent's files folder goes to a bucket of its own in every backend, made on its first push and always encrypted, with one password per agent. The same commands work on it, but pull and clean refuse while the agent runs, or when Link can't tell, so stop it first.

lnk bucket push ~/Link/Agents/work/Home
lnk agent stop --agent work && lnk bucket clean ~/Link/Agents/work/Home

When it moves

An agent's buckets move with it (agents). On a box, it backs up with a key that reaches only its bucket. If its machine is gone, lnk agent restore finds its bucket in your lnk cloud accounts.

Keep an agent's conversations and notes in its bucket

An agent keeps its collections (instructions, skills, memory, conversations, schedule) where its spec prefers, among the places its machine supports (Agent Spec). With bucket, they're objects in its encrypted bucket, under .link/agent/, apart from its files: any machine connected to the bucket reads them as it left them.

# ~/.lnk/agents/work/agent.toml
[storage]
conversations = "bucket"

Memory's index

With memory = "bucket", memory's index is kept there too, and each machine keeps the parts of it it read (~/.lnk/bucket/segments/), so fetching it again downloads only what changed.

A hosted team's folder

Push and pull also discover the owned hosted teams, as the harness plugin answers bucket.folders (Bucket Contract). Their local folder is ~/Link/Teams/<team>/Shared; their permanent bucket-folder ID is qualified as team:<id> so it cannot name an agent's Home.

How it's pushed and pulled

The harness plugin pushes and pulls it (bucket.home) through its hosting adapter, as it does a hosted agent's Home: the same versioned, hashed files stream. Anyone who can talk to a member can receive the shared files it reads; an outside member's owner can read those conversations.

Get files back

pull copies each file from the cheapest backend that has it and checks it. Downloading from AWS or Google Cloud costs about $90 per TB, so it asks before an archive restore or more than 1 GB of charged downloads. It doesn't overwrite a file here that differs, unless you pass --replace.

lnk bucket pull 'photos/2019/**/*.jpg'
lnk bucket pull photos/2019 --backend gcp

See what's where

Each file gets one mark per backend: ✓ checked, A checked in an archive tier, ? not checked, ≠ differs, ✗ missing. tree works offline, and --refresh finds files pushed from another machine.

lnk bucket tree --level 2
lnk bucket tree photos --refresh   # list the backends again first

Use a bucket from another machine

backend connect takes the same options as backend create but makes nothing, and stops if the bucket isn't there. lnk bucket backend list --json on the first machine shows each bucket's name.

lnk bucket backend connect aws --bucket local-link-a1b2c3d4e5f6
lnk bucket backend connect folder --path /Volumes/MyDrive --bucket local-link/personal
lnk bucket backend connect aws --bucket lnk-files-0123 --agent-bucket lnk-agents-0123 --agent main  # asks for the password of the agent

Free up space

clean checks every backend again, now, and deletes the files here only if every backend has a copy with the same checksum. Otherwise it says what's missing where. It's the only Link command that deletes your files here.

lnk bucket clean photos/2019 --dry   # say what would be deleted
lnk bucket clean photos/2019

Archive old files

Archive tiers cost about $1 per TB a month instead of about $20, and reading files back costs money. They exist on AWS (S3 Glacier Deep Archive) and Google Cloud (GCS Archive), and lnk asks before archiving or restoring.

lnk bucket push old/scans --archive

Restores and early deletes

On AWS, pull requests a restore that takes up to 12 hours, so run it again once it's done. Deleting early is charged anyway, for 180 days on AWS and 365 on Google Cloud, and lnk bucket remove and lnk bucket backend delete count. clean works on archived files without a restore, except a file over 200 MiB on AWS, whose copy it reads back (why).

Anyone with the link can open it in a browser for up to 7 days. The files download straight from your backend, never through Local Link.

lnk bucket link create 'photos/2019/*.jpg'                   # prints https://USER.DOMAIN/b/ID
lnk bucket link create notes.pdf --expires 12h --auth
lnk bucket link list                                      # your links, from every machine
lnk bucket link delete k3j2h4g5f6d7s8a9q1w2e3r4            # end one now

--expires, --auth and --backend

--expires <time> is 30m, 12h, 3d, at most 7d, the default. With --auth, visitors need a user:password, which it asks for, or reads from LNK_AUTH: never the command line, which other users can see. --backend <name> links only to copies in that backend.

A link shares only your files in ~/Link/Personal, from AWS and S3 backends, outside archive tiers. It uses what the manifest knows, so run tree --refresh first for files pushed from elsewhere.

Its paths and its end

A file keeps its path from where the pattern starts, so photos/2019/*.jpg shares beach.jpg and photos/2019 shares 2019/beach.jpg. link delete can't recall URLs already handed out, and they work until the link would have expired.

Without the accounts plugin

A link lives on your relay, signed for by the accounts plugin. A release of Link without that plugin has no share links, and lnk bucket link create says so plainly rather than offering to install it, while backups, pushes and pulls work as ever.

Download a share

To download a share with Link installed, pull its URL: it saves the files into the current folder, each checked against its size.

lnk bucket pull --link https://<user>.<domain>/b/0123456789abcdef01234567

Disconnect or delete

Nothing here deletes a file's last copy unless you pass --everywhere.

Remove from your backends

remove deletes the copies in every backend, or only in those you give with --backend, and keeps the files here. It takes only files and folders, and refuses a backend's name.

lnk bucket remove old/drafts --backend gcp

A backend that's full

A backend whose account reads only (past what it may keep) takes nothing more: push says so before sending anything, with the backend's own words on making room, and remove asks the backend to delete the copies itself, so you can free space with the same command.

Disconnect or delete a backend

backend disconnect prints the backend connect command that brings the backend back, and asks only when the backend holds a file's only copy. backend delete deletes the buckets lnk bucket backend create made, your files' and your agents', and on a folder backend Link's folders in it. It asks first, and refuses a path.

lnk bucket backend disconnect gcp   # forget it and its keys, and delete nothing in it
lnk bucket backend delete aws       # delete its buckets and everything in them, then forget it

Use rclone directly

This is rclone itself, with each connected backend as a remote named after it, for what lnk bucket doesn't do, such as mount, sync and serve. Link's safety checks don't apply here, and sync deletes. Run lnk bucket tree --refresh afterwards to catch up.

lnk bucket rclone lsd aws:
lnk bucket rclone mount aws: ~/mnt

Where your settings live

Everything is in ~/.lnk/bucket/: which backends are connected, their keys and encryption passwords, and what went where. On a Mac, the keys and passwords Link writes are in the Keychain instead.

Removing it

lnk plugin remove bucket and lnk uninstall keep this folder, and neither deletes anything in your backends. lnk uninstall --delete buckets deletes it, along with the only copy of the encryption passwords Link made, so save them first.

More about it

Where the rest of what lnk bucket does is written:

  • Every flag is in lnk bucket <command> --help.
  • Security has what's protected and what isn't.
  • Decisions has why it works this way.
  • Troubleshooting has what to do when a command refuses.

Docs

  • Configuration: The files lnk bucket keeps in ~/.lnk/bucket/, and what each one holds
  • Interface: What other programs can rely on from lnk bucket
  • Security: What buckets protect: your files and their copies in your own backends, against losing the last good copy, and the keys to those backends
  • Decisions: Why buckets work the way they do
  • Troubleshooting: What to do when a lnk bucket command refuses or misses files (Your folders, in your backends)