Buckets keep your folders in your backends
Your folders, in your backends. Back up your files to every backend you connect, check every copy, and free the space here once they're safe.
lnk up buckets # install the bucket plugin
lnk bucket backend create aws # a bucket in your aws account, connecting it first if needed
lnk bucket push photos/2019 # ~/Link/Personal/photos/2019 to every backend, then checked
lnk bucket tree --level 2 # what is where, per backend, even offline
lnk bucket clean photos/2019 # delete here, only if every backend has a checked copy
lnk bucket pull 'photos/2019/**/*.jpg' # back from the cheapest backend, checkedWhat it needs
Nothing runs in the background, and your files never pass through Local
Link's servers. You need a Mac or Linux and an account with a backend: AWS,
Google Cloud, any S3 storage, or an external drive. lnk up buckets
adds the plugin. Your backend bills storage and downloads, and share links
also need a Local Link account (lnk auth login).
Its paths
Paths are in ~/Link/Personal or an agent's files folder
(~/Link/Agents/<name>/Home). A relative path starts from the current
folder when you're in one of them, and from ~/Link/Personal otherwise.
pull, remove and link create take globs, so quote them and use ** to
span folders.
Connect a backend
Connect as many backends as you like, and push copies to every one.
lnk bucket backend create aws --profile work # connects that ~/.aws login as the account aws, then makes the bucket
lnk bucket backend create gcp # signs you in with gcloud, in the browser, needs gcloud
lnk bucket backend create gcp --cloud gcp-ci # as the account gcp-ci, a service account for one
lnk bucket backend create s3 --endpoint https://0123abcd.r2.cloudflarestorage.com --key-id AKIAEXAMPLE
lnk bucket backend create folder --path /Volumes/MyDrive # an external drive
lnk bucket backend create b2 # any rclone storage, and rclone asks what it needs
lnk bucket backend list # what is connected, and what each holdsWhat backend create makes
backend create makes a private bucket for your files, once per backend, and
tests the keys. If a test fails after the bucket is made, it deletes the
bucket again, or says which one it couldn't.
AWS and Google Cloud
On AWS and Google Cloud a bucket signs in as a cloud account from
lnk cloud connect,
the same one your boxes there use. backend create connects it first when
there's none, asking what lnk cloud connect asks. A service account
connects there, with its key: lnk cloud connect gcp --name gcp-ci --credential-file key.json.
--encrypt
Add --encrypt to encrypt your files and their names before upload.
Link makes a password and shows it once, and if you lose it, nobody can
read the files.
Back up
push sends files to every backend at once and checks every copy it sent.
It skips files unchanged since their copy was checked, never deletes
anything in a backend, and never replaces a copy that differs from the
file here unless you pass --replace.
lnk bucket push photos/2019
lnk bucket push photos --backend aws -- --bwlimit 10M --transfers 8Back up each agent's files
Each agent's files folder goes to a bucket of its own in every backend,
made on its first push and always encrypted, with one password per
agent. The same commands work on it, but pull and clean refuse
while the agent runs, or when Link can't tell, so stop it first.
lnk bucket push ~/Link/Agents/work/Home
lnk agent stop --agent work && lnk bucket clean ~/Link/Agents/work/HomeWhen it moves
An agent's buckets move with it
(agents).
On a box, it backs up with a key that reaches only its bucket. If its
machine is gone, lnk agent restore finds its bucket in your lnk cloud accounts.
Keep an agent's conversations and notes in its bucket
An agent keeps its collections (instructions, skills, memory,
conversations, schedule) where its spec prefers, among the places its
machine supports (Agent
Spec).
With bucket, they're objects in its encrypted bucket, under
.link/agent/, apart from its files: any machine connected to the
bucket reads them as it left them.
# ~/.lnk/agents/work/agent.toml
[storage]
conversations = "bucket"Memory's index
With memory = "bucket", memory's index is kept there too, and each
machine keeps the parts of it it read (~/.lnk/bucket/segments/), so
fetching it again downloads only what changed.
A hosted team's folder
Push and pull also discover the owned hosted teams, as the harness
plugin answers bucket.folders (Bucket
Contract).
Their local folder is ~/Link/Teams/<team>/Shared; their permanent
bucket-folder ID is qualified as team:<id> so it cannot name an
agent's Home.
How it's pushed and pulled
The harness plugin pushes and pulls it (bucket.home) through its
hosting adapter, as it does a hosted agent's Home: the same versioned,
hashed files stream. Anyone who can talk to a member can receive the
shared files it reads; an outside member's owner can read those
conversations.
Get files back
pull copies each file from the cheapest backend that has it and checks
it. Downloading from AWS or Google Cloud costs about $90 per TB, so it
asks before an archive restore or more than 1 GB of charged downloads.
It doesn't overwrite a file here that differs, unless you pass
--replace.
lnk bucket pull 'photos/2019/**/*.jpg'
lnk bucket pull photos/2019 --backend gcpSee what's where
Each file gets one mark per backend: ✓ checked, A checked in an
archive tier, ? not checked, ≠ differs, ✗ missing. tree works
offline, and --refresh finds files pushed from another machine.
lnk bucket tree --level 2
lnk bucket tree photos --refresh # list the backends again firstUse a bucket from another machine
backend connect takes the same options as backend create but makes nothing, and
stops if the bucket isn't there. lnk bucket backend list --json on the first
machine shows each bucket's name.
lnk bucket backend connect aws --bucket local-link-a1b2c3d4e5f6
lnk bucket backend connect folder --path /Volumes/MyDrive --bucket local-link/personal
lnk bucket backend connect aws --bucket lnk-files-0123 --agent-bucket lnk-agents-0123 --agent main # asks for the password of the agentFree up space
clean checks every backend again, now, and deletes the files here only
if every backend has a copy with the same checksum. Otherwise it says
what's missing where. It's the only Link command that deletes your files
here.
lnk bucket clean photos/2019 --dry # say what would be deleted
lnk bucket clean photos/2019Archive old files
Archive tiers cost about $1 per TB a month instead of about $20, and
reading files back costs money. They exist on AWS (S3 Glacier Deep
Archive) and Google Cloud (GCS Archive), and lnk asks before archiving
or restoring.
lnk bucket push old/scans --archiveRestores and early deletes
On AWS, pull requests a restore that takes up to 12 hours, so run it
again once it's done. Deleting early is charged anyway, for 180 days on
AWS and 365 on Google Cloud, and lnk bucket remove and lnk bucket backend delete count.
clean works on archived files without a restore, except a file over
200 MiB on AWS, whose copy it reads back
(why).
Share files with a link
Anyone with the link can open it in a browser for up to 7 days. The files download straight from your backend, never through Local Link.
lnk bucket link create 'photos/2019/*.jpg' # prints https://USER.DOMAIN/b/ID
lnk bucket link create notes.pdf --expires 12h --auth
lnk bucket link list # your links, from every machine
lnk bucket link delete k3j2h4g5f6d7s8a9q1w2e3r4 # end one now--expires, --auth and --backend
--expires <time> is 30m, 12h, 3d, at most 7d, the default.
With --auth, visitors need a user:password, which it asks for, or
reads from LNK_AUTH: never the command line, which other users can
see. --backend <name> links only to copies in that backend.
What a link shares
A link shares only your files in ~/Link/Personal, from AWS and S3
backends, outside archive tiers. It uses what the manifest knows, so run
tree --refresh first for files pushed from elsewhere.
Its paths and its end
A file keeps its path from where the pattern starts, so
photos/2019/*.jpg shares beach.jpg and photos/2019 shares
2019/beach.jpg. link delete can't recall URLs already handed out, and
they work until the link would have expired.
Without the accounts plugin
A link lives on your relay, signed for by the accounts plugin. A
release of Link without that plugin has no share links, and lnk bucket link create says so plainly rather than offering to install it, while backups,
pushes and pulls work as ever.
Download a share
To download a share with Link installed, pull its URL: it saves the
files into the current folder, each checked against its size.
lnk bucket pull --link https://<user>.<domain>/b/0123456789abcdef01234567Disconnect or delete
Nothing here deletes a file's last copy unless you pass --everywhere.
Remove from your backends
remove deletes the copies in every backend, or only in those you give
with --backend, and keeps the files here. It takes only files and
folders, and refuses a backend's name.
lnk bucket remove old/drafts --backend gcpA backend that's full
A backend whose account reads only (past what it may keep) takes nothing
more: push says so before sending anything, with the backend's own
words on making room, and remove asks the backend to delete the copies
itself, so you can free space with the same command.
Disconnect or delete a backend
backend disconnect prints the backend connect command that brings the
backend back, and asks only when the backend holds a file's only copy.
backend delete deletes
the buckets lnk bucket backend create made, your files' and your agents', and
on a folder backend Link's folders in it. It asks first, and refuses a
path.
lnk bucket backend disconnect gcp # forget it and its keys, and delete nothing in it
lnk bucket backend delete aws # delete its buckets and everything in them, then forget itUse rclone directly
This is rclone itself, with each connected backend as a remote named after
it, for what lnk bucket doesn't do, such as mount, sync and
serve. Link's safety checks don't apply here, and sync deletes. Run
lnk bucket tree --refresh afterwards to catch up.
lnk bucket rclone lsd aws:
lnk bucket rclone mount aws: ~/mntWhere your settings live
Everything is in ~/.lnk/bucket/: which backends are connected, their
keys and encryption passwords, and what went where. On a Mac, the keys
and passwords Link writes are in the Keychain instead.
Removing it
lnk plugin remove bucket and lnk uninstall keep this folder, and
neither deletes anything in your backends. lnk uninstall --delete buckets deletes it, along with the only copy of the encryption
passwords Link made, so save them first.
More about it
Where the rest of what lnk bucket does is written:
- Every flag is in
lnk bucket <command> --help. - Security has what's protected and what isn't.
- Decisions has why it works this way.
- Troubleshooting has what to do when a command refuses.
Docs
- Configuration: The files
lnk bucketkeeps in~/.lnk/bucket/, and what each one holds - Interface: What other programs can rely on from
lnk bucket - Security: What buckets protect: your files and their copies in your own backends, against losing the last good copy, and the keys to those backends
- Decisions: Why buckets work the way they do
- Troubleshooting: What to do when a
lnk bucketcommand refuses or misses files (Your folders, in your backends)