Storage Guide
Your folders, in your clouds. Back up your files to every cloud you connect, check every copy, and free the space here once they're safe.
lnk uplnk storage create awslnk storage push photos/2019lnk storage list --tree --level 2lnk storage clean photos/2019lnk storage pull 'photos/2019/**/*.jpg'Its paths
Paths are in ~/Link/Files or an owner's folder (Back up an
owner's folder). A relative path starts from the current
folder when you're in one of them, and from ~/Link/Files otherwise.
pull and remove take globs, so quote them and use ** to
span folders.
Connect a cloud
Connect as many clouds as you like, and push copies to every one.
lnk storage create aws --profile worklnk storage create gcplnk storage create gcp --cloud gcp-cilnk storage create s3 --endpoint https://0123abcd.r2.cloudflarestorage.com --key-id AKIAEXAMPLElnk storage create folder --path /Volumes/MyDrivelnk storage create b2lnk storage listAnother rclone storage type
Any storage type rclone knows, such as b2 for Backblaze B2, connects
by its type. --setting KEY=VALUE gives one of that type's own
settings, once for each, and rclone asks for the rest, its secrets among
them, so none is typed on the command line.
lnk storage create b2 --setting hard_delete=trueWhat lnk storage create makes
lnk storage create makes a private bucket for your files, once per cloud, and
tests the keys. If a test fails after the bucket is made, it deletes the
bucket again, or says which one it couldn't.
Say yes to a bucket
The cloud bills a bucket for what it stores, so Link asks before making
one, at lnk storage create and at the first lnk storage push that
needs one. --yes answers yes ahead. With no terminal and no --yes,
the command makes nothing and names --yes. A folder cloud, or an
rclone type without buckets, gets a folder, which no cloud bills, so
nothing is asked of it.
lnk storage create gcp --yesSee your clouds
lnk storage list shows a section for each cloud, headed by its name,
type and where it is. Its table has a row for each folder it holds,
your files or an owner's, with the bucket, its encryption, and how many
files it holds and their size. lnk storage show <name> shows one
cloud: its name, type and where it is, then a section for each folder
it holds. Each section has the bucket and its ID, its encryption, where
its password is kept, its name in rclone, its files and size, and when
a copy there last matched the file here. Both print JSON only with
--json (Interface).
lnk storage listlnk storage show awsAWS and Google Cloud
On AWS and Google Cloud a storage cloud is a cloud account from
lnk cloud connect,
the same one your boxes there use, and is named as it. --cloud picks
the account, by default the one named aws or gcp. lnk storage create connects it first when there's none, asking what lnk cloud connect asks. A service account connects there, with its key: lnk cloud connect gcp --name gcp-ci --credential-file key.json. Every
other storage is a remote, named by --name, which --remote picks
in push, pull and remove.
Name the bucket
lnk storage create names each bucket it makes lnk-bucket- and 12
random letters and digits, and takes no name of yours
(Why are bucket names random?).
To use a name you chose, make the bucket in your cloud first, then
connect it. On Google Cloud, gcloud storage buckets create makes one,
and lnk storage connect adds it.
gcloud storage buckets create gs://my-files-0123 --location USlnk storage connect gcp --bucket my-files-0123A name is 3 to 63 lowercase letters, digits, - and ., starting and
ending with a letter or digit, and unique across the whole cloud.
--region and, on Google Cloud, --project say where lnk storage create makes a bucket.
Encrypt your files
With --encrypt, your files and their names are encrypted on this
machine before they're sent, so your cloud provider and anyone else
with access to the bucket hold only unreadable data. Only a machine
with the bucket's encryption password reads them back.
What encrypts them
Link encrypts with rclone's crypt, the encryption layer of the tool
that moves the files (Why rclone's crypt?).
Each file's contents are sealed with XSalsa20-Poly1305, which both
hides them and detects any change, and each name in its path is
encrypted too. File sizes and modification times stay visible to the
cloud.
Choose it when the bucket is made
Encryption is set per bucket when you create or connect it, and stays
as it was set. A bucket that already holds files without encryption
can't be switched over: connecting it with --encrypt is refused,
because no password opens the files already there. Instead, disconnect
that cloud with lnk storage disconnect, which leaves its bucket as it
is, create a new encrypted bucket and push your files again.
Create an encrypted bucket
lnk storage create with --encrypt makes the bucket and its
encryption password, and shows the password once.
lnk storage create gcp --encryptThe password is 32 random letters and digits that Link makes. Save it in a password manager while it's on screen. Once this machine's copy is gone, nobody can read the files without it, Link included.
Encrypt a bucket you named
Connecting a bucket you made yourself with --encrypt asks for a
password, which you choose. Type it at the hidden prompt and keep it,
since every other machine needs it.
lnk storage connect gcp --bucket my-files-0123 --encryptThe bucket should be empty, since a bucket holding files without encryption is refused.
Where the password is kept
Link keeps the password with the cloud's other settings, readable only
by you. On a Mac it's in the Keychain, and elsewhere in
~/.lnk/safe/storage/rclone.conf, obscured but not encrypted
(Encryption passwords). No lnk storage
command shows it again, and lnk storage show names only where it is,
never the password itself.
Every push, pull and tree reads it from there, so you type it
only when connecting.
Reach it from another machine
On another machine, connect the same bucket with --encrypt, and type
the password when asked. lnk storage list on the first machine shows
the bucket's name.
lnk storage connect gcp --bucket lnk-bucket-a1b2c3d4e5f6 --encryptLink checks the password by decrypting a file already in the bucket and refuses one that doesn't open it (Its password). The prompt needs a terminal, so connect each machine by hand once, and its later commands read the saved copy.
Reach it from an agent on a box
An agent's files go to buckets of their own, always encrypted with one
password for that agent, apart from your files' bucket
(Back up an owner's folder). When you move
the agent to a box in Google Cloud with lnk agent move gcp,
its buckets and their password go with it. The box gets a Google Cloud
service account, made through your account, whose key reaches only
that agent's buckets, so a push of the agent's files works there.
lnk agent move gcp --agent workThe box holds none of your cloud accounts, so an agent there never reaches your own files' bucket.
Back up
push sends files to every cloud at once and checks every copy it sent.
It skips files unchanged since their copy was checked, never deletes
anything in a cloud, and never replaces a copy that differs from the
file here unless you pass --replace.
lnk storage push photos/2019lnk storage push photos --cloud aws -- --bwlimit 10M --transfers 8Back up an owner's folder
Another module can say a folder here is one it keeps, such as a group's
shared folder: an owner's folder, its owner named by a stable ID,
<kind>:<id> (Owners).
lnk storage never takes it for your files. Each goes to a bucket of its
own in every cloud, made on its first push after your yes
(Say yes to a bucket) and always encrypted, with
one password per owner. The same commands work on it, but pull and
clean refuse while its owner runs, or when the module that keeps it
can't tell, so stop the owner first.
lnk storage push <folder>lnk storage clean <folder>lnk storage list and show name an owner's bucket by its ID, from
Link's settings, whether or not the module that keeps the owner is here.
When that module fails to say which folders it keeps, the commands that
touch files warn and go on with your files, leaving its folders out
(When a module doesn't answer).
When it moves
An owner's buckets move with its folder: the module that keeps it asks
lnk storage for them by the owner's ID, and puts them in place where
the folder lands (Interface). On a box, its folder backs
up with a key that reaches only its bucket. If its machine is gone, the
buckets are found again by their labels in your lnk cloud accounts.
Objects beside its files
A module may keep objects of its own in an owner's encrypted bucket,
under .link/, apart from its files. lnk storage never pushes, pulls
or lists anything under .link/ as a file, and any machine connected to
the bucket reads those objects as they were left.
A folder a cloud account keeps
An owner's folder may be kept on a cloud account rather than in your
buckets, as the module that keeps it answers storage.folders. Push and
pull reach it too, from and to its folder here, but through that module
(storage.home) and that account, never rclone and never a key here
(Storage
Contract).
Get files back
pull copies each file from the cheapest cloud that has it and checks
it. Downloading from AWS or Google Cloud costs about $90 per TB, so it
asks before an archive restore or more than 1 GB of charged downloads.
It doesn't overwrite a file here that differs, unless you pass
--replace.
lnk storage pull 'photos/2019/**/*.jpg'lnk storage pull photos/2019 --cloud gcpSee what's where
Each file gets one mark per cloud: ✓ checked, A checked in an
archive tier, ? not checked, ≠ differs, ✗ missing. tree works
offline, and --refresh finds files pushed from another machine.
lnk storage list --tree --level 2lnk storage list --tree photos --refresh--dirs-only shows folders alone, like tree -d, and --json prints
the tree for a script.
lnk storage list --tree --dirs-only --level 3lnk storage list --tree --jsonUse a bucket from another machine
lnk storage connect takes the same options as lnk storage create but makes nothing, and
stops if the bucket isn't there. An encrypted bucket also needs
--encrypt and its password (Reach it from another
machine). lnk storage list on the first
machine shows each bucket's name. --owner-bucket adds an owner's
bucket, asking its encryption password; --owner <kind>:<name> says
whose, as the module that keeps that kind names it, and defaults to the
only owner with a folder here.
lnk storage connect aws --bucket lnk-bucket-a1b2c3d4e5f6lnk storage connect folder --path /Volumes/MyDrive --bucket local-link/personallnk storage connect aws --bucket lnk-files-0123 --owner-bucket lnk-owned-0123 --owner group:researchFree up space
clean checks every cloud again, now, and deletes the files here only
if every cloud has a copy with the same checksum. Otherwise it says
what's missing where. It's the only Link command that deletes your files
here.
lnk storage clean photos/2019 --drylnk storage clean photos/2019Archive old files
Archive tiers cost about $1 per TB a month instead of about $20, and
reading files back costs money. They exist on AWS (S3 Glacier Deep
Archive) and Google Cloud (GCS Archive), and lnk asks before archiving
or restoring.
lnk storage push old/scans --archiveRestores and early deletes
On AWS, pull requests a restore that takes up to 12 hours, so run it
again once it's done. Deleting early is charged anyway, for 180 days on
AWS and 365 on Google Cloud, and lnk storage remove and lnk storage delete count.
clean works on archived files without a restore, except a file over
200 MiB on AWS, whose copy it reads back
(why).
Disconnect or delete
Nothing here deletes a file's last copy unless you pass --everywhere.
Remove from your clouds
remove deletes the copies in every cloud, or only in the cloud
accounts you give with --cloud and the remotes you give with
--remote, and keeps the files here. It takes only files and
folders, and refuses a cloud's name.
lnk storage remove old/drafts --cloud gcpA cloud that's full
A cloud whose account reads only (past what it may keep) takes nothing
more: push says so before sending anything, with the cloud's own
words on making room, and remove asks the cloud to delete the copies
itself, so you can free space with the same command.
Disconnect or delete a cloud
lnk storage disconnect prints the lnk storage connect command that brings the
cloud back, and asks only when the cloud holds a file's only copy.
lnk storage delete deletes
the buckets lnk storage create made, your files' and owners', and
on a folder cloud Link's folders in it. It asks first, and refuses a
path.
lnk storage disconnect gcplnk storage delete awsUse rclone directly
This is rclone itself, with each connected cloud as a remote named after
it, for what lnk storage doesn't do, such as mount, sync and
serve. Link's safety checks don't apply here, and sync deletes. Run
lnk storage list --tree --refresh afterwards to catch up.
lnk storage rclone lsd aws:lnk storage rclone mount aws: ~/mntWhere your settings live
Everything is in ~/.lnk/safe/storage/: which clouds are connected, their
keys and encryption passwords, and what went where. On a Mac, the keys
and passwords Link writes are in the Keychain instead.
Removing it
lnk module remove storage and lnk uninstall keep this folder, and
neither deletes anything in your clouds. lnk uninstall --delete buckets deletes it, along with the only copy of the encryption
passwords Link made, so save them first.
More about it
Where the rest of what lnk storage does is written:
- Every flag is in
lnk storage <command> --help. - Security has what's protected and what isn't.
- Decisions has why it works this way.
- Troubleshooting has what to do when a command refuses.