Images, documents and voice notes it can read

Link Harness reads the images, documents and voice notes a message holds, from ACP, the Agent Client Protocol, or from any of Link's connections, and keeps each as a file in the conversation's folder.

Send it images and documents

A prompt on ACP takes a message's images as image blocks and its documents as embedded resources, each in base64, a document named by the last part of its URI. Images are PNG, JPEG, GIF or WebP, and documents PDF or text (plain, Markdown or CSV), at most 10 MiB each and 8 to a message; a photo needs no caption. Anything else, or a part that isn't what its type says, is refused, saying why.

{"sessionId": "link:main", "prompt": [
  {"type": "text", "text": "What does this say?"},
  {"type": "image", "mimeType": "image/png", "data": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=="}]}

A photo or document sent on one of Link's connections (Telegram, Slack, Discord, Signal) reaches it the same way: its connection downloads it and sends it with the message's text, and names to the agent any it left out, a video, and why.

Turning a kind off

Its spec's [connections] images, documents and voice, or a connection's own, turn each kind off: what it doesn't take is named to the agent as left out.

Where each is kept

Each is a file of its own in the conversation's folder, parts/, which its message names ([image: photo.jpg], in conversation.md too). Forking a conversation copies its parts, and lnk agent chat convert takes them along.

How each reaches the model

An image or a PDF reaches a model on Anthropic, OpenAI or OpenRouter as the provider's own image or document; a text document reaches every model as its text, under its name. A model on your computer is told each image or PDF it was sent and left out, since what Link knows of a model doesn't say whether it reads them.

From a tool

A tool's answer may hold images and documents too (MCP's image content, or a resource's blob), and they reach the model the same way: Link File's file__read gives an image or a PDF of its files so.

How many a prompt sends

A prompt sends at most 20 MiB of them, the latest first, and names those before; the window counts an image as 1,600 tokens and a PDF as a token for each 3 of its bytes.

A conversation kept elsewhere

A conversation kept off this machine (gcs, or through the storage bridge) has no folder for them, so its messages say what was left out instead.

Send it a voice note

A voice note is an audio block on ACP, in base64: Ogg, MP3, MP4 audio, AAC, WAV or WebM, as Telegram, Slack, Discord and Signal send one. Before its turn, Link Harness has it heard by a transcription model, and its words begin the message's text: [voice note] buy milk on the way home.

[model.<provider>] transcribe

The model its spec's transcribe names on the agent's own provider hears it, through OpenAI's POST /v1/audio/transcriptions, which a runtime on your computer may serve too. Link's spec names gpt-4o-mini-transcribe for OpenAI; for another provider, set one:

[model.local]
transcribe = "whisper"

Its audio

With none, or one that can't hear it, the voice note is named to the agent as left out, and why. Its audio is kept in parts/ as any part is, and never sent to the model: only its words are.

Docs

  • Decisions: Why Link Harness reads images and documents the way it does