Desktop apps your agent works with a mouse

Your agent works a desktop. It sees a desktop as screenshots and works it with the mouse and keys, so it can use a program that has no command line or API: the programs it starts with Run show on it.

lnk agent tool add desktop    # it calls desktop__screenshot, desktop__click, desktop__type...

Add it

It installs link-desktop if it's missing, picks a display number no X server here uses, makes the display's cookie, and restarts your agent with the tool. lnk agent tool remove desktop takes it away.

What it needs

The desktop is a virtual one, on Linux: an Xvfb display of the agent's own, so it works on a box with no screen and never touches yours. You need the link-desktop plugin and Xvfb (apt install xvfb, or dnf install xorg-x11-server-Xvfb).

Its programs

Each program its run starts gets DISPLAY and XAUTHORITY for that display, and its sandbox reaches that display's socket and cookie, nothing more of /tmp/.X11-unix.

Ask it something

Ask it to "open the spreadsheet in LibreOffice and total column C" and it starts the program with run, takes a screenshot, and clicks and types until it's done.

What it does

Your agent calls it as desktop__<call>. The pixels are the screenshot's.

`screenshot`

An image of the whole desktop, 1280 by 800 pixels, which your agent reads to find where to click.

`click`, `scroll` and `drag`

click moves the mouse to a pixel and clicks: left, right or middle, once or twice. scroll turns the mouse's wheel, at a pixel when given. drag presses at one pixel, moves to another and lets go.

`type` and `key`

type types text where the keyboard's focus is. key presses a key or a combination: Return, Escape, ctrl+c, alt+F4.

How input is sent

Input goes through the X server's XTEST extension, so a program sees what a person's mouse and keyboard would send. A character no key gives, such as € on a US layout, is put on a spare key while it's typed.

The display

The display starts at the first call and ends when the tool does, when its last session has been idle a while: the programs on it end then too. The agent's other sessions share it.

Its cookie is a file only your user reads, and the display takes no client without it, so no other user of this computer sees or draws on it; it listens on its socket only, never on TCP.

By hand

lnk agent tool add desktop is the plugin, the display's cookie, the tool shared by every session of the agent with its calls allowed, and your agent allowed to call it.

lnk plugin add link-desktop
lnk-link-desktop cookie ~/Link/Agents/main/Desktop/auth --display 120
lnk sandbox tool add desktop --shared --allow screenshot --allow click --allow type \
  --allow key --allow scroll --allow drag \
  -- lnk-link-desktop serve --display 120 --cookie ~/Link/Agents/main/Desktop/auth
lnk agent allow --tool desktop

More about it

To work on its code, see Developing Link Desktop, and when the display doesn't start or stays black, Troubleshooting.

Docs

Developing: lnk-link-desktop is Link Desktop: an MCP server over stdio whose calls see and work an Xvfb display, for any harness to reach as a tool
Troubleshooting: What to do when your agent's desktop doesn't start, stays black, or a program can't open it (Link Desktop)
Decisions: Why Link Desktop gives an agent a virtual desktop of its own