Architecture
One desktop, two control planes
Everything else follows from that separation: people come in over a WebSocket, the agent over a local Unix socket, and neither is a guest of the other.
The desktop
Xvfb :0 provides a virtual display with no physical monitor behind it.
Openbox manages window frames and nothing else — it does not draw the
desktop. That job belongs to pcmanfm in desktop mode, which owns the root
window and paints the wallpaper and icons, with lxpanel on top.
Everything runs under supervisord, so any piece restarts on its own
without taking the rest down: xvfb, pulseaudio,
dbus-session, at-spi, openbox,
pcmanfm-desktop, lxpanel and sentineldesk itself.
The media path, outbound
ximagesrc reads the framebuffer and hands it to GStreamer, which runs
inside the Go process through go-gst rather than as a separate
gst-launch child. That is what keeps the encoder reachable while it runs:
bitrate changes, keyframes on demand, and destinations attached and detached without
cutting anything.
The encoder is chosen at startup by running a real probe pipeline for each candidate, in order. The choice reflects what actually works on that host, not what the drivers claim to support.
| Order | Encoder | Hardware | Codec |
|---|---|---|---|
| 1 | nvenc | NVIDIA (CUDA) | H.264 |
| 2 | vaapi | Intel / AMD | H.264 |
| 3 | h264 (x264) | — | H.264 |
| 4 | vp8 | — | VP8 |
Encoded frames leave through an appsink and each RTP packet goes straight
onto the Pion track. On the H.264 encoders they also pass through a tee: in
the normal case nothing is attached to it, and that is where an external destination gets
a copy of the same picture the room is already receiving. Streaming to YouTube does not
mean encoding the screen a second time.
Why the tee sits before h264parse. Each branch parses
for itself: WebRTC wants byte-stream and FLV wants AVC. Asking one parser to produce
both deadlocks the negotiation. Parsing is nearly free — it reads headers, it does not
decode — so the duplication costs nothing next to a second encoder.
Audio
There is no sound card, so PulseAudio loads a null sink named
sentineldesk. Applications play into it and
pulsesrc sentineldesk.monitor captures what they played;
opusenc encodes it and a second appsink feeds the audio track.
The way back: the microphone
The browser's microphone travels the other way, and the return path needs two PulseAudio objects rather than one:
incoming Opus track → appsrc → pulsesink device=sentineldesk_mic
→ null sink "sentineldesk_mic"
→ sentineldesk_mic.monitor
→ module-remap-source "sentineldesk_mic_in"The remap is not decoration. A monitor is how PulseAudio exposes "what a sink is playing", and applications treat it as such: browsers list monitors separately from real inputs, or hide them from the microphone picker altogether. Remapped, the same audio presents as an ordinary capture device — and it is made the system default.
Both objects are created at startup, not on first use. A page enumerates its audio devices when it loads, so a microphone that appears later is missing from the list of the very page that wanted it.
Control plane 1 — people, over WebSocket
WS /ws is the only door. The first frame must be
{type:"auth"}; until it validates there is no SDP offer, no ICE, no
DataChannel. After authentication the same socket carries signalling and presence.
HTTP :8080 serves the client embedded with go:embed and the
file-manager endpoints, and holds no secrets: the ICE configuration and
TURN credentials travel over the already-authenticated WebSocket. The only informational
endpoint is /auth, which says nothing beyond whether a login is required.
There is no /login endpoint, by design: HTTP is not the
authentication gate.
Keyboard and mouse arrive over a DataChannel named input and are injected
into X through XTEST. A browser gamepad becomes a virtual Xbox 360 pad through uinput,
where the host exposes /dev/uinput.
Control plane 2 — the agent, over a local Unix socket
The MCP server listens on /run/user/1000/sentineldesk-mcp.sock, mode
0600, and exposes 114 tools. The AI host spawns
sentineldesk -mcp-stdio with docker exec; that sub-command is a
thin JSON-RPC pipe between stdin/stdout and the socket. Killing the host never takes the
desktop down with it.
Two helper bridges cover what pixels cannot: a11y.py exposes the AT-SPI
accessibility tree (the ui_* tools, which invoke a button by name rather
than by coordinates) and Chromium runs with CDP on port 9222 (the browser_*
tools, which drive the real DOM).
What the agent may do is bounded by a three-level policy — full,
safe, readonly — plus a denylist and an allowlist. The daemon
sets the ceiling through the environment, and each connection can restrict itself further
but never widen.
| Level | What it lets through |
|---|---|
full | Everything (the default) |
safe | Everything except running code or touching the system; as_root is out |
readonly | Observation only: see the screen, read the tree, list things |
The room, and one deliberate asymmetry
Capture happens once and is fanned out to up to four participants
(MAX_VIEWERS), so a second viewer does not cost a second encoder. One person
drives at a time; the rest watch, and control is handed over cooperatively: everyone
arrived with the same credential, so there is no hierarchy to enforce.
The shared encoder's bitrate is the minimum of what each network estimates, because encoding for the best link would drop the worst one.
Each participant's pointer is drawn as a real X window, not as an overlay in each browser. That distinction matters: because the pointer is part of the desktop, it appears in recordings, in screenshots and in every other viewer's stream, instead of existing only in one browser's DOM.
The asymmetry, stated plainly: MCP tools do not go through the
room's control arbitration. They inject straight into XTEST, because the MCP arrives
over the daemon's local socket rather than over the web. Arbitration exists between
browsers. To stop an agent from touching anything, the instrument is its policy
(-mcp-policy readonly), not room control.
Networking
On Linux with network_mode: host, ICE connects directly over UDP with nothing
in between — the fastest path. On macOS and Windows the Docker Desktop VM breaks direct
ICE, so deploy/docker-compose.yml includes coturn as a
relay. For clients behind strict corporate NAT, publish coturn in production too and pass
its URLs through CLIENT_TURN_URLS.
If the server sits behind NAT and does not hold the public IP on its own interface, set
NAT1TO1_IP=<public IP> so ICE advertises the right address.
How the code is organised
cmd/sentineldesk/ wiring only: flags, HTTP, WebSocket, MCP socket
internal/config/ environment configuration
internal/desktop/ X11: input injection, cursor, clipboard, joystick, pointers
internal/media/ GStreamer: pipelines, encoders, recording, upstream audio
internal/stream/ sessions, the shared room, auth, rate limiting, TLS, files
internal/mcp/ the MCP server and its 114 tools
internal/webui/ the browser client, embedded with go:embed
deploy/ Dockerfile, compose, desktop and supervisor config
The backend is a single Go binary (Pion WebRTC + go-gst). Because it uses
CGO to link GStreamer, it cannot be cross-compiled with a bare GOOS/GOARCH:
release binaries are built inside the Debian 13 Docker stage and extracted, so they are
byte-for-byte what the container runs.