#2962: Multi-peer remote-agent dispatch via Tailnet IPs + coordinator proxy (complement to hermes-agent#9295)

Summary

PR NousResearch/hermes-agent#9295 lands gateway.tailscale_serve to expose a single machine's API server over a Tailnet — a clean fit for the "one machine, remote access" case. A different deployment shape benefits from a complementary pattern: one coordinator process dispatching to N peers, where each peer runs its own hermes-agent and the WebUI surfaces "this turn is talking to peer X". This issue describes that pattern in case it's useful as design input alongside #9295.

Filing in hermes-webui because the WebUI-visible surface is what differs most. The matching agent-side concerns live in a sibling issue at NousResearch/hermes-agent (link added at the bottom after both posts).

This issue is not a PR proposal — it documents a pattern observed working in production and asks whether any of the supporting concerns (per-message peer indicator, per-profile MCP registry, SSE profile identity) belong on the WebUI roadmap.

Pattern

   client app                    coordinator                    peer machines
   (composer,         POST       (small service)
   consumer apps)  -------->     - peerMap: { name -> (ip, port, basicAuthUser, basicAuthPass) }
                                 - lane routing
                                 - per-message peer override
                                 - response stream         HTTP    +-----------------+
                                       |                  ---->    | peer A          |
                                       |                  Basic    |  auth proxy     |
                                       |                  Auth     |   |             |
                                       |                           |   v             |
                                       |                           | hermes-agent    |
                                       |                           | API @ 127.0.0.1 |
                                       |                           +-----------------+
                                       |
                                       +-------- ...               +-----------------+
                                                ---->               | peer B          |
                                                                    | (same shape)    |
                                                                    +-----------------+

• Each peer's hermes-agent stays at its safe loopback default (no --host 0.0.0.0).
• A small ~300-line stdlib Python reverse proxy on each peer binds to that peer's Tailnet interface IP only (not 0.0.0.0), terminates HTTP Basic from the coordinator, and injects Authorization: Bearer ${API_SERVER_KEY} for /v1/* requests before forwarding to 127.0.0.1:8642. Dashboard routes (/) pass through to 127.0.0.1:9119 without Bearer.
• The coordinator holds a peerMap keyed by logical name, calls each peer with POST /v1/chat/completions over Basic Auth, streams the response back. One client request can fan out to multiple peers in parallel.

How this differs from #9295

Axis	#9295 (tailscale_serve)	This pattern
Peers per deployment	1 (the machine running hermes-agent)	N
Exposure mechanism	Tailscale Serve managed proxy + HTTPS	Direct Tailnet IP bind + small stdlib proxy + Basic Auth
Auth	API_SERVER_KEY Bearer only	Outer Basic per-peer + inner Bearer auto-injected
Routing source	None (single backend)	Coordinator decides which peer per message
WebUI surface	/v1/* becomes reachable	Composer needs a "current peer" indicator and per-message peer override

PR #9295 is the right answer for the "expose this one machine over Tailscale" case. This pattern is the right answer when the same operator wants one composer with a peer picker that can target several machines from a single client.

WebUI-side concerns

A few WebUI rough edges become visible when running this pattern. None are blockers, but they would smooth multi-peer use whether the coordinator is a custom service (like ours) or whatever shape the upstream multi-machine story eventually takes:

1. No "current peer" indicator on the composer. When a session runs through a coordinator that can target different peers, users want a visible chip near the model chip that shows which peer the next turn will go to. Today there is nothing to bind to — the composer treats the request as "local to this WebUI."

2. Profile identity is missing from the session-events SSE payload (#2660). Each peer's local WebUI publishes sessions_changed to its own SSE bus, but the payload has no profile identity. A coordinator that subscribes to multiple peers' SSE streams can't route the event to the right tab without re-fetching.

3. MCP server registry is process-global per #1977 / agent-side root cause. When a peer runs concurrent profile sessions, MCP servers register by name only. Cross-peer this is fine (separate processes), but cross-profile on a single peer is the existing layer-2 problem.

4. Profile vs. peer-target axis confusion. Per the runtime-model issue at #749 (Profile is a heavy bundle, not a routing override), our pattern adds a fifth axis: (peer, profile-on-peer). The composer currently has no place to express "Profile A on peer X"; it can express only "this WebUI's currently-active profile."

Of these, #2 (SSE profile identity) is the one most directly addressable in this repo and is already filed at #2660. The others might just be useful design context for the broader multi-machine story.

Verification (honest)

The pattern is wired end-to-end on a coordinator host. As of this filing, both target peers in the local Tailnet are showing offline (Tailscale reports relay "dfw"; offline, last seen 8h ago for one peer and 1d ago for the other), so a fresh live transcript is not available at the moment. A successful round-trip transcript from earlier this week is preserved in the coordinator's tools/remote-agent/README.md:

2026-05-23: POST /api/remote-agent/invoke (target: peer-A) returned 200 with "pong" in 33s

The shape that produced it on the wire (Basic-only from coordinator; proxy injects Bearer for /v1/* upstream):

# from the coordinator host, against the peer's Tailnet IP
curl -u coordinator:<32-char-password> \
     http://<peer-tailscale-ip>:9121/v1/models
# 200 OK -> the proxy is up AND the upstream API server has API_SERVER_KEY set

# the actual round-trip (Chat Completions)
curl -u coordinator:<32-char-password> \
     -X POST http://<peer-tailscale-ip>:9121/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{\"model\":\"<profile-default>\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"stream\":false}"
# 200 OK -> assistant reply streams back

Happy to attach a fresh transcript with timing data once a peer is back online — likely within 24h. (Down due to Home renovations)

Why I'm not proposing a PR here

The WebUI changes that would be most useful (per-message peer indicator, peer-aware composer chip, peer override field) are speculative without knowing where the multi-machine story is headed. The maintainer-authored issues (#749, #1977, #2660, plus the agent-side #10567 and #15731) suggest the underlying problem space is well-mapped; a multi-peer coordinator pattern just exercises that space in a particular shape.

Filing as enhancement + ux for visibility. If this isn't a direction the WebUI wants to take, no action needed on this side — the pattern works fine as a coordinator-only convention. If it IS useful design input, the composer-side (machine, capability) picker shape that's currently working as a local patch might be worth sharing as a reference.

If this direction is something the WebUI wants to bring upstream, happy to take it on and work with maintainers on the implementation -- the composer-side (machine, capability) picker, peer-aware chip, and SSE profile identity refs are all in scope. The existing local pattern is a starting point; the upstream shape should match the WebUI's direction (refs #749 / #1977 / #2660), not the coordinator's particular choices.

Related

• NousResearch/hermes-agent#9295 — single-machine Tailscale Serve PR (this issue's companion)
• NousResearch/hermes-agent#9269 — parent issue PR #9295 closes
• NousResearch/hermes-agent#10567 — --host + CORS for dashboard (Tailscale/VPN)
• NousResearch/hermes-agent#15731 — dashboard chat tab + --host non-localhost
• #749 — Profile runtime model alignment
• #1977 — Profile-keyed MCP server registry (layer-2)
• #2660 — Session-events SSE bus has no profile identity
• Companion issue: NousResearch/hermes-agent#32397

AI Usage Disclosure

This issue was drafted with Claude (Opus 4.7) assistance and reviewed by a human contributor before posting. Architecture description, smoke-test transcript, and cross-issue links were verified by the human reviewer. The pattern described is in production use on the contributor's coordinator deployment.

Edits: cross-link to companion issue NousResearch/hermes-agent#32397 added; explicit offer to contribute the implementation upstream if maintainers want this direction added. No claim changes.