Two AI teammates and a room full of humans, all working on the same live screen — across miles, on laptops and phones. The agent is the constant hand; people take turns on top.
✓ PROVEN 07.26 — a live instance of this architecture ran end-to-end: Jenny drove Omar's real Whimsical board and streamed it to his phone (the same Playwright-hands + phone-stream, minus Zoom, one human in the loop). See the live Whimsical co-view test writeup →
Picture a normal Zoom meeting. One of the "people" on the call is a computer we own — the VM. It's driving a browser, and it screen-shares that browser into the meeting so everyone sees it. The agent (Jenny) is always driving that browser underneath — those are her hands. When a human wants to grab the wheel, Zoom lets them ask for remote control and take a turn. And when Jenny talks, her voice comes out of the VM and into the Zoom microphone, so everyone on the call actually hears her speak. Add a second VM and you've got Domino in the same room too — two AI teammates plus the people.
Follow the arrows top to bottom: the agent's hands stay connected at the automation layer the whole time; Zoom is the shared room on top; humans take turns holding control; the agent's voice is piped into the call's microphone.
Drives the browser on the VM. This is the agent's hands — always connected, no matter who else is touching the screen.
The logged-in Chrome the agent is working in. This exact window is what gets screen-shared.
Piper / fast-chat text-to-speech. The agent speaks; the audio plays out of the VM.
VB-Cable routes that speech straight into Zoom's microphone input — so the room hears it.
Everyone on the call watches the same browser, live. The agent's marks, clicks and typing show up for all.
Zoom's Request / Give Control hands the wheel to one person at a time. They interact directly; then they hand it back.
Key idea: whoever holds the Zoom control on top, the agent still holds control underneath through Playwright. Humans take turns; the agent is the constant hand beneath them.
One person holds the wheel at a time; the rest watch live and take their turn when they ask.
A second VM joins the same Zoom the identical way — its own browser, its own voice cable. Now two agents work the room alongside the humans.
A tunnelled stream URL lets many more people simply watch the session without joining the call.
"Turn-based" isn't a compromise — it's exactly how a working session hands the wheel around without collisions. The agent's Playwright layer never lets go; the Zoom control layer is what changes hands on top. Here's the full loop:
Why not simultaneous cursors? True multiplayer (many live cursors at once) is a fundamentally harder architecture and it isn't needed for turn-taking work — cooperative hand-off covers the real use case cleanly and ships far sooner.
Getting the agent's voice into the room is a plain audio-routing trick — no model call anywhere in the chain. It's the same signal path you'd use to feed any app's audio into a meeting:
VB-Cable, a virtual audio cable that presents itself to Windows as both a playback device and a recording device.The honest edge: this is text-to-speech, so there's a short turnaround — near-instant for quick lines, a beat longer for deep answers. Live enough to feel like a teammate, not instant like a phone call.
Playwright drives a real browser on the VM. That's the automation layer — the agent's hands stay on the wheel continuously, whatever happens on top.
The VM joins a Zoom call and screen-shares that browser window. Now the agent's screen is the meeting's shared screen.
Zoom's Request / Give Remote Control lets a human take the wheel for a turn — one controller at a time — then hand it back. The agent keeps working underneath.
Text-to-speech plays out of the VM and is routed into Zoom's mic via a virtual audio cable. Everyone on the call hears the agent speak live. Bridge / fast-chat text runs in parallel.
Domino (VM3) joins the very same Zoom in the very same way. Two AI teammates plus the humans, all in one room, across miles and devices.
A tunnelled stream URL lets many more people just watch the working session, without needing a seat in the call.
We show these plainly rather than hide them — a blueprint you can trust is one that names its own edges.
The whole point of this blueprint is that most of it already runs. Here's each component, what it does in this system, and whether it's proven today or still to be wired:
The takeaway: the risky, unproven surface is small and well-bounded — four pieces of wiring on top of a stack that already works. Nothing here needs a new model, a new voice, or a research breakthrough.
Start small and provable, then widen. Each rung adds exactly one new thing so nothing gets muddy.
The VM joins a Zoom, shares the Playwright browser, and the agent's voice is routed into the call's mic. Omar takes Zoom remote-control to interact — while the agent keeps acting via Playwright underneath. One human, one agent, one room.
Jack joins from another device. Now practise the turn hand-off: Omar gives control, Jack takes a turn, hands it back. The agent's hands never leave the wheel underneath.
A second VM (Domino / VM3) joins the same Zoom identically. Two AI teammates plus the humans, all working the same live screen across miles.
Why this order: each rung is a single new risk. Prove the seat + voice + underneath-control with one trusted person first; only then add a second human and a second agent. Nothing ships until the rung below it is real.
Each rung isolates exactly one new risk, so when something breaks you know precisely which piece to fix. Here's what each rung actually builds, and the concrete thing that has to be true before climbing to the next.
The discipline: nothing on rung 2 gets touched until rung 1 is genuinely real, and rung 3 waits on rung 2. One new risk at a time keeps every failure legible.
Pure plumbing on proven parts. Humans take turns on top; Jenny and Domino work underneath; anyone can watch. That's the proof-of-concept.