NAME
zeughaus-architecture - processes, protocols and execution semantics of zeughaus
DESCRIPTION
zeughaus composes heterogeneous tools (math and string transforms, screen capture, SQLite schemas, LLM conversations, Keras model design, recordings, jobs) as one node graph, edited collaboratively and executed by a headless process that also multiplexes terminals for its editors. This page condenses the architecture document, DESIGN.md in the zeughaus repository; only FUTURE DIRECTIONS describes what is not built.
PROCESSES
Two processes, and the split is not optional. zeughaus, the editor, never executes a node; zeughaus-runner, the runner, is a headless process that does nothing else. So every editor, local or remote, is the same thing: a remote view, and a side effect such as a screen capture fires once per session, not once per open window.
They meet in two places: the store holds the graph document and who runs it, and the link carries everything a pass produces straight from the runner to each editor.
Plugins are one list in each process (Runner::new, App::new), the same plugins in the same order; the editor gates the ones that touch the OS. A node type the runner cannot instantiate never runs; a type the editor cannot instantiate cannot be placed. The catalog is in zeughaus-nodes(7).
THE STORE
The store is SpacetimeDB running the zeughaus-module server module; both binaries reach it through the client in zeughaus-sync. It holds the graph document and runtime presence, and nothing a pass produces.
nodeid,type_id,display_name,x,y,params(the node's settings as one JSON string),parent(the enclosing container,0for the root) andrunner(set on top-level graphs only).edgeid,from_node,from_pin,to_node,to_pin.runtime- One row per connected runner:
connection_id,identity, an auto-incrementedseqandaddr, the URL editors dial.
Reducers are fine-grained: create_node, move_node, set_node_params, rename_node, delete_node, connect_edge, disconnect_edge, join_runtime, announce_endpoint, adopt_root_nodes. Conflicts are last-writer-wins per row. Ids are made process-unique at startup (NodeId::seed_unique), so two editors never collide.
Without a store the runner refuses to start. The editor starts anyway and edits a local scratch graph that nothing computes and that is gone on close unless saved as a .zgh file. Save and load are explicit keys (Ctrl+S, Ctrl+O); the store is never written from a file behind the user's back.
THE LINK
The link is the runner-to-editor protocol over weida (QUIC with mutual TLS), defined in zeughaus-link. Each kind of traffic has its own path on the runner's listener:
/events- Pub/sub. Scalar outputs, cleared pins, node errors, setting rejections and edge traffic.
/snapshot- Req/rep. The current output set for a late-joining editor.
/triggers- Push/pull. Trigger presses from an editor or the CLI to the runner.
/samples- Frames: one standing exchange per node pin (see THE SAMPLE FEED).
/mux- Every terminal exchange (see TERMINALS).
/runs- Req/rep. A job run's files by range: log, exit record, artifacts.
/hold- Req/rep. Holds and releases the runner's job host.
Only bool, int, float and str are values on the wire (zeughaus-core/src/wire.rs). Frames have their own path; every other value (a KerasModel, a Conversation) stays in the runner.
The listener requires a trusted client certificate on every path; the credentials live in the state directory.
WHO EXECUTES
The store decides; nothing is negotiated. Every top-level graph (a graph.sub node with parent 0) names its runner in its runner column: the sha256:<hex> fingerprint of that runner's endpoint. Each runner executes exactly the subtrees of the graphs that name it (in_scope_of in zeughaus-runner/src/runner.rs).
Every runner calls join_runtime and announces its pinned URL, weida://sha256:<fp>@<host>:<port>/, in its runtime row; an editor dials every runner it finds there. Root-level nodes left from a session before graphs had owners are adopted once into a new graph by the runner with the lowest seq (adopt_root_nodes).
Two runners racing for one input pin resolve it identically from the data (occupancy_winner: the larger edge id wins), because arrival order differs per process.
REDIAL AND RESYNC
Redial is weida's, resync is zeughaus's. An editor dials each runtime address once (first_dial in zeughaus/src/transport.rs) under a ReconnectPolicy that starts at 250 ms, doubles to 4 s and never gives up:
What a redial cannot restore, the editor does on PeerEvent::Connected: a fresh /snapshot, reopened feed exchanges, a fresh mux attach. A runner that restarted is a new peer; the new address in its runtime row is what replaces the editor's tasks for it.
TYPES
Pin types are runtime values in zeughaus-core, not compile-time labels:
enum Ty { Any, Bool, Int, Float, Str, List(Arc<Ty>), Option(Arc<Ty>),
Record(Arc<Record>), Opaque(Arc<str>) }
Any connects to anything and is never coerced. The scalars map one to one onto bool, i64, f64 and String; narrower numerics are not pin types, a node converts where it emits. Record is built at runtime (a user-designed schema); Opaque names a plugin's own Rust type (KerasModel, Conversation, db.table).
A Rust type declares its Ty once through Typed::ty(), so the pin declaration and the tag on a Value cannot disagree. Coercion across an edge uses the value's own tag, not the source pin's declaration. Converters are registered per type pair in TypeConverters; the one built-in is Int -> Float. Editor and runner build the same registry, so what may connect and what is coerced agree.
Nodes, pins, settings
A plugin implements ExecutableNode; a DomainPlugin lists and instantiates node types. The catalog is data: a NodeDefinition carries the type id, display name, category, pins, settings and whether the type is a container.
A pin is an input, an output or a field. PinKind::Trigger declares a trigger pin: the node acts when something arrives on it, drawn as a square. PinKind::Sample declares a sample pin, state read whenever the node runs, drawn as a circle. A field pin has PinDirection::Both; a wire between two field pins is not dataflow but a declared relation between two nodes.
Settings are the one way the editor configures a node. A SettingDef names a key, a default, a placeholder and a SettingKind: Text, Multiline, Title or Fields (a name:type row editor). The value is always text, stored in the node's params; the node parses it. A refused value (ZeughausError::InvalidParameter) is published as SettingRejected, and every editor draws the reason under the field.
Some parameters are derived by the editor rather than typed: db_path (from the enclosing db.database), relations (from wires between field pins), renamed_from (a table's previous name). The runner applies them like any other params key.
sync_pins lets a node reshape its inputs from what is wired to it (a variadic merge grows a slot); both processes call it after every connection change.
A source needs a clock: tick_interval() asks the host to run a node periodically, and flow.timer is that clock. Without one upstream, a capture node produces one frame and stops.
EXECUTION
GraphExecutor (zeughaus-runtime) owns the graph and the node instances; every mutation goes through it. It keeps, per edge, the last value and a generation counter (the edge cache), and per node the last output set and the last error. A pass (execute_dirty) runs the nodes of the dirty set in topological order, computed once per graph revision.
Dirty propagation
Propagation is uniform. A changed node marks everything downstream dirty, and every downstream node reruns. A node that must act only on its own trigger asks InputSet::changed(pin), which compares the edge's cache generation with the one the node saw last.
Atomic flush
A node buffers outputs with NodeContext::emit and releases them with flush; a multi-output node never delivers half a result.
Async work
A node that would block (an LLM request, a portal capture) hands async work to its context. The executor marks it pending and holds its downstream back; the host runs the work on a blocking pool and delivers the result with deliver_async_result. A pending node is never dispatched twice.
Failing nodes
A failing node does not fail the pass. Its message is recorded as a node error, its downstream is held back, and everything unrelated runs. Errors reach editors as NodeError and NodeErrorCleared events.
Cycles
A cycle does not fail the pass either. The nodes in it, and everything downstream of it, get one node error and are dropped from the dirty set; the rest runs. Removing a wire or a node that breaks the cycle wakes them.
Relations
Relations are not edges to the executor: Graph::is_dataflow excludes edges with a Both endpoint from ordering and from input sets.
Seeded wires
A new wire is seeded, not recomputed: add_edge copies the source's last output into the edge and dirties only the target's subtree. A node with side effects does not fire again because someone drew a wire.
The runner's loop
The loop in zeughaus-runner/src/runner.rs applies store rows to the executor, diffing parameters against what it last applied so a drag does not rerun a node, serves clocked nodes, runs a pass, publishes what changed and answers /snapshot from the same state.
THE EDITOR
The editor holds one ExecutableNode instance per node, used only for what a node knows about itself: pins, settings, whether it accepts a value, how it reshapes under connections. It does not depend on zeughaus-runtime. Values, errors and setting refusals on screen are what the runner reported, sequence-guarded per pin so a late event never overwrites a fresh one. A pin whose last run produced no value is drawn dim.
Local edits go to the store through an outbox that replays after a reconnect. Settings edits are held back after the last keystroke (DEBOUNCE in zeughaus/src/pending.rs) and flushed on close:
A remote row is applied without echoing back as a reducer call. Edits a window still owes the store are not overwritten by an older shared value for the same key.
Connection rules run in one place, wire_refusal: while a cable is dragged, to decide whether the pin under the cursor is a target, and again when a drop is refused, to say why. A wire that would close a dataflow cycle is refused at the drop.
Containers and subgraphs
Every node carries a parent; 0 is the root graph. A container type (graph.sub, db.database) has no pins of its own: the editor synthesizes them from its direct graph.input and graph.output children (the boundary nodes), named by each child's title. An edge drawn onto a container's pin is stored against the boundary child, so the store and the executor see one flat graph of real nodes; only the editor knows about nesting. Deleting a container deletes its contents, recursively in the reducer and locally in every editor.
A top-level graph is a container too: it is what a graph tab shows, and closing its last tab deletes it. A nested container's open button opens its contents in a tab of its own.
Workspace
The window draws its own titlebar. The tab bar has one section per connected runner, plus Local (no store: the scratch graphs) or Not running (graphs whose runner is not connected). A runner section is that runner's shared workspace: loose tabs and one level of coloured, collapsible groups, each tab a split tree whose leaves are terminal or graph panes. Order, groups, splits and ids come from the runner; active tab, focus, selection and the tab bar's placement are per window. The focused pane's graph is where the palette spawns nodes; keys and palette commands are in zeughaus-keys(7).
Themes
A theme (zeughaus-theme) is an iced::Theme paired with a terminal colour scheme; the scheme is the single source, and the editor's own colours are ANSI slots (a Float pin is green, a Str pin yellow, a Bool pin blue). WezTerm scheme files in <state-dir>/themes/ add to the bundled pack.
DATABASE SCHEMAS
A db.table node (zeughaus-db) is its schema: an editable title and a Fields setting whose rows are field pins. A wire between two field pins is a relation and becomes a FOREIGN KEY in the emitted DDL; the referenced end is the one whose field is named id, else the end the wire was dropped on. The editor only lets equal column types connect.
The wire carries nothing and stays out of execution, which is what makes two tables referencing each other a legal schema and not a cycle. The relations reach the runner as the derived relations parameter.
TERMINALS
The runner is also a terminal mux, in the shape of WezTerm's mux and without its code. portable-pty spawns the child, a pinned wezterm-term parses its output into a canonical screen with stable row indices and change sequence numbers (zeughaus-terminal), and zeughaus-runner/src/mux serves that screen to every editor over /mux.
PTY bytes never leave the runner. What travels is the zeughaus-mux wire model: rows as spans with wire-stable styles, deltas of the rows changed since the sequence number the client holds, and workspace snapshots for the tab and split topology, in bounded postcard frames.
Three exchange kinds
All three ride one pooled QUIC connection, told apart by their first frame.
- control
- One per client. The attach (hello, topology, one head per terminal, so the first paint is one round trip), structural commands with correlated, deduplicated replies, and every later workspace snapshot.
- terminal
- One per attached terminal, full duplex: input up, deltas down.
- row fetch
- A short exchange of its own, so a scrollback page cannot block a keystroke.
A delta is computed per subscriber from the last sequence number that subscriber received and carries every retained row written since: a slow client gets fewer, larger deltas and never a queue. Output is coalesced (COALESCE in zeughaus-runner/src/mux/service.rs):
The client (zeughaus-mux/src/view.rs) applies deltas only at their exact base, pages into holes, and keeps its row store bounded around the viewport.
Lease
The runner owns tab order, splits, ratios, pane and terminal ids; each editor owns its presentation state. Any client may view a terminal; exactly one holds its lease and may type, resize and move the mouse in it. The first client that types acquires an unowned terminal; another takes it with TakeControl (Ctrl+Shift+T). A lease survives a network blink (LEASE_GRACE), so a redial does not turn a shell read-only:
Closing a pane kills its child; closing an editor window does not.
Shims and restart
On unix every terminal's PTY and child live in a shim: the runner binary started as zeughaus-runner shim <state-dir>/terminals/<id>. It detaches into a session of its own, starts the profile from spec.json, tees a job's log, keeps the last 4 MiB of output and serves one session at a time on sock. A new session gets that replay first, then live output, with no gap between them. Dropping a session leaves the shim running; only Close (a closed pane, CloseTerminal) ends it.
The mux writes <state-dir>/workspace.json after every structural change. A starting runner reattaches every saved terminal, rebuilds the workspace around the ones that came back, gives every graph it owns and no pane shows a tab, and closes shims nothing refers to.
SIGUSR1 saves and execs the runner's binary again with the same arguments. The incarnation is new, the terminal and tab ids are the old ones, so an editor keeps its active tab and focus. SIGINT and SIGTERM drain and exit, leaving shells in their shims for the next runner. Windows keeps terminals in the runner process.
Locale
Shells and jobs inherit the runner's environment. The runner fixes its locale before any thread exists (zeughaus-terminal/src/locale.rs): its own locale variables when it has any, else the first readable locale.conf, or on macOS the AppleLocale default; a non-UTF-8 character type gets a UTF-8 LC_CTYPE. The editor's locale plays no part.
Security
The runner's identity (runner.pem) and one client identity (client.pem) live in the state directory, owner-only and written atomically; a PEM that exists but does not parse is an error, never overwritten. The listener requires a trusted client certificate on every path (client.pem plus any clients/*.pem), and a bind on anything but loopback without client trust refuses to start.
Every /mux exchange is authorized by the peer identity weida proved; a terminal id is a name, never a credential. A client creates terminals only from runner-side profiles (the login shell) or attaches ones the runner created for a job; no mux command carries an argv.
THE SAMPLE FEED
Frames never touch the store. A viewer holds one standing exchange per node and pin on /samples (the frame feed): it sends a FeedRequest once, naming the size it draws, and the runner scales the newest frame to a ladder tier of 240, 360, 480, 720 or 1080 lines (box-averaged, at most 4x4 samples per output pixel) and writes FrameHeader-prefixed frames until the viewer stops reading.
Two viewers of similar size share one scaled result. Backpressure is QUIC's: a slow viewer gets fewer frames, always the current one, never a backlog. The runner keeps an authoritative frame registry per pin and an LRU of scaled results keyed by pin, tier and sequence.
JOBS
A job.run node (zeughaus-job) executes a process with a beginning and an end, a CI step or an all-night agent session, as a terminal the runner owns. A failed job is a terminal to attach to, not a log to read.
The node
Settings: command (split like a shell splits words; no shell runs), env, cwd, artifacts (globs relative to cwd), keep_on_failure (default true). Inputs: run (trigger) and cwd (sample; wired, it wins over the setting). Outputs: ok, failed (the exit code, -1 for a signal or a kill) and dir (the run directory), so success and failure each drive their own trigger wire. Full pin and setting reference: zeughaus-nodes(7).
The node refuses while a run is live ("job busy"), while the runner is held, and in a process that does not execute: the editor registers JobPlugin::detached(), which only contributes the catalog entry. The program sees ZEUGHAUS_RUN_DIR and, when the trigger carried text, ZEUGHAUS_PAYLOAD.
A run
Each run gets a run directory, <state-dir>/runs/<id>/; ids continue past whatever is on disk, so a restart never reuses one. The process starts as a terminal with no pane, whose PTY bytes the shim appends to log before they are parsed. When the child exits, exit is written (code, killed, started, finished), declared artifacts are copied under artifacts/, and the outputs are delivered like any async result.
A run that exited 0 closes its terminal; every other outcome keeps it. With keep_on_failure (unix) a /bin/sh wrapper writes a non-zero exit code to <run_dir>/code and execs $SHELL in the same directory and environment, so a failed job is a place to look. --keep-runs (default 50) is how many successful runs stay; failed runs are never pruned. The store holds nothing about runs; /runs serves run files by range from the machine that produced them.
A job's terminal outlives its pane: it is listed as detached while no pane shows it, closing its pane detaches it, and CloseTerminal kills it. flow.all is the fan-in: it fires once every wired input has fired since it last fired.
Hold and drain
/hold (HoldRequest to HoldReply { held, live_runs }) flips the job host's hold flag; the palette offers "Runner / Hold" and "Runner / Release". SIGINT and SIGTERM hold the runner and let it exit once no run is live (drain); a runner with no live runs exits at once.
A restarted runner adopts every reattached run without an exit record (JobHost::adopt_runs): it waits for it, then writes exit and copies the artifacts. The node that started it is gone with the old process, so its ok and failed pins do not fire; adopted runs count as live for a drain.
Triggers and the CLI
A trigger carries an optional payload (TriggerRequest { node_id, payload }) that becomes the node's fire parameter; a bare press sends none. Triggers come through weida only: no HTTP listener and no polling in the runner.
zeughaus-runner trigger and zeughaus-runner hold are the runner binary as a client of /triggers and /hold, with the client identity from the state directory; no store is involved. A run started by the CLI gets a tab in the runner's locked Triggered group; a press from an editor leaves its run detached. See zeughaus-runner(1).
FUTURE DIRECTIONS
None of the following is built:
- Staged deployment of graph versions (draft, staged, deployed).
- Placement of nodes across several runners within one graph. A graph runs on the runner it names, whole; edges between graphs of different runners are not carried. Routing between runners would go over the weida broker, not the store.
- Opt-in per-node capture of results into a database; the recorder plugin writes datasets to disk instead.
- Queue semantics on edges; every edge is last-value. For jobs that means one run at a time per node. A queue depth per node, and per-trigger instantiation of a pipeline subgraph for parallel branch builds, are not built.
- Rich per-type inspection widgets on edges; nodes show text or a frame.
- A browser editor that syncs with the store; the wasm build edits locally.
- Terminal image protocols; the wire model reserves kinds for them.
- Jobs, decided but not built:
- Reading run files in the editor;
/runsis served, nothing calls it. - Webhooks (Gitea) landing on the weida broker once it speaks HTTP, relayed to
/triggers. freeze(SIGSTOPor the cgroup freezer) as a per-node hold policy, and automatic host-state detection (game running, user idle, GPU busy).- Secrets through weida's wrapped-secret flow: one refreshable token per run, child tokens per service ordered through it, everything invalidated when the run ends; nothing secret-shaped in the store or in settings.
- Workspace nodes (checkout, btrfs snapshot) producing the
cwda job runs in, with caches per runner under its state directory. - VM guests as machines with their own runner.
vm/win11/is the reference QEMU lifecycle for a headless Windows 11 guest; booting it from a job is not wired.
- Reading run files in the editor;
SEE ALSO
zeughaus(1), zeughaus-runner(1), zeughaus-install(8), zeughaus-nodes(7), zeughaus-keys(7), zeughaus-files(5), zeughaus-glossary(7)
DESIGN.md, zeughaus-module/src/lib.rs, zeughaus-link/src/lib.rs, zeughaus-runtime, zeughaus-runner/src/runner.rs, zeughaus-runner/src/mux, zeughaus-terminal/src/shim, zeughaus-runner/src/jobs.rs