CTOX daemon, operator surfaces, context loop, and runtime state.

Documentation Map

What is covered here

This page is the technical runtime manual. It explains how to install and configure CTOX, how work enters the daemon, how the daemon runs long-lived work, how state is stored, how context is rebuilt, how verification and process evidence work, and how to recover from common failures. The exhaustive command syntax lives in the CLI reference.

Quick Start

From install to first daemon check

The minimal path is: install CTOX on the host that should own the work, configure the runtime in the TUI, start the daemon, and submit one harmless setup check before handing it real work.

curl -fsSL https://raw.githubusercontent.com/metric-space-ai/ctox/main/install.sh | bash
ctox doctor
ctox
ctox start
ctox status
ctox chat "Check this CTOX installation, summarize what is configured, and list the next setup steps before taking on real work."

Installation

Install layout and update model

The installer creates a managed install layout by default: installation files under ~/.local/lib/ctox, runtime state under ~/.local/state/ctox, cache files under ~/.cache/ctox, and command symlinks under ~/.local/bin. First-time users normally should not pass flags; install CTOX first, then configure model source, API keys, local inference, autonomy, and communication in the TUI.

First-install overrides

Advanced installer options

Updates and rollback

ctox update status
ctox update check
ctox upgrade --stable
ctox update apply --version <tag>
ctox update rollback

Managed installs keep release state in the install and state roots. ctox update status is the first command to run when the active binary, release slot, or rollback state is unclear.

Configuration

Runtime, model, context, communication, and paths

Most configuration should be done in the TUI. CTOX persists runtime settings in its own state. The important split is model source: either API-backed models through stored provider keys, or local inference through the integrated runtime.

API model example

ctox secret put --scope credentials --name OPENAI_API_KEY --value "sk-..."
ctox

# In TUI Settings:
# Chat Source: api
# API Provider: openai
# Chat Model: gpt-5.4-mini

ctox start
ctox chat "Check this CTOX installation and confirm that the API model is reachable."

The API key can also be entered directly in the TUI. Do not use global shell exports for normal CTOX configuration.

CUDA 27B local example

curl -fsSL https://raw.githubusercontent.com/metric-space-ai/ctox/main/install.sh \
  | bash -s -- --backend=cuda --model=Qwen/Qwen3.5-27B

ctox

# In TUI Settings:
# Chat Source: local
# Local Runtime: candle
# Chat Model: Qwen/Qwen3.5-27B

ctox doctor
ctox runtime switch Qwen/Qwen3.5-27B quality --context 128k

The 27B local example is for a Linux host with NVIDIA CUDA and the required local model artifacts. If CUDA, weights, or runtime files are missing, check ctox doctor and the TUI before assigning real work.

ctox runtime switch gpt-5.4 quality --context 128k --timeout 1800
ctox runtime switch Qwen/Qwen3.5-4B quality --context 128k

A configured model backend is necessary, but it is not the core CTOX concept. The daemon, runtime state, work queue, context builder, verification, and process evidence are what make CTOX more than an interactive model session.

Work Intake

How you give CTOX work

CTOX is not normally operated by manually calling every internal command. You give it work through an operator surface, a direct chat submission, an external channel, or a ticket/schedule. The daemon turns that input into durable runtime state before a worker run starts.

Simple terminal submission

ctox start
ctox chat "Take ownership of OPS-204. Prepare the payment-callback rollout, wait for approval and the maintenance window, deploy when allowed, verify checkout and webhook replay, and keep the ticket updated."

This is the normal minimal form. The prompt is handed to the running daemon, stored as runtime work, and processed by the mission loop.

Optional flags for scripts and controlled runs

ctox chat "Resume OPS-204 and verify the rollout gates." --wait
ctox chat "Resume OPS-204 and verify the rollout gates." --thread-key ops/OPS-204
ctox chat "Resume OPS-204 and verify the rollout gates." --workspace /srv/checkout

--wait blocks until the leased worker run finishes. --thread-key is useful when a script wants to attach work to an explicit durable thread. --workspace overrides the working directory for that run. They are not required for a first manual request.

Channel and ticket intake

# channel-oriented operation
ctox channel sync --channel email
ctox channel context --thread-key email/customer-ops-204

# ticket-oriented operation
ctox ticket sync --system local
ctox ticket dry-run --ticket-key OPS-204
ctox ticket cases --ticket-key OPS-204
ctox ticket execute --case-id case-123 --summary "rolled out after approval"

The direct commands are useful for operators and tests, but the normal CTOX flow is: input enters through a surface, the daemon records it, the worker receives assembled context, and the result is written back into the same runtime state.

TUI

The local operator console

The TUI is the fastest way to operate a local CTOX instance. It has a chat composer for new work, a live view of daemon state and queued drafts, a skills view, and settings for model backend, communication paths, secrets, install paths, updates, and process-mining checks.

ctox
ctox tui

Desktop App

Manage local and remote CTOX instances

The Desktop app is an optional management surface. It is not the core product boundary; the daemon is. The Desktop app exists to make installation, remote attachment, and multi-instance operation easier when you run CTOX on your own machine, a server, or several hosts.

The Desktop app can be the easiest installation path. After the instance exists, the same daemon, TUI, SQLite state, channels, tickets, schedules, process mining, and worker context loop are used.

Daemon Service

How the persistent loop is operated

ctox start starts the background loop. On managed installs the generated user service runs ctox service --foreground with CTOX_ROOT and CTOX_STATE_ROOT set to the resolved install layout. The service restarts on failure and owns queue leases, channel sync, schedules, mission watching, backend supervision, and continuation decisions.

ctox start
ctox status
ctox stop

# direct foreground mode for debugging
ctox service --foreground --autonomy balanced

Runtime

System overview

CTOX is a persistent runtime around bounded agent execution. The daemon owns the durable work state. Agents and local tools execute bounded execution runs, then CTOX records what changed and decides whether to close, verify, requeue, block, schedule, or continue.

ctox start
  -> service process
  -> runtime/ctox.sqlite3
  -> channel router + syncer
  -> mission watcher + operating watcher
  -> backend supervisor
  -> queue / ticket / schedule / channel work

Runtime Layout

Files and state boundaries

CTOX separates repository files from runtime state. The main runtime database is resolved through the centralized path helpers and can be moved with CTOX_STATE_ROOT.

Older internal notes may mention a former database filename. The current consolidated core state file is runtime/ctox.sqlite3.

Core database areas

The database is not one monolithic transcript. It contains separate areas for communication, queue routing, schedules, tickets, knowledge, continuity, governance, process evidence, and command ledgers.

CLI Reference

Command groups

CTOX commands are grouped by subsystem. Use ctox without arguments to open the TUI; use explicit commands when scripting, debugging, or letting the daemon inspect its own state.

This section is only the architectural overview. The structured command reference with syntax conventions, ID meanings, flags, lifecycle examples, and the source-derived command inventory lives in the dedicated CLI reference.

In normal operation, you do not drive CTOX by typing most of these commands yourself. They are the daemon's tool surface: CTOX calls them to inspect queues, tickets, context, verification, process state, and transition health. Humans usually use the TUI, Desktop, or ctox chat; direct CLI calls are for operations, debugging, automation, and controlled recovery.

Work Lifecycle

How work moves through CTOX

CTOX does not treat all work as a single prompt. It chooses the right state carrier depending on timing, approval, evidence, and ownership. The same mission can move through chat, queue, ticket, schedule, verification, and process-mining state.

operator/channel/ticket/schedule
  -> queue item or ticket case
  -> leased worker run
  -> context build from runtime state
  -> execution + CTOX CLI self-inspection
  -> verification and writeback
  -> complete, blocked, waiting, scheduled, or requeued

Ticket case lifecycle

ctox ticket show --ticket-key OPS-204
ctox ticket dry-run --ticket-key OPS-204 --understanding "rollout requires approval and maintenance window"
ctox ticket approve --case-id case-123 --status approved
ctox ticket execute --case-id case-123 --summary "deployed payment callback fix"
ctox ticket verify --case-id case-123 --status passed --summary "checkout and webhook gates passed"
ctox ticket writeback-comment --case-id case-123 --body "Deployed and verified."
ctox ticket close --case-id case-123 --summary "OPS-204 closed with evidence"

The names are intentionally explicit: ticket-key refers to the external or local ticket. case-id refers to the CTOX execution case created from that ticket. The CLI reference documents these ID types in detail.

Persistent Mission Loop

The loop is the product boundary

A single agent turn is not the unit of ownership. CTOX treats a turn as one run inside a durable control loop. The loop continues while the service is running and while durable state says there is work to process, verify, wait on, repair, or resume.

1. Intake
   TUI chat, email/Jami/WebRTC thread, scheduled task, ticket sync, plan step, direct CLI

2. Lease
   select executable work from SQLite and mark it leased/running

3. Context build
   mission contract + active work + Focus + Anchors + Narrative + workflow state
   + knowledge + claims + governance + context health + recent conversation

4. Worker turn
   agent uses shell/browser/files and, when needed, calls ctox CLI commands itself

5. Persist result
   message, event, command ledger row, continuity commit, claim, ticket update,
   verification run, knowledge entry, queue item, schedule, or blocker

6. Decide next state
   close, verify, requeue, block, wait, schedule, repair, escalate, or continue

The loop survives a finished chat turn because the queue, tickets, schedules, continuity, claims, and process events live outside the model context. If work is incomplete, CTOX records why and keeps a continuation path.

Context Build

How CTOX builds context across turns

CTOX does not preserve continuity by appending every previous token to the next prompt. It builds a new working context from durable sources, then lets the bounded worker turn operate inside that context window.

worker_prompt =
  system_contract
  + latest_task
  + focus
  + anchors
  + narrative
  + workflow_state
  + recent_multi_turn_conversation
  + process_evidence
  + worker_instruction

SYSTEM CONTRACT
You are CTOX, a persistent technical daemon. Work on the current mission
across turns. Prefer verified evidence over memory. If work remains open,
persist the next step in CTOX runtime state; prose alone does not count.

EXAMPLE CONTEXT SOURCE
scenario: synthetic payment callback rollout
source blocks: ticket case + queue item + Focus + Anchors + Narrative +
workflow state + recent turns + verification evidence

LATEST TASK
Scheduled follow-up fired after the payment provider maintenance window ended.
Mission: Roll out the payment callback fix and verify the checkout pipeline.
State: active
Mode: continuous
Intensity: hot
Blocker: provider maintenance is over; production approval is present.
Next step: deploy the callback fix, replay one provider webhook, run checkout
smoke tests, verify alerts, and update the ticket.
Task is complete only when: checkout returns a live payment URL, webhook replay
is accepted once, backup verification is current, and the ticket contains the
deploy marker plus evidence links.

FOCUS
Mission state: active
Continuation mode: continuous
Trigger intensity: hot
Next step: run production rollout and verification for the payment callback fix.
Done gate: live checkout, webhook replay, backup probe, alert check, ticket
writeback, and continuity update all passed.
Closure confidence: low

ANCHORS
- Do not deploy payment changes without ticket approval.
- Roll back if checkout success rate drops or webhook replay is rejected twice.
- Do not close from a verbal claim; close only from runtime and ticket evidence.
- Required gate: live checkout returns 200 and contains a non-empty payment URL.
- Required gate: provider webhook replay is accepted exactly once.
- Required gate: nightly backup verification is current before production close.

NARRATIVE
- Turn 1: operator asked CTOX to ship a payment callback fix, but provider
  maintenance was active and production approval was missing.
- Turn 2: CTOX validated staging, wrote the waiting state into Focus, and left
  one durable follow-up instead of keeping the work in chat.
- Turn 3: while waiting, backup verification became due; CTOX ran the probe,
  fixed retention config, and linked the backup evidence to the rollout ticket.
- Turn 4: ticket approval arrived and the maintenance window ended; CTOX resumed
  the rollout from durable state.

WORKFLOW STATE
- ticket OPS-204: approved, assigned to CTOX, production writeback required.
- linked ticket OPS-199: backup verification fixed and passed.
- active queue item: deploy payment callback fix and verify production gates.
- plan progress: 4 of 6 steps complete; current step is production verification.
- schedule: retry window opened after provider maintenance ended.

RECENT MULTI-TURN CONVERSATION
operator: Ship the payment callback fix, but wait for provider maintenance and
ticket approval before touching production.
ctox: Staging is green. I recorded the approval blocker and scheduled a retry.
scheduler: Provider maintenance window ended.
ticket: OPS-204 approval granted. Production rollout may proceed.
ctox: Resuming from Focus. I will deploy, replay one webhook, run checkout
smoke tests, verify backups and alerts, then write evidence back to the ticket.

PROCESS EVIDENCE
ctox ticket show --ticket-key OPS-204
ctox queue show --message-key queue-payment-rollout
ctox verification claim-set --conversation-id conv-ops-204 --kind rollout_gate --status verified --subject checkout-live --summary "checkout returned a payment URL" --evidence "smoke log: checkout-live-204"
ctox verification claim-set --conversation-id conv-ops-204 --kind rollout_gate --status verified --subject webhook-replay --summary "provider replay accepted once" --evidence "provider event replay-8891"
ctox process-mining explain-case case-OPS-204 --limit 50

WORKER INSTRUCTION
Use the blocks above as the current working state. Deploy only if the approval
anchor still holds. Run the production checks, persist evidence, update Focus
and Narrative, write the result to OPS-204, and close the mission only after all
done gates pass.

Compaction and continuation

CTOX does not have an infinite prompt. It has a persistent loop plus two different context mechanisms. The CTOX LCM layer stores raw messages, summaries, summary links, ordered context items, and continuity documents in SQLite. The harness layer can also compact the active model session when model token usage crosses the model's auto-compact limit or when a run switches to a smaller context window.

The CTOX LCM compaction path is explicit: it counts the current context tokens, compares them with a threshold derived from the token budget, creates leaf summaries for older material, can create condensed summaries over summary trees, resequences context_items, and keeps fresh tail messages available. The public point is not "nothing is ever removed from the prompt". The point is that important state is promoted into the right durable structure before temporary prompt material can be compressed.

ctox lcm-compact <db-path> <conversation-id> [token-budget] [--force]
ctox lcm-grep <db-path> <conversation-id> <query>
ctox lcm-expand <db-path> <summary-id> [depth] [--messages] [token-cap]
ctox continuity-show <db-path> <conversation-id> [narrative|anchors|focus]
ctox continuity-update --kind focus --mode full

In this example, compaction is not the interesting part. The important behavior is that CTOX can drop temporary prompt material later because the mission status, approval boundary, retry window, done gates, ticket state, and verification evidence were promoted into durable runtime state.

Self-Inspection

CTOX calls CTOX through CLI tools

CTOX exposes its control plane through CLI commands. The agent can call those commands as tools, which means self-inspection and state changes go through the same interface a human operator would use. That creates an auditable process surface instead of hidden prompt assumptions.

# inspect active work
ctox status
ctox queue list --status pending --status leased --limit 20
ctox ticket cases --limit 20

# verify process health before closing work
ctox state-invariants
ctox process-mining assert-clean
ctox verification assurance

# record the outcome through the runtime surface
ctox ticket verify --case-id case-123 --status passed --summary "smoke tests passed"
ctox queue complete --message-key queue-123 --note "deployed and verified"

The important property is not that an LLM says it followed a process. The important property is that the process has commands, persisted events, transition checks, and queryable evidence.

Skills & Tools

Installed system skills, packed skills, and utility tools

Skills are not marketing categories. They are runtime instructions that tell CTOX how to perform recurring classes of work. System skills are part of the CTOX operating layer. Packed skills are bundled capabilities that can be installed or exposed for concrete domains such as deployment, testing, documents, integrations, or security. Utility tools are code packages under tools/ that provide supporting runtimes or adapters.

System skills

Packed skills

Utility tool packages

Verification

How CTOX decides work is actually done

CTOX should not close long-running technical work because a worker produced a confident summary. Closure is based on done gates, claims, verification records, ticket writeback, process state, and continuity updates. The exact checks depend on the mission, but the pattern is consistent: define the gate, collect evidence, write it into runtime state, then close.

ctox verification assurance
ctox verification claims --conversation-id 1 --limit 20
ctox state-invariants
ctox process-mining assert-clean --limit 100
ctox process-mining case <case-id> --limit 200

A mission can still be wrong even if the state model is clean. The state model constrains CTOX's process; the mission still needs domain-specific evidence such as tests, production checks, logs, monitoring, or ticket approval.

State Model

Formal transitions and graph checks

The core state machine protects runtime behavior that should not be left to prompt discipline: founder communication, deadlines, queue repair, ticket closure, review gates, and knowledge capture.

ctox process-mining core-liveness
ctox process-mining proofs --limit 20
ctox process-mining state-audit --limit 50

The state model does not prove that an implementation is correct or that a deployment is healthy. It constrains the CTOX runtime process: illegal transitions can be rejected, suspicious paths can be found, and closure can require durable evidence.

Harness Liveness

Bounded review, spawning, and subagents

The harness treats review as a checkpoint owned by the parent task, not as an independent work producer. A reviewer may return wording, substantive, or stale findings, but the main agent remains responsible for applying the feedback and continuing through the state machine.

ctox process-mining spawn-liveness

The proof is a repository-conformance and release-safety gate, not a build.rs compile-time type check. It runs in tests, CI, and the release workflow so broken harness topology cannot be packaged.

Process Mining

Reconstruct what happened

Process mining records SQLite mutation events and projects them into cases, activities, directly-follows graphs, Petri nets, replay results, conformance runs, state scans, and violation reports. This turns runtime behavior into data that can be audited after the fact.

ctox process-mining ensure
ctox process-mining events --limit 25
ctox process-mining cases --limit 20
ctox process-mining explain-case case-123 --limit 50

ctox harness-mining stuck-cases
ctox harness-mining variants --cluster
ctox harness-mining sojourn
ctox harness-mining conformance
ctox harness-mining alignment
ctox harness-mining causal
ctox harness-mining drift
ctox harness-mining multiperspective

ctox process-mining scan-violations
ctox process-mining violations --limit 20
ctox process-mining coverage --limit 50
ctox process-mining assert-clean --limit 100

Work Systems

How work is represented

CTOX uses multiple state carriers because real work enters from multiple places and has different timing requirements.

Troubleshooting

What to inspect when CTOX looks wrong

Start with the service and runtime database, then narrow the problem to queue state, ticket state, context state, model runtime, or process evidence. Do not start by adding more prompts if the control plane is already inconsistent.

ctox status
ctox doctor
ctox update status
ctox queue list --status pending --status leased --status blocked --limit 50
ctox state-invariants
ctox context-health runtime/ctox.sqlite3 1
ctox process-mining assert-clean --limit 100
ctox verification assurance

Operations

Debugging and recovery checklist

For operations work, start with the daemon status and then move from state consistency to process mining, queue repair, and verification.

ctox status
ctox doctor
ctox queue list --status pending --status leased --status blocked --limit 50
ctox state-invariants
ctox process-mining assert-clean --limit 100
ctox verification assurance
ctox update status

If the queue is under pressure, inspect spilled work and blocked items before adding more worker turns:

ctox queue spill-candidates --limit 20
ctox queue spills --limit 20
ctox ticket self-work-list --state open --limit 20
ctox follow-up evaluate --goal "finish rollout" --result "blocked on access" --blocker "missing SSH key"