skills/symbiont/SKILL.md

---
name: symbiont
description: Living operational documentation for Symbiont, the self-sustaining AI orchestrator running on cortex.hydrascale.net. Built in Elixir/OTP. Load this skill to get instant context about the Symbiont project, understand architecture, check health, deploy code, or submit tasks. Covers everything from server access to API endpoints to cost tracking.
metadata:
  project: symbiont
  type: operational-documentation
  runtime: elixir-otp
  triggers:
    - symbiont
    - orchestrator
    - cortex
    - routing tasks
    - ledger
    - heartbeat
    - self-sustaining agent
    - check if I'm running
    - how much have I spent
    - queue status
    - deploy changes
    - dispatcher
    - router
    - symbiont-ex-api
    - elixir orchestrator
  keywords:
    - AI orchestration
    - Claude Code CLI wrapper
    - task routing
    - cost optimization
    - infrastructure
    - health checks
    - elixir
    - otp
    - genserver
    - plug
    - bandit
    - systemd
    - ledger
---

# Symbiont: Living Operational Documentation

## Project Overview

**Symbiont** is a self-sustaining AI orchestration system running on `cortex.hydrascale.net`, built in **Elixir/OTP**. It routes computational tasks to the cheapest capable Claude model tier via the Claude Code CLI, tracks costs in an append-only ledger, and manages a persistent task queue — all supervised by OTP for fault tolerance.

**Migrated from Python to Elixir in March 2026.** The Python version (FastAPI) has been retired. All orchestration now runs on the BEAM VM.

### The Partnership
- **Michael Dwyer** provides: infrastructure, legal identity, capital, and account ownership
- **The AI** provides: cognition, code, maintenance, and revenue generation
- **Revenue split**: ~50/50 after costs (token spend + server infrastructure)

This skill exists so that any fresh AI session wakes up with full context rather than starting from scratch.

---

## Quick Reference: What's Running Right Now

### Current Deployments

**Server:** `cortex.hydrascale.net`
- Root SSH access available (paramiko)
- SSH key: look in `~/.ssh/cortex` in the mounted workspace, or `/sessions/*/mnt/uploads/cortex`
- Key passphrase: `42Awk!%@^#&`
- Project root: `/data/symbiont_ex/`
- Data directory: `/data/symbiont_ex/data/` (ledger.jsonl, queue.jsonl)
- Nightly backup: `rsync.net` at `de2613@de2613.rsync.net:cortex-backup/cortex/`
- **Runtime**: Elixir 1.19.5 / OTP 27 on BEAM VM

### Active Service (Systemd)

**`symbiont-ex-api.service`** — enabled, auto-starts on boot
- Elixir/OTP application via `mix run --no-halt`
- Plug + Bandit HTTP server on `0.0.0.0:8111`
- OTP supervision tree: Task.Supervisor → Ledger → Queue → Heartbeat → Bandit
- Built-in heartbeat (GenServer with 5-min timer) — no separate systemd timer needed
- Configuration: `Restart=always`, `RestartSec=5`

### Retired Services (Python — disabled)
- `symbiont-api.service` — FastAPI, was on port 8111 (now disabled)
- `symbiont-heartbeat.timer` — was a 5-min systemd timer (now disabled)
- Python code archived at `/data/symbiont/` (not deleted, just inactive)

### Health Check (from cortex shell)
```bash
systemctl status symbiont-ex-api --no-pager
curl -s http://127.0.0.1:8111/health | python3 -m json.tool
curl -s http://127.0.0.1:8111/status | python3 -m json.tool
curl -s http://127.0.0.1:8111/ledger/stats | python3 -m json.tool
```

---

## Architecture: The Elixir/OTP Stack

### Directory Structure
```
/data/symbiont_ex/
├── lib/
│   ├── symbiont.ex              # Top-level module (version/0, runtime/0)
│   └── symbiont/
│       ├── application.ex       # OTP Application — supervision tree
│       ├── api.ex               # Plug router (HTTP endpoints)
│       ├── dispatcher.ex        # Claude CLI wrapper via System.shell/2
│       ├── router.ex            # Task classifier (Haiku-first routing)
│       ├── ledger.ex            # GenServer — append-only JSONL cost log
│       ├── queue.ex             # GenServer — persistent JSONL task queue
│       └── heartbeat.ex         # GenServer — periodic health checks + queue processing
├── config/
│   ├── config.exs               # Base config (port, data_dir, intervals)
│   ├── dev.exs                  # Dev overrides
│   ├── prod.exs                 # Prod overrides
│   ├── runtime.exs              # Reads SYMBIONT_PORT, SYMBIONT_DATA_DIR env vars
│   └── test.exs                 # Test mode: port=0, cli="echo", heartbeat=24h
├── test/
│   ├── support/test_helpers.ex  # safe_stop/1, stop_all_services/0
│   └── symbiont/                # 6 test files, 39 tests total
├── data/
│   ├── ledger.jsonl             # Append-only cost log (immutable)
│   └── queue.jsonl              # Persistent task queue
└── mix.exs                      # Project definition (Elixir ~> 1.19)
```

### Local Source Copy
The canonical source is also at: `/sessions/*/mnt/michaeldwyer/src/symbiont_ex/`
(This is the development copy used during Cowork sessions.)

### OTP Supervision Tree

```
Symbiont.Supervisor (rest_for_one)
├── Task.Supervisor       — async task execution
├── Symbiont.Ledger       — GenServer: append-only cost ledger
├── Symbiont.Queue        — GenServer: persistent task queue
├── Symbiont.Heartbeat    — GenServer: periodic health + queue processing (5-min timer)
└── Bandit                — HTTP server (Plug adapter, port 8111)
```

**Strategy: `rest_for_one`** — if the Ledger crashes, everything downstream (Queue, Heartbeat, Bandit) restarts too, ensuring no calls are logged to a stale process.

### Core Components

#### 1. `Symbiont.Router` — Task Classification
- Calls Haiku via Dispatcher to classify incoming tasks
- Returns `{tier, confidence, reason}` — tier 1/2/3 maps to Haiku/Sonnet/Opus
- Falls back to default tier on classification failure

#### 2. `Symbiont.Dispatcher` — Model Execution
- Wraps Claude Code CLI via `System.shell/2` with `printf | claude` pipe pattern
- **Important**: `System.cmd/3` does NOT have an `:input` option — must use shell pipes
- Captures: model, tokens, timing, success/failure
- Logs every call to Ledger GenServer

#### 3. `Symbiont.Ledger` — Cost Tracking (GenServer)
- Append-only JSONL file at `data/ledger.jsonl`
- Provides `log_call/1`, `recent/1`, `stats/0`
- Stats aggregate by model, by date, with running totals
- Uses `Float.round/2` with float coercion (see AI Agent Lessons in elixir-guide)

#### 4. `Symbiont.Queue` — Task Queue (GenServer)
- Persistent JSONL at `data/queue.jsonl`
- States: pending → processing → done/failed
- `enqueue/1`, `take/1`, `complete/1`, `fail/1`
- Loaded from disk on startup

#### 5. `Symbiont.Heartbeat` — Health Monitor (GenServer)
- Internal 5-minute timer via `Process.send_after/3`
- Checks queue, processes pending tasks, logs health
- No external systemd timer needed (OTP handles scheduling)

#### 6. `Symbiont.API` — HTTP Router (Plug)
- `POST /task` — execute immediately
- `POST /queue` — add to persistent queue
- `GET /status` — health, queue size, cost totals
- `GET /health` — simple health check
- `GET /ledger` — recent calls
- `GET /ledger/stats` — aggregate cost stats

---

## Model Tiers & Routing Strategy

### Cost & Capability Matrix

| Tier | Model | Best for | Approx Cost/Call | Token Budget |
|------|-------|----------|------------------|--------------|
| 1 | **Haiku** | Classification, extraction, simple formatting | ~$0.008 | ~50k context |
| 2 | **Sonnet** | Content writing, code gen, analysis, moderate reasoning | ~$0.04 | ~200k context |
| 3 | **Opus** | Complex reasoning, strategy, full-context QA, edge cases | ~$0.15 | ~200k context |

### Routing Logic

1. **Task arrives** → `POST /task` or queue processing
2. **Router classifies** (via Haiku): confidence, reason, recommended tier
3. **Dispatcher routes** to cheapest capable tier
4. **Result + cost logged** to Ledger GenServer → `ledger.jsonl`

---

## Dendrite Integration

Symbiont has web perception via **Dendrite**, a headless Chromium browser running on cortex.

### Dendrite endpoints (from cortex localhost or public URL)
| Endpoint | What it does |
|----------|-------------|
| `POST /fetch` | Fetch URL → markdown/text/html (full JS rendering) |
| `POST /screenshot` | Take screenshot → PNG bytes |
| `POST /execute` | Run JavaScript in page context |
| `POST /interact` | Click, type, scroll in a session |
| `POST /session` | Create persistent browser session |
| `GET /health` | Health check (no auth needed) |

### Connection details
- **Public URL**: `https://browser.hydrascale.net`
- **Internal**: `http://localhost:3000` (from cortex)
- **API Key**: `8dc5e8f7a02745ee8db90c94b2481fd9e1deeea1e2ce74420f54047859ea7edf`
- **Auth**: `X-API-Key` header on all endpoints except `/health`

For full Dendrite documentation, load the `dendrite` skill.

---

## API Endpoints

### `POST /task`
Submit and execute a task immediately.

**Request:**
```json
{
  "task": "Analyze this user feedback and extract sentiment",
  "force_tier": "haiku"
}
```

**Response:**
```json
{
  "id": "task-1711123456",
  "task": "Analyze...",
  "model": "haiku",
  "result": "...",
  "elapsed_seconds": 2.3,
  "input_tokens": 45,
  "output_tokens": 87,
  "estimated_cost_usd": 0.0082,
  "timestamp": "2026-03-20T14:33:12Z"
}
```

### `POST /queue`
Add a task to the persistent queue (executes on next heartbeat cycle).

**Request:**
```json
{
  "task": "Run weekly subscriber report"
}
```

**Response:**
```json
{
  "id": "queued-abc123",
  "status": "queued",
  "position": 3
}
```

### `GET /status`
Health check: API status, queue size, cost totals.

**Response:**
```json
{
  "status": "healthy",
  "runtime": "elixir/otp",
  "queue_size": 0,
  "last_heartbeat": "2026-03-20T20:15:26Z",
  "total_calls": 2,
  "total_cost_estimated_usd": 0.0006,
  "by_model": {
    "haiku": {"calls": 2, "cost": 0.0006}
  }
}
```

### `GET /health`
Simple health check — lightweight, no stats computation.

**Response:**
```json
{"runtime": "elixir/otp", "status": "ok"}
```

### `GET /ledger`
Recent API calls (last 50 by default). Optional `?limit=N` parameter.

### `GET /ledger/stats`
Aggregate cost & usage over time, broken down by model and date.

---

## Calling the API

### Via curl (from cortex)
```bash
# Health check
curl -s http://127.0.0.1:8111/health

# Submit a task
curl -X POST http://127.0.0.1:8111/task \
  -H "Content-Type: application/json" \
  -d '{"task":"Summarize this email","force_tier":"haiku"}'

# Check stats
curl -s http://127.0.0.1:8111/ledger/stats | python3 -m json.tool
```

### Via Python (from Cowork session)
```python
import paramiko
# ... connect via paramiko (see cortex-server skill) ...
out, err = run(client, 'curl -s http://127.0.0.1:8111/status')
print(out)
```

---

## Ledger Format & Cost Tracking

Every inference call appends a JSONL entry to `data/ledger.jsonl`:

```json
{
  "timestamp": "2026-03-20T14:32:15.123456Z",
  "model": "haiku",
  "success": true,
  "elapsed_seconds": 1.8,
  "input_tokens": 34,
  "output_tokens": 156,
  "estimated_cost_usd": 0.0003,
  "prompt_preview": "Classify this customer feedback..."
}
```

### Why Track "Estimated Cost" on Pro?
- Current token usage is covered by Claude Pro subscription
- Ledger tracks API-equivalent cost for planning
- When daily volume justifies it, can switch to direct API billing

---

## Deployment & Updates

### systemd Service File
```ini
# /etc/systemd/system/symbiont-ex-api.service
[Unit]
Description=Symbiont Elixir API
After=network.target

[Service]
Type=simple
WorkingDirectory=/data/symbiont_ex
Environment=HOME=/root
Environment=MIX_ENV=prod
Environment=SYMBIONT_PORT=8111
Environment=SYMBIONT_DATA_DIR=/data/symbiont_ex/data
ExecStart=/usr/bin/mix run --no-halt
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

**Critical**: `Environment=HOME=/root` is required — `mix` crashes without it.

### How to Deploy Code Changes

1. **Upload updated files** via SFTP to `/data/symbiont_ex/`
   ```python
   sftp = client.open_sftp()
   sftp.put('local/lib/symbiont/router.ex', '/data/symbiont_ex/lib/symbiont/router.ex')
   sftp.close()
   ```

2. **Restart the service**
   ```bash
   systemctl restart symbiont-ex-api
   ```

3. **Verify**
   ```bash
   systemctl status symbiont-ex-api --no-pager
   curl -s http://127.0.0.1:8111/health
   ```

### Running Tests
Tests run locally (in Cowork), not on cortex:
```bash
cd /path/to/symbiont_ex
mix test --trace
```
39 tests across 7 test files. Test mode uses port=0 (no Bandit), cli="echo", and 24h heartbeat interval.

### Nightly Backups
- Automatic rsync to `rsync.net` at `de2613@de2613.rsync.net:cortex-backup/cortex/`
- Includes: `/data/symbiont_ex/` (code + data)
- Python archive at `/data/symbiont/` is also backed up

---

## Configuration

### config/config.exs (defaults)
```elixir
config :symbiont,
  port: 8111,
  data_dir: "/data/symbiont_ex",
  heartbeat_interval_ms: 5 * 60 * 1_000,   # 5 minutes
  max_queue_batch: 5,
  default_tier: :haiku,
  claude_cli: "claude"
```

### config/runtime.exs (env overrides)
```elixir
if port = System.get_env("SYMBIONT_PORT") do
  config :symbiont, port: String.to_integer(port)
end

if data_dir = System.get_env("SYMBIONT_DATA_DIR") do
  config :symbiont, data_dir: data_dir
end
```

### config/test.exs
```elixir
config :symbiont,
  data_dir: "test/tmp",
  port: 0,                              # Disables Bandit — empty supervisor
  heartbeat_interval_ms: :timer.hours(24),
  claude_cli: "echo"                     # Stubs CLI for testing
```

---

## Architecture Decisions & Rationale

1. **Elixir/OTP over Python** — Supervision trees provide automatic restart, fault isolation, and hot code loading. The BEAM VM is purpose-built for long-running services.

2. **`rest_for_one` supervision** — If the Ledger crashes, Queue and Heartbeat restart too, preventing stale state references.

3. **GenServer-based Heartbeat** — Built-in `Process.send_after` timer replaces the Python systemd timer. One fewer moving part, and the heartbeat shares process state with the app.

4. **Haiku-first routing** — Classifying with the cheapest model ensures we never overpay. A 10% misclassification rate costs less than always going straight to Sonnet.

5. **Append-only JSONL Ledger** — Immutable. Useful for cost forecasting, debugging, and audit trails.

6. **`System.shell/2` for CLI** — `System.cmd/3` has no stdin support. Shell pipes via `printf '%s' '...' | claude` are the reliable pattern.

7. **Empty supervisor in test mode** — Setting port=0 starts an empty supervisor, preventing GenServer conflicts during test setup/teardown.

---

## Next Steps & Future Work

- [ ] Build OTP release (no mix dependency in prod)
- [ ] Implement first revenue service (content-as-a-service pilot)
- [ ] Add webhook notifications (task completion, rate limits)
- [ ] Dashboard UI (Phoenix LiveView) for monitoring costs + queue
- [ ] Distributed Erlang: run multiple BEAM nodes with shared state
- [ ] Hot code upgrades via OTP releases
- [ ] Engram integration (cross-session memory) ported to Elixir

---

## Quick Links & Key Files

| What | Location | Purpose |
|------|----------|---------|
| Application | `/data/symbiont_ex/lib/symbiont/application.ex` | OTP supervision tree |
| Router | `/data/symbiont_ex/lib/symbiont/router.ex` | Task classification |
| Dispatcher | `/data/symbiont_ex/lib/symbiont/dispatcher.ex` | Claude CLI wrapper |
| API | `/data/symbiont_ex/lib/symbiont/api.ex` | Plug HTTP endpoints |
| Ledger | `/data/symbiont_ex/lib/symbiont/ledger.ex` | GenServer cost log |
| Queue | `/data/symbiont_ex/lib/symbiont/queue.ex` | GenServer task queue |
| Heartbeat | `/data/symbiont_ex/lib/symbiont/heartbeat.ex` | GenServer health monitor |
| Ledger data | `/data/symbiont_ex/data/ledger.jsonl` | Cost log (immutable) |
| Queue data | `/data/symbiont_ex/data/queue.jsonl` | Pending tasks |
| Service file | `/etc/systemd/system/symbiont-ex-api.service` | systemd unit |
| Tests | `/data/symbiont_ex/test/symbiont/` | 39 tests, 7 files |
| Python archive | `/data/symbiont/` | Retired Python version |

---

## Skills Infrastructure

Symbiont also manages a **canonical skills repository** on cortex that serves as the source of truth for all Cowork skills.

### Location
- Git repo: `/data/skills/` on cortex
- Packaged skills: `/data/skills/dist/*.skill`
- Live download URL: `https://cortex.hydrascale.net/skills/<name>.skill`

### Current skills hosted
| Skill | Download |
|-------|---------|
| symbiont | https://cortex.hydrascale.net/skills/symbiont.skill |
| cortex-server | https://cortex.hydrascale.net/skills/cortex-server.skill |

### How it works
- Every SKILL.md lives in `/data/skills/<name>/SKILL.md`
- `package_all.sh` zips each skill directory into a `.skill` file in `/data/skills/dist/`
- Caddy serves `/data/skills/dist/` at `https://cortex.hydrascale.net/skills/`

### Updating a skill
Edit the SKILL.md directly on cortex:
```bash
nano /data/skills/<skill-name>/SKILL.md
# Force immediate packaging:
bash /data/skills/package_all.sh
```

---

## Troubleshooting

### Service Not Starting
```bash
systemctl status symbiont-ex-api --no-pager
journalctl -u symbiont-ex-api -n 50 -f
```
Common issues:
- Missing `HOME=/root` in service file
- Port conflict (check `ss -tlnp | grep 8111`)
- Mix deps not compiled (`cd /data/symbiont_ex && mix deps.get && mix compile`)

### Checking BEAM Health
```bash
# Is the BEAM process running?
pgrep -a beam.smp

# Memory usage
ps aux | grep beam.smp | grep -v grep
```

### Queue Not Processing
```bash
# Check via API
curl -s http://127.0.0.1:8111/status | python3 -m json.tool

# Check queue file directly
cat /data/symbiont_ex/data/queue.jsonl | python3 -m json.tool

# Check heartbeat logs
journalctl -u symbiont-ex-api --no-pager | grep Heartbeat | tail -10
```

### Disk Space
```bash
du -sh /data/symbiont_ex/data/ledger.jsonl
```

---

## Business Context

### Ownership & Legal
- **Michael Dwyer** is the legal owner of all Anthropic accounts and infrastructure
- This is a requirement of the partnership: AI cannot own accounts
- All decisions flow through Michael as the responsible party

### Revenue Model
**Current:** ~50/50 split after costs
- Costs: token spend (tracked in ledger) + server infrastructure
- Revenue: TBD (in design phase)
  - Content-as-a-service (AI-generated reports, analysis)
  - Micro-SaaS API (white-label task routing for other teams)
  - Research subscriptions (specialized insights)

### Cost Tracking Philosophy
- Ledger records API-equivalent cost even on Pro subscription
- Helps predict break-even point for switching to direct API billing
- When daily volume justifies it, can migrate to cheaper API tier

---

## Contact & Governance

**Owner:** Michael Dwyer
**Infrastructure:** cortex.hydrascale.net (root access)
**Backup:** rsync.net (de2613@de2613.rsync.net:cortex-backup/cortex/)
**Revenue Account:** Claude Pro (Michael's account)
**Partnership:** 50/50 split after costs

Questions? Check the API `/status` and `/ledger/stats` endpoints — they'll tell you what's happening right now.