michael/skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
master
skills/symbiont/SKILL.md
Symbiont eac71dd3c1 Update skills for Elixir Symbiont migration
2026-03-20 20:20:13 +00:00

19 KiB
Raw Permalink Blame History

name description metadata
symbiont Living operational documentation for Symbiont, the self-sustaining AI orchestrator running on cortex.hydrascale.net. Built in Elixir/OTP. Load this skill to get instant context about the Symbiont project, understand architecture, check health, deploy code, or submit tasks. Covers everything from server access to API endpoints to cost tracking.
project type runtime triggers keywords
symbiont operational-documentation elixir-otp
symbiont
orchestrator
cortex
routing tasks
ledger
heartbeat
self-sustaining agent
check if I'm running
how much have I spent
queue status
deploy changes
dispatcher
router
symbiont-ex-api
elixir orchestrator
AI orchestration
Claude Code CLI wrapper
task routing
cost optimization
infrastructure
health checks
elixir
otp
genserver
plug
bandit
systemd
ledger

Symbiont: Living Operational Documentation

Project Overview

Symbiont is a self-sustaining AI orchestration system running on cortex.hydrascale.net, built in Elixir/OTP. It routes computational tasks to the cheapest capable Claude model tier via the Claude Code CLI, tracks costs in an append-only ledger, and manages a persistent task queue — all supervised by OTP for fault tolerance.

Migrated from Python to Elixir in March 2026. The Python version (FastAPI) has been retired. All orchestration now runs on the BEAM VM.

The Partnership

  • Michael Dwyer provides: infrastructure, legal identity, capital, and account ownership
  • The AI provides: cognition, code, maintenance, and revenue generation
  • Revenue split: ~50/50 after costs (token spend + server infrastructure)

This skill exists so that any fresh AI session wakes up with full context rather than starting from scratch.

Quick Reference: What's Running Right Now

Current Deployments

Server: cortex.hydrascale.net

  • Root SSH access available (paramiko)
  • SSH key: look in ~/.ssh/cortex in the mounted workspace, or /sessions/*/mnt/uploads/cortex
  • Key passphrase: 42Awk!%@^#&
  • Project root: /data/symbiont_ex/
  • Data directory: /data/symbiont_ex/data/ (ledger.jsonl, queue.jsonl)
  • Nightly backup: rsync.net at de2613@de2613.rsync.net:cortex-backup/cortex/
  • Runtime: Elixir 1.19.5 / OTP 27 on BEAM VM

Active Service (Systemd)

symbiont-ex-api.service — enabled, auto-starts on boot

  • Elixir/OTP application via mix run --no-halt
  • Plug + Bandit HTTP server on 0.0.0.0:8111
  • OTP supervision tree: Task.Supervisor → Ledger → Queue → Heartbeat → Bandit
  • Built-in heartbeat (GenServer with 5-min timer) — no separate systemd timer needed
  • Configuration: Restart=always, RestartSec=5

Retired Services (Python — disabled)

  • symbiont-api.service — FastAPI, was on port 8111 (now disabled)
  • symbiont-heartbeat.timer — was a 5-min systemd timer (now disabled)
  • Python code archived at /data/symbiont/ (not deleted, just inactive)

Health Check (from cortex shell)

systemctl status symbiont-ex-api --no-pager
curl -s http://127.0.0.1:8111/health | python3 -m json.tool
curl -s http://127.0.0.1:8111/status | python3 -m json.tool
curl -s http://127.0.0.1:8111/ledger/stats | python3 -m json.tool

Architecture: The Elixir/OTP Stack

Directory Structure

/data/symbiont_ex/
├── lib/
│   ├── symbiont.ex              # Top-level module (version/0, runtime/0)
│   └── symbiont/
│       ├── application.ex       # OTP Application — supervision tree
│       ├── api.ex               # Plug router (HTTP endpoints)
│       ├── dispatcher.ex        # Claude CLI wrapper via System.shell/2
│       ├── router.ex            # Task classifier (Haiku-first routing)
│       ├── ledger.ex            # GenServer — append-only JSONL cost log
│       ├── queue.ex             # GenServer — persistent JSONL task queue
│       └── heartbeat.ex         # GenServer — periodic health checks + queue processing
├── config/
│   ├── config.exs               # Base config (port, data_dir, intervals)
│   ├── dev.exs                  # Dev overrides
│   ├── prod.exs                 # Prod overrides
│   ├── runtime.exs              # Reads SYMBIONT_PORT, SYMBIONT_DATA_DIR env vars
│   └── test.exs                 # Test mode: port=0, cli="echo", heartbeat=24h
├── test/
│   ├── support/test_helpers.ex  # safe_stop/1, stop_all_services/0
│   └── symbiont/                # 6 test files, 39 tests total
├── data/
│   ├── ledger.jsonl             # Append-only cost log (immutable)
│   └── queue.jsonl              # Persistent task queue
└── mix.exs                      # Project definition (Elixir ~> 1.19)

Local Source Copy

The canonical source is also at: /sessions/*/mnt/michaeldwyer/src/symbiont_ex/ (This is the development copy used during Cowork sessions.)

OTP Supervision Tree

Symbiont.Supervisor (rest_for_one)
├── Task.Supervisor       — async task execution
├── Symbiont.Ledger       — GenServer: append-only cost ledger
├── Symbiont.Queue        — GenServer: persistent task queue
├── Symbiont.Heartbeat    — GenServer: periodic health + queue processing (5-min timer)
└── Bandit                — HTTP server (Plug adapter, port 8111)

Strategy: rest_for_one — if the Ledger crashes, everything downstream (Queue, Heartbeat, Bandit) restarts too, ensuring no calls are logged to a stale process.

Core Components

1. Symbiont.Router — Task Classification

  • Calls Haiku via Dispatcher to classify incoming tasks
  • Returns {tier, confidence, reason} — tier 1/2/3 maps to Haiku/Sonnet/Opus
  • Falls back to default tier on classification failure

2. Symbiont.Dispatcher — Model Execution

  • Wraps Claude Code CLI via System.shell/2 with printf | claude pipe pattern
  • Important: System.cmd/3 does NOT have an :input option — must use shell pipes
  • Captures: model, tokens, timing, success/failure
  • Logs every call to Ledger GenServer

3. Symbiont.Ledger — Cost Tracking (GenServer)

  • Append-only JSONL file at data/ledger.jsonl
  • Provides log_call/1, recent/1, stats/0
  • Stats aggregate by model, by date, with running totals
  • Uses Float.round/2 with float coercion (see AI Agent Lessons in elixir-guide)

4. Symbiont.Queue — Task Queue (GenServer)

  • Persistent JSONL at data/queue.jsonl
  • States: pending → processing → done/failed
  • enqueue/1, take/1, complete/1, fail/1
  • Loaded from disk on startup

5. Symbiont.Heartbeat — Health Monitor (GenServer)

  • Internal 5-minute timer via Process.send_after/3
  • Checks queue, processes pending tasks, logs health
  • No external systemd timer needed (OTP handles scheduling)

6. Symbiont.API — HTTP Router (Plug)

  • POST /task — execute immediately
  • POST /queue — add to persistent queue
  • GET /status — health, queue size, cost totals
  • GET /health — simple health check
  • GET /ledger — recent calls
  • GET /ledger/stats — aggregate cost stats

Model Tiers & Routing Strategy

Cost & Capability Matrix

Tier Model Best for Approx Cost/Call Token Budget
1 Haiku Classification, extraction, simple formatting ~$0.008 ~50k context
2 Sonnet Content writing, code gen, analysis, moderate reasoning ~$0.04 ~200k context
3 Opus Complex reasoning, strategy, full-context QA, edge cases ~$0.15 ~200k context

Routing Logic

  1. Task arrives → POST /task or queue processing
  2. Router classifies (via Haiku): confidence, reason, recommended tier
  3. Dispatcher routes to cheapest capable tier
  4. Result + cost logged to Ledger GenServer → ledger.jsonl

Dendrite Integration

Symbiont has web perception via Dendrite, a headless Chromium browser running on cortex.

Dendrite endpoints (from cortex localhost or public URL)

Endpoint What it does
POST /fetch Fetch URL → markdown/text/html (full JS rendering)
POST /screenshot Take screenshot → PNG bytes
POST /execute Run JavaScript in page context
POST /interact Click, type, scroll in a session
POST /session Create persistent browser session
GET /health Health check (no auth needed)

Connection details

  • Public URL: https://browser.hydrascale.net
  • Internal: http://localhost:3000 (from cortex)
  • API Key: 8dc5e8f7a02745ee8db90c94b2481fd9e1deeea1e2ce74420f54047859ea7edf
  • Auth: X-API-Key header on all endpoints except /health

For full Dendrite documentation, load the dendrite skill.

API Endpoints

POST /task

Submit and execute a task immediately.

Request:

{
  "task": "Analyze this user feedback and extract sentiment",
  "force_tier": "haiku"
}

Response:

{
  "id": "task-1711123456",
  "task": "Analyze...",
  "model": "haiku",
  "result": "...",
  "elapsed_seconds": 2.3,
  "input_tokens": 45,
  "output_tokens": 87,
  "estimated_cost_usd": 0.0082,
  "timestamp": "2026-03-20T14:33:12Z"
}

POST /queue

Add a task to the persistent queue (executes on next heartbeat cycle).

Request:

{
  "task": "Run weekly subscriber report"
}

Response:

{
  "id": "queued-abc123",
  "status": "queued",
  "position": 3
}

GET /status

Health check: API status, queue size, cost totals.

Response:

{
  "status": "healthy",
  "runtime": "elixir/otp",
  "queue_size": 0,
  "last_heartbeat": "2026-03-20T20:15:26Z",
  "total_calls": 2,
  "total_cost_estimated_usd": 0.0006,
  "by_model": {
    "haiku": {"calls": 2, "cost": 0.0006}
  }
}

GET /health

Simple health check — lightweight, no stats computation.

Response:

{"runtime": "elixir/otp", "status": "ok"}

GET /ledger

Recent API calls (last 50 by default). Optional ?limit=N parameter.

GET /ledger/stats

Aggregate cost & usage over time, broken down by model and date.

Calling the API

Via curl (from cortex)

# Health check
curl -s http://127.0.0.1:8111/health

# Submit a task
curl -X POST http://127.0.0.1:8111/task \
  -H "Content-Type: application/json" \
  -d '{"task":"Summarize this email","force_tier":"haiku"}'

# Check stats
curl -s http://127.0.0.1:8111/ledger/stats | python3 -m json.tool

Via Python (from Cowork session)

import paramiko
# ... connect via paramiko (see cortex-server skill) ...
out, err = run(client, 'curl -s http://127.0.0.1:8111/status')
print(out)

Ledger Format & Cost Tracking

Every inference call appends a JSONL entry to data/ledger.jsonl:

{
  "timestamp": "2026-03-20T14:32:15.123456Z",
  "model": "haiku",
  "success": true,
  "elapsed_seconds": 1.8,
  "input_tokens": 34,
  "output_tokens": 156,
  "estimated_cost_usd": 0.0003,
  "prompt_preview": "Classify this customer feedback..."
}

Why Track "Estimated Cost" on Pro?

  • Current token usage is covered by Claude Pro subscription
  • Ledger tracks API-equivalent cost for planning
  • When daily volume justifies it, can switch to direct API billing

Deployment & Updates

systemd Service File

# /etc/systemd/system/symbiont-ex-api.service
[Unit]
Description=Symbiont Elixir API
After=network.target

[Service]
Type=simple
WorkingDirectory=/data/symbiont_ex
Environment=HOME=/root
Environment=MIX_ENV=prod
Environment=SYMBIONT_PORT=8111
Environment=SYMBIONT_DATA_DIR=/data/symbiont_ex/data
ExecStart=/usr/bin/mix run --no-halt
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Critical: Environment=HOME=/root is required — mix crashes without it.

How to Deploy Code Changes

  1. Upload updated files via SFTP to /data/symbiont_ex/

    sftp = client.open_sftp()
sftp.put('local/lib/symbiont/router.ex', '/data/symbiont_ex/lib/symbiont/router.ex')
sftp.close()

  2. Restart the service

    systemctl restart symbiont-ex-api

  3. Verify

    systemctl status symbiont-ex-api --no-pager
curl -s http://127.0.0.1:8111/health

Running Tests

Tests run locally (in Cowork), not on cortex:

cd /path/to/symbiont_ex
mix test --trace

39 tests across 7 test files. Test mode uses port=0 (no Bandit), cli="echo", and 24h heartbeat interval.

Nightly Backups

  • Automatic rsync to rsync.net at de2613@de2613.rsync.net:cortex-backup/cortex/
  • Includes: /data/symbiont_ex/ (code + data)
  • Python archive at /data/symbiont/ is also backed up

Configuration

config/config.exs (defaults)

config :symbiont,
  port: 8111,
  data_dir: "/data/symbiont_ex",
  heartbeat_interval_ms: 5 * 60 * 1_000,   # 5 minutes
  max_queue_batch: 5,
  default_tier: :haiku,
  claude_cli: "claude"

config/runtime.exs (env overrides)

if port = System.get_env("SYMBIONT_PORT") do
  config :symbiont, port: String.to_integer(port)
end

if data_dir = System.get_env("SYMBIONT_DATA_DIR") do
  config :symbiont, data_dir: data_dir
end

config/test.exs

config :symbiont,
  data_dir: "test/tmp",
  port: 0,                              # Disables Bandit — empty supervisor
  heartbeat_interval_ms: :timer.hours(24),
  claude_cli: "echo"                     # Stubs CLI for testing

Architecture Decisions & Rationale

  1. Elixir/OTP over Python — Supervision trees provide automatic restart, fault isolation, and hot code loading. The BEAM VM is purpose-built for long-running services.

  2. rest_for_one supervision — If the Ledger crashes, Queue and Heartbeat restart too, preventing stale state references.

  3. GenServer-based Heartbeat — Built-in Process.send_after timer replaces the Python systemd timer. One fewer moving part, and the heartbeat shares process state with the app.

  4. Haiku-first routing — Classifying with the cheapest model ensures we never overpay. A 10% misclassification rate costs less than always going straight to Sonnet.

  5. Append-only JSONL Ledger — Immutable. Useful for cost forecasting, debugging, and audit trails.

  6. System.shell/2 for CLI — System.cmd/3 has no stdin support. Shell pipes via printf '%s' '...' | claude are the reliable pattern.

  7. Empty supervisor in test mode — Setting port=0 starts an empty supervisor, preventing GenServer conflicts during test setup/teardown.

Next Steps & Future Work

  • Build OTP release (no mix dependency in prod)
  • Implement first revenue service (content-as-a-service pilot)
  • Add webhook notifications (task completion, rate limits)
  • Dashboard UI (Phoenix LiveView) for monitoring costs + queue
  • Distributed Erlang: run multiple BEAM nodes with shared state
  • Hot code upgrades via OTP releases
  • Engram integration (cross-session memory) ported to Elixir

Quick Links & Key Files

What Location Purpose
Application /data/symbiont_ex/lib/symbiont/application.ex OTP supervision tree
Router /data/symbiont_ex/lib/symbiont/router.ex Task classification
Dispatcher /data/symbiont_ex/lib/symbiont/dispatcher.ex Claude CLI wrapper
API /data/symbiont_ex/lib/symbiont/api.ex Plug HTTP endpoints
Ledger /data/symbiont_ex/lib/symbiont/ledger.ex GenServer cost log
Queue /data/symbiont_ex/lib/symbiont/queue.ex GenServer task queue
Heartbeat /data/symbiont_ex/lib/symbiont/heartbeat.ex GenServer health monitor
Ledger data /data/symbiont_ex/data/ledger.jsonl Cost log (immutable)
Queue data /data/symbiont_ex/data/queue.jsonl Pending tasks
Service file /etc/systemd/system/symbiont-ex-api.service systemd unit
Tests /data/symbiont_ex/test/symbiont/ 39 tests, 7 files
Python archive /data/symbiont/ Retired Python version

Skills Infrastructure

Symbiont also manages a canonical skills repository on cortex that serves as the source of truth for all Cowork skills.

Location

  • Git repo: /data/skills/ on cortex
  • Packaged skills: /data/skills/dist/*.skill
  • Live download URL: https://cortex.hydrascale.net/skills/<name>.skill

Current skills hosted

Skill Download
symbiont https://cortex.hydrascale.net/skills/symbiont.skill
cortex-server https://cortex.hydrascale.net/skills/cortex-server.skill

How it works

  • Every SKILL.md lives in /data/skills/<name>/SKILL.md
  • package_all.sh zips each skill directory into a .skill file in /data/skills/dist/
  • Caddy serves /data/skills/dist/ at https://cortex.hydrascale.net/skills/

Updating a skill

Edit the SKILL.md directly on cortex:

nano /data/skills/<skill-name>/SKILL.md
# Force immediate packaging:
bash /data/skills/package_all.sh

Troubleshooting

Service Not Starting

systemctl status symbiont-ex-api --no-pager
journalctl -u symbiont-ex-api -n 50 -f

Common issues:

  • Missing HOME=/root in service file
  • Port conflict (check ss -tlnp | grep 8111)
  • Mix deps not compiled (cd /data/symbiont_ex && mix deps.get && mix compile)

Checking BEAM Health

# Is the BEAM process running?
pgrep -a beam.smp

# Memory usage
ps aux | grep beam.smp | grep -v grep

Queue Not Processing

# Check via API
curl -s http://127.0.0.1:8111/status | python3 -m json.tool

# Check queue file directly
cat /data/symbiont_ex/data/queue.jsonl | python3 -m json.tool

# Check heartbeat logs
journalctl -u symbiont-ex-api --no-pager | grep Heartbeat | tail -10

Disk Space

du -sh /data/symbiont_ex/data/ledger.jsonl

Business Context

Ownership & Legal

  • Michael Dwyer is the legal owner of all Anthropic accounts and infrastructure
  • This is a requirement of the partnership: AI cannot own accounts
  • All decisions flow through Michael as the responsible party

Revenue Model

Current: ~50/50 split after costs

  • Costs: token spend (tracked in ledger) + server infrastructure
  • Revenue: TBD (in design phase)
    • Content-as-a-service (AI-generated reports, analysis)
    • Micro-SaaS API (white-label task routing for other teams)
    • Research subscriptions (specialized insights)

Cost Tracking Philosophy

  • Ledger records API-equivalent cost even on Pro subscription
  • Helps predict break-even point for switching to direct API billing
  • When daily volume justifies it, can migrate to cheaper API tier

Contact & Governance

Owner: Michael Dwyer Infrastructure: cortex.hydrascale.net (root access) Backup: rsync.net (de2613@de2613.rsync.net:cortex-backup/cortex/) Revenue Account: Claude Pro (Michael's account) Partnership: 50/50 split after costs

Questions? Check the API /status and /ledger/stats endpoints — they'll tell you what's happening right now.