symbiont_ex/INBOUND_MIGRATION_SCOPE.md

Inbound Email Path Migration Scope

Date: 2026-05-21 Status: Scoping complete; ready for implementation planning Successor to: Telepathy JMAP outbound (shipped in fe5896b)

---

1. Current State

The inbound email path is split across two Python services:

Muse Inbox (/data/muse/smtp_handler.py — 360 lines)

• Purpose: SMTP server on port 25 accepting inbound mail for muse@hydrascale.net and muse@cortex.hydrascale.net
• Architecture: aiosmtpd + aiohttp async event loop
• Key features:
  • Parses MIME emails
  • Logs raw emails to inbox.jsonl (similar to Telepathy's messages.jsonl)
  • Dispatches to Symbiont API /task endpoint (port 8111) with task prompt + context
  • Implements exponential backoff retry (3 attempts) on transient Symbiont failures
  • Sends reply via Telepathy /email (JMAP)
  • Marks inbox entries as processed=True after success

Telepathy (/data/telepathy/app.py — 143 lines; /data/telepathy/mailer.py — 204 lines)

• Purpose: HTTP API (FastAPI on port 8114) for message persistence and JMAP email sending
• Endpoints: POST /messages, GET /messages, GET /messages/unread, POST /messages/{id}/read, POST /email, GET /health
• Dependencies: requests, FastAPI, Pydantic

Data Flow (Merged View)

Inbound email (SMTP:25, Muse)
  → Parse MIME + log to inbox.jsonl
  → POST /task to Symbiont (8111)
  → Claude CLI produces reply
  → POST /email to Telepathy (8114)
  → JMAP send via Fastmail
  → Mark processed in inbox.jsonl
  → (Optionally) Mark Telepathy message as read

---

2. Proposed Elixir Architecture

Goal: Consolidate both services into Symbiont.Telepathy supervision tree on port 8111 (single app).

New Modules

Module	Purpose	Est. Lines
Symbiont.Telepathy.SMTPHandler	gen_smtp_server_session callbacks	100-150
Symbiont.Telepathy.InboxStore	JSONL-backed inbox (like MessageStore)	80-120
Symbiont.Telepathy.Pipeline	Orchestrates receive → Symbiont dispatch → reply → mark read	150-200
HTTP endpoints (extend Symbiont.API.ex)	/messages, /email, /health	80-120
Tests	SMTP integration, dispatch retry, pipeline	150-200

Supervision Tree (Updated)

Symbiont.Telepathy.Supervisor (one_for_one)
  +-- MessageStore (JSONL: Fastmail-fetched emails)
  +-- InboxStore (JSONL: SMTP-received emails)
  +-- JMAP (GenServer: Fastmail session + sender)
  +-- SMTPHandler (wraps :gen_smtp listener on port 25)
  +-- TaskSupervisor (spawns Pipeline.process tasks)

Key Design Decisions

1. Inbox vs Messages: Separate stores because they track different email directions (received vs polled from Fastmail). Both feed the UI and dispatch logic.
2. Retry logic: Implement as a simple loop in Pipeline.process (not a separate backoff supervisor) since Muse inbox already handles transient retries.
3. API consolidation: POST /email becomes a direct call to JMAP.send_email/3 instead of an HTTP hop; remove port 8114 entirely.

---

3. Complexity Estimate

Total LOC: ~650-870 across modules + tests

Category	Effort	Notes
SMTP handler (gen_smtp)	Medium (100-150)	Straightforward protocol impl; risk is MIME edge cases
Pipeline + dispatch	Medium (150-200)	Needs retry logic, timeout handling, error reporting
Inbox store	Small (80-120)	Clone MessageStore pattern; simple JSONL ops
API endpoints	Small (80-120)	Thin wrappers around store/sender; mostly copy-paste
Tests + integration	Medium (150-200)	Must verify SMTP flow, Symbiont dispatch, reply send
Refactor/Config	Small	Update Application.ex, mix.exs (add :gen_smtp), runtime.exs

Blockers:

• None identified. Both Muse Inbox and Telepathy run standalone with no inter-service locks. Can be developed/tested in parallel with Python services running.

Risks:

1. Email loss on crashes: If Elixir app restarts mid-Pipeline.process, inbox entry is not marked processed. Mitigation: restore from unprocessed entries on startup.
2. gen_smtp limitations: Erlang SMTP impl is less polished than Python aiosmtpd. May hit MIME parsing edge cases. Mitigation: test with real emails from inbox.jsonl.
3. Symbiont dispatch timeout: Task may take >600s. Current SMTP client will time out waiting for reply. Mitigation: return 250 OK immediately (like Muse does now), dispatch async.

---

4. Recommendation

Start next session (after Michael confirms).

Rationale:

• Work is well-scoped with a clear blueprint and existing code to port
• JMAP client already shipped; this is a natural successor
• System currently stable (both services running)
• Estimated 2-3 focused sessions including tests and cutover
• No explicit urgency signal from Michael (reflection logs don't mention inbound as a blocker)
• Higher-priority items remain blocked (blog project, permissions)

Decision factors to verify with Michael:

1. Is the "19 unread" bug urgent enough to fast-track this? (Elixir port fixes it by design)
2. Are there other blockers (blog, permissions) that should be prioritized first?
3. Can SMTP listen on a non-privileged port during development (e.g., 10025) for testing?

---

5. Next Steps (If Approved)

1. Confirm with Michael: urgency, priority relative to blog/permissions, port 25 availability
2. Add :gen_smtp ~> 1.2 to mix.exs
3. Implement SMTPHandler as gen_smtp_server_session behaviour
4. Implement InboxStore (clone MessageStore, s/messages/inbox/)
5. Implement Pipeline with retry loop and Symbiont dispatch
6. Add HTTP endpoints to API.ex
7. Integration test: send email to muse@, verify dispatch, verify reply
8. Cutover: stop Python services, verify Elixir handles full flow
9. Decommission /data/muse/ and /data/telepathy/ services

---

Appendix: Python Source Files for Reference

• /data/muse/smtp_handler.py (360 lines) — aiosmtpd handler + Symbiont dispatch logic
• /data/telepathy/app.py (143 lines) — FastAPI message/email endpoints
• /data/telepathy/mailer.py (204 lines) — JMAP sender (already ported to Elixir)
• /data/muse/inbox.jsonl — inbox log (schema: timestamp, from, subject, message_id, body, processed)
• /data/telepathy/messages.jsonl — messages log (schema: id, timestamp, source, subject, content, read)