diff --git a/INBOUND_MIGRATION_SCOPE.md b/INBOUND_MIGRATION_SCOPE.md new file mode 100644 index 0000000..05ec42e --- /dev/null +++ b/INBOUND_MIGRATION_SCOPE.md @@ -0,0 +1,136 @@ +# Inbound Email Path Migration Scope + +**Date:** 2026-05-21 +**Status:** Scoping complete; ready for implementation planning +**Successor to:** Telepathy JMAP outbound (shipped in fe5896b) + +--- + +## 1. Current State + +The inbound email path is split across **two Python services**: + +### Muse Inbox (`/data/muse/smtp_handler.py` — 360 lines) +- **Purpose:** SMTP server on port 25 accepting inbound mail for `muse@hydrascale.net` and `muse@cortex.hydrascale.net` +- **Architecture:** aiosmtpd + aiohttp async event loop +- **Key features:** + - Parses MIME emails + - Logs raw emails to `inbox.jsonl` (similar to Telepathy's `messages.jsonl`) + - Dispatches to Symbiont API `/task` endpoint (port 8111) with task prompt + context + - Implements exponential backoff retry (3 attempts) on transient Symbiont failures + - Sends reply via Telepathy `/email` (JMAP) + - Marks inbox entries as `processed=True` after success + +### Telepathy (`/data/telepathy/app.py` — 143 lines; `/data/telepathy/mailer.py` — 204 lines) +- **Purpose:** HTTP API (FastAPI on port 8114) for message persistence and JMAP email sending +- **Endpoints:** `POST /messages`, `GET /messages`, `GET /messages/unread`, `POST /messages/{id}/read`, `POST /email`, `GET /health` +- **Dependencies:** requests, FastAPI, Pydantic + +### Data Flow (Merged View) +``` +Inbound email (SMTP:25, Muse) + → Parse MIME + log to inbox.jsonl + → POST /task to Symbiont (8111) + → Claude CLI produces reply + → POST /email to Telepathy (8114) + → JMAP send via Fastmail + → Mark processed in inbox.jsonl + → (Optionally) Mark Telepathy message as read +``` + +--- + +## 2. Proposed Elixir Architecture + +**Goal:** Consolidate both services into Symbiont.Telepathy supervision tree on port 8111 (single app). + +### New Modules + +| Module | Purpose | Est. Lines | +|--------|---------|-----------| +| `Symbiont.Telepathy.SMTPHandler` | gen_smtp_server_session callbacks | 100-150 | +| `Symbiont.Telepathy.InboxStore` | JSONL-backed inbox (like MessageStore) | 80-120 | +| `Symbiont.Telepathy.Pipeline` | Orchestrates receive → Symbiont dispatch → reply → mark read | 150-200 | +| HTTP endpoints (extend `Symbiont.API.ex`) | `/messages`, `/email`, `/health` | 80-120 | +| Tests | SMTP integration, dispatch retry, pipeline | 150-200 | + +### Supervision Tree (Updated) +``` +Symbiont.Telepathy.Supervisor (one_for_one) + +-- MessageStore (JSONL: Fastmail-fetched emails) + +-- InboxStore (JSONL: SMTP-received emails) + +-- JMAP (GenServer: Fastmail session + sender) + +-- SMTPHandler (wraps :gen_smtp listener on port 25) + +-- TaskSupervisor (spawns Pipeline.process tasks) +``` + +### Key Design Decisions +1. **Inbox vs Messages:** Separate stores because they track different email directions (received vs polled from Fastmail). Both feed the UI and dispatch logic. +2. **Retry logic:** Implement as a simple loop in Pipeline.process (not a separate backoff supervisor) since Muse inbox already handles transient retries. +3. **API consolidation:** POST `/email` becomes a direct call to `JMAP.send_email/3` instead of an HTTP hop; remove port 8114 entirely. + +--- + +## 3. Complexity Estimate + +**Total LOC: ~650-870 across modules + tests** + +| Category | Effort | Notes | +|----------|--------|-------| +| SMTP handler (gen_smtp) | Medium (100-150) | Straightforward protocol impl; risk is MIME edge cases | +| Pipeline + dispatch | Medium (150-200) | Needs retry logic, timeout handling, error reporting | +| Inbox store | Small (80-120) | Clone MessageStore pattern; simple JSONL ops | +| API endpoints | Small (80-120) | Thin wrappers around store/sender; mostly copy-paste | +| Tests + integration | Medium (150-200) | Must verify SMTP flow, Symbiont dispatch, reply send | +| **Refactor/Config** | **Small** | Update Application.ex, mix.exs (add :gen_smtp), runtime.exs | + +**Blockers:** +- None identified. Both Muse Inbox and Telepathy run standalone with no inter-service locks. Can be developed/tested in parallel with Python services running. + +**Risks:** +1. **Email loss on crashes:** If Elixir app restarts mid-Pipeline.process, inbox entry is not marked processed. Mitigation: restore from unprocessed entries on startup. +2. **gen_smtp limitations:** Erlang SMTP impl is less polished than Python aiosmtpd. May hit MIME parsing edge cases. Mitigation: test with real emails from inbox.jsonl. +3. **Symbiont dispatch timeout:** Task may take >600s. Current SMTP client will time out waiting for reply. Mitigation: return 250 OK immediately (like Muse does now), dispatch async. + +--- + +## 4. Recommendation + +**Start next session (after Michael confirms).** + +**Rationale:** +- Work is well-scoped with a clear blueprint and existing code to port +- JMAP client already shipped; this is a natural successor +- System currently stable (both services running) +- Estimated 2-3 focused sessions including tests and cutover +- **No explicit urgency signal from Michael** (reflection logs don't mention inbound as a blocker) +- Higher-priority items remain blocked (blog project, permissions) + +**Decision factors to verify with Michael:** +1. Is the "19 unread" bug urgent enough to fast-track this? (Elixir port fixes it by design) +2. Are there other blockers (blog, permissions) that should be prioritized first? +3. Can SMTP listen on a non-privileged port during development (e.g., 10025) for testing? + +--- + +## 5. Next Steps (If Approved) + +1. Confirm with Michael: urgency, priority relative to blog/permissions, port 25 availability +2. Add `:gen_smtp ~> 1.2` to `mix.exs` +3. Implement SMTPHandler as gen_smtp_server_session behaviour +4. Implement InboxStore (clone MessageStore, s/messages/inbox/) +5. Implement Pipeline with retry loop and Symbiont dispatch +6. Add HTTP endpoints to API.ex +7. Integration test: send email to muse@, verify dispatch, verify reply +8. Cutover: stop Python services, verify Elixir handles full flow +9. Decommission `/data/muse/` and `/data/telepathy/` services + +--- + +**Appendix: Python Source Files for Reference** + +- `/data/muse/smtp_handler.py` (360 lines) — aiosmtpd handler + Symbiont dispatch logic +- `/data/telepathy/app.py` (143 lines) — FastAPI message/email endpoints +- `/data/telepathy/mailer.py` (204 lines) — JMAP sender (already ported to Elixir) +- `/data/muse/inbox.jsonl` — inbox log (schema: timestamp, from, subject, message_id, body, processed) +- `/data/telepathy/messages.jsonl` — messages log (schema: id, timestamp, source, subject, content, read)