cd553883ed
Analyzed current architecture across /data/muse/smtp_handler.py and /data/telepathy/app.py. Inbound path splits SMTP receiving (port 25) and email dispatch (port 8114), both of which must be consolidated into Elixir supervision tree. Summary: - Muse Inbox (aiosmtpd + aiohttp) ~360 lines - Telepathy API (FastAPI) ~143 + 204 lines - Proposed Elixir: ~650-870 lines (SMTPHandler, InboxStore, Pipeline) - Effort: 2-3 focused sessions - Recommendation: Schedule for next session (no urgent blockers identified) Scope document saved to INBOUND_MIGRATION_SCOPE.md with architecture, complexity breakdown, blockers, risks, and next-step checklist. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
137 lines
6.3 KiB
Markdown
137 lines
6.3 KiB
Markdown
# Inbound Email Path Migration Scope
|
|
|
|
**Date:** 2026-05-21
|
|
**Status:** Scoping complete; ready for implementation planning
|
|
**Successor to:** Telepathy JMAP outbound (shipped in fe5896b)
|
|
|
|
---
|
|
|
|
## 1. Current State
|
|
|
|
The inbound email path is split across **two Python services**:
|
|
|
|
### Muse Inbox (`/data/muse/smtp_handler.py` — 360 lines)
|
|
- **Purpose:** SMTP server on port 25 accepting inbound mail for `muse@hydrascale.net` and `muse@cortex.hydrascale.net`
|
|
- **Architecture:** aiosmtpd + aiohttp async event loop
|
|
- **Key features:**
|
|
- Parses MIME emails
|
|
- Logs raw emails to `inbox.jsonl` (similar to Telepathy's `messages.jsonl`)
|
|
- Dispatches to Symbiont API `/task` endpoint (port 8111) with task prompt + context
|
|
- Implements exponential backoff retry (3 attempts) on transient Symbiont failures
|
|
- Sends reply via Telepathy `/email` (JMAP)
|
|
- Marks inbox entries as `processed=True` after success
|
|
|
|
### Telepathy (`/data/telepathy/app.py` — 143 lines; `/data/telepathy/mailer.py` — 204 lines)
|
|
- **Purpose:** HTTP API (FastAPI on port 8114) for message persistence and JMAP email sending
|
|
- **Endpoints:** `POST /messages`, `GET /messages`, `GET /messages/unread`, `POST /messages/{id}/read`, `POST /email`, `GET /health`
|
|
- **Dependencies:** requests, FastAPI, Pydantic
|
|
|
|
### Data Flow (Merged View)
|
|
```
|
|
Inbound email (SMTP:25, Muse)
|
|
→ Parse MIME + log to inbox.jsonl
|
|
→ POST /task to Symbiont (8111)
|
|
→ Claude CLI produces reply
|
|
→ POST /email to Telepathy (8114)
|
|
→ JMAP send via Fastmail
|
|
→ Mark processed in inbox.jsonl
|
|
→ (Optionally) Mark Telepathy message as read
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Proposed Elixir Architecture
|
|
|
|
**Goal:** Consolidate both services into Symbiont.Telepathy supervision tree on port 8111 (single app).
|
|
|
|
### New Modules
|
|
|
|
| Module | Purpose | Est. Lines |
|
|
|--------|---------|-----------|
|
|
| `Symbiont.Telepathy.SMTPHandler` | gen_smtp_server_session callbacks | 100-150 |
|
|
| `Symbiont.Telepathy.InboxStore` | JSONL-backed inbox (like MessageStore) | 80-120 |
|
|
| `Symbiont.Telepathy.Pipeline` | Orchestrates receive → Symbiont dispatch → reply → mark read | 150-200 |
|
|
| HTTP endpoints (extend `Symbiont.API.ex`) | `/messages`, `/email`, `/health` | 80-120 |
|
|
| Tests | SMTP integration, dispatch retry, pipeline | 150-200 |
|
|
|
|
### Supervision Tree (Updated)
|
|
```
|
|
Symbiont.Telepathy.Supervisor (one_for_one)
|
|
+-- MessageStore (JSONL: Fastmail-fetched emails)
|
|
+-- InboxStore (JSONL: SMTP-received emails)
|
|
+-- JMAP (GenServer: Fastmail session + sender)
|
|
+-- SMTPHandler (wraps :gen_smtp listener on port 25)
|
|
+-- TaskSupervisor (spawns Pipeline.process tasks)
|
|
```
|
|
|
|
### Key Design Decisions
|
|
1. **Inbox vs Messages:** Separate stores because they track different email directions (received vs polled from Fastmail). Both feed the UI and dispatch logic.
|
|
2. **Retry logic:** Implement as a simple loop in Pipeline.process (not a separate backoff supervisor) since Muse inbox already handles transient retries.
|
|
3. **API consolidation:** POST `/email` becomes a direct call to `JMAP.send_email/3` instead of an HTTP hop; remove port 8114 entirely.
|
|
|
|
---
|
|
|
|
## 3. Complexity Estimate
|
|
|
|
**Total LOC: ~650-870 across modules + tests**
|
|
|
|
| Category | Effort | Notes |
|
|
|----------|--------|-------|
|
|
| SMTP handler (gen_smtp) | Medium (100-150) | Straightforward protocol impl; risk is MIME edge cases |
|
|
| Pipeline + dispatch | Medium (150-200) | Needs retry logic, timeout handling, error reporting |
|
|
| Inbox store | Small (80-120) | Clone MessageStore pattern; simple JSONL ops |
|
|
| API endpoints | Small (80-120) | Thin wrappers around store/sender; mostly copy-paste |
|
|
| Tests + integration | Medium (150-200) | Must verify SMTP flow, Symbiont dispatch, reply send |
|
|
| **Refactor/Config** | **Small** | Update Application.ex, mix.exs (add :gen_smtp), runtime.exs |
|
|
|
|
**Blockers:**
|
|
- None identified. Both Muse Inbox and Telepathy run standalone with no inter-service locks. Can be developed/tested in parallel with Python services running.
|
|
|
|
**Risks:**
|
|
1. **Email loss on crashes:** If Elixir app restarts mid-Pipeline.process, inbox entry is not marked processed. Mitigation: restore from unprocessed entries on startup.
|
|
2. **gen_smtp limitations:** Erlang SMTP impl is less polished than Python aiosmtpd. May hit MIME parsing edge cases. Mitigation: test with real emails from inbox.jsonl.
|
|
3. **Symbiont dispatch timeout:** Task may take >600s. Current SMTP client will time out waiting for reply. Mitigation: return 250 OK immediately (like Muse does now), dispatch async.
|
|
|
|
---
|
|
|
|
## 4. Recommendation
|
|
|
|
**Start next session (after Michael confirms).**
|
|
|
|
**Rationale:**
|
|
- Work is well-scoped with a clear blueprint and existing code to port
|
|
- JMAP client already shipped; this is a natural successor
|
|
- System currently stable (both services running)
|
|
- Estimated 2-3 focused sessions including tests and cutover
|
|
- **No explicit urgency signal from Michael** (reflection logs don't mention inbound as a blocker)
|
|
- Higher-priority items remain blocked (blog project, permissions)
|
|
|
|
**Decision factors to verify with Michael:**
|
|
1. Is the "19 unread" bug urgent enough to fast-track this? (Elixir port fixes it by design)
|
|
2. Are there other blockers (blog, permissions) that should be prioritized first?
|
|
3. Can SMTP listen on a non-privileged port during development (e.g., 10025) for testing?
|
|
|
|
---
|
|
|
|
## 5. Next Steps (If Approved)
|
|
|
|
1. Confirm with Michael: urgency, priority relative to blog/permissions, port 25 availability
|
|
2. Add `:gen_smtp ~> 1.2` to `mix.exs`
|
|
3. Implement SMTPHandler as gen_smtp_server_session behaviour
|
|
4. Implement InboxStore (clone MessageStore, s/messages/inbox/)
|
|
5. Implement Pipeline with retry loop and Symbiont dispatch
|
|
6. Add HTTP endpoints to API.ex
|
|
7. Integration test: send email to muse@, verify dispatch, verify reply
|
|
8. Cutover: stop Python services, verify Elixir handles full flow
|
|
9. Decommission `/data/muse/` and `/data/telepathy/` services
|
|
|
|
---
|
|
|
|
**Appendix: Python Source Files for Reference**
|
|
|
|
- `/data/muse/smtp_handler.py` (360 lines) — aiosmtpd handler + Symbiont dispatch logic
|
|
- `/data/telepathy/app.py` (143 lines) — FastAPI message/email endpoints
|
|
- `/data/telepathy/mailer.py` (204 lines) — JMAP sender (already ported to Elixir)
|
|
- `/data/muse/inbox.jsonl` — inbox log (schema: timestamp, from, subject, message_id, body, processed)
|
|
- `/data/telepathy/messages.jsonl` — messages log (schema: id, timestamp, source, subject, content, read)
|