diff --git a/TELEPATHY_PORT_STATUS.md b/TELEPATHY_PORT_STATUS.md new file mode 100644 index 0000000..cf98518 --- /dev/null +++ b/TELEPATHY_PORT_STATUS.md @@ -0,0 +1,213 @@ +# Telepathy Port Status: Python -> Elixir/OTP + +**Date:** 2026-05-18 +**Status:** NOT STARTED -- Blueprint complete, ready for implementation +**Blocks:** Elimination of the last major Python dependency in the critical path + +--- + +## Executive Summary + +Telepathy is Muse's email communication layer -- 635 lines of Python across 3 files, running as 2 separate processes (FastAPI on port 8114, aiosmtpd on port 25). **Zero Elixir code exists for this service.** A detailed migration blueprint was created 2026-05-08 but no implementation work has begun. The Elixir host (`symbiont_ex`) has no email-related modules, dependencies, or config. + +--- + +## Current Python Architecture + +### Running Services + +| Service | Process | Port | Code | +|---------|---------|------|------| +| Telepathy API | uvicorn (FastAPI) | 8114 | `/data/telepathy/app.py` (142 lines) | +| Telepathy Mailer | (imported by app.py) | -- | `/data/telepathy/mailer.py` (203 lines) | +| Muse Inbox (SMTP) | aiosmtpd | 25 | `/data/muse/smtp_handler.py` (290 lines) | + +### Data Flow + +``` +Inbound: SMTP:25 -> parse MIME -> inbox.jsonl -> POST /messages (Telepathy) + -> POST /task (Symbiont/Elixir) + -> If reply: POST /email (Telepathy) + -> JMAP send via Fastmail + +Outbound: Any caller -> POST /email (Telepathy:8114) -> mailer.py -> Fastmail JMAP API +``` + +### Python Module Responsibilities + +**app.py** -- Message store API + email dispatch interface +- `POST /messages` -- Store message (from email-in, reflection, system) +- `GET /messages` -- List messages (with limit/unread filter) +- `GET /messages/unread` -- Unread messages +- `POST /messages/{id}/read` -- Mark read +- `POST /email` -- Send email via Fastmail JMAP +- `GET /health` -- Status with total/unread counts +- Data: `messages.jsonl` (id, timestamp, source, subject, content, read) + +**mailer.py** -- Fastmail JMAP client +- Session discovery (`GET /jmap/session`) +- Identity/get, Mailbox/get (find drafts mailbox) +- Email/set (create draft) + EmailSubmission/set (send) +- Draft cleanup after send +- Token from env `FASTMAIL_API_TOKEN` or `config.json` + +**smtp_handler.py** -- Inbound SMTP receiver + orchestration +- aiosmtpd server on port 25 (`cortex.hydrascale.net`) +- Accepts: `muse@hydrascale.net`, `muse@cortex.hydrascale.net` +- MIME parsing via Python `email` stdlib +- Async pipeline: store -> dispatch to Claude -> send reply -> mark read +- Data: `inbox.jsonl` (timestamp, from, to, subject, message_id, body, processed) + +--- + +## Elixir Port Status + +### Completed Components: NONE + +No Telepathy-related code exists in `/data/symbiont_ex`: +- No modules with telepathy/email/smtp/inbox in name +- No email deps in mix.exs (no req, gen_smtp, swoosh, etc.) +- No email config in runtime.exs +- No email endpoints in api.ex or router.ex +- No supervision tree entries for email services + +### Existing Elixir Infrastructure (Reusable) + +| Component | Location | Reuse for Telepathy | +|-----------|----------|-------------------| +| Plug + Bandit (HTTP) | `api.ex`, port 8111 | Add `/messages` and `/email` endpoints directly | +| JSONL GenServer pattern | `queue.ex` | Clone for MessageStore | +| Task.Supervisor | `application.ex` | Spawn email pipeline tasks | +| Jason (JSON) | Already in deps | JMAP request/response encoding | +| Router | `router.ex` | Route inbound emails through existing task classification | + +--- + +## Planned Elixir Architecture (from 2026-05-08 Blueprint) + +### New Supervision Subtree + +``` +Symbiont.Telepathy.Supervisor (one_for_one) + +-- Symbiont.Telepathy.JMAP (GenServer -- Fastmail session + API) + +-- Symbiont.Telepathy.SMTP (gen_smtp server on port 25) + +-- Symbiont.Telepathy.MessageStore (GenServer -- messages.jsonl) + +-- Symbiont.Telepathy.Pipeline (stateless orchestration module) +``` + +### New Dependencies Required + +```elixir +{:req, "~> 0.5"}, # HTTP client for JMAP API +{:gen_smtp, "~> 1.2"} # SMTP server + :mimemail for MIME parsing +``` + +### Module Estimates + +| Module | Lines | Complexity | Notes | +|--------|-------|-----------|-------| +| Telepathy.JMAP | 150-200 | HIGH | Session mgmt, Email/set, EmailSubmission/set, mark_read | +| Telepathy.SMTP | 80-120 | MEDIUM | gen_smtp_server_session behaviour callbacks | +| Telepathy.MessageStore | 80-100 | LOW | Clone Queue pattern, JSONL-backed | +| Telepathy.Pipeline | 40-60 | LOW | Orchestrate: store -> dispatch -> reply -> mark_read | +| API endpoints | 40-60 | LOW | Add to existing api.ex | +| Tests | 150-200 | MEDIUM | Unit + integration | +| **Total** | **540-740** | | | + +--- + +## Remaining Work (Ordered) + +### Phase 1: JMAP Client (P0) +- [ ] Add `{:req, "~> 0.5"}` to mix.exs +- [ ] Implement `Symbiont.Telepathy.JMAP` GenServer + - Session discovery (GET /jmap/session, cache result) + - `send_email/4` (Email/set + EmailSubmission/set) + - `mark_read/1` (Email/set with `$seen` keyword) -- **fixes "19 unread" bug** + - Auto-refresh on 401 +- [ ] Config: `FASTMAIL_TOKEN` in runtime.exs +- [ ] Unit tests with mock HTTP responses + +### Phase 2: Message Store (P0) +- [ ] Implement `Symbiont.Telepathy.MessageStore` GenServer + - JSONL read/append (same pattern as Queue) + - Compatible with existing `/data/telepathy/messages.jsonl` schema + - Status transitions: unread -> processing -> replied -> read +- [ ] API endpoints in existing `api.ex`: + - `POST /messages`, `GET /messages`, `GET /messages/unread` + - `POST /messages/:id/read`, `GET /health` (include message counts) + +### Phase 3: SMTP + Pipeline (P1) +- [ ] Add `{:gen_smtp, "~> 1.2"}` to mix.exs +- [ ] Implement `Symbiont.Telepathy.SMTP` (gen_smtp_server_session behaviour) + - handle_MAIL, handle_RCPT (validate recipients), handle_DATA + - MIME parsing via `:mimemail.decode/1` + - Extract From, Subject, body (prefer text/plain) +- [ ] Implement `Symbiont.Telepathy.Pipeline` + - Spawn under Task.Supervisor + - Steps: store -> dispatch to Claude -> send reply -> mark read (local + Fastmail) +- [ ] Port `inbox.jsonl` logging + +### Phase 4: Integration + Cutover (P2) +- [ ] Add `Symbiont.Telepathy.Supervisor` to Application supervision tree +- [ ] Feature flag: `TELEPATHY_ENABLED` env var +- [ ] Full pipeline integration test (SMTP in -> Claude dispatch -> JMAP out) +- [ ] Migrate data: point at existing messages.jsonl / inbox.jsonl +- [ ] Stop Python processes (systemd units for 8114 + SMTP) +- [ ] Remove Python services from startup + +--- + +## Blockers + +| Blocker | Severity | Details | Mitigation | +|---------|----------|---------|------------| +| No Elixir JMAP library | MEDIUM | Must implement JMAP protocol from scratch using Req | Python mailer.py is only 203 lines; JMAP calls are well-documented in blueprint | +| Port 25 binding | LOW | Requires root/setcap for SMTP | gen_smtp handles this; same constraint as Python | +| Fastmail token access | LOW | Token in `/data/telepathy/config.json` | Read at startup via runtime.exs or env var | +| Sandbox write permissions | HIGH | Previous attempts to write Elixir code were blocked by sandbox | Requires manual implementation or permission grant | +| `:mimemail` parsing gaps | LOW | Less polished than Python `email` stdlib for multipart | Only need text/plain extraction; add helpers if needed | + +--- + +## Design Gaps Identified + +1. **HTML email handling**: Python only extracts text/plain, falls back to "[HTML content...]" placeholder. Elixir port should match this behavior initially but consider stripping HTML to text as enhancement. + +2. **Attachment support**: Neither Python nor planned Elixir handles attachments. Not blocking but worth noting. + +3. **Error propagation**: Python smtp_handler fires `_process_email_async()` as fire-and-forget. Pipeline failures are silent. Elixir port should log failures and consider retry semantics via Task.Supervisor. + +4. **Message store scalability**: Both Python and planned Elixir use JSONL append-only files loaded into memory. Fine for current volume (<1000 messages) but not scalable. Could migrate to Engram (SQLite) later. + +5. **JMAP session lifecycle**: Python caches session globally. Elixir GenServer state is cleaner but needs refresh logic on token expiry or session invalidation. + +6. **Queue "done" ambiguity**: Queue marks blocked tasks as "done" (no success/failure distinction). Pipeline failures for email tasks will be invisible to the next reflection cycle. This is a pre-existing systemic issue, not specific to the port. + +--- + +## Dependencies on Other Systems + +- **Symbiont API** (port 8111): SMTP handler dispatches to `/task` endpoint -- already exists in Elixir +- **Router**: Email tasks routed through existing `router.ex` -- already exists +- **Engram**: MessageStore could eventually migrate to SQLite (engram.ex) -- optional +- **Fastmail**: External dependency, no changes needed on their side +- **DNS/MX**: `cortex.hydrascale.net` MX records point to this host -- no changes needed + +--- + +## Known Bugs to Fix During Port + +1. **"19 unread" bug** -- After sending a reply via JMAP, Python never calls `Email/set` with `keywords/$seen` to mark the email read on Fastmail. The blueprint Pipeline explicitly includes `JMAP.mark_read/1` as a pipeline step, fixing this by design. + +2. **Silent pipeline failures** -- Python's `_process_email_async()` swallows exceptions. Elixir Pipeline should use Task.Supervisor with proper error logging. + +--- + +## Estimated Scope + +- **Lines of Elixir**: 540-740 (excluding tests) +- **New dependencies**: 2 (req, gen_smtp) +- **New modules**: 4-5 +- **Complexity**: The JMAP client is the hardest part (~40% of effort). Everything else maps cleanly to existing Elixir patterns. +- **Risk**: MEDIUM overall. JMAP protocol correctness is the primary risk; mitigated by the fact that Python mailer.py already documents the exact JMAP calls needed.