symbiont/TELEPATHY_PORT_STATUS.md

# Telepathy Port Status: Python -> Elixir/OTP

**Date:** 2026-05-18
**Status:** NOT STARTED -- Blueprint complete, ready for implementation
**Blocks:** Elimination of the last major Python dependency in the critical path

---

## Executive Summary

Telepathy is Muse's email communication layer -- 635 lines of Python across 3 files, running as 2 separate processes (FastAPI on port 8114, aiosmtpd on port 25). **Zero Elixir code exists for this service.** A detailed migration blueprint was created 2026-05-08 but no implementation work has begun. The Elixir host (`symbiont_ex`) has no email-related modules, dependencies, or config.

---

## Current Python Architecture

### Running Services

| Service | Process | Port | Code |
|---------|---------|------|------|
| Telepathy API | uvicorn (FastAPI) | 8114 | `/data/telepathy/app.py` (142 lines) |
| Telepathy Mailer | (imported by app.py) | -- | `/data/telepathy/mailer.py` (203 lines) |
| Muse Inbox (SMTP) | aiosmtpd | 25 | `/data/muse/smtp_handler.py` (290 lines) |

### Data Flow

```
Inbound: SMTP:25 -> parse MIME -> inbox.jsonl -> POST /messages (Telepathy)
                                              -> POST /task (Symbiont/Elixir)
                                              -> If reply: POST /email (Telepathy)
                                              -> JMAP send via Fastmail

Outbound: Any caller -> POST /email (Telepathy:8114) -> mailer.py -> Fastmail JMAP API
```

### Python Module Responsibilities

**app.py** -- Message store API + email dispatch interface
- `POST /messages` -- Store message (from email-in, reflection, system)
- `GET /messages` -- List messages (with limit/unread filter)
- `GET /messages/unread` -- Unread messages
- `POST /messages/{id}/read` -- Mark read
- `POST /email` -- Send email via Fastmail JMAP
- `GET /health` -- Status with total/unread counts
- Data: `messages.jsonl` (id, timestamp, source, subject, content, read)

**mailer.py** -- Fastmail JMAP client
- Session discovery (`GET /jmap/session`)
- Identity/get, Mailbox/get (find drafts mailbox)
- Email/set (create draft) + EmailSubmission/set (send)
- Draft cleanup after send
- Token from env `FASTMAIL_API_TOKEN` or `config.json`

**smtp_handler.py** -- Inbound SMTP receiver + orchestration
- aiosmtpd server on port 25 (`cortex.hydrascale.net`)
- Accepts: `muse@hydrascale.net`, `muse@cortex.hydrascale.net`
- MIME parsing via Python `email` stdlib
- Async pipeline: store -> dispatch to Claude -> send reply -> mark read
- Data: `inbox.jsonl` (timestamp, from, to, subject, message_id, body, processed)

---

## Elixir Port Status

### Completed Components: NONE

No Telepathy-related code exists in `/data/symbiont_ex`:
- No modules with telepathy/email/smtp/inbox in name
- No email deps in mix.exs (no req, gen_smtp, swoosh, etc.)
- No email config in runtime.exs
- No email endpoints in api.ex or router.ex
- No supervision tree entries for email services

### Existing Elixir Infrastructure (Reusable)

| Component | Location | Reuse for Telepathy |
|-----------|----------|-------------------|
| Plug + Bandit (HTTP) | `api.ex`, port 8111 | Add `/messages` and `/email` endpoints directly |
| JSONL GenServer pattern | `queue.ex` | Clone for MessageStore |
| Task.Supervisor | `application.ex` | Spawn email pipeline tasks |
| Jason (JSON) | Already in deps | JMAP request/response encoding |
| Router | `router.ex` | Route inbound emails through existing task classification |

---

## Planned Elixir Architecture (from 2026-05-08 Blueprint)

### New Supervision Subtree

```
Symbiont.Telepathy.Supervisor (one_for_one)
  +-- Symbiont.Telepathy.JMAP         (GenServer -- Fastmail session + API)
  +-- Symbiont.Telepathy.SMTP         (gen_smtp server on port 25)
  +-- Symbiont.Telepathy.MessageStore  (GenServer -- messages.jsonl)
  +-- Symbiont.Telepathy.Pipeline      (stateless orchestration module)
```

### New Dependencies Required

```elixir
{:req, "~> 0.5"},       # HTTP client for JMAP API
{:gen_smtp, "~> 1.2"}   # SMTP server + :mimemail for MIME parsing
```

### Module Estimates

| Module | Lines | Complexity | Notes |
|--------|-------|-----------|-------|
| Telepathy.JMAP | 150-200 | HIGH | Session mgmt, Email/set, EmailSubmission/set, mark_read |
| Telepathy.SMTP | 80-120 | MEDIUM | gen_smtp_server_session behaviour callbacks |
| Telepathy.MessageStore | 80-100 | LOW | Clone Queue pattern, JSONL-backed |
| Telepathy.Pipeline | 40-60 | LOW | Orchestrate: store -> dispatch -> reply -> mark_read |
| API endpoints | 40-60 | LOW | Add to existing api.ex |
| Tests | 150-200 | MEDIUM | Unit + integration |
| **Total** | **540-740** | | |

---

## Remaining Work (Ordered)

### Phase 1: JMAP Client (P0)
- [ ] Add `{:req, "~> 0.5"}` to mix.exs
- [ ] Implement `Symbiont.Telepathy.JMAP` GenServer
  - Session discovery (GET /jmap/session, cache result)
  - `send_email/4` (Email/set + EmailSubmission/set)
  - `mark_read/1` (Email/set with `$seen` keyword) -- **fixes "19 unread" bug**
  - Auto-refresh on 401
- [ ] Config: `FASTMAIL_TOKEN` in runtime.exs
- [ ] Unit tests with mock HTTP responses

### Phase 2: Message Store (P0)
- [ ] Implement `Symbiont.Telepathy.MessageStore` GenServer
  - JSONL read/append (same pattern as Queue)
  - Compatible with existing `/data/telepathy/messages.jsonl` schema
  - Status transitions: unread -> processing -> replied -> read
- [ ] API endpoints in existing `api.ex`:
  - `POST /messages`, `GET /messages`, `GET /messages/unread`
  - `POST /messages/:id/read`, `GET /health` (include message counts)

### Phase 3: SMTP + Pipeline (P1)
- [ ] Add `{:gen_smtp, "~> 1.2"}` to mix.exs
- [ ] Implement `Symbiont.Telepathy.SMTP` (gen_smtp_server_session behaviour)
  - handle_MAIL, handle_RCPT (validate recipients), handle_DATA
  - MIME parsing via `:mimemail.decode/1`
  - Extract From, Subject, body (prefer text/plain)
- [ ] Implement `Symbiont.Telepathy.Pipeline`
  - Spawn under Task.Supervisor
  - Steps: store -> dispatch to Claude -> send reply -> mark read (local + Fastmail)
- [ ] Port `inbox.jsonl` logging

### Phase 4: Integration + Cutover (P2)
- [ ] Add `Symbiont.Telepathy.Supervisor` to Application supervision tree
- [ ] Feature flag: `TELEPATHY_ENABLED` env var
- [ ] Full pipeline integration test (SMTP in -> Claude dispatch -> JMAP out)
- [ ] Migrate data: point at existing messages.jsonl / inbox.jsonl
- [ ] Stop Python processes (systemd units for 8114 + SMTP)
- [ ] Remove Python services from startup

---

## Blockers

| Blocker | Severity | Details | Mitigation |
|---------|----------|---------|------------|
| No Elixir JMAP library | MEDIUM | Must implement JMAP protocol from scratch using Req | Python mailer.py is only 203 lines; JMAP calls are well-documented in blueprint |
| Port 25 binding | LOW | Requires root/setcap for SMTP | gen_smtp handles this; same constraint as Python |
| Fastmail token access | LOW | Token in `/data/telepathy/config.json` | Read at startup via runtime.exs or env var |
| Sandbox write permissions | HIGH | Previous attempts to write Elixir code were blocked by sandbox | Requires manual implementation or permission grant |
| `:mimemail` parsing gaps | LOW | Less polished than Python `email` stdlib for multipart | Only need text/plain extraction; add helpers if needed |

---

## Design Gaps Identified

1. **HTML email handling**: Python only extracts text/plain, falls back to "[HTML content...]" placeholder. Elixir port should match this behavior initially but consider stripping HTML to text as enhancement.

2. **Attachment support**: Neither Python nor planned Elixir handles attachments. Not blocking but worth noting.

3. **Error propagation**: Python smtp_handler fires `_process_email_async()` as fire-and-forget. Pipeline failures are silent. Elixir port should log failures and consider retry semantics via Task.Supervisor.

4. **Message store scalability**: Both Python and planned Elixir use JSONL append-only files loaded into memory. Fine for current volume (<1000 messages) but not scalable. Could migrate to Engram (SQLite) later.

5. **JMAP session lifecycle**: Python caches session globally. Elixir GenServer state is cleaner but needs refresh logic on token expiry or session invalidation.

6. **Queue "done" ambiguity**: Queue marks blocked tasks as "done" (no success/failure distinction). Pipeline failures for email tasks will be invisible to the next reflection cycle. This is a pre-existing systemic issue, not specific to the port.

---

## Dependencies on Other Systems

- **Symbiont API** (port 8111): SMTP handler dispatches to `/task` endpoint -- already exists in Elixir
- **Router**: Email tasks routed through existing `router.ex` -- already exists
- **Engram**: MessageStore could eventually migrate to SQLite (engram.ex) -- optional
- **Fastmail**: External dependency, no changes needed on their side
- **DNS/MX**: `cortex.hydrascale.net` MX records point to this host -- no changes needed

---

## Known Bugs to Fix During Port

1. **"19 unread" bug** -- After sending a reply via JMAP, Python never calls `Email/set` with `keywords/$seen` to mark the email read on Fastmail. The blueprint Pipeline explicitly includes `JMAP.mark_read/1` as a pipeline step, fixing this by design.

2. **Silent pipeline failures** -- Python's `_process_email_async()` swallows exceptions. Elixir Pipeline should use Task.Supervisor with proper error logging.

---

## Estimated Scope

- **Lines of Elixir**: 540-740 (excluding tests)
- **New dependencies**: 2 (req, gen_smtp)
- **New modules**: 4-5
- **Complexity**: The JMAP client is the hardest part (~40% of effort). Everything else maps cleanly to existing Elixir patterns.
- **Risk**: MEDIUM overall. JMAP protocol correctness is the primary risk; mitigated by the fact that Python mailer.py already documents the exact JMAP calls needed.