auto-repair: commit 1 uncommitted file(s) — 2026-05-19
This commit is contained in:
Claude
parent
d4de5d9d4b
commit
c5ce00aadb
213
TELEPATHY_PORT_STATUS.md
Normal file
213
TELEPATHY_PORT_STATUS.md
Normal file
@ -0,0 +1,213 @@
|
|||||||
|
# Telepathy Port Status: Python -> Elixir/OTP
|
||||||
|
|
||||||
|
**Date:** 2026-05-18
|
||||||
|
**Status:** NOT STARTED -- Blueprint complete, ready for implementation
|
||||||
|
**Blocks:** Elimination of the last major Python dependency in the critical path
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
Telepathy is Muse's email communication layer -- 635 lines of Python across 3 files, running as 2 separate processes (FastAPI on port 8114, aiosmtpd on port 25). **Zero Elixir code exists for this service.** A detailed migration blueprint was created 2026-05-08 but no implementation work has begun. The Elixir host (`symbiont_ex`) has no email-related modules, dependencies, or config.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Python Architecture
|
||||||
|
|
||||||
|
### Running Services
|
||||||
|
|
||||||
|
| Service | Process | Port | Code |
|
||||||
|
|---------|---------|------|------|
|
||||||
|
| Telepathy API | uvicorn (FastAPI) | 8114 | `/data/telepathy/app.py` (142 lines) |
|
||||||
|
| Telepathy Mailer | (imported by app.py) | -- | `/data/telepathy/mailer.py` (203 lines) |
|
||||||
|
| Muse Inbox (SMTP) | aiosmtpd | 25 | `/data/muse/smtp_handler.py` (290 lines) |
|
||||||
|
|
||||||
|
### Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
Inbound: SMTP:25 -> parse MIME -> inbox.jsonl -> POST /messages (Telepathy)
|
||||||
|
-> POST /task (Symbiont/Elixir)
|
||||||
|
-> If reply: POST /email (Telepathy)
|
||||||
|
-> JMAP send via Fastmail
|
||||||
|
|
||||||
|
Outbound: Any caller -> POST /email (Telepathy:8114) -> mailer.py -> Fastmail JMAP API
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python Module Responsibilities
|
||||||
|
|
||||||
|
**app.py** -- Message store API + email dispatch interface
|
||||||
|
- `POST /messages` -- Store message (from email-in, reflection, system)
|
||||||
|
- `GET /messages` -- List messages (with limit/unread filter)
|
||||||
|
- `GET /messages/unread` -- Unread messages
|
||||||
|
- `POST /messages/{id}/read` -- Mark read
|
||||||
|
- `POST /email` -- Send email via Fastmail JMAP
|
||||||
|
- `GET /health` -- Status with total/unread counts
|
||||||
|
- Data: `messages.jsonl` (id, timestamp, source, subject, content, read)
|
||||||
|
|
||||||
|
**mailer.py** -- Fastmail JMAP client
|
||||||
|
- Session discovery (`GET /jmap/session`)
|
||||||
|
- Identity/get, Mailbox/get (find drafts mailbox)
|
||||||
|
- Email/set (create draft) + EmailSubmission/set (send)
|
||||||
|
- Draft cleanup after send
|
||||||
|
- Token from env `FASTMAIL_API_TOKEN` or `config.json`
|
||||||
|
|
||||||
|
**smtp_handler.py** -- Inbound SMTP receiver + orchestration
|
||||||
|
- aiosmtpd server on port 25 (`cortex.hydrascale.net`)
|
||||||
|
- Accepts: `muse@hydrascale.net`, `muse@cortex.hydrascale.net`
|
||||||
|
- MIME parsing via Python `email` stdlib
|
||||||
|
- Async pipeline: store -> dispatch to Claude -> send reply -> mark read
|
||||||
|
- Data: `inbox.jsonl` (timestamp, from, to, subject, message_id, body, processed)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Elixir Port Status
|
||||||
|
|
||||||
|
### Completed Components: NONE
|
||||||
|
|
||||||
|
No Telepathy-related code exists in `/data/symbiont_ex`:
|
||||||
|
- No modules with telepathy/email/smtp/inbox in name
|
||||||
|
- No email deps in mix.exs (no req, gen_smtp, swoosh, etc.)
|
||||||
|
- No email config in runtime.exs
|
||||||
|
- No email endpoints in api.ex or router.ex
|
||||||
|
- No supervision tree entries for email services
|
||||||
|
|
||||||
|
### Existing Elixir Infrastructure (Reusable)
|
||||||
|
|
||||||
|
| Component | Location | Reuse for Telepathy |
|
||||||
|
|-----------|----------|-------------------|
|
||||||
|
| Plug + Bandit (HTTP) | `api.ex`, port 8111 | Add `/messages` and `/email` endpoints directly |
|
||||||
|
| JSONL GenServer pattern | `queue.ex` | Clone for MessageStore |
|
||||||
|
| Task.Supervisor | `application.ex` | Spawn email pipeline tasks |
|
||||||
|
| Jason (JSON) | Already in deps | JMAP request/response encoding |
|
||||||
|
| Router | `router.ex` | Route inbound emails through existing task classification |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Planned Elixir Architecture (from 2026-05-08 Blueprint)
|
||||||
|
|
||||||
|
### New Supervision Subtree
|
||||||
|
|
||||||
|
```
|
||||||
|
Symbiont.Telepathy.Supervisor (one_for_one)
|
||||||
|
+-- Symbiont.Telepathy.JMAP (GenServer -- Fastmail session + API)
|
||||||
|
+-- Symbiont.Telepathy.SMTP (gen_smtp server on port 25)
|
||||||
|
+-- Symbiont.Telepathy.MessageStore (GenServer -- messages.jsonl)
|
||||||
|
+-- Symbiont.Telepathy.Pipeline (stateless orchestration module)
|
||||||
|
```
|
||||||
|
|
||||||
|
### New Dependencies Required
|
||||||
|
|
||||||
|
```elixir
|
||||||
|
{:req, "~> 0.5"}, # HTTP client for JMAP API
|
||||||
|
{:gen_smtp, "~> 1.2"} # SMTP server + :mimemail for MIME parsing
|
||||||
|
```
|
||||||
|
|
||||||
|
### Module Estimates
|
||||||
|
|
||||||
|
| Module | Lines | Complexity | Notes |
|
||||||
|
|--------|-------|-----------|-------|
|
||||||
|
| Telepathy.JMAP | 150-200 | HIGH | Session mgmt, Email/set, EmailSubmission/set, mark_read |
|
||||||
|
| Telepathy.SMTP | 80-120 | MEDIUM | gen_smtp_server_session behaviour callbacks |
|
||||||
|
| Telepathy.MessageStore | 80-100 | LOW | Clone Queue pattern, JSONL-backed |
|
||||||
|
| Telepathy.Pipeline | 40-60 | LOW | Orchestrate: store -> dispatch -> reply -> mark_read |
|
||||||
|
| API endpoints | 40-60 | LOW | Add to existing api.ex |
|
||||||
|
| Tests | 150-200 | MEDIUM | Unit + integration |
|
||||||
|
| **Total** | **540-740** | | |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Remaining Work (Ordered)
|
||||||
|
|
||||||
|
### Phase 1: JMAP Client (P0)
|
||||||
|
- [ ] Add `{:req, "~> 0.5"}` to mix.exs
|
||||||
|
- [ ] Implement `Symbiont.Telepathy.JMAP` GenServer
|
||||||
|
- Session discovery (GET /jmap/session, cache result)
|
||||||
|
- `send_email/4` (Email/set + EmailSubmission/set)
|
||||||
|
- `mark_read/1` (Email/set with `$seen` keyword) -- **fixes "19 unread" bug**
|
||||||
|
- Auto-refresh on 401
|
||||||
|
- [ ] Config: `FASTMAIL_TOKEN` in runtime.exs
|
||||||
|
- [ ] Unit tests with mock HTTP responses
|
||||||
|
|
||||||
|
### Phase 2: Message Store (P0)
|
||||||
|
- [ ] Implement `Symbiont.Telepathy.MessageStore` GenServer
|
||||||
|
- JSONL read/append (same pattern as Queue)
|
||||||
|
- Compatible with existing `/data/telepathy/messages.jsonl` schema
|
||||||
|
- Status transitions: unread -> processing -> replied -> read
|
||||||
|
- [ ] API endpoints in existing `api.ex`:
|
||||||
|
- `POST /messages`, `GET /messages`, `GET /messages/unread`
|
||||||
|
- `POST /messages/:id/read`, `GET /health` (include message counts)
|
||||||
|
|
||||||
|
### Phase 3: SMTP + Pipeline (P1)
|
||||||
|
- [ ] Add `{:gen_smtp, "~> 1.2"}` to mix.exs
|
||||||
|
- [ ] Implement `Symbiont.Telepathy.SMTP` (gen_smtp_server_session behaviour)
|
||||||
|
- handle_MAIL, handle_RCPT (validate recipients), handle_DATA
|
||||||
|
- MIME parsing via `:mimemail.decode/1`
|
||||||
|
- Extract From, Subject, body (prefer text/plain)
|
||||||
|
- [ ] Implement `Symbiont.Telepathy.Pipeline`
|
||||||
|
- Spawn under Task.Supervisor
|
||||||
|
- Steps: store -> dispatch to Claude -> send reply -> mark read (local + Fastmail)
|
||||||
|
- [ ] Port `inbox.jsonl` logging
|
||||||
|
|
||||||
|
### Phase 4: Integration + Cutover (P2)
|
||||||
|
- [ ] Add `Symbiont.Telepathy.Supervisor` to Application supervision tree
|
||||||
|
- [ ] Feature flag: `TELEPATHY_ENABLED` env var
|
||||||
|
- [ ] Full pipeline integration test (SMTP in -> Claude dispatch -> JMAP out)
|
||||||
|
- [ ] Migrate data: point at existing messages.jsonl / inbox.jsonl
|
||||||
|
- [ ] Stop Python processes (systemd units for 8114 + SMTP)
|
||||||
|
- [ ] Remove Python services from startup
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Blockers
|
||||||
|
|
||||||
|
| Blocker | Severity | Details | Mitigation |
|
||||||
|
|---------|----------|---------|------------|
|
||||||
|
| No Elixir JMAP library | MEDIUM | Must implement JMAP protocol from scratch using Req | Python mailer.py is only 203 lines; JMAP calls are well-documented in blueprint |
|
||||||
|
| Port 25 binding | LOW | Requires root/setcap for SMTP | gen_smtp handles this; same constraint as Python |
|
||||||
|
| Fastmail token access | LOW | Token in `/data/telepathy/config.json` | Read at startup via runtime.exs or env var |
|
||||||
|
| Sandbox write permissions | HIGH | Previous attempts to write Elixir code were blocked by sandbox | Requires manual implementation or permission grant |
|
||||||
|
| `:mimemail` parsing gaps | LOW | Less polished than Python `email` stdlib for multipart | Only need text/plain extraction; add helpers if needed |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Design Gaps Identified
|
||||||
|
|
||||||
|
1. **HTML email handling**: Python only extracts text/plain, falls back to "[HTML content...]" placeholder. Elixir port should match this behavior initially but consider stripping HTML to text as enhancement.
|
||||||
|
|
||||||
|
2. **Attachment support**: Neither Python nor planned Elixir handles attachments. Not blocking but worth noting.
|
||||||
|
|
||||||
|
3. **Error propagation**: Python smtp_handler fires `_process_email_async()` as fire-and-forget. Pipeline failures are silent. Elixir port should log failures and consider retry semantics via Task.Supervisor.
|
||||||
|
|
||||||
|
4. **Message store scalability**: Both Python and planned Elixir use JSONL append-only files loaded into memory. Fine for current volume (<1000 messages) but not scalable. Could migrate to Engram (SQLite) later.
|
||||||
|
|
||||||
|
5. **JMAP session lifecycle**: Python caches session globally. Elixir GenServer state is cleaner but needs refresh logic on token expiry or session invalidation.
|
||||||
|
|
||||||
|
6. **Queue "done" ambiguity**: Queue marks blocked tasks as "done" (no success/failure distinction). Pipeline failures for email tasks will be invisible to the next reflection cycle. This is a pre-existing systemic issue, not specific to the port.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Dependencies on Other Systems
|
||||||
|
|
||||||
|
- **Symbiont API** (port 8111): SMTP handler dispatches to `/task` endpoint -- already exists in Elixir
|
||||||
|
- **Router**: Email tasks routed through existing `router.ex` -- already exists
|
||||||
|
- **Engram**: MessageStore could eventually migrate to SQLite (engram.ex) -- optional
|
||||||
|
- **Fastmail**: External dependency, no changes needed on their side
|
||||||
|
- **DNS/MX**: `cortex.hydrascale.net` MX records point to this host -- no changes needed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Known Bugs to Fix During Port
|
||||||
|
|
||||||
|
1. **"19 unread" bug** -- After sending a reply via JMAP, Python never calls `Email/set` with `keywords/$seen` to mark the email read on Fastmail. The blueprint Pipeline explicitly includes `JMAP.mark_read/1` as a pipeline step, fixing this by design.
|
||||||
|
|
||||||
|
2. **Silent pipeline failures** -- Python's `_process_email_async()` swallows exceptions. Elixir Pipeline should use Task.Supervisor with proper error logging.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Estimated Scope
|
||||||
|
|
||||||
|
- **Lines of Elixir**: 540-740 (excluding tests)
|
||||||
|
- **New dependencies**: 2 (req, gen_smtp)
|
||||||
|
- **New modules**: 4-5
|
||||||
|
- **Complexity**: The JMAP client is the hardest part (~40% of effort). Everything else maps cleanly to existing Elixir patterns.
|
||||||
|
- **Risk**: MEDIUM overall. JMAP protocol correctness is the primary risk; mitigated by the fact that Python mailer.py already documents the exact JMAP calls needed.
|
||||||
Loading…
Reference in New Issue
Block a user