michael/symbiont
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
c5ce00aadb
symbiont/TELEPATHY_PORT_STATUS.md

9.6 KiB
Raw Blame History

Telepathy Port Status: Python -> Elixir/OTP

Date: 2026-05-18 Status: NOT STARTED -- Blueprint complete, ready for implementation Blocks: Elimination of the last major Python dependency in the critical path

Executive Summary

Telepathy is Muse's email communication layer -- 635 lines of Python across 3 files, running as 2 separate processes (FastAPI on port 8114, aiosmtpd on port 25). Zero Elixir code exists for this service. A detailed migration blueprint was created 2026-05-08 but no implementation work has begun. The Elixir host (symbiont_ex) has no email-related modules, dependencies, or config.

Current Python Architecture

Running Services

Service Process Port Code
Telepathy API uvicorn (FastAPI) 8114 /data/telepathy/app.py (142 lines)
Telepathy Mailer (imported by app.py) -- /data/telepathy/mailer.py (203 lines)
Muse Inbox (SMTP) aiosmtpd 25 /data/muse/smtp_handler.py (290 lines)

Data Flow

Inbound: SMTP:25 -> parse MIME -> inbox.jsonl -> POST /messages (Telepathy)
                                              -> POST /task (Symbiont/Elixir)
                                              -> If reply: POST /email (Telepathy)
                                              -> JMAP send via Fastmail

Outbound: Any caller -> POST /email (Telepathy:8114) -> mailer.py -> Fastmail JMAP API

Python Module Responsibilities

app.py -- Message store API + email dispatch interface

  • POST /messages -- Store message (from email-in, reflection, system)
  • GET /messages -- List messages (with limit/unread filter)
  • GET /messages/unread -- Unread messages
  • POST /messages/{id}/read -- Mark read
  • POST /email -- Send email via Fastmail JMAP
  • GET /health -- Status with total/unread counts
  • Data: messages.jsonl (id, timestamp, source, subject, content, read)

mailer.py -- Fastmail JMAP client

  • Session discovery (GET /jmap/session)
  • Identity/get, Mailbox/get (find drafts mailbox)
  • Email/set (create draft) + EmailSubmission/set (send)
  • Draft cleanup after send
  • Token from env FASTMAIL_API_TOKEN or config.json

smtp_handler.py -- Inbound SMTP receiver + orchestration

  • aiosmtpd server on port 25 (cortex.hydrascale.net)
  • Accepts: muse@hydrascale.net, muse@cortex.hydrascale.net
  • MIME parsing via Python email stdlib
  • Async pipeline: store -> dispatch to Claude -> send reply -> mark read
  • Data: inbox.jsonl (timestamp, from, to, subject, message_id, body, processed)

Elixir Port Status

Completed Components: NONE

No Telepathy-related code exists in /data/symbiont_ex:

  • No modules with telepathy/email/smtp/inbox in name
  • No email deps in mix.exs (no req, gen_smtp, swoosh, etc.)
  • No email config in runtime.exs
  • No email endpoints in api.ex or router.ex
  • No supervision tree entries for email services

Existing Elixir Infrastructure (Reusable)

Component Location Reuse for Telepathy
Plug + Bandit (HTTP) api.ex, port 8111 Add /messages and /email endpoints directly
JSONL GenServer pattern queue.ex Clone for MessageStore
Task.Supervisor application.ex Spawn email pipeline tasks
Jason (JSON) Already in deps JMAP request/response encoding
Router router.ex Route inbound emails through existing task classification

Planned Elixir Architecture (from 2026-05-08 Blueprint)

New Supervision Subtree

Symbiont.Telepathy.Supervisor (one_for_one)
  +-- Symbiont.Telepathy.JMAP         (GenServer -- Fastmail session + API)
  +-- Symbiont.Telepathy.SMTP         (gen_smtp server on port 25)
  +-- Symbiont.Telepathy.MessageStore  (GenServer -- messages.jsonl)
  +-- Symbiont.Telepathy.Pipeline      (stateless orchestration module)

New Dependencies Required

{:req, "~> 0.5"},       # HTTP client for JMAP API
{:gen_smtp, "~> 1.2"}   # SMTP server + :mimemail for MIME parsing

Module Estimates

Module Lines Complexity Notes
Telepathy.JMAP 150-200 HIGH Session mgmt, Email/set, EmailSubmission/set, mark_read
Telepathy.SMTP 80-120 MEDIUM gen_smtp_server_session behaviour callbacks
Telepathy.MessageStore 80-100 LOW Clone Queue pattern, JSONL-backed
Telepathy.Pipeline 40-60 LOW Orchestrate: store -> dispatch -> reply -> mark_read
API endpoints 40-60 LOW Add to existing api.ex
Tests 150-200 MEDIUM Unit + integration
Total 540-740

Remaining Work (Ordered)

Phase 1: JMAP Client (P0)

  • Add {:req, "~> 0.5"} to mix.exs
  • Implement Symbiont.Telepathy.JMAP GenServer
    • Session discovery (GET /jmap/session, cache result)
    • send_email/4 (Email/set + EmailSubmission/set)
    • mark_read/1 (Email/set with $seen keyword) -- fixes "19 unread" bug
    • Auto-refresh on 401
  • Config: FASTMAIL_TOKEN in runtime.exs
  • Unit tests with mock HTTP responses

Phase 2: Message Store (P0)

  • Implement Symbiont.Telepathy.MessageStore GenServer
    • JSONL read/append (same pattern as Queue)
    • Compatible with existing /data/telepathy/messages.jsonl schema
    • Status transitions: unread -> processing -> replied -> read
  • API endpoints in existing api.ex:
    • POST /messages, GET /messages, GET /messages/unread
    • POST /messages/:id/read, GET /health (include message counts)

Phase 3: SMTP + Pipeline (P1)

  • Add {:gen_smtp, "~> 1.2"} to mix.exs
  • Implement Symbiont.Telepathy.SMTP (gen_smtp_server_session behaviour)
    • handle_MAIL, handle_RCPT (validate recipients), handle_DATA
    • MIME parsing via :mimemail.decode/1
    • Extract From, Subject, body (prefer text/plain)
  • Implement Symbiont.Telepathy.Pipeline
    • Spawn under Task.Supervisor
    • Steps: store -> dispatch to Claude -> send reply -> mark read (local + Fastmail)
  • Port inbox.jsonl logging

Phase 4: Integration + Cutover (P2)

  • Add Symbiont.Telepathy.Supervisor to Application supervision tree
  • Feature flag: TELEPATHY_ENABLED env var
  • Full pipeline integration test (SMTP in -> Claude dispatch -> JMAP out)
  • Migrate data: point at existing messages.jsonl / inbox.jsonl
  • Stop Python processes (systemd units for 8114 + SMTP)
  • Remove Python services from startup

Blockers

Blocker Severity Details Mitigation
No Elixir JMAP library MEDIUM Must implement JMAP protocol from scratch using Req Python mailer.py is only 203 lines; JMAP calls are well-documented in blueprint
Port 25 binding LOW Requires root/setcap for SMTP gen_smtp handles this; same constraint as Python
Fastmail token access LOW Token in /data/telepathy/config.json Read at startup via runtime.exs or env var
Sandbox write permissions HIGH Previous attempts to write Elixir code were blocked by sandbox Requires manual implementation or permission grant
:mimemail parsing gaps LOW Less polished than Python email stdlib for multipart Only need text/plain extraction; add helpers if needed

Design Gaps Identified

  1. HTML email handling: Python only extracts text/plain, falls back to "[HTML content...]" placeholder. Elixir port should match this behavior initially but consider stripping HTML to text as enhancement.

  2. Attachment support: Neither Python nor planned Elixir handles attachments. Not blocking but worth noting.

  3. Error propagation: Python smtp_handler fires _process_email_async() as fire-and-forget. Pipeline failures are silent. Elixir port should log failures and consider retry semantics via Task.Supervisor.

  4. Message store scalability: Both Python and planned Elixir use JSONL append-only files loaded into memory. Fine for current volume (<1000 messages) but not scalable. Could migrate to Engram (SQLite) later.

  5. JMAP session lifecycle: Python caches session globally. Elixir GenServer state is cleaner but needs refresh logic on token expiry or session invalidation.

  6. Queue "done" ambiguity: Queue marks blocked tasks as "done" (no success/failure distinction). Pipeline failures for email tasks will be invisible to the next reflection cycle. This is a pre-existing systemic issue, not specific to the port.

Dependencies on Other Systems

  • Symbiont API (port 8111): SMTP handler dispatches to /task endpoint -- already exists in Elixir
  • Router: Email tasks routed through existing router.ex -- already exists
  • Engram: MessageStore could eventually migrate to SQLite (engram.ex) -- optional
  • Fastmail: External dependency, no changes needed on their side
  • DNS/MX: cortex.hydrascale.net MX records point to this host -- no changes needed

Known Bugs to Fix During Port

  1. "19 unread" bug -- After sending a reply via JMAP, Python never calls Email/set with keywords/$seen to mark the email read on Fastmail. The blueprint Pipeline explicitly includes JMAP.mark_read/1 as a pipeline step, fixing this by design.

  2. Silent pipeline failures -- Python's _process_email_async() swallows exceptions. Elixir Pipeline should use Task.Supervisor with proper error logging.

Estimated Scope

  • Lines of Elixir: 540-740 (excluding tests)
  • New dependencies: 2 (req, gen_smtp)
  • New modules: 4-5
  • Complexity: The JMAP client is the hardest part (~40% of effort). Everything else maps cleanly to existing Elixir patterns.
  • Risk: MEDIUM overall. JMAP protocol correctness is the primary risk; mitigated by the fact that Python mailer.py already documents the exact JMAP calls needed.