# Elixir — Comprehensive Development Guide

## Quick Reference

| Item | Value |
|------|-------|
| Current Version | **Elixir 1.19.5** (target for all new code) |
| Required OTP | **OTP 27+** (OTP 28 supported) |
| Phoenix Version | **1.8.5** (latest — requires `:formats` on controllers) |
| Cortex Status | Elixir **1.19.5 / OTP 27 installed** on cortex.hydrascale.net |
| AI Agent Tooling | `usage_rules` hex package (~> 1.2) — **always include** |
| Production Framework | **Ash Framework** for substantial projects |
| Paradigm | Functional, concurrent, fault-tolerant on BEAM VM |

---

## CRITICAL RULES — Read First

1. **Target Elixir 1.19.5** — not 1.15, not 1.17. Use current idioms and features.
2. **Always add `usage_rules`** to every project: `{:usage_rules, "~> 1.1", only: [:dev]}` — it generates `AGENTS.md`/`CLAUDE.md` from dependency docs, giving AI agents rich context about the libraries in use.
3. **Use Ash Framework** for production/substantial projects (not necessarily POCs). Ash provides declarative resources, built-in authorization, and extensions for Postgres, Phoenix, GraphQL, JSON:API.
4. **Cortex is Elixir-first** — Elixir is the primary orchestrator language, with interop to Python (Erlport/ports), Rust (Rustler NIFs), and Zig when needed.
5. **Never use deprecated patterns** from pre-1.16 code (see Breaking Changes section).

---

## Elixir 1.19 — What's New

### Gradual Set-Theoretic Type System

Elixir 1.19 significantly advances the built-in type system. It is **sound**, **gradual**, and **set-theoretic** (types compose via union, intersection, negation).

**Current capabilities (1.19):**
- Type inference from existing code — no annotations required yet (user-provided signatures coming in future releases)
- **Protocol dispatch type checking** — the compiler warns when you pass a value to string interpolation, `for` comprehensions, or other protocol-dispatched operations that don't implement the required protocol
- **Anonymous function type inference** — `fn` literals and `&captures` now propagate types, catching mismatches at compile time
- Types: `atom()`, `binary()`, `integer()`, `float()`, `pid()`, `port()`, `reference()`, `tuple()`, `list()`, `map()`, `function()`
- Compose with: `atom() or integer()` (union), `atom() and integer()` (intersection → `none()`), `atom() and not nil` (difference)
- `dynamic()` type for gradual typing — represents runtime-checked values
- Tuple precision: `{:ok, binary()}`, open tuples with `...`: `{:ok, binary(), ...}`
- Map types: `%{key: value}` for closed maps, `%{optional(atom()) => term()}` for open maps
- Function types: `(integer() -> boolean())`

**What this means in practice:** The compiler now catches real bugs — passing a struct to string interpolation that doesn't implement `String.Chars`, using a non-enumerable in `for`, type mismatches in anonymous function calls. These are warnings today, errors in the future.

### Up to 4x Faster Compilation

Two major improvements:

1. **Lazy module loading** — modules are no longer loaded immediately when defined. The parallel compiler controls both compilation and loading, reducing pressure on the Erlang code server. Reports of 2x+ speedup on large projects.

2. **Parallel dependency compilation** — set `MIX_OS_DEPS_COMPILE_PARTITION_COUNT` env var to partition OS dep compilation across CPU cores.

**Potential regressions:** If you spawn processes during compilation that invoke other project modules, use `Kernel.ParallelCompiler.pmap/2` or `Code.ensure_compiled!/1` before spawning. Also affects `@on_load` callbacks that reference sibling modules.

### Other 1.19 Features

- `min/2` and `max/2` allowed in guards
- `Access.values/0` — traverse all values in a map/keyword
- `String.count/2` — count occurrences of a pattern
- Unicode 17.0.0 support
- Multi-line IEx prompts
- `mix help Mod.fun/arity` — get help for specific functions
- New pretty printing infrastructure
- OpenChain compliance with Source SBoM
- Erlang/OTP 28 support

---

## Breaking Changes Since 1.15

These are critical to know — older books and tutorials will use the old patterns.

### Struct Update Syntax (1.18+)
```elixir
# OLD (1.15) — no longer valid
%User{user | name: "new"}

# NEW (1.18+) — explicit pattern match required
%User{} = user
%User{user | name: "new"}
# Or in one expression:
%User{name: "new"} = Map.merge(user, %{name: "new"})
```
Actually the change is: `%URI{my_uri | path: "/new"}` now requires `my_uri` to match `%URI{}`. If the compiler can't verify it's the right struct type, it warns.

### Regex as Struct Field Defaults (OTP 28)
```elixir
# BROKEN on OTP 28 — regex literals can't be struct field defaults
defmodule MyMod do
  defstruct pattern: ~r/foo/  # Compile error on OTP 28
end

# FIX — use @default_pattern or compute at runtime
defmodule MyMod do
  @default_pattern ~r/foo/
  defstruct pattern: @default_pattern  # Still fails — use nil + init
end
```

### Logger Backends Deprecated
```elixir
# OLD
config :logger, backends: [:console]

# NEW — use LoggerHandler (Erlang's logger)
config :logger, :default_handler, []
config :logger, :default_formatter, format: "$time $metadata[$level] $message\n"
```

### Mix Task Separator
```elixir
# OLD — comma separator
mix do compile, test

# NEW (1.17+) — plus separator
mix do compile + test
```

### mix cli/0 Replaces Multiple Config Keys
```elixir
# OLD
def project do
  [default_task: "phx.server",
   preferred_cli_env: [test: :test],
   preferred_cli_target: [...]]
end

# NEW
def cli do
  [default_task: "phx.server",
   preferred_envs: [test: :test],
   preferred_targets: [...]]
end
```

---

## Core Language Patterns

### The Pipeline Principle
Elixir code flows through transformations via `|>`. Design functions to take the "subject" as the first argument.

```elixir
orders
|> Enum.filter(&(&1.status == :pending))
|> Enum.sort_by(& &1.created_at, DateTime)
|> Enum.map(&process_order/1)
```

### Pattern Matching — The Heart of Elixir
```elixir
# Function head matching — preferred over conditionals
def process(%Order{status: :pending} = order), do: ship(order)
def process(%Order{status: :shipped} = order), do: track(order)
def process(%Order{status: :delivered}), do: :noop

# Pin operator to match against existing values
expected = "hello"
^expected = some_function()  # Asserts equality

# Map/struct matching is partial — only listed keys must match
%{name: name} = %{name: "Michael", age: 42}  # name = "Michael"
```

### With Expressions for Happy-Path Chaining
```elixir
with {:ok, user} <- fetch_user(id),
     {:ok, account} <- fetch_account(user),
     {:ok, balance} <- check_balance(account) do
  {:ok, balance}
else
  {:error, :not_found} -> {:error, "User not found"}
  {:error, :insufficient} -> {:error, "Insufficient funds"}
  error -> {:error, "Unknown: #{inspect(error)}"}
end
```

### Structs and Protocols
```elixir
defmodule Money do
  defstruct [:amount, :currency]

  defimpl String.Chars do
    def to_string(%Money{amount: a, currency: c}), do: "#{a} #{c}"
  end

  defimpl Inspect do
    def inspect(%Money{amount: a, currency: c}, _opts) do
      "#Money<#{a} #{c}>"
    end
  end
end
```

### Behaviours for Contracts
```elixir
defmodule PaymentProvider do
  @callback charge(amount :: integer(), currency :: String.t()) ::
    {:ok, transaction_id :: String.t()} | {:error, reason :: term()}
  @callback refund(transaction_id :: String.t()) ::
    {:ok, term()} | {:error, reason :: term()}
end

defmodule Stripe do
  @behaviour PaymentProvider

  @impl true
  def charge(amount, currency), do: # ...

  @impl true
  def refund(transaction_id), do: # ...
end
```

---

## Concurrency Model — BEAM Processes

### Fundamentals
- **Processes are cheap** — ~2KB initial memory, microseconds to spawn, millions can run concurrently
- **No shared memory** — processes communicate via message passing only
- **Each process has its own heap** — GC is per-process, no stop-the-world pauses
- **Preemptive scheduling** — one scheduler per CPU core, ~4000 reductions then yield
- **Process isolation** — one process crashing doesn't affect others

### Spawning and Messaging
```elixir
# Basic spawn + message passing
pid = spawn(fn ->
  receive do
    {:greet, name} -> IO.puts("Hello, #{name}!")
  end
end)

send(pid, {:greet, "Michael"})

# Linked processes — bidirectional crash propagation
pid = spawn_link(fn -> raise "boom" end)  # Caller also crashes

# Monitored processes — one-directional crash notification
ref = Process.monitor(pid)
receive do
  {:DOWN, ^ref, :process, ^pid, reason} -> IO.puts("Crashed: #{reason}")
end
```

### Task — Structured Concurrency
```elixir
# Fire and forget
Task.start(fn -> send_email(user) end)

# Async/await (linked to caller)
task = Task.async(fn -> expensive_computation() end)
result = Task.await(task, 30_000)  # 30s timeout

# Parallel map
Task.async_stream(urls, &fetch_url/1, max_concurrency: 10, timeout: 15_000)
|> Enum.to_list()
```

### GenServer — Stateful Server Processes
```elixir
defmodule Counter do
  use GenServer

  # Client API
  def start_link(initial \\ 0), do: GenServer.start_link(__MODULE__, initial, name: __MODULE__)
  def increment, do: GenServer.cast(__MODULE__, :increment)
  def get, do: GenServer.call(__MODULE__, :get)

  # Server callbacks
  @impl true
  def init(initial), do: {:ok, initial}

  @impl true
  def handle_cast(:increment, count), do: {:noreply, count + 1}

  @impl true
  def handle_call(:get, _from, count), do: {:reply, count, count}

  @impl true
  def handle_info(:tick, count) do
    IO.puts("Count: #{count}")
    Process.send_after(self(), :tick, 1000)
    {:noreply, count}
  end
end
```

**Key design principle:** GenServer callbacks run sequentially in a single process — this is both the synchronization mechanism and the potential bottleneck. Keep callbacks fast; delegate heavy work to spawned tasks.

### Supervisors — Let It Crash
```elixir
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Order matters — started top to bottom, stopped bottom to top
      MyApp.Repo,                           # Ecto repo
      {MyApp.Cache, []},                    # Custom GenServer
      {Task.Supervisor, name: MyApp.TaskSupervisor},
      MyAppWeb.Endpoint                     # Phoenix endpoint last
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

**Restart strategies:**
- `:one_for_one` — only restart the crashed child (most common)
- `:one_for_all` — restart all children if any crashes (tightly coupled)
- `:rest_for_one` — restart crashed child and all children started after it

### DynamicSupervisor — Runtime Child Management
```elixir
defmodule MyApp.SessionSupervisor do
  use DynamicSupervisor

  def start_link(_), do: DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__)
  def init(:ok), do: DynamicSupervisor.init(strategy: :one_for_one)

  def start_session(session_id) do
    spec = {MyApp.Session, session_id}
    DynamicSupervisor.start_child(__MODULE__, spec)
  end
end
```

---

## OTP Releases & Deployment

### Building a Release
```bash
# Generate release config (Phoenix projects)
mix phx.gen.release

# Build the release
MIX_ENV=prod mix release

# The release is self-contained — no Erlang/Elixir needed on target
_build/prod/rel/my_app/bin/my_app start       # Foreground
_build/prod/rel/my_app/bin/my_app daemon       # Background
_build/prod/rel/my_app/bin/my_app remote       # Attach IEx to running node
_build/prod/rel/my_app/bin/my_app eval "MyApp.Seeds.run()"  # One-off command
```

### Release Configuration Files
- `config/config.exs` — **compile-time** config (before code compiles)
- `config/runtime.exs` — **runtime** config (executed on every boot) — use for env vars, secrets
- `rel/env.sh.eex` — shell environment for the release (VM flags, env vars)
- `rel/vm.args.eex` — Erlang VM flags (node name, cookie, memory limits)

### Docker Multistage Build Pattern
```dockerfile
# === Builder Stage ===
FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20250317 AS builder

RUN apt-get update -y && apt-get install -y build-essential git && apt-get clean
WORKDIR /app

ENV MIX_ENV=prod
RUN mix local.hex --force && mix local.rebar --force

COPY mix.exs mix.lock ./
RUN mix deps.get --only prod && mix deps.compile

COPY config/config.exs config/prod.exs config/
COPY priv priv
COPY lib lib
COPY assets assets  # If Phoenix

RUN mix assets.deploy  # If Phoenix
RUN mix compile
RUN mix release

# === Runner Stage ===
FROM debian:bookworm-slim AS runner

RUN apt-get update -y && \
    apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates && \
    apt-get clean && rm -f /var/lib/apt/lists/*_*

ENV LANG=en_US.UTF-8
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

WORKDIR /app
RUN useradd --create-home app && chown -R app:app /app
USER app

COPY --from=builder --chown=app:app /app/_build/prod/rel/my_app ./

CMD ["bin/my_app", "start"]
```

### Distributed Erlang in Production
```elixir
# In rel/env.sh.eex — set node name and cookie
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE=my_app@${HOSTNAME}
export RELEASE_COOKIE=my_secret_cookie

# Or in runtime.exs
config :my_app, MyApp.Cluster,
  strategy: Cluster.Strategy.DNSPoll,
  config: [
    polling_interval: 5_000,
    query: "my-app.local",
    node_basename: "my_app"
  ]
```

**Key libraries for clustering:** `libcluster` (automatic node discovery), `Horde` (distributed supervisor/registry), `Phoenix.PubSub` (cross-node pub/sub already built into Phoenix).

---

## Monitoring & Instrumentation

### The Elixir Observability Stack
- **PromEx** — Elixir library exposing `/metrics` endpoint for Prometheus scraping. Includes plugins for Phoenix, Ecto, LiveView, BEAM VM, Oban
- **Prometheus** — scrapes and stores time-series metrics
- **Loki** + **Promtail** — log aggregation (Promtail scrapes Docker container logs → Loki stores)
- **Grafana** — visualization dashboards for both metrics and logs
- **Alloy** — OpenTelemetry collector bridging metrics to Prometheus

### Adding PromEx to a Phoenix App
```elixir
# mix.exs
{:prom_ex, "~> 1.9"}

# lib/my_app/prom_ex.ex
defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      {PromEx.Plugins.Phoenix, router: MyAppWeb.Router},
      {PromEx.Plugins.Ecto, repos: [MyApp.Repo]},
      PromEx.Plugins.Oban
    ]
  end

  @impl true
  def dashboards do
    [{:prom_ex, "application.json"}, {:prom_ex, "beam.json"},
     {:prom_ex, "phoenix.json"}, {:prom_ex, "ecto.json"}]
  end
end
```

### Key BEAM Metrics to Watch
- `erlang_vm_process_count` — total BEAM processes (normal: thousands, alarm at millions)
- `erlang_vm_memory_bytes_total` — total VM memory
- `erlang_vm_atom_count` — atoms are never GC'd; watch for leaks
- `phoenix_endpoint_stop_duration_milliseconds` — request latency
- `ecto_repo_query_duration_milliseconds` — database query time

---

## usage_rules — AI Agent Documentation

**Always include in every project.** This is non-negotiable.

```elixir
# mix.exs deps
{:usage_rules, "~> 1.1", only: [:dev]}

# mix.exs project config
def project do
  [
    # ... other config
    usage_rules: [
      packages: :all,  # Or specific: ["phoenix", "ecto", ~r/ash/]
      output: :agents_md,  # Generates AGENTS.md
      mode: :linked  # Or :inlined for single-file
    ]
  ]
end
```

**Key commands:**
```bash
mix usage_rules.gen          # Generate AGENTS.md from deps
mix usage_rules.search_docs  # Search hex documentation
mix usage_rules.gen_skill    # Generate SKILL.md for Cowork
```

This consolidates `usage-rules.md` files from all dependencies into a single reference document, giving any AI agent working on the project full context about library APIs, patterns, and conventions.

---

## Ash Framework — Production Backend

Ash is a **declarative, resource-oriented** framework for building Elixir backends. Use it for substantial projects — it handles data layer, authorization, validation, and API generation.

### Core Concepts
- **Resources** — the central abstraction (like models but richer): attributes, actions, relationships, calculations, aggregates, policies
- **Domains** — organizational containers that group related resources and define the public API
- **Actions** — CRUD + custom actions defined declaratively on resources
- **Data Layers** — pluggable persistence (AshPostgres, AshSqlite, ETS for dev)
- **Policies** — declarative authorization rules on resources/actions
- **Extensions** — AshPhoenix, AshGraphql, AshJsonApi, AshAuthentication, AshOban

### Quick Start
```bash
mix igniter.new my_app --install ash,ash_postgres,ash_phoenix
```

### Resource Example
```elixir
defmodule MyApp.Blog.Post do
  use Ash.Resource,
    domain: MyApp.Blog,
    data_layer: AshPostgres.DataLayer

  postgres do
    table "posts"
    repo MyApp.Repo
  end

  attributes do
    uuid_primary_key :id
    attribute :title, :string, allow_nil?: false
    attribute :body, :string, allow_nil?: false
    attribute :status, :atom, constraints: [one_of: [:draft, :published]], default: :draft
    timestamps()
  end

  actions do
    defaults [:read, :destroy]

    create :create do
      accept [:title, :body]
    end

    update :publish do
      change set_attribute(:status, :published)
      change set_attribute(:published_at, &DateTime.utc_now/0)
    end
  end

  relationships do
    belongs_to :author, MyApp.Accounts.User
    has_many :comments, MyApp.Blog.Comment
  end

  policies do
    policy action_type(:create) do
      authorize_if actor_attribute_equals(:role, :author)
    end
    policy action_type(:read) do
      authorize_if always()
    end
  end
end
```

### Domain Example
```elixir
defmodule MyApp.Blog do
  use Ash.Domain

  resources do
    resource MyApp.Blog.Post
    resource MyApp.Blog.Comment
  end
end
```

### Using Resources
```elixir
# Create
MyApp.Blog.Post
|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"})
|> Ash.create!()

# Read with filters
MyApp.Blog.Post
|> Ash.Query.filter(status == :published)
|> Ash.Query.sort(inserted_at: :desc)
|> Ash.Query.limit(10)
|> Ash.read!()

# Custom action
post |> Ash.Changeset.for_update(:publish) |> Ash.update!()
```

---

## Mix Project Structure

```
my_app/
├── config/
│   ├── config.exs          # Compile-time config
│   ├── dev.exs
│   ├── prod.exs
│   ├── runtime.exs         # Runtime config (secrets, env vars)
│   └── test.exs
├── lib/
│   ├── my_app/
│   │   ├── application.ex  # OTP Application + supervisor tree
│   │   ├── repo.ex         # Ecto Repo
│   │   └── ...             # Domain modules
│   └── my_app_web/         # Phoenix web layer (if applicable)
│       ├── endpoint.ex
│       ├── router.ex
│       ├── controllers/
│       ├── live/            # LiveView modules
│       └── components/
├── priv/
│   ├── repo/migrations/
│   └── static/
├── test/
│   ├── support/
│   └── ...
├── mix.exs
├── mix.lock
├── .formatter.exs
└── AGENTS.md               # Generated by usage_rules
```

---

## Testing

```elixir
# test/my_app/blog_test.exs
defmodule MyApp.BlogTest do
  use MyApp.DataCase, async: true  # Async when no shared state

  describe "creating posts" do
    test "creates with valid attributes" do
      assert {:ok, post} = Blog.create_post(%{title: "Hi", body: "World"})
      assert post.title == "Hi"
      assert post.status == :draft
    end

    test "fails without title" do
      assert {:error, changeset} = Blog.create_post(%{body: "World"})
      assert "can't be blank" in errors_on(changeset).title
    end
  end
end
```

**Testing philosophy:** Use `async: true` wherever possible. Ecto's SQL Sandbox allows concurrent test execution. For GenServer tests, start under a test supervisor. For external services, use `Mox` for behaviour-based mocking.

---

## Phoenix Framework — v1.8.5 (Current)

Phoenix is the web framework for Elixir. Version 1.8.5 is current. It provides MVC controllers, real-time LiveView, Channels for WebSocket communication, and comprehensive tooling for authentication, testing, and deployment.

### What's New in Phoenix 1.8

- **Scopes** in generators — secure data access by default (e.g., `current_user` automatically applied)
- **Magic links** (passwordless auth) and **"sudo mode"** in `mix phx.gen.auth`
- **daisyUI** integration — light/dark/system mode support out of the box
- **Simplified layouts** — single `root.html.heex` wraps everything; dynamic layouts are function components
- **`use Phoenix.Controller` now requires `:formats`** — must specify `formats: [:html]` or `formats: [:html, :json]`
- **Updated security headers** — `content-security-policy` with `base-uri 'self'; frame-ancestors 'self'`; dropped deprecated `x-frame-options` and `x-download-options`
- **`config` variable removed** from `Phoenix.Endpoint` — use `Application.compile_env/3` instead
- **Deprecated**: `:namespace`, `:put_default_views`, layouts without modules, `:trailing_slash` in router

### Project Setup

```bash
# New project (Phoenix Express — quickest path)
curl https://new.phoenixframework.org/my_app | sh

# Traditional setup
mix phx.new my_app
mix phx.new my_app --no-ecto    # Without database
mix phx.new my_app --no-html    # API only
mix phx.new my_app --database sqlite3  # SQLite instead of Postgres
```

### Directory Structure

```
my_app/
├── lib/
│   ├── my_app/              # Business logic (contexts, schemas)
│   │   ├── application.ex   # Supervision tree
│   │   ├── repo.ex          # Ecto Repo
│   │   └── catalog/         # Context: Catalog
│   │       ├── product.ex   # Schema
│   │       └── catalog.ex   # Context module
│   └── my_app_web/          # Web layer
│       ├── endpoint.ex      # HTTP entry point
│       ├── router.ex        # Routes + pipelines
│       ├── components/
│       │   ├── core_components.ex  # Shared UI components
│       │   └── layouts.ex          # Layout components
│       ├── controllers/
│       │   ├── page_controller.ex
│       │   └── page_html/          # Templates for controller
│       │       └── home.html.heex
│       └── live/             # LiveView modules
│           └── counter_live.ex
├── config/
│   ├── config.exs            # Compile-time config
│   ├── dev.exs / prod.exs
│   └── runtime.exs           # Runtime config (env vars, secrets)
├── priv/
│   ├── repo/migrations/
│   └── static/               # Static assets
├── test/
│   ├── support/
│   │   ├── conn_case.ex      # Test helpers for controllers
│   │   └── data_case.ex      # Test helpers for data layer
│   └── my_app_web/
└── mix.exs
```

### Router & Pipelines

```elixir
defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_live_flash
    plug :put_root_layout, html: {MyAppWeb.Layouts, :root}
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", MyAppWeb do
    pipe_through :browser

    get "/", PageController, :home
    live "/counter", CounterLive         # LiveView route
    resources "/products", ProductController  # RESTful CRUD
  end

  scope "/api", MyAppWeb do
    pipe_through :api

    resources "/items", ItemController, except: [:new, :edit]
  end
end
```

### Verified Routes — `~p` Sigil

**Always use `~p` instead of string paths.** Compile-time verified against your router.

```elixir
# In templates (HEEx)
~H"""
<a href={~p"/products/#{product}"}>View</a>
<a href={~p"/products/#{product}/edit"}>Edit</a>
"""

# In controllers
redirect(conn, to: ~p"/products/#{product}")

# With query params
~p"/products?page=#{page}&sort=name"
~p"/products?#{%{page: 1, sort: "name"}}"

# URL generation (includes host)
url(~p"/products/#{product}")
# => "https://example.com/products/42"
```

### Controllers (1.8 Style)

```elixir
defmodule MyAppWeb.ProductController do
  use MyAppWeb, :controller

  # REQUIRED in 1.8: specify formats
  # This is typically set in MyAppWeb :controller function

  def index(conn, _params) do
    products = Catalog.list_products()
    render(conn, :index, products: products)
  end

  def show(conn, %{"id" => id}) do
    product = Catalog.get_product!(id)
    render(conn, :show, product: product)
  end

  def create(conn, %{"product" => product_params}) do
    case Catalog.create_product(product_params) do
      {:ok, product} ->
        conn
        |> put_flash(:info, "Product created.")
        |> redirect(to: ~p"/products/#{product}")

      {:error, %Ecto.Changeset{} = changeset} ->
        render(conn, :new, changeset: changeset)
    end
  end
end
```

**View module naming:** For `ProductController`, Phoenix looks for `ProductHTML` (for `:html` format) and `ProductJSON` (for `:json` format).

```elixir
# lib/my_app_web/controllers/product_html.ex
defmodule MyAppWeb.ProductHTML do
  use MyAppWeb, :html
  embed_templates "product_html/*"  # Loads .heex files from directory
end

# lib/my_app_web/controllers/product_json.ex
defmodule MyAppWeb.ProductJSON do
  def index(%{products: products}), do: %{data: for(p <- products, do: data(p))}
  def show(%{product: product}), do: %{data: data(product)}
  defp data(product), do: %{id: product.id, title: product.title, price: product.price}
end
```

### Components and HEEx

```elixir
defmodule MyAppWeb.CoreComponents do
  use Phoenix.Component

  # Declare attributes with types and docs
  attr :type, :string, default: "button"
  attr :class, :string, default: nil
  attr :rest, :global  # Passes through all other HTML attrs
  slot :inner_block, required: true

  def button(assigns) do
    ~H"""
    <button type={@type} class={["btn", @class]} {@rest}>
      {render_slot(@inner_block)}
    </button>
    """
  end

  # Table component with slots
  attr :rows, :list, required: true
  slot :col, required: true do
    attr :label, :string
  end

  def table(assigns) do
    ~H"""
    <table>
      <thead>
        <tr>
          <th :for={col <- @col}>{col[:label]}</th>
        </tr>
      </thead>
      <tbody>
        <tr :for={row <- @rows}>
          <td :for={col <- @col}>{render_slot(col, row)}</td>
        </tr>
      </tbody>
    </table>
    """
  end
end
```

**HEEx syntax notes:**
- `{@var}` — render assign (curly braces, not `<%= %>`)
- `:if={condition}` — conditional rendering on any tag
- `:for={item <- list}` — iteration on any tag
- `<.component_name />` — call function component with dot notation
- `<Component.name />` — call remote component with module name
- `{render_slot(@inner_block)}` — render slot content
- `<:slot_name>content</:slot_name>` — named slot content

### LiveView

```elixir
defmodule MyAppWeb.SearchLive do
  use MyAppWeb, :live_view

  def mount(_params, _session, socket) do
    {:ok, assign(socket, query: "", results: [])}
  end

  def handle_params(%{"q" => query}, _uri, socket) do
    {:noreply, assign(socket, query: query, results: search(query))}
  end

  def handle_params(_params, _uri, socket), do: {:noreply, socket}

  def handle_event("search", %{"query" => query}, socket) do
    {:noreply,
     socket
     |> assign(query: query, results: search(query))
     |> push_patch(to: ~p"/search?q=#{query}")}
  end

  def render(assigns) do
    ~H"""
    <Layouts.app flash={@flash}>
      <form phx-submit="search">
        <input name="query" value={@query} phx-debounce="300" />
        <button type="submit">Search</button>
      </form>

      <div :for={result <- @results} class="result">
        <h3>{result.title}</h3>
        <p>{result.summary}</p>
      </div>
    </Layouts.app>
    """
  end

  defp search(query), do: MyApp.Search.find(query)
end
```

**LiveView lifecycle:** `mount/3` → `handle_params/3` → `render/1`. Events via `handle_event/3`. Server pushes via `handle_info/2`.

**Key patterns:**
- `assign/2,3` — set socket assigns
- `push_navigate/2` — navigate to new LiveView (full mount)
- `push_patch/2` — update URL without full remount (calls `handle_params`)
- `push_event/3` — push event to client JS hooks
- `stream/3,4` — efficient list rendering for large collections (inserts/deletes without re-rendering entire list)
- `async_assign/3` + `assign_async/3` — async data loading with loading/error states

### Channels — Real-Time WebSocket

```elixir
# In endpoint.ex
socket "/socket", MyAppWeb.UserSocket,
  websocket: true,
  longpoll: false

# lib/my_app_web/channels/user_socket.ex
defmodule MyAppWeb.UserSocket do
  use Phoenix.Socket

  channel "room:*", MyAppWeb.RoomChannel

  def connect(%{"token" => token}, socket, _connect_info) do
    case Phoenix.Token.verify(socket, "user auth", token, max_age: 86400) do
      {:ok, user_id} -> {:ok, assign(socket, :user_id, user_id)}
      {:error, _} -> :error
    end
  end

  def id(socket), do: "users_socket:#{socket.assigns.user_id}"
end

# lib/my_app_web/channels/room_channel.ex
defmodule MyAppWeb.RoomChannel do
  use MyAppWeb, :channel

  def join("room:" <> room_id, _params, socket) do
    {:ok, assign(socket, :room_id, room_id)}
  end

  def handle_in("new_msg", %{"body" => body}, socket) do
    broadcast!(socket, "new_msg", %{
      body: body,
      user_id: socket.assigns.user_id
    })
    {:noreply, socket}
  end
end
```

**Force disconnect all sessions for a user:**
```elixir
MyAppWeb.Endpoint.broadcast("users_socket:#{user.id}", "disconnect", %{})
```

### Contexts — Business Logic Boundary

Contexts are plain Elixir modules that encapsulate data access and business rules. They are the API between your web layer and your domain.

```elixir
# Generate with: mix phx.gen.context Catalog Product products title:string price:decimal
defmodule MyApp.Catalog do
  import Ecto.Query
  alias MyApp.Repo
  alias MyApp.Catalog.Product

  def list_products do
    Repo.all(Product)
  end

  def get_product!(id), do: Repo.get!(Product, id)

  def create_product(attrs \\ %{}) do
    %Product{}
    |> Product.changeset(attrs)
    |> Repo.insert()
  end

  def update_product(%Product{} = product, attrs) do
    product
    |> Product.changeset(attrs)
    |> Repo.update()
  end

  def delete_product(%Product{} = product) do
    Repo.delete(product)
  end

  def change_product(%Product{} = product, attrs \\ %{}) do
    Product.changeset(product, attrs)
  end
end
```

**Context design principles:**
- One context per bounded domain (Catalog, Accounts, Orders)
- Contexts own their schemas — other contexts reference by ID, not struct
- Cross-context calls go through the public context API, never access another context's Repo directly
- Contexts can nest related schemas (Comments under Posts)

### Authentication — `mix phx.gen.auth`

```bash
mix phx.gen.auth Accounts User users
```

Phoenix 1.8 generates:
- **Magic links** (passwordless) — email-based login links
- **"Sudo mode"** — re-authentication for sensitive actions
- Session-based auth with secure token handling
- Email confirmation and password reset flows
- `require_authenticated_user` plug for protected routes

### Scopes (New in 1.8)

Scopes make secure data access the default in generators. When you generate resources with a scope, all queries are automatically filtered by the scoped user.

```bash
mix phx.gen.live Posts Post posts title body:text --scope current_user
```

This generates code that automatically passes `current_user` to context functions, ensuring users only see their own data.

### Ecto — Database Layer

```elixir
# Schema
defmodule MyApp.Catalog.Product do
  use Ecto.Schema
  import Ecto.Changeset

  schema "products" do
    field :title, :string
    field :price, :decimal
    field :status, Ecto.Enum, values: [:draft, :published, :archived]
    has_many :reviews, MyApp.Reviews.Review
    belongs_to :category, MyApp.Catalog.Category
    timestamps(type: :utc_datetime)
  end

  def changeset(product, attrs) do
    product
    |> cast(attrs, [:title, :price, :status, :category_id])
    |> validate_required([:title, :price])
    |> validate_number(:price, greater_than: 0)
    |> unique_constraint(:title)
    |> foreign_key_constraint(:category_id)
  end
end

# Queries
import Ecto.Query

# Composable queries
def published(query \\ Product) do
  from p in query, where: p.status == :published
end

def recent(query, days \\ 7) do
  from p in query, where: p.inserted_at > ago(^days, "day")
end

def with_reviews(query) do
  from p in query, preload: [:reviews]
end

# Usage: Product |> published() |> recent(30) |> with_reviews() |> Repo.all()
```

### Telemetry — Built-in Observability

Phoenix 1.8 includes a Telemetry supervisor that tracks request duration, Ecto query times, and VM metrics out of the box.

```elixir
# lib/my_app_web/telemetry.ex (auto-generated)
defmodule MyAppWeb.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def metrics do
    [
      summary("phoenix.endpoint.stop.duration", unit: {:native, :millisecond}),
      summary("phoenix.router_dispatch.stop.duration", tags: [:route], unit: {:native, :millisecond}),
      summary("my_app.repo.query.total_time", unit: {:native, :millisecond}),
      summary("vm.memory.total", unit: {:byte, :kilobyte}),
      summary("vm.total_run_queue_lengths.total"),
      summary("vm.total_run_queue_lengths.cpu"),
    ]
  end
end
```

Integrates with **PromEx** for Prometheus/Grafana dashboards (see Monitoring section).

### Phoenix Security Best Practices

**Never pass untrusted input to:** `Code.eval_string/3`, `:os.cmd/2`, `System.cmd/3`, `System.shell/2`, `:erlang.binary_to_term/2`

**Ecto prevents SQL injection by default** — the query DSL parameterizes all inputs. Only `Ecto.Adapters.SQL.query/4` with raw string interpolation is vulnerable.

**Safe deserialization:**
```elixir
# UNSAFE — even with :safe flag
:erlang.binary_to_term(user_input, [:safe])

# SAFE — prevents executable terms
Plug.Crypto.non_executable_binary_to_term(user_input, [:safe])
```

**CSRF protection** is built into the `:browser` pipeline via `protect_from_forgery`. **Content Security Policy** is set by `put_secure_browser_headers`.

### Testing Phoenix

```elixir
# Controller test
defmodule MyAppWeb.ProductControllerTest do
  use MyAppWeb.ConnCase

  test "GET /products", %{conn: conn} do
    conn = get(conn, ~p"/products")
    assert html_response(conn, 200) =~ "Products"
  end

  test "POST /products creates product", %{conn: conn} do
    conn = post(conn, ~p"/products", product: %{title: "Widget", price: 9.99})
    assert redirected_to(conn) =~ ~p"/products/"
  end
end

# LiveView test
defmodule MyAppWeb.CounterLiveTest do
  use MyAppWeb.ConnCase
  import Phoenix.LiveViewTest

  test "increments counter", %{conn: conn} do
    {:ok, view, html} = live(conn, ~p"/counter")
    assert html =~ "Count: 0"

    assert view
           |> element("button", "+1")
           |> render_click() =~ "Count: 1"
  end
end

# Channel test
defmodule MyAppWeb.RoomChannelTest do
  use MyAppWeb.ChannelCase

  test "broadcasts new messages" do
    {:ok, _, socket} = subscribe_and_join(socket(MyAppWeb.UserSocket), MyAppWeb.RoomChannel, "room:lobby")

    push(socket, "new_msg", %{"body" => "hello"})
    assert_broadcast "new_msg", %{body: "hello"}
  end
end
```

### Phoenix Generators Cheat Sheet

```bash
# HTML CRUD (controller + views + templates + context + schema + migration)
mix phx.gen.html Catalog Product products title:string price:decimal

# LiveView CRUD
mix phx.gen.live Catalog Product products title:string price:decimal

# JSON API
mix phx.gen.json Catalog Product products title:string price:decimal

# Context + schema only (no web layer)
mix phx.gen.context Catalog Product products title:string price:decimal

# Schema + migration only
mix phx.gen.schema Product products title:string price:decimal

# Authentication
mix phx.gen.auth Accounts User users

# Channel
mix phx.gen.channel Room

# Presence
mix phx.gen.presence

# Release files (Dockerfile, release.ex, overlay scripts)
mix phx.gen.release
mix phx.gen.release --docker  # Include Dockerfile
```

### Phoenix Release & Deployment

```bash
# Generate release infrastructure
mix phx.gen.release --docker

# Build release
MIX_ENV=prod mix deps.get --only prod
MIX_ENV=prod mix compile
MIX_ENV=prod mix assets.deploy
MIX_ENV=prod mix release

# Release commands
_build/prod/rel/my_app/bin/server     # Start with Phoenix server
_build/prod/rel/my_app/bin/migrate    # Run migrations
_build/prod/rel/my_app/bin/my_app remote  # Attach IEx console
```

**Runtime config (`config/runtime.exs`):**
```elixir
import Config

if config_env() == :prod do
  database_url = System.fetch_env!("DATABASE_URL")
  secret_key_base = System.fetch_env!("SECRET_KEY_BASE")

  config :my_app, MyApp.Repo,
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10")

  config :my_app, MyAppWeb.Endpoint,
    url: [host: System.fetch_env!("PHX_HOST"), port: 443, scheme: "https"],
    http: [ip: {0, 0, 0, 0}, port: String.to_integer(System.get_env("PORT") || "4000")],
    secret_key_base: secret_key_base
end
```

---

## Common Libraries

| Library | Purpose | Hex |
|---------|---------|-----|
| Phoenix | Web framework + LiveView | `{:phoenix, "~> 1.8"}` |
| Ecto | Database wrapper + query DSL | `{:ecto_sql, "~> 3.12"}` |
| Ash | Declarative resource framework | `{:ash, "~> 3.0"}` |
| Oban | Background job processing | `{:oban, "~> 2.18"}` |
| Req | HTTP client (modern, composable) | `{:req, "~> 0.5"}` |
| Jason | JSON encoding/decoding | `{:jason, "~> 1.4"}` |
| Swoosh | Email sending | `{:swoosh, "~> 1.16"}` |
| ExUnit | Testing (built-in) | — |
| Mox | Mock behaviours for testing | `{:mox, "~> 1.1", only: :test}` |
| Credo | Static analysis / linting | `{:credo, "~> 1.7", only: [:dev, :test]}` |
| Dialyxir | Static typing via Dialyzer | `{:dialyxir, "~> 1.4", only: [:dev, :test]}` |
| libcluster | Automatic BEAM node clustering | `{:libcluster, "~> 3.4"}` |
| Horde | Distributed supervisor/registry | `{:horde, "~> 0.9"}` |
| Nx | Numerical computing / tensors | `{:nx, "~> 0.9"}` |
| Bumblebee | Pre-trained ML models on BEAM | `{:bumblebee, "~> 0.6"}` |
| Broadway | Data ingestion pipelines | `{:broadway, "~> 1.1"}` |
| PromEx | Prometheus metrics for Elixir | `{:prom_ex, "~> 1.9"}` |
| Finch | HTTP client (low-level, pooled) | `{:finch, "~> 0.19"}` |
| usage_rules | AI agent docs from deps | `{:usage_rules, "~> 1.1", only: :dev}` |

---

## Interop — Elixir as Orchestrator

### Python via Ports/Erlport
```elixir
# Using erlport for bidirectional Python calls
{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}])
result = :python.call(pid, :my_module, :my_function, [arg1, arg2])
:python.stop(pid)
```

### Rust via Rustler NIFs
```elixir
# mix.exs
{:rustler, "~> 0.34"}

# lib/my_nif.ex
defmodule MyApp.NativeSort do
  use Rustler, otp_app: :my_app, crate: "native_sort"

  # NIF stubs — replaced at load time by Rust implementations
  def sort(_list), do: :erlang.nif_error(:nif_not_loaded)
end
```

### System Commands via Ports
```elixir
# One-shot command
{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output])

# Long-running port
port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status])
send(port, {self(), {:command, "process\n"}})
receive do
  {^port, {:data, data}} -> handle_response(data)
end
```

---

## Cortex Deployment Story

### Current State (March 2026)
- Cortex runs Ubuntu 24.04 with Caddy as web server
- **Elixir 1.19.5 / OTP 27** installed via Erlang Solutions + GitHub releases
- Symbiont Elixir service running on port 8111 (Python Symbiont retired)
- systemd unit: `symbiont-ex-api.service`

### Proven Installation Steps for Ubuntu 24.04

**Important**: Ubuntu 24.04 repos ship ancient Elixir 1.14 / OTP 25. Do NOT use `apt install elixir`.

```bash
# Step 1: Remove Ubuntu's outdated Erlang/Elixir packages
apt-get remove -y erlang-base elixir erlang-dev erlang-parsetools erlang-syntax-tools
apt-get autoremove -y

# Step 2: Install Erlang from Erlang Solutions
wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
dpkg -i erlang-solutions_2.0_all.deb
apt-get update
apt-get install -y esl-erlang   # Installs OTP 27.x

# Step 3: Install precompiled Elixir from GitHub releases
# Check latest: curl -sL https://api.github.com/repos/elixir-lang/elixir/releases?per_page=5 | grep tag_name
ELIXIR_VERSION="1.19.5"
cd /opt
wget https://github.com/elixir-lang/elixir/releases/download/v${ELIXIR_VERSION}/elixir-otp-27.zip
unzip elixir-otp-27.zip -d elixir-${ELIXIR_VERSION}

# Step 4: Symlink binaries
for bin in elixir elixirc iex mix; do
  ln -sf /opt/elixir-${ELIXIR_VERSION}/bin/${bin} /usr/local/bin/${bin}
done

# Step 5: Verify
elixir --version   # Should show Elixir 1.19.5 (compiled with Erlang/OTP 27)
```

### Upgrading Elixir

To find the latest stable version:
```bash
curl -sL https://api.github.com/repos/elixir-lang/elixir/releases?per_page=10 | grep tag_name
```
Look for the highest non-rc tag. Then repeat Steps 3-5 above with the new version number.

### systemd Service Template
```ini
[Unit]
Description=Symbiont Elixir API
After=network.target

[Service]
Type=simple
# CRITICAL: mix requires HOME to be set
Environment=HOME=/root
Environment=MIX_ENV=prod
WorkingDirectory=/root/symbiont_ex
ExecStart=/usr/local/bin/mix run --no-halt
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

### BEAMOps Principles for Cortex
From "Engineering Elixir Applications" — the deployment philosophy:

1. **Environment Integrity** — identical builds dev/staging/prod via Docker + releases
2. **Infrastructure as Code** — Caddy config, systemd units, backup scripts all version-controlled
3. **OTP Releases** — self-contained, no runtime deps, `bin/my_app start`
4. **Distributed Erlang** — nodes discover each other, share state via PubSub, global registry
5. **Instrumentation** — PromEx + Prometheus + Grafana + Loki for full observability
6. **Health Checks + Rollbacks** — Docker health checks trigger automatic rollback on failed deploys
7. **Zero-downtime deploys** — rolling updates via Docker Swarm or `mix release` hot upgrades

### CI Pipeline for Elixir (GitHub Actions)
```yaml
name: Elixir CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      db:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.19.5'
          otp-version: '27.3'
      - run: mix deps.get
      - run: mix compile --warnings-as-errors
      - run: mix format --check-formatted
      - run: mix credo --strict
      - run: mix test
      - run: mix deps.unlock --check-unused
```

---

## Anti-Patterns to Avoid

### Process Anti-Patterns
- **GenServer as code organization** — don't wrap pure functions in a GenServer. Use modules.
- **Agent for complex state** — if you need more than get/update, use GenServer directly.
- **Spawning unsupervised processes** — always use `Task.Supervisor` or link to a supervisor.

### Code Anti-Patterns
- **Primitive obsession** — use structs, not bare maps, for domain concepts.
- **Boolean parameters** — use atoms or keyword options: `format: :json` not `json: true`.
- **Large modules** — split by concern, not by entity type. Domain logic, web layer, workers.
- **String keys in internal maps** — use atoms internally, strings only at boundaries (JSON, forms).

### Design Anti-Patterns
- **Monolithic contexts** — Phoenix contexts should be small, focused. Split `Accounts` from `Authentication`.
- **God GenServer** — one process handling all state for the app. Distribute responsibility.
- **Synchronous calls to slow services** — use `Task.async` + `Task.await` with timeouts.

---

## Quick Recipes

### HTTP Request with Req
```elixir
Req.get!("https://api.example.com/data",
  headers: [{"authorization", "Bearer #{token}"}],
  receive_timeout: 15_000
).body
```

### JSON Encode/Decode
```elixir
Jason.encode!(%{name: "Michael", role: :admin})
Jason.decode!(~s({"name": "Michael"}), keys: :atoms)
```

### Ecto Query
```elixir
from p in Post,
  where: p.status == :published,
  where: p.inserted_at > ago(7, "day"),
  order_by: [desc: p.inserted_at],
  limit: 10,
  preload: [:author, :comments]
```

### Background Job with Oban
```elixir
defmodule MyApp.Workers.EmailWorker do
  use Oban.Worker, queue: :mailers, max_attempts: 3

  @impl true
  def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do
    MyApp.Mailer.send(to, template)
  end
end

# Enqueue
%{to: "user@example.com", template: "welcome"}
|> MyApp.Workers.EmailWorker.new(schedule_in: 60)
|> Oban.insert!()
```

### LiveView Component
```elixir
defmodule MyAppWeb.CounterLive do
  use MyAppWeb, :live_view

  def mount(_params, _session, socket) do
    {:ok, assign(socket, count: 0)}
  end

  def handle_event("increment", _, socket) do
    {:noreply, update(socket, :count, &(&1 + 1))}
  end

  def render(assigns) do
    ~H"""
    <div>
      <h1>Count: {@count}</h1>
      <button phx-click="increment">+1</button>
    </div>
    """
  end
end
```

---

## AI Agent Lessons Learned (Symbiont Migration, March 2026)

Hard-won lessons from building the Elixir Symbiont orchestrator. These are things Claude (and other AI agents) got wrong or didn't know — preserved here so we don't repeat them.

### `System.cmd/3` Does NOT Have an `:input` Option

This is a persistent hallucination. **No version of Elixir** (1.14 through 1.19) supports passing stdin via `System.cmd/3`. The `:input` option simply does not exist.

**Wrong** (will silently ignore the option or error):
```elixir
System.cmd("claude", ["-p", "--model", "haiku"], input: prompt)
```

**Correct** — use `System.shell/2` with a pipe:
```elixir
escaped = prompt |> String.replace("'", "'\\''")
{output, exit_code} = System.shell("printf '%s' '#{escaped}' | claude -p --model haiku 2>&1")
```

Or use Erlang Ports directly for full stdin/stdout control:
```elixir
port = Port.open({:spawn_executable, "/usr/local/bin/claude"}, [:binary, :exit_status, args: ["-p"]])
Port.command(port, prompt)
Port.command(port, :eof)  # signal end of input
```

### `Float.round/2` Requires a Float Argument

`Float.round(0, 4)` crashes with `FunctionClauseError` because `0` is an integer, not a float. This commonly bites when summing an empty list — `Enum.sum([])` returns `0` (integer), not `0.0`.

**Wrong**:
```elixir
entries |> Enum.map(& &1["cost"]) |> Enum.sum() |> Float.round(4)
# Crashes when entries is empty!
```

**Correct** — use a float accumulator:
```elixir
entries
|> Enum.reduce(0.0, fn entry, acc -> acc + to_float(entry["cost"]) end)
|> Float.round(4)

defp to_float(nil), do: 0.0
defp to_float(n) when is_float(n), do: n
defp to_float(n) when is_integer(n), do: n * 1.0
defp to_float(_), do: 0.0
```

### Heredoc Closing `"""` Must Be on Its Own Line

Module attributes with heredocs are tricky. The closing `"""` cannot share a line with content.

**Wrong** (syntax error):
```elixir
@prompt """
Classify this task: """
```

**Correct** — use string concatenation for prompts with trailing content:
```elixir
@prompt "Classify this task. Respond with JSON: " <>
  ~s({"tier": 1|2|3, "reason": "brief explanation"}\n\n) <>
  "Task: "
```

### OTP Application Supervisors vs. Test Isolation

When your `application.ex` starts GenServers in the supervision tree, those processes auto-start when `mix test` runs. Tests that call `start_link` for the same named process will crash with `{:error, {:already_started, pid}}`.

**Solution**: Start an empty supervisor in test mode:
```elixir
# application.ex
def start(_type, _args) do
  if Application.get_env(:symbiont, :port) == 0 do
    Supervisor.start_link([], strategy: :one_for_one, name: Symbiont.Supervisor)
  else
    start_full()
  end
end
```

```elixir
# config/test.exs
config :symbiont, port: 0  # Signals test mode
```

Then each test's `setup` block starts only the processes it needs, with `on_exit` cleanup:
```elixir
setup do
  safe_stop(Symbiont.Ledger)
  {:ok, _} = Symbiont.Ledger.start_link(data_dir: tmp_dir)
  on_exit(fn -> safe_stop(Symbiont.Ledger) end)
end

def safe_stop(name) do
  case Process.whereis(name) do
    nil -> :ok
    pid -> try do GenServer.stop(pid) catch :exit, _ -> :ok end
  end
end
```

### `use Plug.Test` Is Deprecated

In modern Plug (1.15+), `use Plug.Test` emits a deprecation warning. Replace with explicit imports:

```elixir
# Old (deprecated)
use Plug.Test

# New
import Plug.Test    # for conn/2, conn/3
import Plug.Conn    # for put_req_header/3, etc.
```

### Ubuntu 24.04 Ships Ancient Elixir

Ubuntu's apt repos have Elixir 1.14 and OTP 25. These are years behind and missing critical features. **Never** use `apt install elixir` on Ubuntu. See the "Proven Installation Steps" section above for the correct approach.

Key gotcha: Ubuntu's `erlang-base` package conflicts with `esl-erlang` from Erlang Solutions. You must `apt-get remove` all Ubuntu erlang packages before installing `esl-erlang`.

### systemd Needs `Environment=HOME=/root`

`mix` and other Elixir tooling require the `HOME` environment variable. systemd services don't inherit it. Without `Environment=HOME=/root` in the unit file, services crash with "could not find the user home."

### How to Find the Real Latest Stable Elixir Version

Don't trust AI training data for version numbers. Query the source of truth:

```bash
curl -sL https://api.github.com/repos/elixir-lang/elixir/releases?per_page=10 | grep tag_name
```

Look for the highest version that does NOT contain `-rc` or `-dev`. As of March 2026, that's `v1.19.5`.

---

## Resources

- [Elixir Official Docs](https://hexdocs.pm/elixir/) — always check 1.19.5 version
- [Ash Framework Docs](https://hexdocs.pm/ash/) — resource-oriented patterns
- [Phoenix HexDocs](https://hexdocs.pm/phoenix/) — web framework
- [Elixir Forum](https://elixirforum.com/) — community Q&A
- [Elixir School](https://elixirschool.com/) — learning resource
- "Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes above)
- "Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns