# Elixir — Comprehensive Development Guide

## Quick Reference

| Item | Value |
|------|-------|
| Current Version | **Elixir 1.19.5** (target for all new code) |
| Required OTP | **OTP 27+** (OTP 28 supported) |
| Cortex Status | Elixir **not yet installed** on cortex.hydrascale.net |
| AI Agent Tooling | `usage_rules` hex package (~> 1.2) — **always include** |
| Production Framework | **Ash Framework** for substantial projects |
| Paradigm | Functional, concurrent, fault-tolerant on BEAM VM |

---

## CRITICAL RULES — Read First

1. **Target Elixir 1.19.5** — not 1.15, not 1.17. Use current idioms and features.
2. **Always add `usage_rules`** to every project: `{:usage_rules, "~> 1.1", only: [:dev]}` — it generates `AGENTS.md`/`CLAUDE.md` from dependency docs, giving AI agents rich context about the libraries in use.
3. **Use Ash Framework** for production/substantial projects (not necessarily POCs). Ash provides declarative resources, built-in authorization, and extensions for Postgres, Phoenix, GraphQL, JSON:API.
4. **Cortex is Elixir-first** — Elixir is the primary orchestrator language, with interop to Python (Erlport/ports), Rust (Rustler NIFs), and Zig when needed.
5. **Never use deprecated patterns** from pre-1.16 code (see Breaking Changes section).

---

## Elixir 1.19 — What's New

### Gradual Set-Theoretic Type System

Elixir 1.19 significantly advances the built-in type system. It is **sound**, **gradual**, and **set-theoretic** (types compose via union, intersection, negation).

**Current capabilities (1.19):**
- Type inference from existing code — no annotations required yet (user-provided signatures coming in future releases)
- **Protocol dispatch type checking** — the compiler warns when you pass a value to string interpolation, `for` comprehensions, or other protocol-dispatched operations that don't implement the required protocol
- **Anonymous function type inference** — `fn` literals and `&captures` now propagate types, catching mismatches at compile time
- Types: `atom()`, `binary()`, `integer()`, `float()`, `pid()`, `port()`, `reference()`, `tuple()`, `list()`, `map()`, `function()`
- Compose with: `atom() or integer()` (union), `atom() and integer()` (intersection → `none()`), `atom() and not nil` (difference)
- `dynamic()` type for gradual typing — represents runtime-checked values
- Tuple precision: `{:ok, binary()}`, open tuples with `...`: `{:ok, binary(), ...}`
- Map types: `%{key: value}` for closed maps, `%{optional(atom()) => term()}` for open maps
- Function types: `(integer() -> boolean())`

**What this means in practice:** The compiler now catches real bugs — passing a struct to string interpolation that doesn't implement `String.Chars`, using a non-enumerable in `for`, type mismatches in anonymous function calls. These are warnings today, errors in the future.

### Up to 4x Faster Compilation

Two major improvements:

1. **Lazy module loading** — modules are no longer loaded immediately when defined. The parallel compiler controls both compilation and loading, reducing pressure on the Erlang code server. Reports of 2x+ speedup on large projects.

2. **Parallel dependency compilation** — set `MIX_OS_DEPS_COMPILE_PARTITION_COUNT` env var to partition OS dep compilation across CPU cores.

**Potential regressions:** If you spawn processes during compilation that invoke other project modules, use `Kernel.ParallelCompiler.pmap/2` or `Code.ensure_compiled!/1` before spawning. Also affects `@on_load` callbacks that reference sibling modules.

### Other 1.19 Features

- `min/2` and `max/2` allowed in guards
- `Access.values/0` — traverse all values in a map/keyword
- `String.count/2` — count occurrences of a pattern
- Unicode 17.0.0 support
- Multi-line IEx prompts
- `mix help Mod.fun/arity` — get help for specific functions
- New pretty printing infrastructure
- OpenChain compliance with Source SBoM
- Erlang/OTP 28 support

---

## Breaking Changes Since 1.15

These are critical to know — older books and tutorials will use the old patterns.

### Struct Update Syntax (1.18+)
```elixir
# OLD (1.15) — no longer valid
%User{user | name: "new"}

# NEW (1.18+) — explicit pattern match required
%User{} = user
%User{user | name: "new"}
# Or in one expression:
%User{name: "new"} = Map.merge(user, %{name: "new"})
```
Actually the change is: `%URI{my_uri | path: "/new"}` now requires `my_uri` to match `%URI{}`. If the compiler can't verify it's the right struct type, it warns.

### Regex as Struct Field Defaults (OTP 28)
```elixir
# BROKEN on OTP 28 — regex literals can't be struct field defaults
defmodule MyMod do
  defstruct pattern: ~r/foo/  # Compile error on OTP 28
end

# FIX — use @default_pattern or compute at runtime
defmodule MyMod do
  @default_pattern ~r/foo/
  defstruct pattern: @default_pattern  # Still fails — use nil + init
end
```

### Logger Backends Deprecated
```elixir
# OLD
config :logger, backends: [:console]

# NEW — use LoggerHandler (Erlang's logger)
config :logger, :default_handler, []
config :logger, :default_formatter, format: "$time $metadata[$level] $message\n"
```

### Mix Task Separator
```elixir
# OLD — comma separator
mix do compile, test

# NEW (1.17+) — plus separator
mix do compile + test
```

### mix cli/0 Replaces Multiple Config Keys
```elixir
# OLD
def project do
  [default_task: "phx.server",
   preferred_cli_env: [test: :test],
   preferred_cli_target: [...]]
end

# NEW
def cli do
  [default_task: "phx.server",
   preferred_envs: [test: :test],
   preferred_targets: [...]]
end
```

---

## Core Language Patterns

### The Pipeline Principle
Elixir code flows through transformations via `|>`. Design functions to take the "subject" as the first argument.

```elixir
orders
|> Enum.filter(&(&1.status == :pending))
|> Enum.sort_by(& &1.created_at, DateTime)
|> Enum.map(&process_order/1)
```

### Pattern Matching — The Heart of Elixir
```elixir
# Function head matching — preferred over conditionals
def process(%Order{status: :pending} = order), do: ship(order)
def process(%Order{status: :shipped} = order), do: track(order)
def process(%Order{status: :delivered}), do: :noop

# Pin operator to match against existing values
expected = "hello"
^expected = some_function()  # Asserts equality

# Map/struct matching is partial — only listed keys must match
%{name: name} = %{name: "Michael", age: 42}  # name = "Michael"
```

### With Expressions for Happy-Path Chaining
```elixir
with {:ok, user} <- fetch_user(id),
     {:ok, account} <- fetch_account(user),
     {:ok, balance} <- check_balance(account) do
  {:ok, balance}
else
  {:error, :not_found} -> {:error, "User not found"}
  {:error, :insufficient} -> {:error, "Insufficient funds"}
  error -> {:error, "Unknown: #{inspect(error)}"}
end
```

### Structs and Protocols
```elixir
defmodule Money do
  defstruct [:amount, :currency]

  defimpl String.Chars do
    def to_string(%Money{amount: a, currency: c}), do: "#{a} #{c}"
  end

  defimpl Inspect do
    def inspect(%Money{amount: a, currency: c}, _opts) do
      "#Money<#{a} #{c}>"
    end
  end
end
```

### Behaviours for Contracts
```elixir
defmodule PaymentProvider do
  @callback charge(amount :: integer(), currency :: String.t()) ::
    {:ok, transaction_id :: String.t()} | {:error, reason :: term()}
  @callback refund(transaction_id :: String.t()) ::
    {:ok, term()} | {:error, reason :: term()}
end

defmodule Stripe do
  @behaviour PaymentProvider

  @impl true
  def charge(amount, currency), do: # ...

  @impl true
  def refund(transaction_id), do: # ...
end
```

---

## Concurrency Model — BEAM Processes

### Fundamentals
- **Processes are cheap** — ~2KB initial memory, microseconds to spawn, millions can run concurrently
- **No shared memory** — processes communicate via message passing only
- **Each process has its own heap** — GC is per-process, no stop-the-world pauses
- **Preemptive scheduling** — one scheduler per CPU core, ~4000 reductions then yield
- **Process isolation** — one process crashing doesn't affect others

### Spawning and Messaging
```elixir
# Basic spawn + message passing
pid = spawn(fn ->
  receive do
    {:greet, name} -> IO.puts("Hello, #{name}!")
  end
end)

send(pid, {:greet, "Michael"})

# Linked processes — bidirectional crash propagation
pid = spawn_link(fn -> raise "boom" end)  # Caller also crashes

# Monitored processes — one-directional crash notification
ref = Process.monitor(pid)
receive do
  {:DOWN, ^ref, :process, ^pid, reason} -> IO.puts("Crashed: #{reason}")
end
```

### Task — Structured Concurrency
```elixir
# Fire and forget
Task.start(fn -> send_email(user) end)

# Async/await (linked to caller)
task = Task.async(fn -> expensive_computation() end)
result = Task.await(task, 30_000)  # 30s timeout

# Parallel map
Task.async_stream(urls, &fetch_url/1, max_concurrency: 10, timeout: 15_000)
|> Enum.to_list()
```

### GenServer — Stateful Server Processes
```elixir
defmodule Counter do
  use GenServer

  # Client API
  def start_link(initial \\ 0), do: GenServer.start_link(__MODULE__, initial, name: __MODULE__)
  def increment, do: GenServer.cast(__MODULE__, :increment)
  def get, do: GenServer.call(__MODULE__, :get)

  # Server callbacks
  @impl true
  def init(initial), do: {:ok, initial}

  @impl true
  def handle_cast(:increment, count), do: {:noreply, count + 1}

  @impl true
  def handle_call(:get, _from, count), do: {:reply, count, count}

  @impl true
  def handle_info(:tick, count) do
    IO.puts("Count: #{count}")
    Process.send_after(self(), :tick, 1000)
    {:noreply, count}
  end
end
```

**Key design principle:** GenServer callbacks run sequentially in a single process — this is both the synchronization mechanism and the potential bottleneck. Keep callbacks fast; delegate heavy work to spawned tasks.

### Supervisors — Let It Crash
```elixir
defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Order matters — started top to bottom, stopped bottom to top
      MyApp.Repo,                           # Ecto repo
      {MyApp.Cache, []},                    # Custom GenServer
      {Task.Supervisor, name: MyApp.TaskSupervisor},
      MyAppWeb.Endpoint                     # Phoenix endpoint last
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
```

**Restart strategies:**
- `:one_for_one` — only restart the crashed child (most common)
- `:one_for_all` — restart all children if any crashes (tightly coupled)
- `:rest_for_one` — restart crashed child and all children started after it

### DynamicSupervisor — Runtime Child Management
```elixir
defmodule MyApp.SessionSupervisor do
  use DynamicSupervisor

  def start_link(_), do: DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__)
  def init(:ok), do: DynamicSupervisor.init(strategy: :one_for_one)

  def start_session(session_id) do
    spec = {MyApp.Session, session_id}
    DynamicSupervisor.start_child(__MODULE__, spec)
  end
end
```

---

## OTP Releases & Deployment

### Building a Release
```bash
# Generate release config (Phoenix projects)
mix phx.gen.release

# Build the release
MIX_ENV=prod mix release

# The release is self-contained — no Erlang/Elixir needed on target
_build/prod/rel/my_app/bin/my_app start       # Foreground
_build/prod/rel/my_app/bin/my_app daemon       # Background
_build/prod/rel/my_app/bin/my_app remote       # Attach IEx to running node
_build/prod/rel/my_app/bin/my_app eval "MyApp.Seeds.run()"  # One-off command
```

### Release Configuration Files
- `config/config.exs` — **compile-time** config (before code compiles)
- `config/runtime.exs` — **runtime** config (executed on every boot) — use for env vars, secrets
- `rel/env.sh.eex` — shell environment for the release (VM flags, env vars)
- `rel/vm.args.eex` — Erlang VM flags (node name, cookie, memory limits)

### Docker Multistage Build Pattern
```dockerfile
# === Builder Stage ===
FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20250317 AS builder

RUN apt-get update -y && apt-get install -y build-essential git && apt-get clean
WORKDIR /app

ENV MIX_ENV=prod
RUN mix local.hex --force && mix local.rebar --force

COPY mix.exs mix.lock ./
RUN mix deps.get --only prod && mix deps.compile

COPY config/config.exs config/prod.exs config/
COPY priv priv
COPY lib lib
COPY assets assets  # If Phoenix

RUN mix assets.deploy  # If Phoenix
RUN mix compile
RUN mix release

# === Runner Stage ===
FROM debian:bookworm-slim AS runner

RUN apt-get update -y && \
    apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates && \
    apt-get clean && rm -f /var/lib/apt/lists/*_*

ENV LANG=en_US.UTF-8
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

WORKDIR /app
RUN useradd --create-home app && chown -R app:app /app
USER app

COPY --from=builder --chown=app:app /app/_build/prod/rel/my_app ./

CMD ["bin/my_app", "start"]
```

### Distributed Erlang in Production
```elixir
# In rel/env.sh.eex — set node name and cookie
export RELEASE_DISTRIBUTION=name
export RELEASE_NODE=my_app@${HOSTNAME}
export RELEASE_COOKIE=my_secret_cookie

# Or in runtime.exs
config :my_app, MyApp.Cluster,
  strategy: Cluster.Strategy.DNSPoll,
  config: [
    polling_interval: 5_000,
    query: "my-app.local",
    node_basename: "my_app"
  ]
```

**Key libraries for clustering:** `libcluster` (automatic node discovery), `Horde` (distributed supervisor/registry), `Phoenix.PubSub` (cross-node pub/sub already built into Phoenix).

---

## Monitoring & Instrumentation

### The Elixir Observability Stack
- **PromEx** — Elixir library exposing `/metrics` endpoint for Prometheus scraping. Includes plugins for Phoenix, Ecto, LiveView, BEAM VM, Oban
- **Prometheus** — scrapes and stores time-series metrics
- **Loki** + **Promtail** — log aggregation (Promtail scrapes Docker container logs → Loki stores)
- **Grafana** — visualization dashboards for both metrics and logs
- **Alloy** — OpenTelemetry collector bridging metrics to Prometheus

### Adding PromEx to a Phoenix App
```elixir
# mix.exs
{:prom_ex, "~> 1.9"}

# lib/my_app/prom_ex.ex
defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      {PromEx.Plugins.Phoenix, router: MyAppWeb.Router},
      {PromEx.Plugins.Ecto, repos: [MyApp.Repo]},
      PromEx.Plugins.Oban
    ]
  end

  @impl true
  def dashboards do
    [{:prom_ex, "application.json"}, {:prom_ex, "beam.json"},
     {:prom_ex, "phoenix.json"}, {:prom_ex, "ecto.json"}]
  end
end
```

### Key BEAM Metrics to Watch
- `erlang_vm_process_count` — total BEAM processes (normal: thousands, alarm at millions)
- `erlang_vm_memory_bytes_total` — total VM memory
- `erlang_vm_atom_count` — atoms are never GC'd; watch for leaks
- `phoenix_endpoint_stop_duration_milliseconds` — request latency
- `ecto_repo_query_duration_milliseconds` — database query time

---

## usage_rules — AI Agent Documentation

**Always include in every project.** This is non-negotiable.

```elixir
# mix.exs deps
{:usage_rules, "~> 1.1", only: [:dev]}

# mix.exs project config
def project do
  [
    # ... other config
    usage_rules: [
      packages: :all,  # Or specific: ["phoenix", "ecto", ~r/ash/]
      output: :agents_md,  # Generates AGENTS.md
      mode: :linked  # Or :inlined for single-file
    ]
  ]
end
```

**Key commands:**
```bash
mix usage_rules.gen          # Generate AGENTS.md from deps
mix usage_rules.search_docs  # Search hex documentation
mix usage_rules.gen_skill    # Generate SKILL.md for Cowork
```

This consolidates `usage-rules.md` files from all dependencies into a single reference document, giving any AI agent working on the project full context about library APIs, patterns, and conventions.

---

## Ash Framework — Production Backend

Ash is a **declarative, resource-oriented** framework for building Elixir backends. Use it for substantial projects — it handles data layer, authorization, validation, and API generation.

### Core Concepts
- **Resources** — the central abstraction (like models but richer): attributes, actions, relationships, calculations, aggregates, policies
- **Domains** — organizational containers that group related resources and define the public API
- **Actions** — CRUD + custom actions defined declaratively on resources
- **Data Layers** — pluggable persistence (AshPostgres, AshSqlite, ETS for dev)
- **Policies** — declarative authorization rules on resources/actions
- **Extensions** — AshPhoenix, AshGraphql, AshJsonApi, AshAuthentication, AshOban

### Quick Start
```bash
mix igniter.new my_app --install ash,ash_postgres,ash_phoenix
```

### Resource Example
```elixir
defmodule MyApp.Blog.Post do
  use Ash.Resource,
    domain: MyApp.Blog,
    data_layer: AshPostgres.DataLayer

  postgres do
    table "posts"
    repo MyApp.Repo
  end

  attributes do
    uuid_primary_key :id
    attribute :title, :string, allow_nil?: false
    attribute :body, :string, allow_nil?: false
    attribute :status, :atom, constraints: [one_of: [:draft, :published]], default: :draft
    timestamps()
  end

  actions do
    defaults [:read, :destroy]

    create :create do
      accept [:title, :body]
    end

    update :publish do
      change set_attribute(:status, :published)
      change set_attribute(:published_at, &DateTime.utc_now/0)
    end
  end

  relationships do
    belongs_to :author, MyApp.Accounts.User
    has_many :comments, MyApp.Blog.Comment
  end

  policies do
    policy action_type(:create) do
      authorize_if actor_attribute_equals(:role, :author)
    end
    policy action_type(:read) do
      authorize_if always()
    end
  end
end
```

### Domain Example
```elixir
defmodule MyApp.Blog do
  use Ash.Domain

  resources do
    resource MyApp.Blog.Post
    resource MyApp.Blog.Comment
  end
end
```

### Using Resources
```elixir
# Create
MyApp.Blog.Post
|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"})
|> Ash.create!()

# Read with filters
MyApp.Blog.Post
|> Ash.Query.filter(status == :published)
|> Ash.Query.sort(inserted_at: :desc)
|> Ash.Query.limit(10)
|> Ash.read!()

# Custom action
post |> Ash.Changeset.for_update(:publish) |> Ash.update!()
```

---

## Mix Project Structure

```
my_app/
├── config/
│   ├── config.exs          # Compile-time config
│   ├── dev.exs
│   ├── prod.exs
│   ├── runtime.exs         # Runtime config (secrets, env vars)
│   └── test.exs
├── lib/
│   ├── my_app/
│   │   ├── application.ex  # OTP Application + supervisor tree
│   │   ├── repo.ex         # Ecto Repo
│   │   └── ...             # Domain modules
│   └── my_app_web/         # Phoenix web layer (if applicable)
│       ├── endpoint.ex
│       ├── router.ex
│       ├── controllers/
│       ├── live/            # LiveView modules
│       └── components/
├── priv/
│   ├── repo/migrations/
│   └── static/
├── test/
│   ├── support/
│   └── ...
├── mix.exs
├── mix.lock
├── .formatter.exs
└── AGENTS.md               # Generated by usage_rules
```

---

## Testing

```elixir
# test/my_app/blog_test.exs
defmodule MyApp.BlogTest do
  use MyApp.DataCase, async: true  # Async when no shared state

  describe "creating posts" do
    test "creates with valid attributes" do
      assert {:ok, post} = Blog.create_post(%{title: "Hi", body: "World"})
      assert post.title == "Hi"
      assert post.status == :draft
    end

    test "fails without title" do
      assert {:error, changeset} = Blog.create_post(%{body: "World"})
      assert "can't be blank" in errors_on(changeset).title
    end
  end
end
```

**Testing philosophy:** Use `async: true` wherever possible. Ecto's SQL Sandbox allows concurrent test execution. For GenServer tests, start under a test supervisor. For external services, use `Mox` for behaviour-based mocking.

---

## Common Libraries

| Library | Purpose | Hex |
|---------|---------|-----|
| Phoenix | Web framework + LiveView | `{:phoenix, "~> 1.7"}` |
| Ecto | Database wrapper + query DSL | `{:ecto_sql, "~> 3.12"}` |
| Ash | Declarative resource framework | `{:ash, "~> 3.0"}` |
| Oban | Background job processing | `{:oban, "~> 2.18"}` |
| Req | HTTP client (modern, composable) | `{:req, "~> 0.5"}` |
| Jason | JSON encoding/decoding | `{:jason, "~> 1.4"}` |
| Swoosh | Email sending | `{:swoosh, "~> 1.16"}` |
| ExUnit | Testing (built-in) | — |
| Mox | Mock behaviours for testing | `{:mox, "~> 1.1", only: :test}` |
| Credo | Static analysis / linting | `{:credo, "~> 1.7", only: [:dev, :test]}` |
| Dialyxir | Static typing via Dialyzer | `{:dialyxir, "~> 1.4", only: [:dev, :test]}` |
| libcluster | Automatic BEAM node clustering | `{:libcluster, "~> 3.4"}` |
| Horde | Distributed supervisor/registry | `{:horde, "~> 0.9"}` |
| Nx | Numerical computing / tensors | `{:nx, "~> 0.9"}` |
| Bumblebee | Pre-trained ML models on BEAM | `{:bumblebee, "~> 0.6"}` |
| Broadway | Data ingestion pipelines | `{:broadway, "~> 1.1"}` |
| PromEx | Prometheus metrics for Elixir | `{:prom_ex, "~> 1.9"}` |
| Finch | HTTP client (low-level, pooled) | `{:finch, "~> 0.19"}` |
| usage_rules | AI agent docs from deps | `{:usage_rules, "~> 1.1", only: :dev}` |

---

## Interop — Elixir as Orchestrator

### Python via Ports/Erlport
```elixir
# Using erlport for bidirectional Python calls
{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}])
result = :python.call(pid, :my_module, :my_function, [arg1, arg2])
:python.stop(pid)
```

### Rust via Rustler NIFs
```elixir
# mix.exs
{:rustler, "~> 0.34"}

# lib/my_nif.ex
defmodule MyApp.NativeSort do
  use Rustler, otp_app: :my_app, crate: "native_sort"

  # NIF stubs — replaced at load time by Rust implementations
  def sort(_list), do: :erlang.nif_error(:nif_not_loaded)
end
```

### System Commands via Ports
```elixir
# One-shot command
{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output])

# Long-running port
port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status])
send(port, {self(), {:command, "process\n"}})
receive do
  {^port, {:data, data}} -> handle_response(data)
end
```

---

## Cortex Deployment Story

### Current State
- Cortex runs Ubuntu 24.04 with Caddy as web server
- Elixir is **not yet installed** — needs `asdf` or direct install
- Docker is **planned but not installed** — needed for containerized Elixir apps
- Symbiont (Python) is the current orchestrator — Elixir will gradually take over

### Installation Plan for Cortex
```bash
# Option 1: asdf (recommended — manages multiple versions)
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0
echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
asdf plugin add erlang
asdf plugin add elixir
asdf install erlang 27.3
asdf install elixir 1.19.5-otp-27
asdf global erlang 27.3
asdf global elixir 1.19.5-otp-27

# Option 2: Direct from Erlang Solutions
wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
dpkg -i erlang-solutions_2.0_all.deb
apt-get update && apt-get install esl-erlang elixir
```

### BEAMOps Principles for Cortex
From "Engineering Elixir Applications" — the deployment philosophy:

1. **Environment Integrity** — identical builds dev/staging/prod via Docker + releases
2. **Infrastructure as Code** — Caddy config, systemd units, backup scripts all version-controlled
3. **OTP Releases** — self-contained, no runtime deps, `bin/my_app start`
4. **Distributed Erlang** — nodes discover each other, share state via PubSub, global registry
5. **Instrumentation** — PromEx + Prometheus + Grafana + Loki for full observability
6. **Health Checks + Rollbacks** — Docker health checks trigger automatic rollback on failed deploys
7. **Zero-downtime deploys** — rolling updates via Docker Swarm or `mix release` hot upgrades

### CI Pipeline for Elixir (GitHub Actions)
```yaml
name: Elixir CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      db:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
        ports: ['5432:5432']
    steps:
      - uses: actions/checkout@v4
      - uses: erlef/setup-beam@v1
        with:
          elixir-version: '1.19.5'
          otp-version: '27.3'
      - run: mix deps.get
      - run: mix compile --warnings-as-errors
      - run: mix format --check-formatted
      - run: mix credo --strict
      - run: mix test
      - run: mix deps.unlock --check-unused
```

---

## Anti-Patterns to Avoid

### Process Anti-Patterns
- **GenServer as code organization** — don't wrap pure functions in a GenServer. Use modules.
- **Agent for complex state** — if you need more than get/update, use GenServer directly.
- **Spawning unsupervised processes** — always use `Task.Supervisor` or link to a supervisor.

### Code Anti-Patterns
- **Primitive obsession** — use structs, not bare maps, for domain concepts.
- **Boolean parameters** — use atoms or keyword options: `format: :json` not `json: true`.
- **Large modules** — split by concern, not by entity type. Domain logic, web layer, workers.
- **String keys in internal maps** — use atoms internally, strings only at boundaries (JSON, forms).

### Design Anti-Patterns
- **Monolithic contexts** — Phoenix contexts should be small, focused. Split `Accounts` from `Authentication`.
- **God GenServer** — one process handling all state for the app. Distribute responsibility.
- **Synchronous calls to slow services** — use `Task.async` + `Task.await` with timeouts.

---

## Quick Recipes

### HTTP Request with Req
```elixir
Req.get!("https://api.example.com/data",
  headers: [{"authorization", "Bearer #{token}"}],
  receive_timeout: 15_000
).body
```

### JSON Encode/Decode
```elixir
Jason.encode!(%{name: "Michael", role: :admin})
Jason.decode!(~s({"name": "Michael"}), keys: :atoms)
```

### Ecto Query
```elixir
from p in Post,
  where: p.status == :published,
  where: p.inserted_at > ago(7, "day"),
  order_by: [desc: p.inserted_at],
  limit: 10,
  preload: [:author, :comments]
```

### Background Job with Oban
```elixir
defmodule MyApp.Workers.EmailWorker do
  use Oban.Worker, queue: :mailers, max_attempts: 3

  @impl true
  def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do
    MyApp.Mailer.send(to, template)
  end
end

# Enqueue
%{to: "user@example.com", template: "welcome"}
|> MyApp.Workers.EmailWorker.new(schedule_in: 60)
|> Oban.insert!()
```

### LiveView Component
```elixir
defmodule MyAppWeb.CounterLive do
  use MyAppWeb, :live_view

  def mount(_params, _session, socket) do
    {:ok, assign(socket, count: 0)}
  end

  def handle_event("increment", _, socket) do
    {:noreply, update(socket, :count, &(&1 + 1))}
  end

  def render(assigns) do
    ~H"""
    <div>
      <h1>Count: {@count}</h1>
      <button phx-click="increment">+1</button>
    </div>
    """
  end
end
```

---

## Resources

- [Elixir Official Docs](https://hexdocs.pm/elixir/) — always check 1.19.5 version
- [Ash Framework Docs](https://hexdocs.pm/ash/) — resource-oriented patterns
- [Phoenix HexDocs](https://hexdocs.pm/phoenix/) — web framework
- [Elixir Forum](https://elixirforum.com/) — community Q&A
- [Elixir School](https://elixirschool.com/) — learning resource
- "Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes above)
- "Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns