skills/elixir/elixir-part4-ecosystem.md

# Elixir Part 4: Ecosystem, Production & Deployment

## Ash Framework — Declarative Resource Modeling

For production/substantial projects, Ash provides a declarative, resource-oriented approach that eliminates boilerplate while remaining extensible.

```elixir
# mix.exs deps
{:ash, "~> 3.0"},
{:ash_postgres, "~> 2.0"},    # Ecto/Postgres data layer
{:ash_phoenix, "~> 2.0"},     # Phoenix integration
{:ash_graphql, "~> 1.0"},     # Auto-generated GraphQL API
{:ash_json_api, "~> 1.0"},    # Auto-generated JSON:API
```

### Resource Definition

```elixir
defmodule MyApp.Blog.Post do
  use Ash.Resource,
    domain: MyApp.Blog,
    data_layer: AshPostgres.DataLayer

  postgres do
    table "posts"
    repo MyApp.Repo
  end

  attributes do
    uuid_primary_key :id
    attribute :title, :string, allow_nil?: false
    attribute :body, :string, allow_nil?: false
    attribute :status, :atom, constraints: [one_of: [:draft, :published, :archived]], default: :draft
    timestamps()
  end

  relationships do
    belongs_to :author, MyApp.Accounts.User
    has_many :comments, MyApp.Blog.Comment
  end

  actions do
    defaults [:read, :destroy]

    create :create do
      accept [:title, :body]
      change relate_actor(:author)
    end

    update :publish do
      change set_attribute(:status, :published)
    end
  end

  policies do
    policy action_type(:read) do
      authorize_if always()
    end

    policy action_type([:create, :update, :destroy]) do
      authorize_if actor_attribute_equals(:role, :admin)
    end
  end
end
```

### Domain Definition

```elixir
defmodule MyApp.Blog do
  use Ash.Domain

  resources do
    resource MyApp.Blog.Post
    resource MyApp.Blog.Comment
  end
end
```

### Using Ash Resources

```elixir
# Create
MyApp.Blog.Post
|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"}, actor: current_user)
|> Ash.create!()

# Read with filters
MyApp.Blog.Post
|> Ash.Query.filter(status == :published)
|> Ash.Query.sort(inserted_at: :desc)
|> Ash.read!()

# Custom action
post |> Ash.Changeset.for_update(:publish) |> Ash.update!()
```

**When to use Ash vs plain Phoenix:** Ash excels when you need consistent authorization, multi-tenancy, auto-generated APIs, or complex domain logic. For simple CRUD apps or learning, plain Phoenix contexts are fine.

---

## Jido — Multi-Agent Orchestration Framework

Jido (~> 2.1) is the premier Elixir framework for building and managing agents of all types — not just LLMs. Vision: 10,000 agents per user on the BEAM.

```elixir
# mix.exs deps
{:jido, "~> 2.1"},
{:jido_ai, "~> 0.x"},        # AI/LLM integration
{:jido_action, "~> 0.x"},    # Composable actions
{:jido_signal, "~> 0.x"},    # CloudEvents-based messaging
```

### Core Concepts

**Agents are pure functional structs** — immutable, no side effects in the agent itself. Side effects are described as Directives.

```elixir
defmodule MyAgent do
  use Jido.Agent,
    name: "my_agent",
    description: "Does useful things",
    actions: [FetchData, ProcessData, NotifyUser],
    schema: [
      model: [type: :string, default: "gpt-4"],
      temperature: [type: :float, default: 0.7]
    ]
end

# Create and command
{:ok, agent} = MyAgent.new()
{agent, directives} = MyAgent.cmd(agent, %Signal{type: "task.assigned", data: %{task: "analyze"}})
# directives are typed effects: Emit, Spawn, Schedule, Stop
```

### Actions — Composable Units

Actions are the building blocks. Each has a schema, validates input, and returns output.

```elixir
defmodule FetchData do
  use Jido.Action,
    name: "fetch_data",
    description: "Fetches data from an API",
    schema: [
      url: [type: :string, required: true],
      timeout: [type: :integer, default: 5000]
    ]

  @impl true
  def run(params, _context) do
    case Req.get(params.url, receive_timeout: params.timeout) do
      {:ok, %{status: 200, body: body}} -> {:ok, %{data: body}}
      {:ok, %{status: status}} -> {:error, "HTTP #{status}"}
      {:error, reason} -> {:error, reason}
    end
  end
end

# Actions expose themselves as AI tools
FetchData.to_tool()  # Returns tool spec for LLM function calling
```

### Signals — CloudEvents v1.0.2 Messaging

```elixir
signal = %Jido.Signal{
  type: "task.completed",
  source: "agent:worker_1",
  data: %{result: "analysis complete"},
  subject: "task:123"
}
```

### Directives — Typed Effect Descriptions

Agents don't perform side effects directly. They return Directives describing what should happen:

| Directive | Effect |
|-----------|--------|
| `Emit` | Send a signal to another agent/system |
| `Spawn` | Create a new child agent |
| `Schedule` | Schedule a future action |
| `Stop` | Terminate the agent |

### AgentServer — Runtime Process

`AgentServer` wraps an agent in a GenServer for real-time operation:

```elixir
{:ok, pid} = Jido.AgentServer.start_link(agent: MyAgent, id: "worker-1")
Jido.AgentServer.signal(pid, %Signal{type: "task.start", data: %{...}})
```

### When to Use Jido

- Multi-agent systems where agents communicate via signals
- LLM-powered agents that need tool calling (via `to_tool/0`)
- Systems requiring parent-child agent hierarchies
- Workflows with complex state machines (Jido includes FSM strategy)
- Any scenario targeting high agent density (thousands per node)

---

## OTP Releases & Docker

### OTP Release

```elixir
# mix.exs
def project do
  [
    app: :my_app,
    releases: [
      my_app: [
        include_executables_for: [:unix],
        steps: [:assemble, :tar]
      ]
    ]
  ]
end
```

### Docker Multistage Build

```dockerfile
# Build stage
FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20240904 AS build
RUN apt-get update && apt-get install -y build-essential git
WORKDIR /app
ENV MIX_ENV=prod

COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV && mix deps.compile

COPY config/config.exs config/prod.exs config/
COPY lib lib
COPY priv priv
COPY assets assets

RUN mix assets.deploy && mix compile && mix release

# Runtime stage
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y libstdc++6 openssl libncurses5 locales
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8

WORKDIR /app
COPY --from=build /app/_build/prod/rel/my_app ./

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:4000/health || exit 1

CMD ["bin/server"]
```

---

## Distributed Erlang & Clustering

```elixir
# libcluster config (config/runtime.exs)
config :libcluster,
  topologies: [
    local: [
      strategy: Cluster.Strategy.Gossip
    ],
    # OR for Docker/K8s:
    k8s: [
      strategy: Cluster.Strategy.Kubernetes,
      config: [
        mode: :hostname,
        kubernetes_namespace: "default",
        kubernetes_selector: "app=my_app"
      ]
    ]
  ]

# Distributed PubSub (built into Phoenix)
Phoenix.PubSub.broadcast(MyApp.PubSub, "events", {:new_order, order})
Phoenix.PubSub.subscribe(MyApp.PubSub, "events")

# Horde for distributed supervisor/registry
{:horde, "~> 0.9"}
```

---

## Monitoring & Observability

### PromEx — Prometheus Metrics for Elixir

```elixir
# mix.exs
{:prom_ex, "~> 1.9"}

# lib/my_app/prom_ex.ex
defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      {PromEx.Plugins.Phoenix, router: MyAppWeb.Router, endpoint: MyAppWeb.Endpoint},
      {PromEx.Plugins.Ecto, repos: [MyApp.Repo]},
      PromEx.Plugins.Oban
    ]
  end

  @impl true
  def dashboards do
    [
      {:prom_ex, "application.json"},
      {:prom_ex, "beam.json"},
      {:prom_ex, "phoenix.json"},
      {:prom_ex, "ecto.json"}
    ]
  end
end
```

### Full Observability Stack (BEAMOps)

From "Engineering Elixir Applications":

| Component | Role |
|-----------|------|
| **PromEx** | Expose BEAM/Phoenix/Ecto metrics as Prometheus endpoints |
| **Prometheus** | Scrape and store time-series metrics |
| **Grafana** | Dashboards and alerting |
| **Loki** | Log aggregation (like Prometheus but for logs) |
| **Promtail/Alloy** | Ship logs from containers to Loki |

**Health checks + automatic rollback:** Docker `HEALTHCHECK` triggers rollback if the new container fails health checks within the start period. Use `docker system prune` automation to prevent disk bloat.

---

## CI/CD — Cortex-Native Pipeline

Instead of GitHub Actions, we run CI/CD directly on cortex using shell scripts, systemd, and git hooks. This keeps the entire workflow within our ecosystem.

### Git Push Hook — Trigger on Push

On cortex, set up a bare repo with a `post-receive` hook:

```bash
# On cortex: create bare repo
mkdir -p /data/repos/my_app.git && cd /data/repos/my_app.git
git init --bare

# post-receive hook
cat > hooks/post-receive << 'HOOK'
#!/bin/bash
set -euo pipefail

WORK_DIR="/data/builds/my_app"
LOG="/var/log/ci/my_app-$(date +%Y%m%d-%H%M%S).log"
mkdir -p /var/log/ci "$WORK_DIR"

echo "=== CI triggered at $(date) ===" | tee "$LOG"

# Checkout latest
GIT_WORK_TREE="$WORK_DIR" git checkout -f main 2>&1 | tee -a "$LOG"

# Run CI pipeline
cd "$WORK_DIR"
exec /data/scripts/ci-pipeline.sh "$WORK_DIR" 2>&1 | tee -a "$LOG"
HOOK
chmod +x hooks/post-receive
```

### CI Pipeline Script

```bash
#!/bin/bash
# /data/scripts/ci-pipeline.sh
set -euo pipefail
APP_DIR="$1"
cd "$APP_DIR"

echo "--- Dependencies ---"
mix deps.get

echo "--- Compile (warnings as errors) ---"
mix compile --warnings-as-errors

echo "--- Format check ---"
mix format --check-formatted

echo "--- Static analysis ---"
mix credo --strict

echo "--- Tests ---"
MIX_ENV=test mix test

echo "--- Unused deps check ---"
mix deps.unlock --check-unused

echo "--- Build release ---"
MIX_ENV=prod mix release --overwrite

echo "=== CI PASSED ==="

# Optional: auto-deploy on success
# /data/scripts/deploy.sh "$APP_DIR"
```

### Deploy Script

```bash
#!/bin/bash
# /data/scripts/deploy.sh
set -euo pipefail
APP_DIR="$1"
APP_NAME=$(basename "$APP_DIR")
RELEASE_DIR="/data/releases/$APP_NAME"

echo "--- Deploying $APP_NAME ---"

# Stop current
systemctl stop "$APP_NAME" 2>/dev/null || true

# Copy release
mkdir -p "$RELEASE_DIR"
cp -r "$APP_DIR/_build/prod/rel/$APP_NAME/"* "$RELEASE_DIR/"

# Run migrations
"$RELEASE_DIR/bin/migrate"

# Start
systemctl start "$APP_NAME"

# Health check with rollback
sleep 5
if ! curl -sf http://localhost:4000/health > /dev/null; then
  echo "!!! Health check failed — rolling back"
  systemctl stop "$APP_NAME"
  # Restore previous release from backup
  cp -r "$RELEASE_DIR.prev/"* "$RELEASE_DIR/"
  systemctl start "$APP_NAME"
  exit 1
fi

echo "=== Deploy SUCCESS ==="
```

### Systemd Service Template

```ini
# /etc/systemd/system/my_app.service
[Unit]
Description=MyApp Elixir Service
After=network.target postgresql.service

[Service]
Type=exec
User=deploy
Environment=MIX_ENV=prod
Environment=PORT=4000
Environment=PHX_HOST=myapp.hydrascale.net
EnvironmentFile=/data/releases/my_app/.env
ExecStart=/data/releases/my_app/bin/server
ExecStop=/data/releases/my_app/bin/my_app stop
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

### Push from Dev Machine

```bash
# Add cortex as a git remote
git remote add cortex root@cortex.hydrascale.net:/data/repos/my_app.git

# Push triggers CI → optional auto-deploy
git push cortex main
```

### Symbiont Integration

For orchestration via Symbiont, the CI can report status:

```bash
# At end of ci-pipeline.sh
curl -s -X POST http://localhost:8080/api/tasks/ci-report \
  -H "Content-Type: application/json" \
  -d "{\"app\": \"$APP_NAME\", \"status\": \"passed\", \"commit\": \"$(git rev-parse HEAD)\"}"
```

---

## Interop — Elixir as Orchestrator

### Python via Ports/Erlport
```elixir
# Using erlport for bidirectional Python calls
{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}])
result = :python.call(pid, :my_module, :my_function, [arg1, arg2])
:python.stop(pid)
```

### Rust via Rustler NIFs
```elixir
# mix.exs
{:rustler, "~> 0.34"}

# lib/my_nif.ex
defmodule MyApp.NativeSort do
  use Rustler, otp_app: :my_app, crate: "native_sort"

  # NIF stubs — replaced at load time by Rust implementations
  def sort(_list), do: :erlang.nif_error(:nif_not_loaded)
end
```

### System Commands via Ports
```elixir
# One-shot command
{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output])

# Long-running port
port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status])
send(port, {self(), {:command, "process\n"}})
receive do
  {^port, {:data, data}} -> handle_response(data)
end
```

---

## Cortex Deployment Story

### Current State
- Cortex runs Ubuntu 24.04 with Caddy as web server
- Elixir is **not yet installed** — needs `asdf` or direct install
- Docker is **planned but not installed** — needed for containerized Elixir apps
- Symbiont (Python) is the current orchestrator — Elixir will gradually take over

### Installation Plan for Cortex
```bash
# Option 1: asdf (recommended — manages multiple versions)
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0
echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
asdf plugin add erlang
asdf plugin add elixir
asdf install erlang 27.3
asdf install elixir 1.19.5-otp-27
asdf global erlang 27.3
asdf global elixir 1.19.5-otp-27

# Option 2: Direct from Erlang Solutions
wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
dpkg -i erlang-solutions_2.0_all.deb
apt-get update && apt-get install esl-erlang elixir
```

### BEAMOps Principles for Cortex

From "Engineering Elixir Applications" — the deployment philosophy:

1. **Environment Integrity** — identical builds dev/staging/prod via Docker + releases
2. **Infrastructure as Code** — Caddy config, systemd units, backup scripts all version-controlled
3. **OTP Releases** — self-contained, no runtime deps, `bin/my_app start`
4. **Distributed Erlang** — nodes discover each other, share state via PubSub, global registry
5. **Instrumentation** — PromEx + Prometheus + Grafana + Loki for full observability
6. **Health Checks + Rollbacks** — Docker health checks trigger automatic rollback on failed deploys
7. **Zero-downtime deploys** — rolling updates via Docker Swarm or `mix release` hot upgrades

---

## Anti-Patterns to Avoid

### Process Anti-Patterns
- **GenServer as code organization** — don't wrap pure functions in a GenServer. Use modules.
- **Agent for complex state** — if you need more than get/update, use GenServer directly.
- **Spawning unsupervised processes** — always use `Task.Supervisor` or link to a supervisor.

### Code Anti-Patterns
- **Primitive obsession** — use structs, not bare maps, for domain concepts.
- **Boolean parameters** — use atoms or keyword options: `format: :json` not `json: true`.
- **Large modules** — split by concern, not by entity type. Domain logic, web layer, workers.
- **String keys in internal maps** — use atoms internally, strings only at boundaries (JSON, forms).

### Design Anti-Patterns
- **Monolithic contexts** — Phoenix contexts should be small, focused. Split `Accounts` from `Authentication`.
- **God GenServer** — one process handling all state for the app. Distribute responsibility.
- **Synchronous calls to slow services** — use `Task.async` + `Task.await` with timeouts.

---

## Quick Recipes

### HTTP Request with Req
```elixir
Req.get!("https://api.example.com/data",
  headers: [{"authorization", "Bearer #{token}"}],
  receive_timeout: 15_000
).body
```

### JSON Encode/Decode
```elixir
Jason.encode!(%{name: "Michael", role: :admin})
Jason.decode!(~s({"name": "Michael"}), keys: :atoms)
```

### Background Job with Oban
```elixir
defmodule MyApp.Workers.EmailWorker do
  use Oban.Worker, queue: :mailers, max_attempts: 3

  @impl true
  def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do
    MyApp.Mailer.send(to, template)
  end
end

# Enqueue
%{to: "user@example.com", template: "welcome"}
|> MyApp.Workers.EmailWorker.new(schedule_in: 60)
|> Oban.insert!()
```

---

## Resources

- [Elixir Official Docs](https://hexdocs.pm/elixir/) — always check 1.19.5 version
- [Ash Framework Docs](https://hexdocs.pm/ash/) — resource-oriented patterns
- [Phoenix HexDocs](https://hexdocs.pm/phoenix/) — web framework
- [Jido Docs](https://hexdocs.pm/jido/) — multi-agent orchestration
- [Elixir Forum](https://elixirforum.com/) — community Q&A
- "Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes)
- "Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns