Symbiont aa4f55e8f1 Deep dive: comprehensive OTP, Ecto, cortex-native CI/CD

2026-03-20 17:08:36 +00:00

16 KiB

Raw Permalink Blame History

Elixir Part 4: Ecosystem, Production & Deployment

Ash Framework — Declarative Resource Modeling

For production/substantial projects, Ash provides a declarative, resource-oriented approach that eliminates boilerplate while remaining extensible.

# mix.exs deps
{:ash, "~> 3.0"},
{:ash_postgres, "~> 2.0"},    # Ecto/Postgres data layer
{:ash_phoenix, "~> 2.0"},     # Phoenix integration
{:ash_graphql, "~> 1.0"},     # Auto-generated GraphQL API
{:ash_json_api, "~> 1.0"},    # Auto-generated JSON:API

Resource Definition

defmodule MyApp.Blog.Post do
  use Ash.Resource,
    domain: MyApp.Blog,
    data_layer: AshPostgres.DataLayer

  postgres do
    table "posts"
    repo MyApp.Repo
  end

  attributes do
    uuid_primary_key :id
    attribute :title, :string, allow_nil?: false
    attribute :body, :string, allow_nil?: false
    attribute :status, :atom, constraints: [one_of: [:draft, :published, :archived]], default: :draft
    timestamps()
  end

  relationships do
    belongs_to :author, MyApp.Accounts.User
    has_many :comments, MyApp.Blog.Comment
  end

  actions do
    defaults [:read, :destroy]

    create :create do
      accept [:title, :body]
      change relate_actor(:author)
    end

    update :publish do
      change set_attribute(:status, :published)
    end
  end

  policies do
    policy action_type(:read) do
      authorize_if always()
    end

    policy action_type([:create, :update, :destroy]) do
      authorize_if actor_attribute_equals(:role, :admin)
    end
  end
end

Domain Definition

defmodule MyApp.Blog do
  use Ash.Domain

  resources do
    resource MyApp.Blog.Post
    resource MyApp.Blog.Comment
  end
end

Using Ash Resources

# Create
MyApp.Blog.Post
|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"}, actor: current_user)
|> Ash.create!()

# Read with filters
MyApp.Blog.Post
|> Ash.Query.filter(status == :published)
|> Ash.Query.sort(inserted_at: :desc)
|> Ash.read!()

# Custom action
post |> Ash.Changeset.for_update(:publish) |> Ash.update!()

When to use Ash vs plain Phoenix: Ash excels when you need consistent authorization, multi-tenancy, auto-generated APIs, or complex domain logic. For simple CRUD apps or learning, plain Phoenix contexts are fine.

Jido — Multi-Agent Orchestration Framework

Jido (~> 2.1) is the premier Elixir framework for building and managing agents of all types — not just LLMs. Vision: 10,000 agents per user on the BEAM.

# mix.exs deps
{:jido, "~> 2.1"},
{:jido_ai, "~> 0.x"},        # AI/LLM integration
{:jido_action, "~> 0.x"},    # Composable actions
{:jido_signal, "~> 0.x"},    # CloudEvents-based messaging

Core Concepts

Agents are pure functional structs — immutable, no side effects in the agent itself. Side effects are described as Directives.

defmodule MyAgent do
  use Jido.Agent,
    name: "my_agent",
    description: "Does useful things",
    actions: [FetchData, ProcessData, NotifyUser],
    schema: [
      model: [type: :string, default: "gpt-4"],
      temperature: [type: :float, default: 0.7]
    ]
end

# Create and command
{:ok, agent} = MyAgent.new()
{agent, directives} = MyAgent.cmd(agent, %Signal{type: "task.assigned", data: %{task: "analyze"}})
# directives are typed effects: Emit, Spawn, Schedule, Stop

Actions — Composable Units

Actions are the building blocks. Each has a schema, validates input, and returns output.

defmodule FetchData do
  use Jido.Action,
    name: "fetch_data",
    description: "Fetches data from an API",
    schema: [
      url: [type: :string, required: true],
      timeout: [type: :integer, default: 5000]
    ]

  @impl true
  def run(params, _context) do
    case Req.get(params.url, receive_timeout: params.timeout) do
      {:ok, %{status: 200, body: body}} -> {:ok, %{data: body}}
      {:ok, %{status: status}} -> {:error, "HTTP #{status}"}
      {:error, reason} -> {:error, reason}
    end
  end
end

# Actions expose themselves as AI tools
FetchData.to_tool()  # Returns tool spec for LLM function calling

Signals — CloudEvents v1.0.2 Messaging

signal = %Jido.Signal{
  type: "task.completed",
  source: "agent:worker_1",
  data: %{result: "analysis complete"},
  subject: "task:123"
}

Directives — Typed Effect Descriptions

Agents don't perform side effects directly. They return Directives describing what should happen:

Directive	Effect
`Emit`	Send a signal to another agent/system
`Spawn`	Create a new child agent
`Schedule`	Schedule a future action
`Stop`	Terminate the agent

AgentServer — Runtime Process

AgentServer wraps an agent in a GenServer for real-time operation:

{:ok, pid} = Jido.AgentServer.start_link(agent: MyAgent, id: "worker-1")
Jido.AgentServer.signal(pid, %Signal{type: "task.start", data: %{...}})

When to Use Jido

Multi-agent systems where agents communicate via signals
LLM-powered agents that need tool calling (via to_tool/0)
Systems requiring parent-child agent hierarchies
Workflows with complex state machines (Jido includes FSM strategy)
Any scenario targeting high agent density (thousands per node)

OTP Releases & Docker

OTP Release

# mix.exs
def project do
  [
    app: :my_app,
    releases: [
      my_app: [
        include_executables_for: [:unix],
        steps: [:assemble, :tar]
      ]
    ]
  ]
end

Docker Multistage Build

# Build stage
FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20240904 AS build
RUN apt-get update && apt-get install -y build-essential git
WORKDIR /app
ENV MIX_ENV=prod

COPY mix.exs mix.lock ./
RUN mix deps.get --only $MIX_ENV && mix deps.compile

COPY config/config.exs config/prod.exs config/
COPY lib lib
COPY priv priv
COPY assets assets

RUN mix assets.deploy && mix compile && mix release

# Runtime stage
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y libstdc++6 openssl libncurses5 locales
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
ENV LANG=en_US.UTF-8

WORKDIR /app
COPY --from=build /app/_build/prod/rel/my_app ./

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD curl -f http://localhost:4000/health || exit 1

CMD ["bin/server"]

Distributed Erlang & Clustering

# libcluster config (config/runtime.exs)
config :libcluster,
  topologies: [
    local: [
      strategy: Cluster.Strategy.Gossip
    ],
    # OR for Docker/K8s:
    k8s: [
      strategy: Cluster.Strategy.Kubernetes,
      config: [
        mode: :hostname,
        kubernetes_namespace: "default",
        kubernetes_selector: "app=my_app"
      ]
    ]
  ]

# Distributed PubSub (built into Phoenix)
Phoenix.PubSub.broadcast(MyApp.PubSub, "events", {:new_order, order})
Phoenix.PubSub.subscribe(MyApp.PubSub, "events")

# Horde for distributed supervisor/registry
{:horde, "~> 0.9"}

Monitoring & Observability

PromEx — Prometheus Metrics for Elixir

# mix.exs
{:prom_ex, "~> 1.9"}

# lib/my_app/prom_ex.ex
defmodule MyApp.PromEx do
  use PromEx, otp_app: :my_app

  @impl true
  def plugins do
    [
      PromEx.Plugins.Application,
      PromEx.Plugins.Beam,
      {PromEx.Plugins.Phoenix, router: MyAppWeb.Router, endpoint: MyAppWeb.Endpoint},
      {PromEx.Plugins.Ecto, repos: [MyApp.Repo]},
      PromEx.Plugins.Oban
    ]
  end

  @impl true
  def dashboards do
    [
      {:prom_ex, "application.json"},
      {:prom_ex, "beam.json"},
      {:prom_ex, "phoenix.json"},
      {:prom_ex, "ecto.json"}
    ]
  end
end

Full Observability Stack (BEAMOps)

From "Engineering Elixir Applications":

Component	Role
PromEx	Expose BEAM/Phoenix/Ecto metrics as Prometheus endpoints
Prometheus	Scrape and store time-series metrics
Grafana	Dashboards and alerting
Loki	Log aggregation (like Prometheus but for logs)
Promtail/Alloy	Ship logs from containers to Loki

Health checks + automatic rollback: Docker HEALTHCHECK triggers rollback if the new container fails health checks within the start period. Use docker system prune automation to prevent disk bloat.

CI/CD — Cortex-Native Pipeline

Instead of GitHub Actions, we run CI/CD directly on cortex using shell scripts, systemd, and git hooks. This keeps the entire workflow within our ecosystem.

Git Push Hook — Trigger on Push

On cortex, set up a bare repo with a post-receive hook:

# On cortex: create bare repo
mkdir -p /data/repos/my_app.git && cd /data/repos/my_app.git
git init --bare

# post-receive hook
cat > hooks/post-receive << 'HOOK'
#!/bin/bash
set -euo pipefail

WORK_DIR="/data/builds/my_app"
LOG="/var/log/ci/my_app-$(date +%Y%m%d-%H%M%S).log"
mkdir -p /var/log/ci "$WORK_DIR"

echo "=== CI triggered at $(date) ===" | tee "$LOG"

# Checkout latest
GIT_WORK_TREE="$WORK_DIR" git checkout -f main 2>&1 | tee -a "$LOG"

# Run CI pipeline
cd "$WORK_DIR"
exec /data/scripts/ci-pipeline.sh "$WORK_DIR" 2>&1 | tee -a "$LOG"
HOOK
chmod +x hooks/post-receive

CI Pipeline Script

#!/bin/bash
# /data/scripts/ci-pipeline.sh
set -euo pipefail
APP_DIR="$1"
cd "$APP_DIR"

echo "--- Dependencies ---"
mix deps.get

echo "--- Compile (warnings as errors) ---"
mix compile --warnings-as-errors

echo "--- Format check ---"
mix format --check-formatted

echo "--- Static analysis ---"
mix credo --strict

echo "--- Tests ---"
MIX_ENV=test mix test

echo "--- Unused deps check ---"
mix deps.unlock --check-unused

echo "--- Build release ---"
MIX_ENV=prod mix release --overwrite

echo "=== CI PASSED ==="

# Optional: auto-deploy on success
# /data/scripts/deploy.sh "$APP_DIR"

Deploy Script

#!/bin/bash
# /data/scripts/deploy.sh
set -euo pipefail
APP_DIR="$1"
APP_NAME=$(basename "$APP_DIR")
RELEASE_DIR="/data/releases/$APP_NAME"

echo "--- Deploying $APP_NAME ---"

# Stop current
systemctl stop "$APP_NAME" 2>/dev/null || true

# Copy release
mkdir -p "$RELEASE_DIR"
cp -r "$APP_DIR/_build/prod/rel/$APP_NAME/"* "$RELEASE_DIR/"

# Run migrations
"$RELEASE_DIR/bin/migrate"

# Start
systemctl start "$APP_NAME"

# Health check with rollback
sleep 5
if ! curl -sf http://localhost:4000/health > /dev/null; then
  echo "!!! Health check failed — rolling back"
  systemctl stop "$APP_NAME"
  # Restore previous release from backup
  cp -r "$RELEASE_DIR.prev/"* "$RELEASE_DIR/"
  systemctl start "$APP_NAME"
  exit 1
fi

echo "=== Deploy SUCCESS ==="

Systemd Service Template

# /etc/systemd/system/my_app.service
[Unit]
Description=MyApp Elixir Service
After=network.target postgresql.service

[Service]
Type=exec
User=deploy
Environment=MIX_ENV=prod
Environment=PORT=4000
Environment=PHX_HOST=myapp.hydrascale.net
EnvironmentFile=/data/releases/my_app/.env
ExecStart=/data/releases/my_app/bin/server
ExecStop=/data/releases/my_app/bin/my_app stop
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Push from Dev Machine

# Add cortex as a git remote
git remote add cortex root@cortex.hydrascale.net:/data/repos/my_app.git

# Push triggers CI → optional auto-deploy
git push cortex main

Symbiont Integration

For orchestration via Symbiont, the CI can report status:

# At end of ci-pipeline.sh
curl -s -X POST http://localhost:8080/api/tasks/ci-report \
  -H "Content-Type: application/json" \
  -d "{\"app\": \"$APP_NAME\", \"status\": \"passed\", \"commit\": \"$(git rev-parse HEAD)\"}"

Interop — Elixir as Orchestrator

Python via Ports/Erlport

# Using erlport for bidirectional Python calls
{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}])
result = :python.call(pid, :my_module, :my_function, [arg1, arg2])
:python.stop(pid)

Rust via Rustler NIFs

# mix.exs
{:rustler, "~> 0.34"}

# lib/my_nif.ex
defmodule MyApp.NativeSort do
  use Rustler, otp_app: :my_app, crate: "native_sort"

  # NIF stubs — replaced at load time by Rust implementations
  def sort(_list), do: :erlang.nif_error(:nif_not_loaded)
end

System Commands via Ports

# One-shot command
{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output])

# Long-running port
port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status])
send(port, {self(), {:command, "process\n"}})
receive do
  {^port, {:data, data}} -> handle_response(data)
end

Cortex Deployment Story

Current State

Cortex runs Ubuntu 24.04 with Caddy as web server
Elixir is not yet installed — needs asdf or direct install
Docker is planned but not installed — needed for containerized Elixir apps
Symbiont (Python) is the current orchestrator — Elixir will gradually take over

Installation Plan for Cortex

# Option 1: asdf (recommended — manages multiple versions)
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0
echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc
asdf plugin add erlang
asdf plugin add elixir
asdf install erlang 27.3
asdf install elixir 1.19.5-otp-27
asdf global erlang 27.3
asdf global elixir 1.19.5-otp-27

# Option 2: Direct from Erlang Solutions
wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb
dpkg -i erlang-solutions_2.0_all.deb
apt-get update && apt-get install esl-erlang elixir

BEAMOps Principles for Cortex

From "Engineering Elixir Applications" — the deployment philosophy:

Environment Integrity — identical builds dev/staging/prod via Docker + releases
Infrastructure as Code — Caddy config, systemd units, backup scripts all version-controlled
OTP Releases — self-contained, no runtime deps, bin/my_app start
Distributed Erlang — nodes discover each other, share state via PubSub, global registry
Instrumentation — PromEx + Prometheus + Grafana + Loki for full observability
Health Checks + Rollbacks — Docker health checks trigger automatic rollback on failed deploys
Zero-downtime deploys — rolling updates via Docker Swarm or mix release hot upgrades

Anti-Patterns to Avoid

Process Anti-Patterns

GenServer as code organization — don't wrap pure functions in a GenServer. Use modules.
Agent for complex state — if you need more than get/update, use GenServer directly.
Spawning unsupervised processes — always use Task.Supervisor or link to a supervisor.

Code Anti-Patterns

Primitive obsession — use structs, not bare maps, for domain concepts.
Boolean parameters — use atoms or keyword options: format: :json not json: true.
Large modules — split by concern, not by entity type. Domain logic, web layer, workers.
String keys in internal maps — use atoms internally, strings only at boundaries (JSON, forms).

Design Anti-Patterns

Monolithic contexts — Phoenix contexts should be small, focused. Split Accounts from Authentication.
God GenServer — one process handling all state for the app. Distribute responsibility.
Synchronous calls to slow services — use Task.async + Task.await with timeouts.

Quick Recipes

HTTP Request with Req

Req.get!("https://api.example.com/data",
  headers: [{"authorization", "Bearer #{token}"}],
  receive_timeout: 15_000
).body

JSON Encode/Decode

Jason.encode!(%{name: "Michael", role: :admin})
Jason.decode!(~s({"name": "Michael"}), keys: :atoms)

Background Job with Oban

defmodule MyApp.Workers.EmailWorker do
  use Oban.Worker, queue: :mailers, max_attempts: 3

  @impl true
  def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do
    MyApp.Mailer.send(to, template)
  end
end

# Enqueue
%{to: "user@example.com", template: "welcome"}
|> MyApp.Workers.EmailWorker.new(schedule_in: 60)
|> Oban.insert!()

Resources

Elixir Official Docs — always check 1.19.5 version
Ash Framework Docs — resource-oriented patterns
Phoenix HexDocs — web framework
Jido Docs — multi-agent orchestration
Elixir Forum — community Q&A
"Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes)
"Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns

16 KiB Raw Permalink Blame History