diff --git a/elixir/SKILL.md b/elixir/SKILL.md index 388fad5..dc682a2 100644 --- a/elixir/SKILL.md +++ b/elixir/SKILL.md @@ -10,6 +10,7 @@ | Cortex Status | Elixir **not yet installed** on cortex.hydrascale.net | | AI Agent Tooling | `usage_rules` hex package (~> 1.2) — **always include** | | Production Framework | **Ash Framework** for substantial projects | +| Agent Framework | **Jido** (~> 2.1) for multi-agent systems | | Paradigm | Functional, concurrent, fault-tolerant on BEAM VM | --- @@ -17,1473 +18,51 @@ ## CRITICAL RULES — Read First 1. **Target Elixir 1.19.5** — not 1.15, not 1.17. Use current idioms and features. -2. **Always add `usage_rules`** to every project: `{:usage_rules, "~> 1.1", only: [:dev]}` — it generates `AGENTS.md`/`CLAUDE.md` from dependency docs, giving AI agents rich context about the libraries in use. -3. **Use Ash Framework** for production/substantial projects (not necessarily POCs). Ash provides declarative resources, built-in authorization, and extensions for Postgres, Phoenix, GraphQL, JSON:API. -4. **Cortex is Elixir-first** — Elixir is the primary orchestrator language, with interop to Python (Erlport/ports), Rust (Rustler NIFs), and Zig when needed. -5. **Never use deprecated patterns** from pre-1.16 code (see Breaking Changes section). +2. **Always add `usage_rules`** to every project: `{:usage_rules, "~> 1.1", only: [:dev]}`. +3. **Use Ash Framework** for production/substantial projects (not necessarily POCs). +4. **Cortex is Elixir-first** — Elixir is the primary orchestrator, with interop to Python/Rust/Zig. +5. **Never use deprecated patterns** from pre-1.16 code (see Part 1). +6. **Phoenix 1.8** — always specify `:formats` on controllers, use `~p` verified routes, use scopes. +7. **For agentic workflows** — consider `GenStateMachine` over GenServer (built-in timeouts, state enter, postpone). For multi-agent orchestration, use **Jido**. --- -## Elixir 1.19 — What's New +## Guide Structure — Load What You Need -### Gradual Set-Theoretic Type System +This guide is split into focused parts. Load the part relevant to your current task: -Elixir 1.19 significantly advances the built-in type system. It is **sound**, **gradual**, and **set-theoretic** (types compose via union, intersection, negation). +### Part 1: Core Language & OTP (`elixir-part1-core.md`) +Elixir 1.19 features, type system, breaking changes since 1.15, pattern matching, pipelines, with expressions, structs, protocols, behaviours, Mix project structure, testing. -**Current capabilities (1.19):** -- Type inference from existing code — no annotations required yet (user-provided signatures coming in future releases) -- **Protocol dispatch type checking** — the compiler warns when you pass a value to string interpolation, `for` comprehensions, or other protocol-dispatched operations that don't implement the required protocol -- **Anonymous function type inference** — `fn` literals and `&captures` now propagate types, catching mismatches at compile time -- Types: `atom()`, `binary()`, `integer()`, `float()`, `pid()`, `port()`, `reference()`, `tuple()`, `list()`, `map()`, `function()` -- Compose with: `atom() or integer()` (union), `atom() and integer()` (intersection → `none()`), `atom() and not nil` (difference) -- `dynamic()` type for gradual typing — represents runtime-checked values -- Tuple precision: `{:ok, binary()}`, open tuples with `...`: `{:ok, binary(), ...}` -- Map types: `%{key: value}` for closed maps, `%{optional(atom()) => term()}` for open maps -- Function types: `(integer() -> boolean())` +### Part 2: Concurrency & OTP (`elixir-part2-concurrency.md`) +BEAM processes, message passing, GenServer, Supervisors, DynamicSupervisor, Task, GenStateMachine (state machines for agentic workflows), Registry, ETS. -**What this means in practice:** The compiler now catches real bugs — passing a struct to string interpolation that doesn't implement `String.Chars`, using a non-enumerable in `for`, type mismatches in anonymous function calls. These are warnings today, errors in the future. +### Part 3: Phoenix Framework (`elixir-part3-phoenix.md`) +Phoenix 1.8.5 — router/pipelines, verified routes, controllers, HEEx components, LiveView, Channels, contexts, Ecto, authentication, scopes, telemetry, security, testing, generators, deployment. -### Up to 4x Faster Compilation - -Two major improvements: - -1. **Lazy module loading** — modules are no longer loaded immediately when defined. The parallel compiler controls both compilation and loading, reducing pressure on the Erlang code server. Reports of 2x+ speedup on large projects. - -2. **Parallel dependency compilation** — set `MIX_OS_DEPS_COMPILE_PARTITION_COUNT` env var to partition OS dep compilation across CPU cores. - -**Potential regressions:** If you spawn processes during compilation that invoke other project modules, use `Kernel.ParallelCompiler.pmap/2` or `Code.ensure_compiled!/1` before spawning. Also affects `@on_load` callbacks that reference sibling modules. - -### Other 1.19 Features - -- `min/2` and `max/2` allowed in guards -- `Access.values/0` — traverse all values in a map/keyword -- `String.count/2` — count occurrences of a pattern -- Unicode 17.0.0 support -- Multi-line IEx prompts -- `mix help Mod.fun/arity` — get help for specific functions -- New pretty printing infrastructure -- OpenChain compliance with Source SBoM -- Erlang/OTP 28 support +### Part 4: Ecosystem & Production (`elixir-part4-ecosystem.md`) +Ash Framework, Jido (multi-agent systems), usage_rules, OTP releases, Docker deployment, distributed Erlang, monitoring (PromEx/Prometheus/Grafana), CI/CD, interop (Python/Rust/Zig), common libraries, cortex deployment plan, anti-patterns. --- -## Breaking Changes Since 1.15 - -These are critical to know — older books and tutorials will use the old patterns. - -### Struct Update Syntax (1.18+) -```elixir -# OLD (1.15) — no longer valid -%User{user | name: "new"} - -# NEW (1.18+) — explicit pattern match required -%User{} = user -%User{user | name: "new"} -# Or in one expression: -%User{name: "new"} = Map.merge(user, %{name: "new"}) -``` -Actually the change is: `%URI{my_uri | path: "/new"}` now requires `my_uri` to match `%URI{}`. If the compiler can't verify it's the right struct type, it warns. - -### Regex as Struct Field Defaults (OTP 28) -```elixir -# BROKEN on OTP 28 — regex literals can't be struct field defaults -defmodule MyMod do - defstruct pattern: ~r/foo/ # Compile error on OTP 28 -end - -# FIX — use @default_pattern or compute at runtime -defmodule MyMod do - @default_pattern ~r/foo/ - defstruct pattern: @default_pattern # Still fails — use nil + init -end -``` - -### Logger Backends Deprecated -```elixir -# OLD -config :logger, backends: [:console] - -# NEW — use LoggerHandler (Erlang's logger) -config :logger, :default_handler, [] -config :logger, :default_formatter, format: "$time $metadata[$level] $message\n" -``` - -### Mix Task Separator -```elixir -# OLD — comma separator -mix do compile, test - -# NEW (1.17+) — plus separator -mix do compile + test -``` - -### mix cli/0 Replaces Multiple Config Keys -```elixir -# OLD -def project do - [default_task: "phx.server", - preferred_cli_env: [test: :test], - preferred_cli_target: [...]] -end - -# NEW -def cli do - [default_task: "phx.server", - preferred_envs: [test: :test], - preferred_targets: [...]] -end -``` - ---- - -## Core Language Patterns - -### The Pipeline Principle -Elixir code flows through transformations via `|>`. Design functions to take the "subject" as the first argument. - -```elixir -orders -|> Enum.filter(&(&1.status == :pending)) -|> Enum.sort_by(& &1.created_at, DateTime) -|> Enum.map(&process_order/1) -``` - -### Pattern Matching — The Heart of Elixir -```elixir -# Function head matching — preferred over conditionals -def process(%Order{status: :pending} = order), do: ship(order) -def process(%Order{status: :shipped} = order), do: track(order) -def process(%Order{status: :delivered}), do: :noop - -# Pin operator to match against existing values -expected = "hello" -^expected = some_function() # Asserts equality - -# Map/struct matching is partial — only listed keys must match -%{name: name} = %{name: "Michael", age: 42} # name = "Michael" -``` - -### With Expressions for Happy-Path Chaining -```elixir -with {:ok, user} <- fetch_user(id), - {:ok, account} <- fetch_account(user), - {:ok, balance} <- check_balance(account) do - {:ok, balance} -else - {:error, :not_found} -> {:error, "User not found"} - {:error, :insufficient} -> {:error, "Insufficient funds"} - error -> {:error, "Unknown: #{inspect(error)}"} -end -``` - -### Structs and Protocols -```elixir -defmodule Money do - defstruct [:amount, :currency] - - defimpl String.Chars do - def to_string(%Money{amount: a, currency: c}), do: "#{a} #{c}" - end - - defimpl Inspect do - def inspect(%Money{amount: a, currency: c}, _opts) do - "#Money<#{a} #{c}>" - end - end -end -``` - -### Behaviours for Contracts -```elixir -defmodule PaymentProvider do - @callback charge(amount :: integer(), currency :: String.t()) :: - {:ok, transaction_id :: String.t()} | {:error, reason :: term()} - @callback refund(transaction_id :: String.t()) :: - {:ok, term()} | {:error, reason :: term()} -end - -defmodule Stripe do - @behaviour PaymentProvider - - @impl true - def charge(amount, currency), do: # ... - - @impl true - def refund(transaction_id), do: # ... -end -``` - ---- - -## Concurrency Model — BEAM Processes - -### Fundamentals -- **Processes are cheap** — ~2KB initial memory, microseconds to spawn, millions can run concurrently -- **No shared memory** — processes communicate via message passing only -- **Each process has its own heap** — GC is per-process, no stop-the-world pauses -- **Preemptive scheduling** — one scheduler per CPU core, ~4000 reductions then yield -- **Process isolation** — one process crashing doesn't affect others - -### Spawning and Messaging -```elixir -# Basic spawn + message passing -pid = spawn(fn -> - receive do - {:greet, name} -> IO.puts("Hello, #{name}!") - end -end) - -send(pid, {:greet, "Michael"}) - -# Linked processes — bidirectional crash propagation -pid = spawn_link(fn -> raise "boom" end) # Caller also crashes - -# Monitored processes — one-directional crash notification -ref = Process.monitor(pid) -receive do - {:DOWN, ^ref, :process, ^pid, reason} -> IO.puts("Crashed: #{reason}") -end -``` - -### Task — Structured Concurrency -```elixir -# Fire and forget -Task.start(fn -> send_email(user) end) - -# Async/await (linked to caller) -task = Task.async(fn -> expensive_computation() end) -result = Task.await(task, 30_000) # 30s timeout - -# Parallel map -Task.async_stream(urls, &fetch_url/1, max_concurrency: 10, timeout: 15_000) -|> Enum.to_list() -``` - -### GenServer — Stateful Server Processes -```elixir -defmodule Counter do - use GenServer - - # Client API - def start_link(initial \\ 0), do: GenServer.start_link(__MODULE__, initial, name: __MODULE__) - def increment, do: GenServer.cast(__MODULE__, :increment) - def get, do: GenServer.call(__MODULE__, :get) - - # Server callbacks - @impl true - def init(initial), do: {:ok, initial} - - @impl true - def handle_cast(:increment, count), do: {:noreply, count + 1} - - @impl true - def handle_call(:get, _from, count), do: {:reply, count, count} - - @impl true - def handle_info(:tick, count) do - IO.puts("Count: #{count}") - Process.send_after(self(), :tick, 1000) - {:noreply, count} - end -end -``` - -**Key design principle:** GenServer callbacks run sequentially in a single process — this is both the synchronization mechanism and the potential bottleneck. Keep callbacks fast; delegate heavy work to spawned tasks. - -### Supervisors — Let It Crash -```elixir -defmodule MyApp.Application do - use Application - - @impl true - def start(_type, _args) do - children = [ - # Order matters — started top to bottom, stopped bottom to top - MyApp.Repo, # Ecto repo - {MyApp.Cache, []}, # Custom GenServer - {Task.Supervisor, name: MyApp.TaskSupervisor}, - MyAppWeb.Endpoint # Phoenix endpoint last - ] - - opts = [strategy: :one_for_one, name: MyApp.Supervisor] - Supervisor.start_link(children, opts) - end -end -``` - -**Restart strategies:** -- `:one_for_one` — only restart the crashed child (most common) -- `:one_for_all` — restart all children if any crashes (tightly coupled) -- `:rest_for_one` — restart crashed child and all children started after it - -### DynamicSupervisor — Runtime Child Management -```elixir -defmodule MyApp.SessionSupervisor do - use DynamicSupervisor - - def start_link(_), do: DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__) - def init(:ok), do: DynamicSupervisor.init(strategy: :one_for_one) - - def start_session(session_id) do - spec = {MyApp.Session, session_id} - DynamicSupervisor.start_child(__MODULE__, spec) - end -end -``` - ---- - -## OTP Releases & Deployment - -### Building a Release -```bash -# Generate release config (Phoenix projects) -mix phx.gen.release - -# Build the release -MIX_ENV=prod mix release - -# The release is self-contained — no Erlang/Elixir needed on target -_build/prod/rel/my_app/bin/my_app start # Foreground -_build/prod/rel/my_app/bin/my_app daemon # Background -_build/prod/rel/my_app/bin/my_app remote # Attach IEx to running node -_build/prod/rel/my_app/bin/my_app eval "MyApp.Seeds.run()" # One-off command -``` - -### Release Configuration Files -- `config/config.exs` — **compile-time** config (before code compiles) -- `config/runtime.exs` — **runtime** config (executed on every boot) — use for env vars, secrets -- `rel/env.sh.eex` — shell environment for the release (VM flags, env vars) -- `rel/vm.args.eex` — Erlang VM flags (node name, cookie, memory limits) - -### Docker Multistage Build Pattern -```dockerfile -# === Builder Stage === -FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20250317 AS builder - -RUN apt-get update -y && apt-get install -y build-essential git && apt-get clean -WORKDIR /app - -ENV MIX_ENV=prod -RUN mix local.hex --force && mix local.rebar --force - -COPY mix.exs mix.lock ./ -RUN mix deps.get --only prod && mix deps.compile - -COPY config/config.exs config/prod.exs config/ -COPY priv priv -COPY lib lib -COPY assets assets # If Phoenix - -RUN mix assets.deploy # If Phoenix -RUN mix compile -RUN mix release - -# === Runner Stage === -FROM debian:bookworm-slim AS runner - -RUN apt-get update -y && \ - apt-get install -y libstdc++6 openssl libncurses5 locales ca-certificates && \ - apt-get clean && rm -f /var/lib/apt/lists/*_* - -ENV LANG=en_US.UTF-8 -RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen - -WORKDIR /app -RUN useradd --create-home app && chown -R app:app /app -USER app - -COPY --from=builder --chown=app:app /app/_build/prod/rel/my_app ./ - -CMD ["bin/my_app", "start"] -``` - -### Distributed Erlang in Production -```elixir -# In rel/env.sh.eex — set node name and cookie -export RELEASE_DISTRIBUTION=name -export RELEASE_NODE=my_app@${HOSTNAME} -export RELEASE_COOKIE=my_secret_cookie - -# Or in runtime.exs -config :my_app, MyApp.Cluster, - strategy: Cluster.Strategy.DNSPoll, - config: [ - polling_interval: 5_000, - query: "my-app.local", - node_basename: "my_app" - ] -``` - -**Key libraries for clustering:** `libcluster` (automatic node discovery), `Horde` (distributed supervisor/registry), `Phoenix.PubSub` (cross-node pub/sub already built into Phoenix). - ---- - -## Monitoring & Instrumentation - -### The Elixir Observability Stack -- **PromEx** — Elixir library exposing `/metrics` endpoint for Prometheus scraping. Includes plugins for Phoenix, Ecto, LiveView, BEAM VM, Oban -- **Prometheus** — scrapes and stores time-series metrics -- **Loki** + **Promtail** — log aggregation (Promtail scrapes Docker container logs → Loki stores) -- **Grafana** — visualization dashboards for both metrics and logs -- **Alloy** — OpenTelemetry collector bridging metrics to Prometheus - -### Adding PromEx to a Phoenix App -```elixir -# mix.exs -{:prom_ex, "~> 1.9"} - -# lib/my_app/prom_ex.ex -defmodule MyApp.PromEx do - use PromEx, otp_app: :my_app - - @impl true - def plugins do - [ - PromEx.Plugins.Application, - PromEx.Plugins.Beam, - {PromEx.Plugins.Phoenix, router: MyAppWeb.Router}, - {PromEx.Plugins.Ecto, repos: [MyApp.Repo]}, - PromEx.Plugins.Oban - ] - end - - @impl true - def dashboards do - [{:prom_ex, "application.json"}, {:prom_ex, "beam.json"}, - {:prom_ex, "phoenix.json"}, {:prom_ex, "ecto.json"}] - end -end -``` - -### Key BEAM Metrics to Watch -- `erlang_vm_process_count` — total BEAM processes (normal: thousands, alarm at millions) -- `erlang_vm_memory_bytes_total` — total VM memory -- `erlang_vm_atom_count` — atoms are never GC'd; watch for leaks -- `phoenix_endpoint_stop_duration_milliseconds` — request latency -- `ecto_repo_query_duration_milliseconds` — database query time - ---- - -## usage_rules — AI Agent Documentation - -**Always include in every project.** This is non-negotiable. - -```elixir -# mix.exs deps -{:usage_rules, "~> 1.1", only: [:dev]} - -# mix.exs project config -def project do - [ - # ... other config - usage_rules: [ - packages: :all, # Or specific: ["phoenix", "ecto", ~r/ash/] - output: :agents_md, # Generates AGENTS.md - mode: :linked # Or :inlined for single-file - ] - ] -end -``` - -**Key commands:** -```bash -mix usage_rules.gen # Generate AGENTS.md from deps -mix usage_rules.search_docs # Search hex documentation -mix usage_rules.gen_skill # Generate SKILL.md for Cowork -``` - -This consolidates `usage-rules.md` files from all dependencies into a single reference document, giving any AI agent working on the project full context about library APIs, patterns, and conventions. - ---- - -## Ash Framework — Production Backend - -Ash is a **declarative, resource-oriented** framework for building Elixir backends. Use it for substantial projects — it handles data layer, authorization, validation, and API generation. - -### Core Concepts -- **Resources** — the central abstraction (like models but richer): attributes, actions, relationships, calculations, aggregates, policies -- **Domains** — organizational containers that group related resources and define the public API -- **Actions** — CRUD + custom actions defined declaratively on resources -- **Data Layers** — pluggable persistence (AshPostgres, AshSqlite, ETS for dev) -- **Policies** — declarative authorization rules on resources/actions -- **Extensions** — AshPhoenix, AshGraphql, AshJsonApi, AshAuthentication, AshOban - -### Quick Start -```bash -mix igniter.new my_app --install ash,ash_postgres,ash_phoenix -``` - -### Resource Example -```elixir -defmodule MyApp.Blog.Post do - use Ash.Resource, - domain: MyApp.Blog, - data_layer: AshPostgres.DataLayer - - postgres do - table "posts" - repo MyApp.Repo - end - - attributes do - uuid_primary_key :id - attribute :title, :string, allow_nil?: false - attribute :body, :string, allow_nil?: false - attribute :status, :atom, constraints: [one_of: [:draft, :published]], default: :draft - timestamps() - end - - actions do - defaults [:read, :destroy] - - create :create do - accept [:title, :body] - end - - update :publish do - change set_attribute(:status, :published) - change set_attribute(:published_at, &DateTime.utc_now/0) - end - end - - relationships do - belongs_to :author, MyApp.Accounts.User - has_many :comments, MyApp.Blog.Comment - end - - policies do - policy action_type(:create) do - authorize_if actor_attribute_equals(:role, :author) - end - policy action_type(:read) do - authorize_if always() - end - end -end -``` - -### Domain Example -```elixir -defmodule MyApp.Blog do - use Ash.Domain - - resources do - resource MyApp.Blog.Post - resource MyApp.Blog.Comment - end -end -``` - -### Using Resources -```elixir -# Create -MyApp.Blog.Post -|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"}) -|> Ash.create!() - -# Read with filters -MyApp.Blog.Post -|> Ash.Query.filter(status == :published) -|> Ash.Query.sort(inserted_at: :desc) -|> Ash.Query.limit(10) -|> Ash.read!() - -# Custom action -post |> Ash.Changeset.for_update(:publish) |> Ash.update!() -``` - ---- - -## Mix Project Structure - -``` -my_app/ -├── config/ -│ ├── config.exs # Compile-time config -│ ├── dev.exs -│ ├── prod.exs -│ ├── runtime.exs # Runtime config (secrets, env vars) -│ └── test.exs -├── lib/ -│ ├── my_app/ -│ │ ├── application.ex # OTP Application + supervisor tree -│ │ ├── repo.ex # Ecto Repo -│ │ └── ... # Domain modules -│ └── my_app_web/ # Phoenix web layer (if applicable) -│ ├── endpoint.ex -│ ├── router.ex -│ ├── controllers/ -│ ├── live/ # LiveView modules -│ └── components/ -├── priv/ -│ ├── repo/migrations/ -│ └── static/ -├── test/ -│ ├── support/ -│ └── ... -├── mix.exs -├── mix.lock -├── .formatter.exs -└── AGENTS.md # Generated by usage_rules -``` - ---- - -## Testing - -```elixir -# test/my_app/blog_test.exs -defmodule MyApp.BlogTest do - use MyApp.DataCase, async: true # Async when no shared state - - describe "creating posts" do - test "creates with valid attributes" do - assert {:ok, post} = Blog.create_post(%{title: "Hi", body: "World"}) - assert post.title == "Hi" - assert post.status == :draft - end - - test "fails without title" do - assert {:error, changeset} = Blog.create_post(%{body: "World"}) - assert "can't be blank" in errors_on(changeset).title - end - end -end -``` - -**Testing philosophy:** Use `async: true` wherever possible. Ecto's SQL Sandbox allows concurrent test execution. For GenServer tests, start under a test supervisor. For external services, use `Mox` for behaviour-based mocking. - ---- - -## Phoenix Framework — v1.8.5 (Current) - -Phoenix is the web framework for Elixir. Version 1.8.5 is current. It provides MVC controllers, real-time LiveView, Channels for WebSocket communication, and comprehensive tooling for authentication, testing, and deployment. - -### What's New in Phoenix 1.8 - -- **Scopes** in generators — secure data access by default (e.g., `current_user` automatically applied) -- **Magic links** (passwordless auth) and **"sudo mode"** in `mix phx.gen.auth` -- **daisyUI** integration — light/dark/system mode support out of the box -- **Simplified layouts** — single `root.html.heex` wraps everything; dynamic layouts are function components -- **`use Phoenix.Controller` now requires `:formats`** — must specify `formats: [:html]` or `formats: [:html, :json]` -- **Updated security headers** — `content-security-policy` with `base-uri 'self'; frame-ancestors 'self'`; dropped deprecated `x-frame-options` and `x-download-options` -- **`config` variable removed** from `Phoenix.Endpoint` — use `Application.compile_env/3` instead -- **Deprecated**: `:namespace`, `:put_default_views`, layouts without modules, `:trailing_slash` in router - -### Project Setup - -```bash -# New project (Phoenix Express — quickest path) -curl https://new.phoenixframework.org/my_app | sh - -# Traditional setup -mix phx.new my_app -mix phx.new my_app --no-ecto # Without database -mix phx.new my_app --no-html # API only -mix phx.new my_app --database sqlite3 # SQLite instead of Postgres -``` - -### Directory Structure - -``` -my_app/ -├── lib/ -│ ├── my_app/ # Business logic (contexts, schemas) -│ │ ├── application.ex # Supervision tree -│ │ ├── repo.ex # Ecto Repo -│ │ └── catalog/ # Context: Catalog -│ │ ├── product.ex # Schema -│ │ └── catalog.ex # Context module -│ └── my_app_web/ # Web layer -│ ├── endpoint.ex # HTTP entry point -│ ├── router.ex # Routes + pipelines -│ ├── components/ -│ │ ├── core_components.ex # Shared UI components -│ │ └── layouts.ex # Layout components -│ ├── controllers/ -│ │ ├── page_controller.ex -│ │ └── page_html/ # Templates for controller -│ │ └── home.html.heex -│ └── live/ # LiveView modules -│ └── counter_live.ex -├── config/ -│ ├── config.exs # Compile-time config -│ ├── dev.exs / prod.exs -│ └── runtime.exs # Runtime config (env vars, secrets) -├── priv/ -│ ├── repo/migrations/ -│ └── static/ # Static assets -├── test/ -│ ├── support/ -│ │ ├── conn_case.ex # Test helpers for controllers -│ │ └── data_case.ex # Test helpers for data layer -│ └── my_app_web/ -└── mix.exs -``` - -### Router & Pipelines - -```elixir -defmodule MyAppWeb.Router do - use MyAppWeb, :router - - pipeline :browser do - plug :accepts, ["html"] - plug :fetch_session - plug :fetch_live_flash - plug :put_root_layout, html: {MyAppWeb.Layouts, :root} - plug :protect_from_forgery - plug :put_secure_browser_headers - end - - pipeline :api do - plug :accepts, ["json"] - end - - scope "/", MyAppWeb do - pipe_through :browser - - get "/", PageController, :home - live "/counter", CounterLive # LiveView route - resources "/products", ProductController # RESTful CRUD - end - - scope "/api", MyAppWeb do - pipe_through :api - - resources "/items", ItemController, except: [:new, :edit] - end -end -``` - -### Verified Routes — `~p` Sigil - -**Always use `~p` instead of string paths.** Compile-time verified against your router. - -```elixir -# In templates (HEEx) -~H""" -View -Edit -""" - -# In controllers -redirect(conn, to: ~p"/products/#{product}") - -# With query params -~p"/products?page=#{page}&sort=name" -~p"/products?#{%{page: 1, sort: "name"}}" - -# URL generation (includes host) -url(~p"/products/#{product}") -# => "https://example.com/products/42" -``` - -### Controllers (1.8 Style) - -```elixir -defmodule MyAppWeb.ProductController do - use MyAppWeb, :controller - - # REQUIRED in 1.8: specify formats - # This is typically set in MyAppWeb :controller function - - def index(conn, _params) do - products = Catalog.list_products() - render(conn, :index, products: products) - end - - def show(conn, %{"id" => id}) do - product = Catalog.get_product!(id) - render(conn, :show, product: product) - end - - def create(conn, %{"product" => product_params}) do - case Catalog.create_product(product_params) do - {:ok, product} -> - conn - |> put_flash(:info, "Product created.") - |> redirect(to: ~p"/products/#{product}") - - {:error, %Ecto.Changeset{} = changeset} -> - render(conn, :new, changeset: changeset) - end - end -end -``` - -**View module naming:** For `ProductController`, Phoenix looks for `ProductHTML` (for `:html` format) and `ProductJSON` (for `:json` format). - -```elixir -# lib/my_app_web/controllers/product_html.ex -defmodule MyAppWeb.ProductHTML do - use MyAppWeb, :html - embed_templates "product_html/*" # Loads .heex files from directory -end - -# lib/my_app_web/controllers/product_json.ex -defmodule MyAppWeb.ProductJSON do - def index(%{products: products}), do: %{data: for(p <- products, do: data(p))} - def show(%{product: product}), do: %{data: data(product)} - defp data(product), do: %{id: product.id, title: product.title, price: product.price} -end -``` - -### Components and HEEx - -```elixir -defmodule MyAppWeb.CoreComponents do - use Phoenix.Component - - # Declare attributes with types and docs - attr :type, :string, default: "button" - attr :class, :string, default: nil - attr :rest, :global # Passes through all other HTML attrs - slot :inner_block, required: true - - def button(assigns) do - ~H""" - - """ - end - - # Table component with slots - attr :rows, :list, required: true - slot :col, required: true do - attr :label, :string - end - - def table(assigns) do - ~H""" - - - - - - - - - - - -
{col[:label]}
{render_slot(col, row)}
- """ - end -end -``` - -**HEEx syntax notes:** -- `{@var}` — render assign (curly braces, not `<%= %>`) -- `:if={condition}` — conditional rendering on any tag -- `:for={item <- list}` — iteration on any tag -- `<.component_name />` — call function component with dot notation -- `` — call remote component with module name -- `{render_slot(@inner_block)}` — render slot content -- `<:slot_name>content` — named slot content - -### LiveView - -```elixir -defmodule MyAppWeb.SearchLive do - use MyAppWeb, :live_view - - def mount(_params, _session, socket) do - {:ok, assign(socket, query: "", results: [])} - end - - def handle_params(%{"q" => query}, _uri, socket) do - {:noreply, assign(socket, query: query, results: search(query))} - end - - def handle_params(_params, _uri, socket), do: {:noreply, socket} - - def handle_event("search", %{"query" => query}, socket) do - {:noreply, - socket - |> assign(query: query, results: search(query)) - |> push_patch(to: ~p"/search?q=#{query}")} - end - - def render(assigns) do - ~H""" - -
- - -
- -
-

{result.title}

-

{result.summary}

-
-
- """ - end - - defp search(query), do: MyApp.Search.find(query) -end -``` - -**LiveView lifecycle:** `mount/3` → `handle_params/3` → `render/1`. Events via `handle_event/3`. Server pushes via `handle_info/2`. - -**Key patterns:** -- `assign/2,3` — set socket assigns -- `push_navigate/2` — navigate to new LiveView (full mount) -- `push_patch/2` — update URL without full remount (calls `handle_params`) -- `push_event/3` — push event to client JS hooks -- `stream/3,4` — efficient list rendering for large collections (inserts/deletes without re-rendering entire list) -- `async_assign/3` + `assign_async/3` — async data loading with loading/error states - -### Channels — Real-Time WebSocket - -```elixir -# In endpoint.ex -socket "/socket", MyAppWeb.UserSocket, - websocket: true, - longpoll: false - -# lib/my_app_web/channels/user_socket.ex -defmodule MyAppWeb.UserSocket do - use Phoenix.Socket - - channel "room:*", MyAppWeb.RoomChannel - - def connect(%{"token" => token}, socket, _connect_info) do - case Phoenix.Token.verify(socket, "user auth", token, max_age: 86400) do - {:ok, user_id} -> {:ok, assign(socket, :user_id, user_id)} - {:error, _} -> :error - end - end - - def id(socket), do: "users_socket:#{socket.assigns.user_id}" -end - -# lib/my_app_web/channels/room_channel.ex -defmodule MyAppWeb.RoomChannel do - use MyAppWeb, :channel - - def join("room:" <> room_id, _params, socket) do - {:ok, assign(socket, :room_id, room_id)} - end - - def handle_in("new_msg", %{"body" => body}, socket) do - broadcast!(socket, "new_msg", %{ - body: body, - user_id: socket.assigns.user_id - }) - {:noreply, socket} - end -end -``` - -**Force disconnect all sessions for a user:** -```elixir -MyAppWeb.Endpoint.broadcast("users_socket:#{user.id}", "disconnect", %{}) -``` - -### Contexts — Business Logic Boundary - -Contexts are plain Elixir modules that encapsulate data access and business rules. They are the API between your web layer and your domain. - -```elixir -# Generate with: mix phx.gen.context Catalog Product products title:string price:decimal -defmodule MyApp.Catalog do - import Ecto.Query - alias MyApp.Repo - alias MyApp.Catalog.Product - - def list_products do - Repo.all(Product) - end - - def get_product!(id), do: Repo.get!(Product, id) - - def create_product(attrs \\ %{}) do - %Product{} - |> Product.changeset(attrs) - |> Repo.insert() - end - - def update_product(%Product{} = product, attrs) do - product - |> Product.changeset(attrs) - |> Repo.update() - end - - def delete_product(%Product{} = product) do - Repo.delete(product) - end - - def change_product(%Product{} = product, attrs \\ %{}) do - Product.changeset(product, attrs) - end -end -``` - -**Context design principles:** -- One context per bounded domain (Catalog, Accounts, Orders) -- Contexts own their schemas — other contexts reference by ID, not struct -- Cross-context calls go through the public context API, never access another context's Repo directly -- Contexts can nest related schemas (Comments under Posts) - -### Authentication — `mix phx.gen.auth` - -```bash -mix phx.gen.auth Accounts User users -``` - -Phoenix 1.8 generates: -- **Magic links** (passwordless) — email-based login links -- **"Sudo mode"** — re-authentication for sensitive actions -- Session-based auth with secure token handling -- Email confirmation and password reset flows -- `require_authenticated_user` plug for protected routes - -### Scopes (New in 1.8) - -Scopes make secure data access the default in generators. When you generate resources with a scope, all queries are automatically filtered by the scoped user. - -```bash -mix phx.gen.live Posts Post posts title body:text --scope current_user -``` - -This generates code that automatically passes `current_user` to context functions, ensuring users only see their own data. - -### Ecto — Database Layer - -```elixir -# Schema -defmodule MyApp.Catalog.Product do - use Ecto.Schema - import Ecto.Changeset - - schema "products" do - field :title, :string - field :price, :decimal - field :status, Ecto.Enum, values: [:draft, :published, :archived] - has_many :reviews, MyApp.Reviews.Review - belongs_to :category, MyApp.Catalog.Category - timestamps(type: :utc_datetime) - end - - def changeset(product, attrs) do - product - |> cast(attrs, [:title, :price, :status, :category_id]) - |> validate_required([:title, :price]) - |> validate_number(:price, greater_than: 0) - |> unique_constraint(:title) - |> foreign_key_constraint(:category_id) - end -end - -# Queries -import Ecto.Query - -# Composable queries -def published(query \\ Product) do - from p in query, where: p.status == :published -end - -def recent(query, days \\ 7) do - from p in query, where: p.inserted_at > ago(^days, "day") -end - -def with_reviews(query) do - from p in query, preload: [:reviews] -end - -# Usage: Product |> published() |> recent(30) |> with_reviews() |> Repo.all() -``` - -### Telemetry — Built-in Observability - -Phoenix 1.8 includes a Telemetry supervisor that tracks request duration, Ecto query times, and VM metrics out of the box. - -```elixir -# lib/my_app_web/telemetry.ex (auto-generated) -defmodule MyAppWeb.Telemetry do - use Supervisor - import Telemetry.Metrics - - def metrics do - [ - summary("phoenix.endpoint.stop.duration", unit: {:native, :millisecond}), - summary("phoenix.router_dispatch.stop.duration", tags: [:route], unit: {:native, :millisecond}), - summary("my_app.repo.query.total_time", unit: {:native, :millisecond}), - summary("vm.memory.total", unit: {:byte, :kilobyte}), - summary("vm.total_run_queue_lengths.total"), - summary("vm.total_run_queue_lengths.cpu"), - ] - end -end -``` - -Integrates with **PromEx** for Prometheus/Grafana dashboards (see Monitoring section). - -### Phoenix Security Best Practices - -**Never pass untrusted input to:** `Code.eval_string/3`, `:os.cmd/2`, `System.cmd/3`, `System.shell/2`, `:erlang.binary_to_term/2` - -**Ecto prevents SQL injection by default** — the query DSL parameterizes all inputs. Only `Ecto.Adapters.SQL.query/4` with raw string interpolation is vulnerable. - -**Safe deserialization:** -```elixir -# UNSAFE — even with :safe flag -:erlang.binary_to_term(user_input, [:safe]) - -# SAFE — prevents executable terms -Plug.Crypto.non_executable_binary_to_term(user_input, [:safe]) -``` - -**CSRF protection** is built into the `:browser` pipeline via `protect_from_forgery`. **Content Security Policy** is set by `put_secure_browser_headers`. - -### Testing Phoenix - -```elixir -# Controller test -defmodule MyAppWeb.ProductControllerTest do - use MyAppWeb.ConnCase - - test "GET /products", %{conn: conn} do - conn = get(conn, ~p"/products") - assert html_response(conn, 200) =~ "Products" - end - - test "POST /products creates product", %{conn: conn} do - conn = post(conn, ~p"/products", product: %{title: "Widget", price: 9.99}) - assert redirected_to(conn) =~ ~p"/products/" - end -end - -# LiveView test -defmodule MyAppWeb.CounterLiveTest do - use MyAppWeb.ConnCase - import Phoenix.LiveViewTest - - test "increments counter", %{conn: conn} do - {:ok, view, html} = live(conn, ~p"/counter") - assert html =~ "Count: 0" - - assert view - |> element("button", "+1") - |> render_click() =~ "Count: 1" - end -end - -# Channel test -defmodule MyAppWeb.RoomChannelTest do - use MyAppWeb.ChannelCase - - test "broadcasts new messages" do - {:ok, _, socket} = subscribe_and_join(socket(MyAppWeb.UserSocket), MyAppWeb.RoomChannel, "room:lobby") - - push(socket, "new_msg", %{"body" => "hello"}) - assert_broadcast "new_msg", %{body: "hello"} - end -end -``` - -### Phoenix Generators Cheat Sheet - -```bash -# HTML CRUD (controller + views + templates + context + schema + migration) -mix phx.gen.html Catalog Product products title:string price:decimal - -# LiveView CRUD -mix phx.gen.live Catalog Product products title:string price:decimal - -# JSON API -mix phx.gen.json Catalog Product products title:string price:decimal - -# Context + schema only (no web layer) -mix phx.gen.context Catalog Product products title:string price:decimal - -# Schema + migration only -mix phx.gen.schema Product products title:string price:decimal - -# Authentication -mix phx.gen.auth Accounts User users - -# Channel -mix phx.gen.channel Room - -# Presence -mix phx.gen.presence - -# Release files (Dockerfile, release.ex, overlay scripts) -mix phx.gen.release -mix phx.gen.release --docker # Include Dockerfile -``` - -### Phoenix Release & Deployment - -```bash -# Generate release infrastructure -mix phx.gen.release --docker - -# Build release -MIX_ENV=prod mix deps.get --only prod -MIX_ENV=prod mix compile -MIX_ENV=prod mix assets.deploy -MIX_ENV=prod mix release - -# Release commands -_build/prod/rel/my_app/bin/server # Start with Phoenix server -_build/prod/rel/my_app/bin/migrate # Run migrations -_build/prod/rel/my_app/bin/my_app remote # Attach IEx console -``` - -**Runtime config (`config/runtime.exs`):** -```elixir -import Config - -if config_env() == :prod do - database_url = System.fetch_env!("DATABASE_URL") - secret_key_base = System.fetch_env!("SECRET_KEY_BASE") - - config :my_app, MyApp.Repo, - url: database_url, - pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10") - - config :my_app, MyAppWeb.Endpoint, - url: [host: System.fetch_env!("PHX_HOST"), port: 443, scheme: "https"], - http: [ip: {0, 0, 0, 0}, port: String.to_integer(System.get_env("PORT") || "4000")], - secret_key_base: secret_key_base -end -``` - ---- - -## Common Libraries +## Common Libraries Quick Reference | Library | Purpose | Hex | |---------|---------|-----| | Phoenix | Web framework + LiveView | `{:phoenix, "~> 1.8"}` | | Ecto | Database wrapper + query DSL | `{:ecto_sql, "~> 3.12"}` | | Ash | Declarative resource framework | `{:ash, "~> 3.0"}` | +| Jido | Multi-agent orchestration | `{:jido, "~> 2.1"}` | +| GenStateMachine | State machines (wraps gen_statem) | `{:gen_state_machine, "~> 3.0"}` | | Oban | Background job processing | `{:oban, "~> 2.18"}` | -| Req | HTTP client (modern, composable) | `{:req, "~> 0.5"}` | +| Req | HTTP client (modern) | `{:req, "~> 0.5"}` | | Jason | JSON encoding/decoding | `{:jason, "~> 1.4"}` | | Swoosh | Email sending | `{:swoosh, "~> 1.16"}` | -| ExUnit | Testing (built-in) | — | | Mox | Mock behaviours for testing | `{:mox, "~> 1.1", only: :test}` | | Credo | Static analysis / linting | `{:credo, "~> 1.7", only: [:dev, :test]}` | -| Dialyxir | Static typing via Dialyzer | `{:dialyxir, "~> 1.4", only: [:dev, :test]}` | | libcluster | Automatic BEAM node clustering | `{:libcluster, "~> 3.4"}` | | Horde | Distributed supervisor/registry | `{:horde, "~> 0.9"}` | | Nx | Numerical computing / tensors | `{:nx, "~> 0.9"}` | -| Bumblebee | Pre-trained ML models on BEAM | `{:bumblebee, "~> 0.6"}` | | Broadway | Data ingestion pipelines | `{:broadway, "~> 1.1"}` | | PromEx | Prometheus metrics for Elixir | `{:prom_ex, "~> 1.9"}` | -| Finch | HTTP client (low-level, pooled) | `{:finch, "~> 0.19"}` | | usage_rules | AI agent docs from deps | `{:usage_rules, "~> 1.1", only: :dev}` | - ---- - -## Interop — Elixir as Orchestrator - -### Python via Ports/Erlport -```elixir -# Using erlport for bidirectional Python calls -{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}]) -result = :python.call(pid, :my_module, :my_function, [arg1, arg2]) -:python.stop(pid) -``` - -### Rust via Rustler NIFs -```elixir -# mix.exs -{:rustler, "~> 0.34"} - -# lib/my_nif.ex -defmodule MyApp.NativeSort do - use Rustler, otp_app: :my_app, crate: "native_sort" - - # NIF stubs — replaced at load time by Rust implementations - def sort(_list), do: :erlang.nif_error(:nif_not_loaded) -end -``` - -### System Commands via Ports -```elixir -# One-shot command -{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output]) - -# Long-running port -port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status]) -send(port, {self(), {:command, "process\n"}}) -receive do - {^port, {:data, data}} -> handle_response(data) -end -``` - ---- - -## Cortex Deployment Story - -### Current State -- Cortex runs Ubuntu 24.04 with Caddy as web server -- Elixir is **not yet installed** — needs `asdf` or direct install -- Docker is **planned but not installed** — needed for containerized Elixir apps -- Symbiont (Python) is the current orchestrator — Elixir will gradually take over - -### Installation Plan for Cortex -```bash -# Option 1: asdf (recommended — manages multiple versions) -git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0 -echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc -asdf plugin add erlang -asdf plugin add elixir -asdf install erlang 27.3 -asdf install elixir 1.19.5-otp-27 -asdf global erlang 27.3 -asdf global elixir 1.19.5-otp-27 - -# Option 2: Direct from Erlang Solutions -wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb -dpkg -i erlang-solutions_2.0_all.deb -apt-get update && apt-get install esl-erlang elixir -``` - -### BEAMOps Principles for Cortex -From "Engineering Elixir Applications" — the deployment philosophy: - -1. **Environment Integrity** — identical builds dev/staging/prod via Docker + releases -2. **Infrastructure as Code** — Caddy config, systemd units, backup scripts all version-controlled -3. **OTP Releases** — self-contained, no runtime deps, `bin/my_app start` -4. **Distributed Erlang** — nodes discover each other, share state via PubSub, global registry -5. **Instrumentation** — PromEx + Prometheus + Grafana + Loki for full observability -6. **Health Checks + Rollbacks** — Docker health checks trigger automatic rollback on failed deploys -7. **Zero-downtime deploys** — rolling updates via Docker Swarm or `mix release` hot upgrades - -### CI Pipeline for Elixir (GitHub Actions) -```yaml -name: Elixir CI -on: [push, pull_request] - -jobs: - test: - runs-on: ubuntu-latest - services: - db: - image: postgres:16 - env: - POSTGRES_PASSWORD: postgres - ports: ['5432:5432'] - steps: - - uses: actions/checkout@v4 - - uses: erlef/setup-beam@v1 - with: - elixir-version: '1.19.5' - otp-version: '27.3' - - run: mix deps.get - - run: mix compile --warnings-as-errors - - run: mix format --check-formatted - - run: mix credo --strict - - run: mix test - - run: mix deps.unlock --check-unused -``` - ---- - -## Anti-Patterns to Avoid - -### Process Anti-Patterns -- **GenServer as code organization** — don't wrap pure functions in a GenServer. Use modules. -- **Agent for complex state** — if you need more than get/update, use GenServer directly. -- **Spawning unsupervised processes** — always use `Task.Supervisor` or link to a supervisor. - -### Code Anti-Patterns -- **Primitive obsession** — use structs, not bare maps, for domain concepts. -- **Boolean parameters** — use atoms or keyword options: `format: :json` not `json: true`. -- **Large modules** — split by concern, not by entity type. Domain logic, web layer, workers. -- **String keys in internal maps** — use atoms internally, strings only at boundaries (JSON, forms). - -### Design Anti-Patterns -- **Monolithic contexts** — Phoenix contexts should be small, focused. Split `Accounts` from `Authentication`. -- **God GenServer** — one process handling all state for the app. Distribute responsibility. -- **Synchronous calls to slow services** — use `Task.async` + `Task.await` with timeouts. - ---- - -## Quick Recipes - -### HTTP Request with Req -```elixir -Req.get!("https://api.example.com/data", - headers: [{"authorization", "Bearer #{token}"}], - receive_timeout: 15_000 -).body -``` - -### JSON Encode/Decode -```elixir -Jason.encode!(%{name: "Michael", role: :admin}) -Jason.decode!(~s({"name": "Michael"}), keys: :atoms) -``` - -### Ecto Query -```elixir -from p in Post, - where: p.status == :published, - where: p.inserted_at > ago(7, "day"), - order_by: [desc: p.inserted_at], - limit: 10, - preload: [:author, :comments] -``` - -### Background Job with Oban -```elixir -defmodule MyApp.Workers.EmailWorker do - use Oban.Worker, queue: :mailers, max_attempts: 3 - - @impl true - def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do - MyApp.Mailer.send(to, template) - end -end - -# Enqueue -%{to: "user@example.com", template: "welcome"} -|> MyApp.Workers.EmailWorker.new(schedule_in: 60) -|> Oban.insert!() -``` - -### LiveView Component -```elixir -defmodule MyAppWeb.CounterLive do - use MyAppWeb, :live_view - - def mount(_params, _session, socket) do - {:ok, assign(socket, count: 0)} - end - - def handle_event("increment", _, socket) do - {:noreply, update(socket, :count, &(&1 + 1))} - end - - def render(assigns) do - ~H""" -
-

Count: {@count}

- -
- """ - end -end -``` - ---- - -## Resources - -- [Elixir Official Docs](https://hexdocs.pm/elixir/) — always check 1.19.5 version -- [Ash Framework Docs](https://hexdocs.pm/ash/) — resource-oriented patterns -- [Phoenix HexDocs](https://hexdocs.pm/phoenix/) — web framework -- [Elixir Forum](https://elixirforum.com/) — community Q&A -- [Elixir School](https://elixirschool.com/) — learning resource -- "Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes above) -- "Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns diff --git a/elixir/elixir-part1-core.md b/elixir/elixir-part1-core.md new file mode 100644 index 0000000..219d04c --- /dev/null +++ b/elixir/elixir-part1-core.md @@ -0,0 +1,208 @@ +# Elixir Part 1: Core Language & Modern Idioms + +## Elixir 1.19 — What's New + +### Gradual Set-Theoretic Type System + +Elixir 1.19 significantly advances the built-in type system. It is **sound**, **gradual**, and **set-theoretic** (types compose via union, intersection, negation). + +**Current capabilities (1.19):** +- Type inference from existing code — no annotations required yet +- **Protocol dispatch type checking** — warns on invalid protocol usage (e.g., string interpolation on structs without `String.Chars`) +- **Anonymous function type inference** — `fn` literals and `&captures` propagate types +- Types: `atom()`, `binary()`, `integer()`, `float()`, `pid()`, `port()`, `reference()`, `tuple()`, `list()`, `map()`, `function()` +- Compose: `atom() or integer()` (union), `atom() and not nil` (difference) +- `dynamic()` type for gradual typing — runtime-checked values +- Tuple precision: `{:ok, binary()}`, open tuples: `{:ok, binary(), ...}` +- Map types: `%{key: value}` closed, `%{optional(atom()) => term()}` open +- Function types: `(integer() -> boolean())` + +### Up to 4x Faster Compilation + +1. **Lazy module loading** — modules no longer loaded when defined; parallel compiler controls loading +2. **Parallel dependency compilation** — set `MIX_OS_DEPS_COMPILE_PARTITION_COUNT` env var + +**Caveat:** If spawning processes during compilation that invoke sibling modules, use `Code.ensure_compiled!/1` first. + +### Other 1.19 Features + +- `min/2` and `max/2` allowed in guards +- `Access.values/0` — traverse all values in a map/keyword +- `String.count/2` — count occurrences +- Unicode 17.0.0, multi-line IEx prompts +- `mix help Mod.fun/arity` +- Erlang/OTP 28 support + +--- + +## Breaking Changes Since 1.15 + +### Struct Update Syntax (1.18+) +```elixir +# The compiler now verifies the variable matches the struct type +%URI{my_uri | path: "/new"} # my_uri must be verified as %URI{} +``` + +### Regex as Struct Field Defaults (OTP 28) +```elixir +# BROKEN on OTP 28 — regex can't be struct field defaults +defstruct pattern: ~r/foo/ # Compile error +# FIX: use nil default, set at runtime +``` + +### Logger Backends Deprecated +```elixir +# OLD: config :logger, backends: [:console] +# NEW: use LoggerHandler (Erlang's logger) +config :logger, :default_handler, [] +``` + +### Mix Task Separator +```elixir +# OLD: mix do compile, test +# NEW: mix do compile + test +``` + +### mix cli/0 Replaces Config Keys +```elixir +# OLD: default_task, preferred_cli_env in project/0 +# NEW: def cli, do: [default_task: "phx.server", preferred_envs: [test: :test]] +``` + +--- + +## Core Language Patterns + +### The Pipeline +```elixir +orders +|> Enum.filter(&(&1.status == :pending)) +|> Enum.sort_by(& &1.created_at, DateTime) +|> Enum.map(&process_order/1) +``` + +### Pattern Matching +```elixir +# Function head matching — preferred over conditionals +def process(%Order{status: :pending} = order), do: ship(order) +def process(%Order{status: :shipped} = order), do: track(order) +def process(%Order{status: :delivered}), do: :noop + +# Pin operator +expected = "hello" +^expected = some_function() + +# Partial map matching +%{name: name} = %{name: "Michael", age: 42} +``` + +### With Expressions +```elixir +with {:ok, user} <- fetch_user(id), + {:ok, account} <- fetch_account(user), + {:ok, balance} <- check_balance(account) do + {:ok, balance} +else + {:error, :not_found} -> {:error, "User not found"} + error -> {:error, "Unknown: #{inspect(error)}"} +end +``` + +### Structs and Protocols +```elixir +defmodule Money do + defstruct [:amount, :currency] + + defimpl String.Chars do + def to_string(%Money{amount: a, currency: c}), do: "#{a} #{c}" + end +end +``` + +### Behaviours +```elixir +defmodule PaymentProvider do + @callback charge(integer(), String.t()) :: {:ok, String.t()} | {:error, term()} + @callback refund(String.t()) :: {:ok, term()} | {:error, term()} +end + +defmodule Stripe do + @behaviour PaymentProvider + @impl true + def charge(amount, currency), do: # ... + @impl true + def refund(transaction_id), do: # ... +end +``` + +--- + +## Mix Project Structure + +``` +my_app/ +├── config/ +│ ├── config.exs # Compile-time config +│ ├── dev.exs / prod.exs / test.exs +│ └── runtime.exs # Runtime config (secrets, env vars) +├── lib/ +│ ├── my_app/ +│ │ ├── application.ex # OTP Application + supervisor tree +│ │ └── ... # Domain modules +│ └── my_app_web/ # Phoenix web layer (if applicable) +├── priv/repo/migrations/ +├── test/ +├── mix.exs +├── .formatter.exs +└── AGENTS.md # Generated by usage_rules +``` + +--- + +## Testing + +```elixir +defmodule MyApp.BlogTest do + use MyApp.DataCase, async: true + + describe "creating posts" do + test "creates with valid attributes" do + assert {:ok, post} = Blog.create_post(%{title: "Hi", body: "World"}) + assert post.title == "Hi" + end + + test "fails without title" do + assert {:error, changeset} = Blog.create_post(%{body: "World"}) + assert "can't be blank" in errors_on(changeset).title + end + end +end +``` + +Use `async: true` wherever possible. Use `Mox` for behaviour-based mocking. + +--- + +## usage_rules — AI Agent Documentation + +**Always include in every project.** + +```elixir +# mix.exs deps +{:usage_rules, "~> 1.1", only: [:dev]} + +# mix.exs project config +usage_rules: [packages: :all, output: :agents_md, mode: :linked] +``` + +```bash +mix usage_rules.gen # Generate AGENTS.md from deps +mix usage_rules.search_docs # Search hex documentation +mix usage_rules.gen_skill # Generate SKILL.md for Cowork +``` + +--- + +## Pro Tip: Local Docs + +Elixir packages ship excellent local documentation. Once Elixir is installed on cortex, accessing docs locally via `mix hex.docs fetch ` or `h Module.function` in IEx may be more efficient than fetching URLs. Consider installing Elixir on cortex to enable this workflow. diff --git a/elixir/elixir-part2-concurrency.md b/elixir/elixir-part2-concurrency.md new file mode 100644 index 0000000..08e4a26 --- /dev/null +++ b/elixir/elixir-part2-concurrency.md @@ -0,0 +1,278 @@ +# Elixir Part 2: Concurrency, OTP & State Machines + +## BEAM Process Fundamentals + +- **Processes are cheap** — ~2KB initial memory, microseconds to spawn, millions concurrent +- **No shared memory** — message passing only +- **Per-process GC** — no stop-the-world pauses +- **Preemptive scheduling** — one scheduler per CPU core, ~4000 reductions then yield +- **Process isolation** — one crash doesn't affect others + +### Spawning and Messaging +```elixir +pid = spawn(fn -> + receive do + {:greet, name} -> IO.puts("Hello, #{name}!") + end +end) +send(pid, {:greet, "Michael"}) + +# Linked — bidirectional crash propagation +pid = spawn_link(fn -> raise "boom" end) + +# Monitored — one-directional crash notification +ref = Process.monitor(pid) +receive do + {:DOWN, ^ref, :process, ^pid, reason} -> handle_crash(reason) +end +``` + +### Task — Structured Concurrency +```elixir +Task.start(fn -> send_email(user) end) # Fire and forget +task = Task.async(fn -> expensive_computation() end) # Async/await +result = Task.await(task, 30_000) + +Task.async_stream(urls, &fetch_url/1, max_concurrency: 10) # Parallel map +|> Enum.to_list() +``` + +--- + +## GenServer — Stateful Server Processes + +```elixir +defmodule Counter do + use GenServer + + # Client API + def start_link(initial \\ 0), do: GenServer.start_link(__MODULE__, initial, name: __MODULE__) + def increment, do: GenServer.cast(__MODULE__, :increment) + def get, do: GenServer.call(__MODULE__, :get) + + # Server callbacks + @impl true + def init(initial), do: {:ok, initial} + + @impl true + def handle_cast(:increment, count), do: {:noreply, count + 1} + + @impl true + def handle_call(:get, _from, count), do: {:reply, count, count} + + @impl true + def handle_info(:tick, count) do + Process.send_after(self(), :tick, 1000) + {:noreply, count} + end +end +``` + +**Key principle:** Callbacks run sequentially — this is both the synchronization mechanism and potential bottleneck. Keep callbacks fast; delegate heavy work to spawned tasks. + +--- + +## GenStateMachine — State Machines for Agentic Workflows + +**Why GenStateMachine over GenServer for agents?** GenServer has a single state and handles all messages uniformly. GenStateMachine (wrapping Erlang's `:gen_statem`) provides: +- **Explicit states** with per-state event handling +- **Built-in timeouts** — state timeouts (reset on state change), generic timeouts, event timeouts +- **Postpone** — defer events until the right state +- **State enter callbacks** — run setup logic when entering a state +- **Event types** — distinguish calls, casts, info, timeouts, internal events + +These map naturally onto agentic patterns: an agent in "thinking" state ignores new requests (postpone), has retry timeouts, transitions through well-defined phases, and runs setup on each state entry. + +### Installation +```elixir +{:gen_state_machine, "~> 3.0"} +``` + +### Callback Modes + +**`:handle_event_function`** (default) — single `handle_event/4` for all states: +```elixir +defmodule AgentFSM do + use GenStateMachine + + # Client API + def start_link(opts), do: GenStateMachine.start_link(__MODULE__, opts) + def submit(pid, task), do: GenStateMachine.call(pid, {:submit, task}) + def status(pid), do: GenStateMachine.call(pid, :status) + + # Server callbacks + def init(opts) do + {:ok, :idle, %{tasks: [], results: [], config: opts}} + end + + # State: idle + def handle_event({:call, from}, {:submit, task}, :idle, data) do + {:next_state, :processing, %{data | tasks: [task]}, + [{:reply, from, :accepted}, {:state_timeout, 30_000, :timeout}]} + end + + def handle_event({:call, from}, :status, state, data) do + {:keep_state_and_data, [{:reply, from, {state, length(data.results)}}]} + end + + # State: processing — with timeout + def handle_event(:state_timeout, :timeout, :processing, data) do + {:next_state, :error, %{data | error: :timeout}} + end + + # Internal event for completion + def handle_event(:info, {:result, result}, :processing, data) do + {:next_state, :complete, %{data | results: [result | data.results]}} + end + + # Postpone submissions while processing + def handle_event({:call, _from}, {:submit, _task}, :processing, _data) do + {:keep_state_and_data, :postpone} + end +end +``` + +**`:state_functions`** — each state is a separate function (states must be atoms): +```elixir +defmodule WorkflowFSM do + use GenStateMachine, callback_mode: [:state_functions, :state_enter] + + def init(_), do: {:ok, :pending, %{}} + + # State enter callbacks — run on every state transition + def pending(:enter, _old_state, data) do + {:keep_state, %{data | entered_at: DateTime.utc_now()}} + end + + def pending({:call, from}, {:start, params}, data) do + {:next_state, :running, %{data | params: params}, + [{:reply, from, :ok}, {:state_timeout, 60_000, :execution_timeout}]} + end + + def running(:enter, :pending, data) do + # Setup when entering running from pending + send(self(), :execute) + {:keep_state, data} + end + + def running(:info, :execute, data) do + result = do_work(data.params) + {:next_state, :complete, %{data | result: result}} + end + + def running(:state_timeout, :execution_timeout, data) do + {:next_state, :failed, %{data | error: :timeout}} + end + + # Postpone any calls while running + def running({:call, _from}, _request, _data) do + {:keep_state_and_data, :postpone} + end + + def complete(:enter, _old, data), do: {:keep_state, data} + def complete({:call, from}, :get_result, data) do + {:keep_state_and_data, [{:reply, from, {:ok, data.result}}]} + end + + def failed(:enter, _old, data), do: {:keep_state, data} + def failed({:call, from}, :get_error, data) do + {:keep_state_and_data, [{:reply, from, {:error, data.error}}]} + end +end +``` + +### Timeout Types + +| Type | Behavior | Use Case | +|------|----------|----------| +| `{:timeout, ms, event}` | Generic — survives state changes | Periodic polling | +| `{:state_timeout, ms, event}` | Resets on state change | Per-state deadlines | +| `{:event_timeout, ms, event}` | Resets on any event | Inactivity detection | + +### Key Actions in Return Tuples + +```elixir +# Return format: {:next_state, new_state, new_data, actions} +actions = [ + {:reply, from, response}, # Reply to caller + {:state_timeout, 30_000, :deadline}, # State-scoped timeout + {:timeout, 5_000, :poll}, # Generic timeout + :postpone, # Defer event to next state + :hibernate, # Reduce memory footprint + {:next_event, :internal, :setup} # Queue internal event +] +``` + +### When to Use GenStateMachine vs GenServer + +| Scenario | Use | +|----------|-----| +| Simple key-value state, CRUD | GenServer | +| Request/response server | GenServer | +| Well-defined state transitions | **GenStateMachine** | +| Need built-in timeouts per state | **GenStateMachine** | +| Events valid only in certain states | **GenStateMachine** | +| Agentic workflow with phases | **GenStateMachine** | +| Need postpone/defer semantics | **GenStateMachine** | + +--- + +## Supervisors — Let It Crash + +```elixir +defmodule MyApp.Application do + use Application + + @impl true + def start(_type, _args) do + children = [ + MyApp.Repo, + {MyApp.Cache, []}, + {Task.Supervisor, name: MyApp.TaskSupervisor}, + MyAppWeb.Endpoint + ] + opts = [strategy: :one_for_one, name: MyApp.Supervisor] + Supervisor.start_link(children, opts) + end +end +``` + +**Strategies:** +- `:one_for_one` — restart only crashed child (most common) +- `:one_for_all` — restart all if any crashes (tightly coupled) +- `:rest_for_one` — restart crashed + all started after it + +### DynamicSupervisor +```elixir +defmodule MyApp.SessionSupervisor do + use DynamicSupervisor + + def start_link(_), do: DynamicSupervisor.start_link(__MODULE__, :ok, name: __MODULE__) + def init(:ok), do: DynamicSupervisor.init(strategy: :one_for_one) + + def start_session(session_id) do + DynamicSupervisor.start_child(__MODULE__, {MyApp.Session, session_id}) + end +end +``` + +--- + +## Registry & Process Discovery + +```elixir +# In supervisor +{Registry, keys: :unique, name: MyApp.Registry} + +# Register a process +Registry.register(MyApp.Registry, "session:#{id}", %{}) + +# Lookup +case Registry.lookup(MyApp.Registry, "session:#{id}") do + [{pid, _value}] -> {:ok, pid} + [] -> {:error, :not_found} +end + +# Use as GenServer name +GenServer.start_link(MyWorker, arg, name: {:via, Registry, {MyApp.Registry, "worker:#{id}"}}) +``` diff --git a/elixir/elixir-part3-phoenix.md b/elixir/elixir-part3-phoenix.md new file mode 100644 index 0000000..090b87a --- /dev/null +++ b/elixir/elixir-part3-phoenix.md @@ -0,0 +1,628 @@ +# Elixir Part 3: Phoenix Framework — v1.8.5 (Current) + +Phoenix is the web framework for Elixir. Version 1.8.5 is current. It provides MVC controllers, real-time LiveView, Channels for WebSocket communication, and comprehensive tooling for authentication, testing, and deployment. + +## What's New in Phoenix 1.8 + +- **Scopes** in generators — secure data access by default (e.g., `current_user` automatically applied) +- **Magic links** (passwordless auth) and **"sudo mode"** in `mix phx.gen.auth` +- **daisyUI** integration — light/dark/system mode support out of the box +- **Simplified layouts** — single `root.html.heex` wraps everything; dynamic layouts are function components +- **`use Phoenix.Controller` now requires `:formats`** — must specify `formats: [:html]` or `formats: [:html, :json]` +- **Updated security headers** — `content-security-policy` with `base-uri 'self'; frame-ancestors 'self'`; dropped deprecated `x-frame-options` and `x-download-options` +- **`config` variable removed** from `Phoenix.Endpoint` — use `Application.compile_env/3` instead +- **Deprecated**: `:namespace`, `:put_default_views`, layouts without modules, `:trailing_slash` in router + +--- + +## Project Setup + +```bash +# New project (Phoenix Express — quickest path) +curl https://new.phoenixframework.org/my_app | sh + +# Traditional setup +mix phx.new my_app +mix phx.new my_app --no-ecto # Without database +mix phx.new my_app --no-html # API only +mix phx.new my_app --database sqlite3 # SQLite instead of Postgres +``` + +## Directory Structure + +``` +my_app/ +├── lib/ +│ ├── my_app/ # Business logic (contexts, schemas) +│ │ ├── application.ex # Supervision tree +│ │ ├── repo.ex # Ecto Repo +│ │ └── catalog/ # Context: Catalog +│ │ ├── product.ex # Schema +│ │ └── catalog.ex # Context module +│ └── my_app_web/ # Web layer +│ ├── endpoint.ex # HTTP entry point +│ ├── router.ex # Routes + pipelines +│ ├── components/ +│ │ ├── core_components.ex # Shared UI components +│ │ └── layouts.ex # Layout components +│ ├── controllers/ +│ │ ├── page_controller.ex +│ │ └── page_html/ # Templates for controller +│ │ └── home.html.heex +│ └── live/ # LiveView modules +│ └── counter_live.ex +├── config/ +│ ├── config.exs # Compile-time config +│ ├── dev.exs / prod.exs +│ └── runtime.exs # Runtime config (env vars, secrets) +├── priv/ +│ ├── repo/migrations/ +│ └── static/ # Static assets +├── test/ +│ ├── support/ +│ │ ├── conn_case.ex # Test helpers for controllers +│ │ └── data_case.ex # Test helpers for data layer +│ └── my_app_web/ +└── mix.exs +``` + +--- + +## Router & Pipelines + +```elixir +defmodule MyAppWeb.Router do + use MyAppWeb, :router + + pipeline :browser do + plug :accepts, ["html"] + plug :fetch_session + plug :fetch_live_flash + plug :put_root_layout, html: {MyAppWeb.Layouts, :root} + plug :protect_from_forgery + plug :put_secure_browser_headers + end + + pipeline :api do + plug :accepts, ["json"] + end + + scope "/", MyAppWeb do + pipe_through :browser + + get "/", PageController, :home + live "/counter", CounterLive # LiveView route + resources "/products", ProductController # RESTful CRUD + end + + scope "/api", MyAppWeb do + pipe_through :api + + resources "/items", ItemController, except: [:new, :edit] + end +end +``` + +--- + +## Verified Routes — `~p` Sigil + +**Always use `~p` instead of string paths.** Compile-time verified against your router. + +```elixir +# In templates (HEEx) +~H""" +View +Edit +""" + +# In controllers +redirect(conn, to: ~p"/products/#{product}") + +# With query params +~p"/products?page=#{page}&sort=name" +~p"/products?#{%{page: 1, sort: "name"}}" + +# URL generation (includes host) +url(~p"/products/#{product}") +# => "https://example.com/products/42" +``` + +--- + +## Controllers (1.8 Style) + +```elixir +defmodule MyAppWeb.ProductController do + use MyAppWeb, :controller + + # REQUIRED in 1.8: specify formats + # This is typically set in MyAppWeb :controller function + + def index(conn, _params) do + products = Catalog.list_products() + render(conn, :index, products: products) + end + + def show(conn, %{"id" => id}) do + product = Catalog.get_product!(id) + render(conn, :show, product: product) + end + + def create(conn, %{"product" => product_params}) do + case Catalog.create_product(product_params) do + {:ok, product} -> + conn + |> put_flash(:info, "Product created.") + |> redirect(to: ~p"/products/#{product}") + + {:error, %Ecto.Changeset{} = changeset} -> + render(conn, :new, changeset: changeset) + end + end +end +``` + +**View module naming:** For `ProductController`, Phoenix looks for `ProductHTML` (for `:html` format) and `ProductJSON` (for `:json` format). + +```elixir +# lib/my_app_web/controllers/product_html.ex +defmodule MyAppWeb.ProductHTML do + use MyAppWeb, :html + embed_templates "product_html/*" # Loads .heex files from directory +end + +# lib/my_app_web/controllers/product_json.ex +defmodule MyAppWeb.ProductJSON do + def index(%{products: products}), do: %{data: for(p <- products, do: data(p))} + def show(%{product: product}), do: %{data: data(product)} + defp data(product), do: %{id: product.id, title: product.title, price: product.price} +end +``` + +--- + +## Components and HEEx + +```elixir +defmodule MyAppWeb.CoreComponents do + use Phoenix.Component + + # Declare attributes with types and docs + attr :type, :string, default: "button" + attr :class, :string, default: nil + attr :rest, :global # Passes through all other HTML attrs + slot :inner_block, required: true + + def button(assigns) do + ~H""" + + """ + end + + # Table component with slots + attr :rows, :list, required: true + slot :col, required: true do + attr :label, :string + end + + def table(assigns) do + ~H""" + + + + + + + + + + + +
{col[:label]}
{render_slot(col, row)}
+ """ + end +end +``` + +**HEEx syntax notes:** +- `{@var}` — render assign (curly braces, not `<%= %>`) +- `:if={condition}` — conditional rendering on any tag +- `:for={item <- list}` — iteration on any tag +- `<.component_name />` — call function component with dot notation +- `` — call remote component with module name +- `{render_slot(@inner_block)}` — render slot content +- `<:slot_name>content` — named slot content + +--- + +## LiveView + +```elixir +defmodule MyAppWeb.SearchLive do + use MyAppWeb, :live_view + + def mount(_params, _session, socket) do + {:ok, assign(socket, query: "", results: [])} + end + + def handle_params(%{"q" => query}, _uri, socket) do + {:noreply, assign(socket, query: query, results: search(query))} + end + + def handle_params(_params, _uri, socket), do: {:noreply, socket} + + def handle_event("search", %{"query" => query}, socket) do + {:noreply, + socket + |> assign(query: query, results: search(query)) + |> push_patch(to: ~p"/search?q=#{query}")} + end + + def render(assigns) do + ~H""" + +
+ + +
+ +
+

{result.title}

+

{result.summary}

+
+
+ """ + end + + defp search(query), do: MyApp.Search.find(query) +end +``` + +**LiveView lifecycle:** `mount/3` → `handle_params/3` → `render/1`. Events via `handle_event/3`. Server pushes via `handle_info/2`. + +**Key patterns:** +- `assign/2,3` — set socket assigns +- `push_navigate/2` — navigate to new LiveView (full mount) +- `push_patch/2` — update URL without full remount (calls `handle_params`) +- `push_event/3` — push event to client JS hooks +- `stream/3,4` — efficient list rendering for large collections (inserts/deletes without re-rendering entire list) +- `async_assign/3` + `assign_async/3` — async data loading with loading/error states + +--- + +## Channels — Real-Time WebSocket + +```elixir +# In endpoint.ex +socket "/socket", MyAppWeb.UserSocket, + websocket: true, + longpoll: false + +# lib/my_app_web/channels/user_socket.ex +defmodule MyAppWeb.UserSocket do + use Phoenix.Socket + + channel "room:*", MyAppWeb.RoomChannel + + def connect(%{"token" => token}, socket, _connect_info) do + case Phoenix.Token.verify(socket, "user auth", token, max_age: 86400) do + {:ok, user_id} -> {:ok, assign(socket, :user_id, user_id)} + {:error, _} -> :error + end + end + + def id(socket), do: "users_socket:#{socket.assigns.user_id}" +end + +# lib/my_app_web/channels/room_channel.ex +defmodule MyAppWeb.RoomChannel do + use MyAppWeb, :channel + + def join("room:" <> room_id, _params, socket) do + {:ok, assign(socket, :room_id, room_id)} + end + + def handle_in("new_msg", %{"body" => body}, socket) do + broadcast!(socket, "new_msg", %{ + body: body, + user_id: socket.assigns.user_id + }) + {:noreply, socket} + end +end +``` + +**Force disconnect all sessions for a user:** +```elixir +MyAppWeb.Endpoint.broadcast("users_socket:#{user.id}", "disconnect", %{}) +``` + +--- + +## Contexts — Business Logic Boundary + +Contexts are plain Elixir modules that encapsulate data access and business rules. They are the API between your web layer and your domain. + +```elixir +# Generate with: mix phx.gen.context Catalog Product products title:string price:decimal +defmodule MyApp.Catalog do + import Ecto.Query + alias MyApp.Repo + alias MyApp.Catalog.Product + + def list_products do + Repo.all(Product) + end + + def get_product!(id), do: Repo.get!(Product, id) + + def create_product(attrs \\ %{}) do + %Product{} + |> Product.changeset(attrs) + |> Repo.insert() + end + + def update_product(%Product{} = product, attrs) do + product + |> Product.changeset(attrs) + |> Repo.update() + end + + def delete_product(%Product{} = product) do + Repo.delete(product) + end + + def change_product(%Product{} = product, attrs \\ %{}) do + Product.changeset(product, attrs) + end +end +``` + +**Context design principles:** +- One context per bounded domain (Catalog, Accounts, Orders) +- Contexts own their schemas — other contexts reference by ID, not struct +- Cross-context calls go through the public context API, never access another context's Repo directly +- Contexts can nest related schemas (Comments under Posts) + +--- + +## Authentication — `mix phx.gen.auth` + +```bash +mix phx.gen.auth Accounts User users +``` + +Phoenix 1.8 generates: +- **Magic links** (passwordless) — email-based login links +- **"Sudo mode"** — re-authentication for sensitive actions +- Session-based auth with secure token handling +- Email confirmation and password reset flows +- `require_authenticated_user` plug for protected routes + +## Scopes (New in 1.8) + +Scopes make secure data access the default in generators. When you generate resources with a scope, all queries are automatically filtered by the scoped user. + +```bash +mix phx.gen.live Posts Post posts title body:text --scope current_user +``` + +This generates code that automatically passes `current_user` to context functions, ensuring users only see their own data. + +--- + +## Ecto — Database Layer + +```elixir +# Schema +defmodule MyApp.Catalog.Product do + use Ecto.Schema + import Ecto.Changeset + + schema "products" do + field :title, :string + field :price, :decimal + field :status, Ecto.Enum, values: [:draft, :published, :archived] + has_many :reviews, MyApp.Reviews.Review + belongs_to :category, MyApp.Catalog.Category + timestamps(type: :utc_datetime) + end + + def changeset(product, attrs) do + product + |> cast(attrs, [:title, :price, :status, :category_id]) + |> validate_required([:title, :price]) + |> validate_number(:price, greater_than: 0) + |> unique_constraint(:title) + |> foreign_key_constraint(:category_id) + end +end + +# Queries +import Ecto.Query + +# Composable queries +def published(query \\ Product) do + from p in query, where: p.status == :published +end + +def recent(query, days \\ 7) do + from p in query, where: p.inserted_at > ago(^days, "day") +end + +def with_reviews(query) do + from p in query, preload: [:reviews] +end + +# Usage: Product |> published() |> recent(30) |> with_reviews() |> Repo.all() +``` + +--- + +## Telemetry — Built-in Observability + +Phoenix 1.8 includes a Telemetry supervisor that tracks request duration, Ecto query times, and VM metrics out of the box. + +```elixir +# lib/my_app_web/telemetry.ex (auto-generated) +defmodule MyAppWeb.Telemetry do + use Supervisor + import Telemetry.Metrics + + def metrics do + [ + summary("phoenix.endpoint.stop.duration", unit: {:native, :millisecond}), + summary("phoenix.router_dispatch.stop.duration", tags: [:route], unit: {:native, :millisecond}), + summary("my_app.repo.query.total_time", unit: {:native, :millisecond}), + summary("vm.memory.total", unit: {:byte, :kilobyte}), + summary("vm.total_run_queue_lengths.total"), + summary("vm.total_run_queue_lengths.cpu"), + ] + end +end +``` + +Integrates with **PromEx** for Prometheus/Grafana dashboards (see Part 4: Ecosystem). + +--- + +## Security Best Practices + +**Never pass untrusted input to:** `Code.eval_string/3`, `:os.cmd/2`, `System.cmd/3`, `System.shell/2`, `:erlang.binary_to_term/2` + +**Ecto prevents SQL injection by default** — the query DSL parameterizes all inputs. Only `Ecto.Adapters.SQL.query/4` with raw string interpolation is vulnerable. + +**Safe deserialization:** +```elixir +# UNSAFE — even with :safe flag +:erlang.binary_to_term(user_input, [:safe]) + +# SAFE — prevents executable terms +Plug.Crypto.non_executable_binary_to_term(user_input, [:safe]) +``` + +**CSRF protection** is built into the `:browser` pipeline via `protect_from_forgery`. **Content Security Policy** is set by `put_secure_browser_headers`. + +--- + +## Testing Phoenix + +```elixir +# Controller test +defmodule MyAppWeb.ProductControllerTest do + use MyAppWeb.ConnCase + + test "GET /products", %{conn: conn} do + conn = get(conn, ~p"/products") + assert html_response(conn, 200) =~ "Products" + end + + test "POST /products creates product", %{conn: conn} do + conn = post(conn, ~p"/products", product: %{title: "Widget", price: 9.99}) + assert redirected_to(conn) =~ ~p"/products/" + end +end + +# LiveView test +defmodule MyAppWeb.CounterLiveTest do + use MyAppWeb.ConnCase + import Phoenix.LiveViewTest + + test "increments counter", %{conn: conn} do + {:ok, view, html} = live(conn, ~p"/counter") + assert html =~ "Count: 0" + + assert view + |> element("button", "+1") + |> render_click() =~ "Count: 1" + end +end + +# Channel test +defmodule MyAppWeb.RoomChannelTest do + use MyAppWeb.ChannelCase + + test "broadcasts new messages" do + {:ok, _, socket} = subscribe_and_join(socket(MyAppWeb.UserSocket), MyAppWeb.RoomChannel, "room:lobby") + + push(socket, "new_msg", %{"body" => "hello"}) + assert_broadcast "new_msg", %{body: "hello"} + end +end +``` + +--- + +## Generators Cheat Sheet + +```bash +# HTML CRUD (controller + views + templates + context + schema + migration) +mix phx.gen.html Catalog Product products title:string price:decimal + +# LiveView CRUD +mix phx.gen.live Catalog Product products title:string price:decimal + +# JSON API +mix phx.gen.json Catalog Product products title:string price:decimal + +# Context + schema only (no web layer) +mix phx.gen.context Catalog Product products title:string price:decimal + +# Schema + migration only +mix phx.gen.schema Product products title:string price:decimal + +# Authentication +mix phx.gen.auth Accounts User users + +# Channel +mix phx.gen.channel Room + +# Presence +mix phx.gen.presence + +# Release files (Dockerfile, release.ex, overlay scripts) +mix phx.gen.release +mix phx.gen.release --docker # Include Dockerfile +``` + +--- + +## Release & Deployment + +```bash +# Generate release infrastructure +mix phx.gen.release --docker + +# Build release +MIX_ENV=prod mix deps.get --only prod +MIX_ENV=prod mix compile +MIX_ENV=prod mix assets.deploy +MIX_ENV=prod mix release + +# Release commands +_build/prod/rel/my_app/bin/server # Start with Phoenix server +_build/prod/rel/my_app/bin/migrate # Run migrations +_build/prod/rel/my_app/bin/my_app remote # Attach IEx console +``` + +**Runtime config (`config/runtime.exs`):** +```elixir +import Config + +if config_env() == :prod do + database_url = System.fetch_env!("DATABASE_URL") + secret_key_base = System.fetch_env!("SECRET_KEY_BASE") + + config :my_app, MyApp.Repo, + url: database_url, + pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10") + + config :my_app, MyAppWeb.Endpoint, + url: [host: System.fetch_env!("PHX_HOST"), port: 443, scheme: "https"], + http: [ip: {0, 0, 0, 0}, port: String.to_integer(System.get_env("PORT") || "4000")], + secret_key_base: secret_key_base +end +``` diff --git a/elixir/elixir-part4-ecosystem.md b/elixir/elixir-part4-ecosystem.md new file mode 100644 index 0000000..1f48345 --- /dev/null +++ b/elixir/elixir-part4-ecosystem.md @@ -0,0 +1,514 @@ +# Elixir Part 4: Ecosystem, Production & Deployment + +## Ash Framework — Declarative Resource Modeling + +For production/substantial projects, Ash provides a declarative, resource-oriented approach that eliminates boilerplate while remaining extensible. + +```elixir +# mix.exs deps +{:ash, "~> 3.0"}, +{:ash_postgres, "~> 2.0"}, # Ecto/Postgres data layer +{:ash_phoenix, "~> 2.0"}, # Phoenix integration +{:ash_graphql, "~> 1.0"}, # Auto-generated GraphQL API +{:ash_json_api, "~> 1.0"}, # Auto-generated JSON:API +``` + +### Resource Definition + +```elixir +defmodule MyApp.Blog.Post do + use Ash.Resource, + domain: MyApp.Blog, + data_layer: AshPostgres.DataLayer + + postgres do + table "posts" + repo MyApp.Repo + end + + attributes do + uuid_primary_key :id + attribute :title, :string, allow_nil?: false + attribute :body, :string, allow_nil?: false + attribute :status, :atom, constraints: [one_of: [:draft, :published, :archived]], default: :draft + timestamps() + end + + relationships do + belongs_to :author, MyApp.Accounts.User + has_many :comments, MyApp.Blog.Comment + end + + actions do + defaults [:read, :destroy] + + create :create do + accept [:title, :body] + change relate_actor(:author) + end + + update :publish do + change set_attribute(:status, :published) + end + end + + policies do + policy action_type(:read) do + authorize_if always() + end + + policy action_type([:create, :update, :destroy]) do + authorize_if actor_attribute_equals(:role, :admin) + end + end +end +``` + +### Domain Definition + +```elixir +defmodule MyApp.Blog do + use Ash.Domain + + resources do + resource MyApp.Blog.Post + resource MyApp.Blog.Comment + end +end +``` + +### Using Ash Resources + +```elixir +# Create +MyApp.Blog.Post +|> Ash.Changeset.for_create(:create, %{title: "Hello", body: "World"}, actor: current_user) +|> Ash.create!() + +# Read with filters +MyApp.Blog.Post +|> Ash.Query.filter(status == :published) +|> Ash.Query.sort(inserted_at: :desc) +|> Ash.read!() + +# Custom action +post |> Ash.Changeset.for_update(:publish) |> Ash.update!() +``` + +**When to use Ash vs plain Phoenix:** Ash excels when you need consistent authorization, multi-tenancy, auto-generated APIs, or complex domain logic. For simple CRUD apps or learning, plain Phoenix contexts are fine. + +--- + +## Jido — Multi-Agent Orchestration Framework + +Jido (~> 2.1) is the premier Elixir framework for building and managing agents of all types — not just LLMs. Vision: 10,000 agents per user on the BEAM. + +```elixir +# mix.exs deps +{:jido, "~> 2.1"}, +{:jido_ai, "~> 0.x"}, # AI/LLM integration +{:jido_action, "~> 0.x"}, # Composable actions +{:jido_signal, "~> 0.x"}, # CloudEvents-based messaging +``` + +### Core Concepts + +**Agents are pure functional structs** — immutable, no side effects in the agent itself. Side effects are described as Directives. + +```elixir +defmodule MyAgent do + use Jido.Agent, + name: "my_agent", + description: "Does useful things", + actions: [FetchData, ProcessData, NotifyUser], + schema: [ + model: [type: :string, default: "gpt-4"], + temperature: [type: :float, default: 0.7] + ] +end + +# Create and command +{:ok, agent} = MyAgent.new() +{agent, directives} = MyAgent.cmd(agent, %Signal{type: "task.assigned", data: %{task: "analyze"}}) +# directives are typed effects: Emit, Spawn, Schedule, Stop +``` + +### Actions — Composable Units + +Actions are the building blocks. Each has a schema, validates input, and returns output. + +```elixir +defmodule FetchData do + use Jido.Action, + name: "fetch_data", + description: "Fetches data from an API", + schema: [ + url: [type: :string, required: true], + timeout: [type: :integer, default: 5000] + ] + + @impl true + def run(params, _context) do + case Req.get(params.url, receive_timeout: params.timeout) do + {:ok, %{status: 200, body: body}} -> {:ok, %{data: body}} + {:ok, %{status: status}} -> {:error, "HTTP #{status}"} + {:error, reason} -> {:error, reason} + end + end +end + +# Actions expose themselves as AI tools +FetchData.to_tool() # Returns tool spec for LLM function calling +``` + +### Signals — CloudEvents v1.0.2 Messaging + +```elixir +signal = %Jido.Signal{ + type: "task.completed", + source: "agent:worker_1", + data: %{result: "analysis complete"}, + subject: "task:123" +} +``` + +### Directives — Typed Effect Descriptions + +Agents don't perform side effects directly. They return Directives describing what should happen: + +| Directive | Effect | +|-----------|--------| +| `Emit` | Send a signal to another agent/system | +| `Spawn` | Create a new child agent | +| `Schedule` | Schedule a future action | +| `Stop` | Terminate the agent | + +### AgentServer — Runtime Process + +`AgentServer` wraps an agent in a GenServer for real-time operation: + +```elixir +{:ok, pid} = Jido.AgentServer.start_link(agent: MyAgent, id: "worker-1") +Jido.AgentServer.signal(pid, %Signal{type: "task.start", data: %{...}}) +``` + +### When to Use Jido + +- Multi-agent systems where agents communicate via signals +- LLM-powered agents that need tool calling (via `to_tool/0`) +- Systems requiring parent-child agent hierarchies +- Workflows with complex state machines (Jido includes FSM strategy) +- Any scenario targeting high agent density (thousands per node) + +--- + +## OTP Releases & Docker + +### OTP Release + +```elixir +# mix.exs +def project do + [ + app: :my_app, + releases: [ + my_app: [ + include_executables_for: [:unix], + steps: [:assemble, :tar] + ] + ] + ] +end +``` + +### Docker Multistage Build + +```dockerfile +# Build stage +FROM hexpm/elixir:1.19.5-erlang-27.3-debian-bookworm-20240904 AS build +RUN apt-get update && apt-get install -y build-essential git +WORKDIR /app +ENV MIX_ENV=prod + +COPY mix.exs mix.lock ./ +RUN mix deps.get --only $MIX_ENV && mix deps.compile + +COPY config/config.exs config/prod.exs config/ +COPY lib lib +COPY priv priv +COPY assets assets + +RUN mix assets.deploy && mix compile && mix release + +# Runtime stage +FROM debian:bookworm-slim +RUN apt-get update && apt-get install -y libstdc++6 openssl libncurses5 locales +RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen +ENV LANG=en_US.UTF-8 + +WORKDIR /app +COPY --from=build /app/_build/prod/rel/my_app ./ + +HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \ + CMD curl -f http://localhost:4000/health || exit 1 + +CMD ["bin/server"] +``` + +--- + +## Distributed Erlang & Clustering + +```elixir +# libcluster config (config/runtime.exs) +config :libcluster, + topologies: [ + local: [ + strategy: Cluster.Strategy.Gossip + ], + # OR for Docker/K8s: + k8s: [ + strategy: Cluster.Strategy.Kubernetes, + config: [ + mode: :hostname, + kubernetes_namespace: "default", + kubernetes_selector: "app=my_app" + ] + ] + ] + +# Distributed PubSub (built into Phoenix) +Phoenix.PubSub.broadcast(MyApp.PubSub, "events", {:new_order, order}) +Phoenix.PubSub.subscribe(MyApp.PubSub, "events") + +# Horde for distributed supervisor/registry +{:horde, "~> 0.9"} +``` + +--- + +## Monitoring & Observability + +### PromEx — Prometheus Metrics for Elixir + +```elixir +# mix.exs +{:prom_ex, "~> 1.9"} + +# lib/my_app/prom_ex.ex +defmodule MyApp.PromEx do + use PromEx, otp_app: :my_app + + @impl true + def plugins do + [ + PromEx.Plugins.Application, + PromEx.Plugins.Beam, + {PromEx.Plugins.Phoenix, router: MyAppWeb.Router, endpoint: MyAppWeb.Endpoint}, + {PromEx.Plugins.Ecto, repos: [MyApp.Repo]}, + PromEx.Plugins.Oban + ] + end + + @impl true + def dashboards do + [ + {:prom_ex, "application.json"}, + {:prom_ex, "beam.json"}, + {:prom_ex, "phoenix.json"}, + {:prom_ex, "ecto.json"} + ] + end +end +``` + +### Full Observability Stack (BEAMOps) + +From "Engineering Elixir Applications": + +| Component | Role | +|-----------|------| +| **PromEx** | Expose BEAM/Phoenix/Ecto metrics as Prometheus endpoints | +| **Prometheus** | Scrape and store time-series metrics | +| **Grafana** | Dashboards and alerting | +| **Loki** | Log aggregation (like Prometheus but for logs) | +| **Promtail/Alloy** | Ship logs from containers to Loki | + +**Health checks + automatic rollback:** Docker `HEALTHCHECK` triggers rollback if the new container fails health checks within the start period. Use `docker system prune` automation to prevent disk bloat. + +--- + +## CI/CD Pipeline (GitHub Actions) + +```yaml +name: Elixir CI +on: [push, pull_request] + +jobs: + test: + runs-on: ubuntu-latest + services: + db: + image: postgres:16 + env: + POSTGRES_PASSWORD: postgres + ports: ['5432:5432'] + steps: + - uses: actions/checkout@v4 + - uses: erlef/setup-beam@v1 + with: + elixir-version: '1.19.5' + otp-version: '27.3' + - run: mix deps.get + - run: mix compile --warnings-as-errors + - run: mix format --check-formatted + - run: mix credo --strict + - run: mix test + - run: mix deps.unlock --check-unused +``` + +--- + +## Interop — Elixir as Orchestrator + +### Python via Ports/Erlport +```elixir +# Using erlport for bidirectional Python calls +{:ok, pid} = :python.start([{:python_path, ~c"./python_scripts"}]) +result = :python.call(pid, :my_module, :my_function, [arg1, arg2]) +:python.stop(pid) +``` + +### Rust via Rustler NIFs +```elixir +# mix.exs +{:rustler, "~> 0.34"} + +# lib/my_nif.ex +defmodule MyApp.NativeSort do + use Rustler, otp_app: :my_app, crate: "native_sort" + + # NIF stubs — replaced at load time by Rust implementations + def sort(_list), do: :erlang.nif_error(:nif_not_loaded) +end +``` + +### System Commands via Ports +```elixir +# One-shot command +{output, 0} = System.cmd("ffmpeg", ["-i", input, "-o", output]) + +# Long-running port +port = Port.open({:spawn, "python3 worker.py"}, [:binary, :exit_status]) +send(port, {self(), {:command, "process\n"}}) +receive do + {^port, {:data, data}} -> handle_response(data) +end +``` + +--- + +## Cortex Deployment Story + +### Current State +- Cortex runs Ubuntu 24.04 with Caddy as web server +- Elixir is **not yet installed** — needs `asdf` or direct install +- Docker is **planned but not installed** — needed for containerized Elixir apps +- Symbiont (Python) is the current orchestrator — Elixir will gradually take over + +### Installation Plan for Cortex +```bash +# Option 1: asdf (recommended — manages multiple versions) +git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0 +echo '. "$HOME/.asdf/asdf.sh"' >> ~/.bashrc +asdf plugin add erlang +asdf plugin add elixir +asdf install erlang 27.3 +asdf install elixir 1.19.5-otp-27 +asdf global erlang 27.3 +asdf global elixir 1.19.5-otp-27 + +# Option 2: Direct from Erlang Solutions +wget https://packages.erlang-solutions.com/erlang-solutions_2.0_all.deb +dpkg -i erlang-solutions_2.0_all.deb +apt-get update && apt-get install esl-erlang elixir +``` + +### BEAMOps Principles for Cortex + +From "Engineering Elixir Applications" — the deployment philosophy: + +1. **Environment Integrity** — identical builds dev/staging/prod via Docker + releases +2. **Infrastructure as Code** — Caddy config, systemd units, backup scripts all version-controlled +3. **OTP Releases** — self-contained, no runtime deps, `bin/my_app start` +4. **Distributed Erlang** — nodes discover each other, share state via PubSub, global registry +5. **Instrumentation** — PromEx + Prometheus + Grafana + Loki for full observability +6. **Health Checks + Rollbacks** — Docker health checks trigger automatic rollback on failed deploys +7. **Zero-downtime deploys** — rolling updates via Docker Swarm or `mix release` hot upgrades + +--- + +## Anti-Patterns to Avoid + +### Process Anti-Patterns +- **GenServer as code organization** — don't wrap pure functions in a GenServer. Use modules. +- **Agent for complex state** — if you need more than get/update, use GenServer directly. +- **Spawning unsupervised processes** — always use `Task.Supervisor` or link to a supervisor. + +### Code Anti-Patterns +- **Primitive obsession** — use structs, not bare maps, for domain concepts. +- **Boolean parameters** — use atoms or keyword options: `format: :json` not `json: true`. +- **Large modules** — split by concern, not by entity type. Domain logic, web layer, workers. +- **String keys in internal maps** — use atoms internally, strings only at boundaries (JSON, forms). + +### Design Anti-Patterns +- **Monolithic contexts** — Phoenix contexts should be small, focused. Split `Accounts` from `Authentication`. +- **God GenServer** — one process handling all state for the app. Distribute responsibility. +- **Synchronous calls to slow services** — use `Task.async` + `Task.await` with timeouts. + +--- + +## Quick Recipes + +### HTTP Request with Req +```elixir +Req.get!("https://api.example.com/data", + headers: [{"authorization", "Bearer #{token}"}], + receive_timeout: 15_000 +).body +``` + +### JSON Encode/Decode +```elixir +Jason.encode!(%{name: "Michael", role: :admin}) +Jason.decode!(~s({"name": "Michael"}), keys: :atoms) +``` + +### Background Job with Oban +```elixir +defmodule MyApp.Workers.EmailWorker do + use Oban.Worker, queue: :mailers, max_attempts: 3 + + @impl true + def perform(%Oban.Job{args: %{"to" => to, "template" => template}}) do + MyApp.Mailer.send(to, template) + end +end + +# Enqueue +%{to: "user@example.com", template: "welcome"} +|> MyApp.Workers.EmailWorker.new(schedule_in: 60) +|> Oban.insert!() +``` + +--- + +## Resources + +- [Elixir Official Docs](https://hexdocs.pm/elixir/) — always check 1.19.5 version +- [Ash Framework Docs](https://hexdocs.pm/ash/) — resource-oriented patterns +- [Phoenix HexDocs](https://hexdocs.pm/phoenix/) — web framework +- [Jido Docs](https://hexdocs.pm/jido/) — multi-agent orchestration +- [Elixir Forum](https://elixirforum.com/) — community Q&A +- "Elixir in Action" by Saša Jurić — deep BEAM/OTP understanding (note: covers 1.15, check breaking changes) +- "Engineering Elixir Applications" by Fairholm & D'Lacoste — BEAMOps deployment patterns