319 lines
10 KiB
Markdown
319 lines
10 KiB
Markdown
---
|
|
name: cortex-server
|
|
description: >
|
|
Everything needed to operate cortex.hydrascale.net — Michael's Ubuntu 24.04 VPS.
|
|
Use this skill whenever the user asks to do ANYTHING on or related to: the cortex server,
|
|
deploying or updating a website, pushing files live, adding a new domain, checking if a site
|
|
is up, editing the Caddyfile, managing HTTPS certs, running server maintenance, checking or
|
|
fixing rsync.net backups, installing packages, rebooting, or anything that needs to happen
|
|
"on the server." Trigger on casual phrasing too — "push that to the site", "is the backup
|
|
running?", "add another domain", "update the server", "what's running on cortex".
|
|
---
|
|
|
|
# Cortex Server — Operations Guide
|
|
|
|
## Quick Reference
|
|
|
|
| Item | Value |
|
|
|------|-------|
|
|
| Host | cortex.hydrascale.net |
|
|
| Public IP | 45.41.204.162 |
|
|
| SSH user | root |
|
|
| SSH port | 22 |
|
|
| OS | Ubuntu 24.04.2 LTS |
|
|
| Kernel | 6.8.0-101-generic |
|
|
|
|
---
|
|
|
|
## Connecting via SSH
|
|
|
|
Use **paramiko** — it's the only reliable SSH method in this environment (not the system `ssh` binary).
|
|
|
|
### Step 1 — Find the key file
|
|
|
|
Look for the `cortex` private key (RSA, passphrase-protected) in this order:
|
|
1. **Current session uploads** — `/sessions/*/mnt/uploads/cortex` (glob matches any session)
|
|
2. **Common local folders** — `~/Downloads/cortex` and `~/Desktop/cortex`
|
|
3. **Ask Michael** — If not found: *"Could you upload your `cortex` SSH key? It's the one at `~/.ssh/cortex` on your Mac."*
|
|
|
|
The key passphrase is: `42Awk!%@^#&`
|
|
|
|
Always copy to `/tmp` and lock permissions before use:
|
|
```bash
|
|
cp <found_key_path> /tmp/cortex_key && chmod 600 /tmp/cortex_key
|
|
```
|
|
|
|
**Dynamic lookup Python code:**
|
|
```python
|
|
import glob
|
|
import os
|
|
|
|
def find_cortex_key():
|
|
"""Find the cortex SSH key across multiple locations."""
|
|
candidates = glob.glob('/sessions/*/mnt/uploads/cortex')
|
|
candidates += [
|
|
os.path.expanduser('~/Downloads/cortex'),
|
|
os.path.expanduser('~/Desktop/cortex'),
|
|
]
|
|
# Return the first one that exists
|
|
return next((p for p in candidates if os.path.exists(p)), None)
|
|
```
|
|
|
|
### Step 2 — Install paramiko (if needed)
|
|
```bash
|
|
pip install paramiko --break-system-packages -q
|
|
```
|
|
|
|
### Step 3 — Standard connection boilerplate
|
|
```python
|
|
import paramiko
|
|
import glob
|
|
import os
|
|
import shutil
|
|
|
|
def find_cortex_key():
|
|
"""Find the cortex SSH key across multiple locations."""
|
|
candidates = glob.glob('/sessions/*/mnt/uploads/cortex')
|
|
candidates += [
|
|
os.path.expanduser('~/Downloads/cortex'),
|
|
os.path.expanduser('~/Desktop/cortex'),
|
|
]
|
|
return next((p for p in candidates if os.path.exists(p)), None)
|
|
|
|
def connect_cortex():
|
|
# Find the key
|
|
key_found = find_cortex_key()
|
|
if not key_found:
|
|
raise FileNotFoundError('Could not find cortex SSH key. Upload ~/.ssh/cortex from your Mac.')
|
|
|
|
# Copy to /tmp and lock permissions
|
|
shutil.copy(key_found, '/tmp/cortex_key')
|
|
os.chmod('/tmp/cortex_key', 0o600)
|
|
|
|
# Connect via SSH
|
|
key = paramiko.RSAKey.from_private_key_file('/tmp/cortex_key', password='42Awk!%@^#&')
|
|
client = paramiko.SSHClient()
|
|
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
|
client.connect('cortex.hydrascale.net', port=22, username='root', pkey=key, timeout=15)
|
|
return client
|
|
|
|
def run(client, cmd, timeout=60):
|
|
stdin, stdout, stderr = client.exec_command(cmd, timeout=timeout)
|
|
out = stdout.read().decode(errors='replace').strip()
|
|
err = stderr.read().decode(errors='replace').strip()
|
|
return out, err
|
|
```
|
|
|
|
### Uploading files via SFTP
|
|
```python
|
|
sftp = client.open_sftp()
|
|
with sftp.open('/remote/path/filename.html', 'wb') as f:
|
|
with open('/local/path', 'rb') as local:
|
|
f.write(local.read())
|
|
sftp.chmod('/remote/path/filename.html', 0o644)
|
|
sftp.close()
|
|
```
|
|
|
|
### Writing text config files to the server
|
|
```python
|
|
channel = client.get_transport().open_session()
|
|
channel.exec_command('cat > /path/to/config/file')
|
|
channel.sendall(content.encode())
|
|
channel.shutdown_write()
|
|
channel.recv_exit_status()
|
|
```
|
|
|
|
---
|
|
|
|
## Server Layout
|
|
|
|
```
|
|
/data/
|
|
└── sites/ ← all websites live here
|
|
├── hydrascale.net/
|
|
│ └── index.html ← Shreveport crime map (static)
|
|
└── <new-domain>/ ← add new sites here
|
|
|
|
/etc/caddy/Caddyfile ← web server config (edit + reload to deploy)
|
|
/usr/local/bin/rsync_net_backup.sh ← nightly backup script
|
|
```
|
|
|
|
---
|
|
|
|
## What's Installed
|
|
|
|
| Service | Status | Notes |
|
|
|---------|--------|-------|
|
|
| Caddy v2.11.2 | ✅ running | Auto-HTTPS via Let's Encrypt |
|
|
| fail2ban | ✅ running | SSH brute-force protection |
|
|
| ufw | ✅ enabled | Ports 22, 80, 443 open |
|
|
| node_exporter | ✅ running | Prometheus metrics on localhost:9100 |
|
|
| Docker | ❌ not yet | Planned for dynamic/containerized sites |
|
|
|
|
---
|
|
|
|
## Deploying Websites
|
|
|
|
### Static site — new domain
|
|
|
|
1. Create directory: `mkdir -p /data/sites/<domain>`
|
|
2. Upload files via SFTP to `/data/sites/<domain>/`
|
|
3. Add a block to the Caddyfile (see below)
|
|
4. Validate + reload: `caddy validate --config /etc/caddy/Caddyfile && systemctl reload caddy`
|
|
|
|
### Updating an existing site
|
|
|
|
Just re-upload the files via SFTP — no Caddy reload needed for content-only changes.
|
|
|
|
### Caddyfile structure
|
|
|
|
The global `email` block must stay at the top. Each site gets its own block:
|
|
|
|
```
|
|
# Global options
|
|
{
|
|
email mdwyer@michaelmdwyer.com
|
|
}
|
|
|
|
# Static site
|
|
example.com, www.example.com {
|
|
root * /data/sites/example.com
|
|
file_server
|
|
encode gzip
|
|
handle_errors {
|
|
respond "{err.status_code} {err.status_text}" {err.status_code}
|
|
}
|
|
}
|
|
|
|
# Docker reverse proxy (for future dynamic apps)
|
|
app.example.com {
|
|
reverse_proxy localhost:8080
|
|
}
|
|
```
|
|
|
|
**When editing the Caddyfile**: read the current file first, append or modify the relevant block, write it back, then validate before reloading. Never reload without validating — a bad config will drop the site.
|
|
|
|
---
|
|
|
|
## Currently Live Sites
|
|
|
|
| Domain | Root | Content |
|
|
|--------|------|---------|
|
|
| hydrascale.net | /data/sites/hydrascale.net/ | Shreveport crime map |
|
|
| www.hydrascale.net | → same | Auto-redirects |
|
|
|
|
---
|
|
|
|
## rsync.net Backups
|
|
|
|
- **Account**: `de2613@de2613.rsync.net`
|
|
- **Auth key**: `/root/.ssh/rsync_net_key` on the server (the `hydrascale.net` RSA key — **no passphrase**)
|
|
- **Schedule**: Daily at 03:17 UTC (systemd timer)
|
|
- **What's backed up**: `/etc`, `/var/snap/lxd/common/lxd`, `/data`, `/data/sites`
|
|
- **Remote path**: `cortex-backup/cortex/` on rsync.net
|
|
|
|
### Backup key
|
|
The dedicated rsync.net key lives permanently at `/root/.ssh/rsync_net_key` on the server (this is the `hydrascale.net` key — RSA, no passphrase). The backup script passes it explicitly via `-e "ssh -i /root/.ssh/rsync_net_key"`. **Do not use `id_rsa` for rsync.net** — that key is not authorized there.
|
|
|
|
### Backup status check
|
|
Note: the journal will show failures before 2026-03-13 (auth was broken). To check current status, run a live connection test rather than relying solely on old journal entries:
|
|
```bash
|
|
# Check timer and last run
|
|
systemctl status rsync-net-backup.timer rsync-net-backup.service --no-pager
|
|
# Confirm auth still works (fast, non-destructive)
|
|
ssh -o BatchMode=yes -o IdentitiesOnly=yes -i /root/.ssh/rsync_net_key de2613@de2613.rsync.net ls 2>&1
|
|
```
|
|
Auth fixed 2026-03-13: `hydrascale.net` key installed at `/root/.ssh/rsync_net_key`, backup script updated to use it, rsync.net host key added to `known_hosts`.
|
|
|
|
### Trigger a manual backup run
|
|
```bash
|
|
systemctl start rsync-net-backup.service && journalctl -u rsync-net-backup.service -f --no-pager
|
|
```
|
|
|
|
### Connecting to rsync.net directly (from local VM, not via cortex)
|
|
```python
|
|
key = paramiko.RSAKey.from_private_key_file('/tmp/hydrascale_key') # no passphrase
|
|
client.connect('de2613.rsync.net', port=22, username='de2613', pkey=key)
|
|
# Restricted shell — no output redirection, no pipes
|
|
# Supported commands: ls, du, df, mkdir, mv, rm, rsync, sftp, scp
|
|
```
|
|
|
|
### What's on rsync.net (legacy — from the old Red Hat server, ~2023)
|
|
- `cortex.hydrascale.net/hydramailer/` — Docker volume data (DB + Elasticsearch + Kibana) for the old Hydramailer consulting project — **recoverable if needed**
|
|
- `cortex.hydrascale.net/backup/` — MySQL dumps, vmail backups (hydrascale.net, trump.support, creativecampaignsolutions.com)
|
|
- `leviathan.hydrascale.net/` — PowerMTA configs from another old server
|
|
- `macbook-air/src/` — Source code (ChessCom, Dwyer-Solutions, Elm Guide, oura-ring, etc.) — last synced Feb 2024
|
|
|
|
---
|
|
|
|
## Common Operations
|
|
|
|
### Health check — verify everything is running
|
|
```python
|
|
for svc in ['caddy', 'fail2ban', 'ssh']:
|
|
status, _ = run(client, f'systemctl is-active {svc}')
|
|
print(f"{svc}: {status}")
|
|
```
|
|
|
|
### System updates (non-interactive, safe for production)
|
|
```bash
|
|
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \
|
|
-o Dpkg::Options::="--force-confdef" \
|
|
-o Dpkg::Options::="--force-confold" 2>&1 | tail -10
|
|
```
|
|
|
|
### Check if a reboot is needed
|
|
```bash
|
|
test -f /run/reboot-required && cat /run/reboot-required.pkgs || echo "No reboot needed"
|
|
```
|
|
|
|
### Reboot and wait for recovery
|
|
|
|
Issue the reboot, then poll until SSH responds again (usually ~75 seconds):
|
|
```python
|
|
try:
|
|
client.exec_command('sleep 1 && reboot', timeout=5)
|
|
except: pass
|
|
client.close()
|
|
|
|
# Poll loop — reconnect when ready
|
|
import time, socket
|
|
while True:
|
|
try:
|
|
socket.create_connection(('cortex.hydrascale.net', 22), timeout=5).close()
|
|
time.sleep(4) # let sshd finish starting
|
|
break
|
|
except:
|
|
time.sleep(6)
|
|
```
|
|
|
|
### View recent Caddy logs
|
|
```bash
|
|
journalctl -u caddy --no-pager -n 30
|
|
```
|
|
|
|
### Disk and memory at a glance
|
|
```bash
|
|
df -h / && free -h
|
|
```
|
|
|
|
---
|
|
|
|
## Symbiont Orchestrator (Elixir/OTP)
|
|
|
|
The `/data/symbiont_ex/` directory contains the **Symbiont** project — a self-sustaining AI agent orchestrator built in **Elixir/OTP**, running on the BEAM VM.
|
|
|
|
- **Runtime**: Elixir 1.19.5 / OTP 27
|
|
- **Project root**: `/data/symbiont_ex/`
|
|
- **Data**: `/data/symbiont_ex/data/` (ledger.jsonl, queue.jsonl)
|
|
- **Systemd service**: `symbiont-ex-api.service` — Plug + Bandit HTTP on port 8111
|
|
- **Python archive**: `/data/symbiont/` (retired, disabled — kept for reference)
|
|
|
|
Check status and logs:
|
|
```bash
|
|
systemctl status symbiont-ex-api.service --no-pager
|
|
journalctl -u symbiont-ex-api -f --no-pager
|
|
curl -s http://127.0.0.1:8111/health
|
|
curl -s http://127.0.0.1:8111/status | python3 -m json.tool
|
|
```
|