Tau.Acuvim/docs/deployment-guide.md

# Production Deployment Guide

This guide covers deploying the Tau Acuvim console application, PostgreSQL database, and Mosquitto MQTT broker to a production server using Docker Compose.

---

## 1. Server Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 2 cores | 4 cores |
| RAM | 4 GB | 8 GB |
| Storage | 50 GB SSD | 100 GB SSD |
| OS | Ubuntu 22.04 LTS | Ubuntu 24.04 LTS |
| Network | Static public IP | Static IP + domain name |

The server must have a stable internet connection accessible to ESP32 devices over both WiFi and cellular networks.

---

## 2. Prerequisites

Install Docker and Docker Compose on the server:

```bash
# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Verify
docker --version
docker compose version
```

Install Certbot for SSL certificates:

```bash
sudo apt update
sudo apt install certbot python3-certbot-nginx nginx -y
```

---

## 3. Domain and DNS

Configure a DNS A record pointing your domain to the server's public IP:

```
console.example.com  ->  203.0.113.50
```

If you plan to use a subdomain for the MQTT broker's WebSocket interface, add that as well:

```
mqtt.example.com     ->  203.0.113.50
```

---

## 4. Directory Structure

Create the deployment directory on the server:

```bash
sudo mkdir -p /opt/acuvim
sudo mkdir -p /opt/acuvim/mosquitto/config
sudo mkdir -p /opt/acuvim/mosquitto/data
sudo mkdir -p /opt/acuvim/mosquitto/log
sudo mkdir -p /opt/acuvim/mosquitto/certs
sudo mkdir -p /opt/acuvim/firmware
sudo mkdir -p /opt/acuvim/backups
sudo mkdir -p /opt/acuvim/nginx
```

---

## 5. Environment Configuration

Create the production environment file at `/opt/acuvim/.env`:

```bash
# Database
POSTGRES_DB=acuvim
POSTGRES_USER=acuvim
POSTGRES_PASSWORD=<generate-strong-password-here>

# Console Application
ConnectionStrings__DefaultConnection=Host=db;Database=acuvim;Username=acuvim;Password=<same-password-as-above>
Jwt__Secret=<generate-256-bit-secret-minimum-32-characters>
Jwt__Issuer=Tau.Acuvim.Console
Jwt__ExpiryHours=24
Firmware__BaseUrl=https://console.example.com
Firmware__StoragePath=/app/firmware

# MQTT
Mqtt__Broker=mqtt
Mqtt__Port=1883
Mqtt__Username=console
Mqtt__Password=<generate-mqtt-console-password>
Mqtt__TopicPrefix=acuvim
Mqtt__ClientId=acuvim-console
```

Generate strong passwords:

```bash
# Generate a random 32-character password
openssl rand -base64 32

# Generate a 256-bit JWT secret
openssl rand -base64 48
```

Restrict file permissions:

```bash
chmod 600 /opt/acuvim/.env
```

---

## 6. Mosquitto MQTT Broker Configuration

### 6.1 Production Configuration

Create `/opt/acuvim/mosquitto/config/mosquitto.conf`:

```
# ===========================================
# Mosquitto Production Configuration
# ===========================================

# Persistence
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
log_type error
log_type warning
log_type notice

# -------------------------------------------
# Listener 1: MQTTS (TLS on port 8883)
# -------------------------------------------
listener 8883
cafile /mosquitto/certs/ca.crt
certfile /mosquitto/certs/server.crt
keyfile /mosquitto/certs/server.key
tls_version tlsv1.2
require_certificate false

# -------------------------------------------
# Listener 2: Internal MQTT (port 1883)
# Only accessible within Docker network
# -------------------------------------------
listener 1883 0.0.0.0
# Internal listener for console <-> broker communication
# Not exposed to the host network in production

# -------------------------------------------
# Listener 3: WebSockets over TLS (port 9883)
# -------------------------------------------
listener 9883
protocol websockets
cafile /mosquitto/certs/ca.crt
certfile /mosquitto/certs/server.crt
keyfile /mosquitto/certs/server.key
tls_version tlsv1.2

# -------------------------------------------
# Authentication
# -------------------------------------------
allow_anonymous false
password_file /mosquitto/config/passwd
acl_file /mosquitto/config/acl

# -------------------------------------------
# Connection limits
# -------------------------------------------
max_connections 500
max_inflight_messages 20
max_queued_messages 1000
message_size_limit 10240
```

### 6.2 Per-Device Authentication

Create the password file using `mosquitto_passwd`:

```bash
# Create the initial password file with the console user
docker run --rm -v /opt/acuvim/mosquitto/config:/mosquitto/config \
  eclipse-mosquitto:2 \
  mosquitto_passwd -c -b /mosquitto/config/passwd console <console-mqtt-password>

# Add device users (one per device)
docker run --rm -v /opt/acuvim/mosquitto/config:/mosquitto/config \
  eclipse-mosquitto:2 \
  mosquitto_passwd -b /mosquitto/config/passwd ACV-AABBCCDDEEFF <device-password>
```

Repeat the second command for each device being deployed. The username should match the device_id.

### 6.3 ACL Configuration

Create `/opt/acuvim/mosquitto/config/acl`:

```
# ===========================================
# Mosquitto ACL Configuration
# ===========================================

# Console application: full access to all topics
user console
topic readwrite acuvim/#
topic readwrite devices/#

# Per-device access pattern:
# Each device can only access its own topics and the registration topic.
# The %u placeholder is replaced with the connecting username (device_id).

pattern readwrite acuvim/%u/#
pattern write devices/register
```

This ensures that device `ACV-AABBCCDDEEFF` can only publish and subscribe to topics under `acuvim/ACV-AABBCCDDEEFF/` and can publish to `devices/register`. The console user has unrestricted access.

### 6.4 TLS Certificates for MQTT

Option A -- Use Let's Encrypt (same certificate as the web server):

```bash
# After obtaining certificates with Certbot (see Section 8),
# copy them to the Mosquitto certs directory:
sudo cp /etc/letsencrypt/live/console.example.com/fullchain.pem \
  /opt/acuvim/mosquitto/certs/server.crt
sudo cp /etc/letsencrypt/live/console.example.com/privkey.pem \
  /opt/acuvim/mosquitto/certs/server.key
sudo cp /etc/letsencrypt/live/console.example.com/chain.pem \
  /opt/acuvim/mosquitto/certs/ca.crt
sudo chown 1883:1883 /opt/acuvim/mosquitto/certs/*
```

Option B -- Self-signed certificates (for internal/testing deployments):

```bash
# Generate CA
openssl req -new -x509 -days 3650 -extensions v3_ca \
  -keyout /opt/acuvim/mosquitto/certs/ca.key \
  -out /opt/acuvim/mosquitto/certs/ca.crt \
  -subj "/CN=Acuvim MQTT CA"

# Generate server key and CSR
openssl genrsa -out /opt/acuvim/mosquitto/certs/server.key 2048
openssl req -new \
  -key /opt/acuvim/mosquitto/certs/server.key \
  -out /opt/acuvim/mosquitto/certs/server.csr \
  -subj "/CN=mqtt.example.com"

# Sign server certificate
openssl x509 -req -days 3650 \
  -in /opt/acuvim/mosquitto/certs/server.csr \
  -CA /opt/acuvim/mosquitto/certs/ca.crt \
  -CAkey /opt/acuvim/mosquitto/certs/ca.key \
  -CAcreateserial \
  -out /opt/acuvim/mosquitto/certs/server.crt

# Set ownership
sudo chown -R 1883:1883 /opt/acuvim/mosquitto/certs/
```

If using self-signed certificates, the CA certificate (`ca.crt`) must be embedded in the ESP32 firmware for server verification.

---

## 7. Production Docker Compose

Create `/opt/acuvim/docker-compose.yml`:

```yaml
services:
  console:
    build: .
    restart: unless-stopped
    ports:
      - "127.0.0.1:5000:5000"
    env_file:
      - .env
    depends_on:
      db:
        condition: service_healthy
      mqtt:
        condition: service_started
    volumes:
      - firmware-data:/app/firmware
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

  db:
    image: postgres:16
    restart: unless-stopped
    env_file:
      - .env
    volumes:
      - postgres-data:/var/lib/postgresql/data
    # Do NOT expose port 5432 to the host in production
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U acuvim"]
      interval: 5s
      timeout: 5s
      retries: 5

  mqtt:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - "8883:8883"
      - "9883:9883"
    volumes:
      - /opt/acuvim/mosquitto/config:/mosquitto/config:ro
      - /opt/acuvim/mosquitto/data:/mosquitto/data
      - /opt/acuvim/mosquitto/log:/mosquitto/log
      - /opt/acuvim/mosquitto/certs:/mosquitto/certs:ro

volumes:
  postgres-data:
  firmware-data:
```

Key differences from the development docker-compose.yml:

- Console port is bound to `127.0.0.1` only (nginx handles external traffic).
- PostgreSQL port `5432` is not exposed to the host.
- MQTT exposes TLS port `8883` and WSS port `9883` instead of plaintext `1883` and `9001`.
- Environment variables are loaded from `.env` file.
- All services have `restart: unless-stopped`.
- Health checks are configured.

---

## 8. SSL/TLS with Let's Encrypt and Nginx

### 8.1 Obtain SSL Certificate

```bash
sudo certbot certonly --nginx -d console.example.com
```

### 8.2 Nginx Reverse Proxy Configuration

Create `/etc/nginx/sites-available/acuvim`:

```nginx
# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name console.example.com;
    return 301 https://$server_name$request_uri;
}

# HTTPS reverse proxy to console application
server {
    listen 443 ssl http2;
    server_name console.example.com;

    ssl_certificate /etc/letsencrypt/live/console.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/console.example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS
    add_header Strict-Transport-Security "max-age=63072000" always;

    # Security headers
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options DENY;
    add_header X-XSS-Protection "1; mode=block";
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';";

    # Max upload size (firmware binaries)
    client_max_body_size 10M;

    # API and frontend
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # SignalR WebSocket support
    location /hubs/ {
        proxy_pass http://127.0.0.1:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
    }

    # Firmware download (allow larger timeout for OTA)
    location /api/firmware/download/ {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300;
    }
}
```

Enable the site and reload nginx:

```bash
sudo ln -s /etc/nginx/sites-available/acuvim /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```

### 8.3 Auto-Renew Certificates

Certbot installs a systemd timer by default. Verify it is active:

```bash
sudo systemctl status certbot.timer
```

Add a renewal hook to copy certificates to Mosquitto and restart it:

```bash
sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy
```

Create `/etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh`:

```bash
#!/bin/bash
cp /etc/letsencrypt/live/console.example.com/fullchain.pem /opt/acuvim/mosquitto/certs/server.crt
cp /etc/letsencrypt/live/console.example.com/privkey.pem /opt/acuvim/mosquitto/certs/server.key
cp /etc/letsencrypt/live/console.example.com/chain.pem /opt/acuvim/mosquitto/certs/ca.crt
chown 1883:1883 /opt/acuvim/mosquitto/certs/*
cd /opt/acuvim && docker compose restart mqtt
```

```bash
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh
```

---

## 9. PostgreSQL Security

### 9.1 Change Default Passwords

Never use `secret` as the database password in production. Update the password in `/opt/acuvim/.env` and ensure the `ConnectionStrings__DefaultConnection` matches.

### 9.2 Enable SSL for PostgreSQL (Optional)

For additional security between the console container and the database container, configure PostgreSQL SSL. This is generally not required when both containers are on the same Docker network, but is recommended for compliance-sensitive deployments.

Add to the `db` service in `docker-compose.yml`:

```yaml
  db:
    image: postgres:16
    command: >
      -c ssl=on
      -c ssl_cert_file=/var/lib/postgresql/server.crt
      -c ssl_key_file=/var/lib/postgresql/server.key      
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - /opt/acuvim/db-certs/server.crt:/var/lib/postgresql/server.crt:ro
      - /opt/acuvim/db-certs/server.key:/var/lib/postgresql/server.key:ro
```

Update the connection string to require SSL:

```
ConnectionStrings__DefaultConnection=Host=db;Database=acuvim;Username=acuvim;Password=<password>;SSL Mode=Require;Trust Server Certificate=true
```

### 9.3 Restrict Network Access

The PostgreSQL container should NOT expose port 5432 to the host. The production `docker-compose.yml` above omits the `ports` directive for the `db` service, so it is only accessible from other containers on the same Docker network.

---

## 10. Database Backups

### 10.1 Backup Script

Create `/opt/acuvim/backup.sh`:

```bash
#!/bin/bash
set -e

BACKUP_DIR="/opt/acuvim/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/acuvim_${TIMESTAMP}.sql.gz"
RETENTION_DAYS=30

# Run pg_dump inside the running container and compress
docker compose -f /opt/acuvim/docker-compose.yml exec -T db \
  pg_dump -U acuvim -d acuvim --clean --if-exists | gzip > "${BACKUP_FILE}"

# Verify backup is not empty
if [ ! -s "${BACKUP_FILE}" ]; then
  echo "ERROR: Backup file is empty"
  rm -f "${BACKUP_FILE}"
  exit 1
fi

echo "Backup created: ${BACKUP_FILE} ($(du -h "${BACKUP_FILE}" | cut -f1))"

# Remove backups older than retention period
find "${BACKUP_DIR}" -name "acuvim_*.sql.gz" -mtime +${RETENTION_DAYS} -delete
echo "Cleaned up backups older than ${RETENTION_DAYS} days"
```

```bash
chmod +x /opt/acuvim/backup.sh
```

### 10.2 Schedule with Cron

```bash
# Run daily at 02:00 AM
crontab -e
```

Add the line:

```
0 2 * * * /opt/acuvim/backup.sh >> /opt/acuvim/backups/backup.log 2>&1
```

### 10.3 Restore from Backup

```bash
gunzip -c /opt/acuvim/backups/acuvim_20260101_020000.sql.gz | \
  docker compose -f /opt/acuvim/docker-compose.yml exec -T db \
  psql -U acuvim -d acuvim
```

### 10.4 Off-Site Backup (Optional)

Sync backups to a remote location:

```bash
# Using rsync to a remote server
rsync -avz /opt/acuvim/backups/ backup-server:/backups/acuvim/

# Or using rclone to S3-compatible storage
rclone sync /opt/acuvim/backups/ s3:acuvim-backups/
```

---

## 11. Firewall Configuration

Configure UFW to only allow necessary ports:

```bash
# Allow SSH (adjust port if non-standard)
sudo ufw allow 22/tcp

# Allow HTTPS (console web interface)
sudo ufw allow 443/tcp

# Allow MQTTS (device connections)
sudo ufw allow 8883/tcp

# Allow HTTP temporarily for Let's Encrypt challenges
sudo ufw allow 80/tcp

# Enable firewall
sudo ufw enable
sudo ufw status
```

Expected open ports:

| Port | Protocol | Purpose |
|------|----------|---------|
| 22 | TCP | SSH administration |
| 80 | TCP | HTTP (redirects to 443) |
| 443 | TCP | HTTPS (console web UI + API) |
| 8883 | TCP | MQTTS (device TLS connections) |

Ports that must NOT be exposed:

| Port | Reason |
|------|--------|
| 1883 | Plaintext MQTT (internal Docker network only) |
| 5000 | Console app (accessed via nginx reverse proxy) |
| 5432 | PostgreSQL (internal Docker network only) |
| 9001 | MQTT WebSocket plaintext (not used in production) |

---

## 12. Deployment Steps

### 12.1 Initial Deployment

```bash
# 1. Clone the repository to the server
cd /opt/acuvim
git clone <repository-url> .

# 2. Configure environment variables
cp .env.example .env
nano .env
# Set all production values (passwords, JWT secret, domain)

# 3. Set up Mosquitto configuration
# Copy mosquitto.conf, passwd, and acl files as described in Section 6

# 4. Obtain SSL certificates
sudo certbot certonly --nginx -d console.example.com

# 5. Copy certificates to Mosquitto
sudo /etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh

# 6. Build and start the stack
docker compose build
docker compose up -d

# 7. Apply database migrations
docker compose exec console dotnet ef database update

# 8. Create the initial admin user
docker compose exec console dotnet run -- seed-admin

# 9. Verify all services are running
docker compose ps
docker compose logs --tail=50
```

### 12.2 Updating the Application

```bash
cd /opt/acuvim

# Pull latest code
git pull

# Rebuild and restart the console (zero-downtime is not guaranteed)
docker compose build console
docker compose up -d console

# Apply any new database migrations
docker compose exec console dotnet ef database update
```

### 12.3 Viewing Logs

```bash
# All services
docker compose logs -f

# Console application only
docker compose logs -f console

# MQTT broker
docker compose logs -f mqtt

# Database
docker compose logs -f db

# Mosquitto log file
tail -f /opt/acuvim/mosquitto/log/mosquitto.log
```

---

## 13. Monitoring

### 13.1 Health Check Endpoint

The console application exposes a health check at `GET /health`. Use this with external monitoring services (UptimeRobot, Pingdom, or a self-hosted solution).

```bash
curl -s https://console.example.com/health
```

Expected response when healthy:

```json
{
  "status": "healthy",
  "checks": {
    "database": "healthy",
    "mqtt": "connected",
    "uptime": "3d 14h 22m"
  }
}
```

### 13.2 Docker Health Status

```bash
# Check container health status
docker compose ps

# Watch for container restarts
docker events --filter 'event=restart'
```

### 13.3 Disk Space Monitoring

Set up a simple disk space alert:

```bash
# Add to crontab: alert if disk usage exceeds 85%
0 */6 * * * df -h / | awk 'NR==2 && int($5)>85 {print "DISK WARNING: "$5" used"}' | mail -s "Acuvim Disk Alert" admin@example.com
```

### 13.4 Mosquitto Monitoring

Monitor the MQTT broker by subscribing to the system topic:

```bash
mosquitto_sub -h console.example.com -p 8883 \
  --cafile /opt/acuvim/mosquitto/certs/ca.crt \
  -u console -P <password> \
  -t '$SYS/#' -v
```

Key system topics:

| Topic | Description |
|-------|-------------|
| `$SYS/broker/clients/connected` | Number of currently connected clients |
| `$SYS/broker/messages/received` | Total messages received |
| `$SYS/broker/messages/sent` | Total messages sent |
| `$SYS/broker/load/messages/received/1min` | Message rate (1-minute average) |

### 13.5 Database Size Monitoring

```bash
docker compose exec db psql -U acuvim -d acuvim -c "
SELECT
  pg_size_pretty(pg_database_size('acuvim')) AS database_size,
  (SELECT count(*) FROM telemetry_records) AS telemetry_rows,
  (SELECT count(*) FROM devices) AS device_count,
  (SELECT count(*) FROM alerts WHERE resolved_at IS NULL) AS active_alerts;
"
```

---

## 14. Maintenance

### 14.1 Adding a New Device to MQTT

When commissioning a new device, add its MQTT credentials:

```bash
docker compose exec mqtt mosquitto_passwd -b /mosquitto/config/passwd ACV-<NEW_DEVICE_ID> <device-password>
docker compose restart mqtt
```

Also add a specific ACL entry if needed, or rely on the pattern-based ACL.

### 14.2 Telemetry Data Retention

The console application can be configured to automatically purge telemetry data older than a defined retention period. Set in `.env`:

```
Telemetry__RetentionDays=365
Telemetry__CleanupIntervalHours=24
```

For manual cleanup:

```bash
docker compose exec db psql -U acuvim -d acuvim -c "
DELETE FROM telemetry_records WHERE timestamp < NOW() - INTERVAL '365 days';
VACUUM ANALYZE telemetry_records;
"
```

### 14.3 Certificate Renewal

Let's Encrypt certificates auto-renew via the certbot timer. Verify the renewal process:

```bash
sudo certbot renew --dry-run
```

---

## 15. Production Checklist

### Infrastructure

- [ ] Server provisioned (minimum 2 CPU, 4 GB RAM, 50 GB SSD)
- [ ] Domain name configured with DNS A record
- [ ] Docker and Docker Compose installed
- [ ] SSL/TLS certificate obtained (Let's Encrypt)
- [ ] Nginx reverse proxy configured with HTTPS
- [ ] Firewall configured (ports 22, 80, 443, 8883)

### Database

- [ ] PostgreSQL running with strong password
- [ ] Database migrations applied
- [ ] Backup cron job scheduled and tested
- [ ] Database port NOT exposed to host

### MQTT Broker

- [ ] TLS enabled on port 8883
- [ ] Anonymous access disabled
- [ ] Console user created with password
- [ ] ACL file configured with per-device restrictions
- [ ] Plaintext port 1883 restricted to Docker internal network

### Console Application

- [ ] Environment variables set (JWT secret, DB password, MQTT credentials)
- [ ] Console port bound to 127.0.0.1 only (behind nginx)
- [ ] Admin user created
- [ ] CORS restricted to production domain
- [ ] Health check endpoint responding
- [ ] Firmware storage directory mounted

### Monitoring

- [ ] Health check endpoint monitored externally
- [ ] Disk space monitoring configured
- [ ] Backup verification process tested
- [ ] Log rotation configured