Tau.Acuvim/docs/deployment-guide.md

Production Deployment Guide

This guide covers deploying the Tau Acuvim console application, PostgreSQL database, and Mosquitto MQTT broker to a production server using Docker Compose.

---

1. Server Requirements

Resource	Minimum	Recommended
CPU	2 cores	4 cores
RAM	4 GB	8 GB
Storage	50 GB SSD	100 GB SSD
OS	Ubuntu 22.04 LTS	Ubuntu 24.04 LTS
Network	Static public IP	Static IP + domain name

The server must have a stable internet connection accessible to ESP32 devices over both WiFi and cellular networks.

---

2. Prerequisites

Install Docker and Docker Compose on the server:

# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Verify
docker --version
docker compose version

Install Certbot for SSL certificates:

sudo apt update
sudo apt install certbot python3-certbot-nginx nginx -y

---

3. Domain and DNS

Configure a DNS A record pointing your domain to the server's public IP:

console.example.com  ->  203.0.113.50

If you plan to use a subdomain for the MQTT broker's WebSocket interface, add that as well:

mqtt.example.com     ->  203.0.113.50

---

4. Directory Structure

Create the deployment directory on the server:

sudo mkdir -p /opt/acuvim
sudo mkdir -p /opt/acuvim/mosquitto/config
sudo mkdir -p /opt/acuvim/mosquitto/data
sudo mkdir -p /opt/acuvim/mosquitto/log
sudo mkdir -p /opt/acuvim/mosquitto/certs
sudo mkdir -p /opt/acuvim/firmware
sudo mkdir -p /opt/acuvim/backups
sudo mkdir -p /opt/acuvim/nginx

---

5. Environment Configuration

Create the production environment file at /opt/acuvim/.env:

# Database
POSTGRES_DB=acuvim
POSTGRES_USER=acuvim
POSTGRES_PASSWORD=<generate-strong-password-here>

# Console Application
ConnectionStrings__DefaultConnection=Host=db;Database=acuvim;Username=acuvim;Password=<same-password-as-above>
Jwt__Secret=<generate-256-bit-secret-minimum-32-characters>
Jwt__Issuer=Tau.Acuvim.Console
Jwt__ExpiryHours=24
Firmware__BaseUrl=https://console.example.com
Firmware__StoragePath=/app/firmware

# MQTT
Mqtt__Broker=mqtt
Mqtt__Port=1883
Mqtt__Username=console
Mqtt__Password=<generate-mqtt-console-password>
Mqtt__TopicPrefix=acuvim
Mqtt__ClientId=acuvim-console

Generate strong passwords:

# Generate a random 32-character password
openssl rand -base64 32

# Generate a 256-bit JWT secret
openssl rand -base64 48

Restrict file permissions:

chmod 600 /opt/acuvim/.env

---

6. Mosquitto MQTT Broker Configuration

6.1 Production Configuration

Create /opt/acuvim/mosquitto/config/mosquitto.conf:

# ===========================================
# Mosquitto Production Configuration
# ===========================================

# Persistence
persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
log_type error
log_type warning
log_type notice

# -------------------------------------------
# Listener 1: MQTTS (TLS on port 8883)
# -------------------------------------------
listener 8883
cafile /mosquitto/certs/ca.crt
certfile /mosquitto/certs/server.crt
keyfile /mosquitto/certs/server.key
tls_version tlsv1.2
require_certificate false

# -------------------------------------------
# Listener 2: Internal MQTT (port 1883)
# Only accessible within Docker network
# -------------------------------------------
listener 1883 0.0.0.0
# Internal listener for console <-> broker communication
# Not exposed to the host network in production

# -------------------------------------------
# Listener 3: WebSockets over TLS (port 9883)
# -------------------------------------------
listener 9883
protocol websockets
cafile /mosquitto/certs/ca.crt
certfile /mosquitto/certs/server.crt
keyfile /mosquitto/certs/server.key
tls_version tlsv1.2

# -------------------------------------------
# Authentication
# -------------------------------------------
allow_anonymous false
password_file /mosquitto/config/passwd
acl_file /mosquitto/config/acl

# -------------------------------------------
# Connection limits
# -------------------------------------------
max_connections 500
max_inflight_messages 20
max_queued_messages 1000
message_size_limit 10240

6.2 Per-Device Authentication

Create the password file using mosquitto_passwd:

# Create the initial password file with the console user
docker run --rm -v /opt/acuvim/mosquitto/config:/mosquitto/config \
  eclipse-mosquitto:2 \
  mosquitto_passwd -c -b /mosquitto/config/passwd console <console-mqtt-password>

# Add device users (one per device)
docker run --rm -v /opt/acuvim/mosquitto/config:/mosquitto/config \
  eclipse-mosquitto:2 \
  mosquitto_passwd -b /mosquitto/config/passwd ACV-AABBCCDDEEFF <device-password>

Repeat the second command for each device being deployed. The username should match the device_id.

6.3 ACL Configuration

Create /opt/acuvim/mosquitto/config/acl:

# ===========================================
# Mosquitto ACL Configuration
# ===========================================

# Console application: full access to all topics
user console
topic readwrite acuvim/#
topic readwrite devices/#

# Per-device access pattern:
# Each device can only access its own topics and the registration topic.
# The %u placeholder is replaced with the connecting username (device_id).

pattern readwrite acuvim/%u/#
pattern write devices/register

This ensures that device ACV-AABBCCDDEEFF can only publish and subscribe to topics under acuvim/ACV-AABBCCDDEEFF/ and can publish to devices/register. The console user has unrestricted access.

6.4 TLS Certificates for MQTT

Option A -- Use Let's Encrypt (same certificate as the web server):

# After obtaining certificates with Certbot (see Section 8),
# copy them to the Mosquitto certs directory:
sudo cp /etc/letsencrypt/live/console.example.com/fullchain.pem \
  /opt/acuvim/mosquitto/certs/server.crt
sudo cp /etc/letsencrypt/live/console.example.com/privkey.pem \
  /opt/acuvim/mosquitto/certs/server.key
sudo cp /etc/letsencrypt/live/console.example.com/chain.pem \
  /opt/acuvim/mosquitto/certs/ca.crt
sudo chown 1883:1883 /opt/acuvim/mosquitto/certs/*

Option B -- Self-signed certificates (for internal/testing deployments):

# Generate CA
openssl req -new -x509 -days 3650 -extensions v3_ca \
  -keyout /opt/acuvim/mosquitto/certs/ca.key \
  -out /opt/acuvim/mosquitto/certs/ca.crt \
  -subj "/CN=Acuvim MQTT CA"

# Generate server key and CSR
openssl genrsa -out /opt/acuvim/mosquitto/certs/server.key 2048
openssl req -new \
  -key /opt/acuvim/mosquitto/certs/server.key \
  -out /opt/acuvim/mosquitto/certs/server.csr \
  -subj "/CN=mqtt.example.com"

# Sign server certificate
openssl x509 -req -days 3650 \
  -in /opt/acuvim/mosquitto/certs/server.csr \
  -CA /opt/acuvim/mosquitto/certs/ca.crt \
  -CAkey /opt/acuvim/mosquitto/certs/ca.key \
  -CAcreateserial \
  -out /opt/acuvim/mosquitto/certs/server.crt

# Set ownership
sudo chown -R 1883:1883 /opt/acuvim/mosquitto/certs/

If using self-signed certificates, the CA certificate (ca.crt) must be embedded in the ESP32 firmware for server verification.

---

7. Production Docker Compose

Create /opt/acuvim/docker-compose.yml:

services:
  console:
    build: .
    restart: unless-stopped
    ports:
      - "127.0.0.1:5000:5000"
    env_file:
      - .env
    depends_on:
      db:
        condition: service_healthy
      mqtt:
        condition: service_started
    volumes:
      - firmware-data:/app/firmware
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

  db:
    image: postgres:16
    restart: unless-stopped
    env_file:
      - .env
    volumes:
      - postgres-data:/var/lib/postgresql/data
    # Do NOT expose port 5432 to the host in production
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U acuvim"]
      interval: 5s
      timeout: 5s
      retries: 5

  mqtt:
    image: eclipse-mosquitto:2
    restart: unless-stopped
    ports:
      - "8883:8883"
      - "9883:9883"
    volumes:
      - /opt/acuvim/mosquitto/config:/mosquitto/config:ro
      - /opt/acuvim/mosquitto/data:/mosquitto/data
      - /opt/acuvim/mosquitto/log:/mosquitto/log
      - /opt/acuvim/mosquitto/certs:/mosquitto/certs:ro

volumes:
  postgres-data:
  firmware-data:

Key differences from the development docker-compose.yml:

• Console port is bound to 127.0.0.1 only (nginx handles external traffic).
• PostgreSQL port 5432 is not exposed to the host.
• MQTT exposes TLS port 8883 and WSS port 9883 instead of plaintext 1883 and 9001.
• Environment variables are loaded from .env file.
• All services have restart: unless-stopped.
• Health checks are configured.

---

8. SSL/TLS with Let's Encrypt and Nginx

8.1 Obtain SSL Certificate

sudo certbot certonly --nginx -d console.example.com

8.2 Nginx Reverse Proxy Configuration

Create /etc/nginx/sites-available/acuvim:

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name console.example.com;
    return 301 https://$server_name$request_uri;
}

# HTTPS reverse proxy to console application
server {
    listen 443 ssl http2;
    server_name console.example.com;

    ssl_certificate /etc/letsencrypt/live/console.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/console.example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    # HSTS
    add_header Strict-Transport-Security "max-age=63072000" always;

    # Security headers
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options DENY;
    add_header X-XSS-Protection "1; mode=block";
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';";

    # Max upload size (firmware binaries)
    client_max_body_size 10M;

    # API and frontend
    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # SignalR WebSocket support
    location /hubs/ {
        proxy_pass http://127.0.0.1:5000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 86400;
    }

    # Firmware download (allow larger timeout for OTA)
    location /api/firmware/download/ {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300;
    }
}

Enable the site and reload nginx:

sudo ln -s /etc/nginx/sites-available/acuvim /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

8.3 Auto-Renew Certificates

Certbot installs a systemd timer by default. Verify it is active:

sudo systemctl status certbot.timer

Add a renewal hook to copy certificates to Mosquitto and restart it:

sudo mkdir -p /etc/letsencrypt/renewal-hooks/deploy

Create /etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh:

#!/bin/bash
cp /etc/letsencrypt/live/console.example.com/fullchain.pem /opt/acuvim/mosquitto/certs/server.crt
cp /etc/letsencrypt/live/console.example.com/privkey.pem /opt/acuvim/mosquitto/certs/server.key
cp /etc/letsencrypt/live/console.example.com/chain.pem /opt/acuvim/mosquitto/certs/ca.crt
chown 1883:1883 /opt/acuvim/mosquitto/certs/*
cd /opt/acuvim && docker compose restart mqtt

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh

---

9. PostgreSQL Security

9.1 Change Default Passwords

Never use secret as the database password in production. Update the password in /opt/acuvim/.env and ensure the ConnectionStrings__DefaultConnection matches.

9.2 Enable SSL for PostgreSQL (Optional)

For additional security between the console container and the database container, configure PostgreSQL SSL. This is generally not required when both containers are on the same Docker network, but is recommended for compliance-sensitive deployments.

Add to the db service in docker-compose.yml:

  db:
    image: postgres:16
    command: >
      -c ssl=on
      -c ssl_cert_file=/var/lib/postgresql/server.crt
      -c ssl_key_file=/var/lib/postgresql/server.key      
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - /opt/acuvim/db-certs/server.crt:/var/lib/postgresql/server.crt:ro
      - /opt/acuvim/db-certs/server.key:/var/lib/postgresql/server.key:ro

Update the connection string to require SSL:

ConnectionStrings__DefaultConnection=Host=db;Database=acuvim;Username=acuvim;Password=<password>;SSL Mode=Require;Trust Server Certificate=true

9.3 Restrict Network Access

The PostgreSQL container should NOT expose port 5432 to the host. The production docker-compose.yml above omits the ports directive for the db service, so it is only accessible from other containers on the same Docker network.

---

10. Database Backups

10.1 Backup Script

Create /opt/acuvim/backup.sh:

#!/bin/bash
set -e

BACKUP_DIR="/opt/acuvim/backups"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="${BACKUP_DIR}/acuvim_${TIMESTAMP}.sql.gz"
RETENTION_DAYS=30

# Run pg_dump inside the running container and compress
docker compose -f /opt/acuvim/docker-compose.yml exec -T db \
  pg_dump -U acuvim -d acuvim --clean --if-exists | gzip > "${BACKUP_FILE}"

# Verify backup is not empty
if [ ! -s "${BACKUP_FILE}" ]; then
  echo "ERROR: Backup file is empty"
  rm -f "${BACKUP_FILE}"
  exit 1
fi

echo "Backup created: ${BACKUP_FILE} ($(du -h "${BACKUP_FILE}" | cut -f1))"

# Remove backups older than retention period
find "${BACKUP_DIR}" -name "acuvim_*.sql.gz" -mtime +${RETENTION_DAYS} -delete
echo "Cleaned up backups older than ${RETENTION_DAYS} days"

chmod +x /opt/acuvim/backup.sh

10.2 Schedule with Cron

# Run daily at 02:00 AM
crontab -e

Add the line:

0 2 * * * /opt/acuvim/backup.sh >> /opt/acuvim/backups/backup.log 2>&1

10.3 Restore from Backup

gunzip -c /opt/acuvim/backups/acuvim_20260101_020000.sql.gz | \
  docker compose -f /opt/acuvim/docker-compose.yml exec -T db \
  psql -U acuvim -d acuvim

10.4 Off-Site Backup (Optional)

Sync backups to a remote location:

# Using rsync to a remote server
rsync -avz /opt/acuvim/backups/ backup-server:/backups/acuvim/

# Or using rclone to S3-compatible storage
rclone sync /opt/acuvim/backups/ s3:acuvim-backups/

---

11. Firewall Configuration

Configure UFW to only allow necessary ports:

# Allow SSH (adjust port if non-standard)
sudo ufw allow 22/tcp

# Allow HTTPS (console web interface)
sudo ufw allow 443/tcp

# Allow MQTTS (device connections)
sudo ufw allow 8883/tcp

# Allow HTTP temporarily for Let's Encrypt challenges
sudo ufw allow 80/tcp

# Enable firewall
sudo ufw enable
sudo ufw status

Expected open ports:

Port	Protocol	Purpose
22	TCP	SSH administration
80	TCP	HTTP (redirects to 443)
443	TCP	HTTPS (console web UI + API)
8883	TCP	MQTTS (device TLS connections)

Ports that must NOT be exposed:

Port	Reason
1883	Plaintext MQTT (internal Docker network only)
5000	Console app (accessed via nginx reverse proxy)
5432	PostgreSQL (internal Docker network only)
9001	MQTT WebSocket plaintext (not used in production)

---

12. Deployment Steps

12.1 Initial Deployment

# 1. Clone the repository to the server
cd /opt/acuvim
git clone <repository-url> .

# 2. Configure environment variables
cp .env.example .env
nano .env
# Set all production values (passwords, JWT secret, domain)

# 3. Set up Mosquitto configuration
# Copy mosquitto.conf, passwd, and acl files as described in Section 6

# 4. Obtain SSL certificates
sudo certbot certonly --nginx -d console.example.com

# 5. Copy certificates to Mosquitto
sudo /etc/letsencrypt/renewal-hooks/deploy/acuvim-mqtt.sh

# 6. Build and start the stack
docker compose build
docker compose up -d

# 7. Apply database migrations
docker compose exec console dotnet ef database update

# 8. Create the initial admin user
docker compose exec console dotnet run -- seed-admin

# 9. Verify all services are running
docker compose ps
docker compose logs --tail=50

12.2 Updating the Application

cd /opt/acuvim

# Pull latest code
git pull

# Rebuild and restart the console (zero-downtime is not guaranteed)
docker compose build console
docker compose up -d console

# Apply any new database migrations
docker compose exec console dotnet ef database update

12.3 Viewing Logs

# All services
docker compose logs -f

# Console application only
docker compose logs -f console

# MQTT broker
docker compose logs -f mqtt

# Database
docker compose logs -f db

# Mosquitto log file
tail -f /opt/acuvim/mosquitto/log/mosquitto.log

---

13. Monitoring

13.1 Health Check Endpoint

The console application exposes a health check at GET /health. Use this with external monitoring services (UptimeRobot, Pingdom, or a self-hosted solution).

curl -s https://console.example.com/health

Expected response when healthy:

{
  "status": "healthy",
  "checks": {
    "database": "healthy",
    "mqtt": "connected",
    "uptime": "3d 14h 22m"
  }
}

13.2 Docker Health Status

# Check container health status
docker compose ps

# Watch for container restarts
docker events --filter 'event=restart'

13.3 Disk Space Monitoring

Set up a simple disk space alert:

# Add to crontab: alert if disk usage exceeds 85%
0 */6 * * * df -h / | awk 'NR==2 && int($5)>85 {print "DISK WARNING: "$5" used"}' | mail -s "Acuvim Disk Alert" admin@example.com

13.4 Mosquitto Monitoring

Monitor the MQTT broker by subscribing to the system topic:

mosquitto_sub -h console.example.com -p 8883 \
  --cafile /opt/acuvim/mosquitto/certs/ca.crt \
  -u console -P <password> \
  -t '$SYS/#' -v

Key system topics:

Topic	Description
$SYS/broker/clients/connected	Number of currently connected clients
$SYS/broker/messages/received	Total messages received
$SYS/broker/messages/sent	Total messages sent
$SYS/broker/load/messages/received/1min	Message rate (1-minute average)

13.5 Database Size Monitoring

docker compose exec db psql -U acuvim -d acuvim -c "
SELECT
  pg_size_pretty(pg_database_size('acuvim')) AS database_size,
  (SELECT count(*) FROM telemetry_records) AS telemetry_rows,
  (SELECT count(*) FROM devices) AS device_count,
  (SELECT count(*) FROM alerts WHERE resolved_at IS NULL) AS active_alerts;
"

---

14. Maintenance

14.1 Adding a New Device to MQTT

When commissioning a new device, add its MQTT credentials:

docker compose exec mqtt mosquitto_passwd -b /mosquitto/config/passwd ACV-<NEW_DEVICE_ID> <device-password>
docker compose restart mqtt

Also add a specific ACL entry if needed, or rely on the pattern-based ACL.

14.2 Telemetry Data Retention

The console application can be configured to automatically purge telemetry data older than a defined retention period. Set in .env:

Telemetry__RetentionDays=365
Telemetry__CleanupIntervalHours=24

For manual cleanup:

docker compose exec db psql -U acuvim -d acuvim -c "
DELETE FROM telemetry_records WHERE timestamp < NOW() - INTERVAL '365 days';
VACUUM ANALYZE telemetry_records;
"

14.3 Certificate Renewal

Let's Encrypt certificates auto-renew via the certbot timer. Verify the renewal process:

sudo certbot renew --dry-run

---

15. Production Checklist

Infrastructure

• Server provisioned (minimum 2 CPU, 4 GB RAM, 50 GB SSD)
• Domain name configured with DNS A record
• Docker and Docker Compose installed
• SSL/TLS certificate obtained (Let's Encrypt)
• Nginx reverse proxy configured with HTTPS
• Firewall configured (ports 22, 80, 443, 8883)

Database

• PostgreSQL running with strong password
• Database migrations applied
• Backup cron job scheduled and tested
• Database port NOT exposed to host

MQTT Broker

• TLS enabled on port 8883
• Anonymous access disabled
• Console user created with password
• ACL file configured with per-device restrictions
• Plaintext port 1883 restricted to Docker internal network

Console Application

• Environment variables set (JWT secret, DB password, MQTT credentials)
• Console port bound to 127.0.0.1 only (behind nginx)
• Admin user created
• CORS restricted to production domain
• Health check endpoint responding
• Firmware storage directory mounted

Monitoring

• Health check endpoint monitored externally
• Disk space monitoring configured
• Backup verification process tested
• Log rotation configured