Tau.Acuvim/docs/acuvim-spec-08.md

# Phase 8: Console Application Backend

## Objective

Build the server-side backend for the console application that manages the ESP32 device fleet. The backend handles device registration, configuration management, telemetry ingestion, OTA firmware distribution, health monitoring, and exposes a REST API for the frontend (Phase 9).

## Prerequisites

- Phase 7 complete (device firmware feature-complete)
- Server/VPS for hosting (Linux recommended)
- Domain name (optional, can use IP for development)
- PostgreSQL database
- Mosquitto MQTT broker

## Deliverables

1. .NET 10 Web API application
2. PostgreSQL database schema and migrations
3. MQTT client bridge (subscribe to device topics, publish commands)
4. REST API for frontend and external integrations
5. Firmware binary hosting and version management
6. Authentication and authorization
7. Deployment configuration (Docker)

---

## 8.1 Technology Stack

| Component | Technology | Justification |
|-----------|-----------|---------------|
| Runtime | .NET 10 | Strong typing, high performance, built-in DI |
| Web Framework | ASP.NET Core Minimal API | Lightweight, fast, less boilerplate |
| Database | PostgreSQL 16 | Robust, JSON support, time-series extensions |
| ORM | Entity Framework Core 8 | Migrations, LINQ queries, strong typing |
| MQTT Client | MQTTnet | .NET-native, high-performance MQTT v5 client |
| Auth | ASP.NET Identity + JWT | Built-in user management, token-based API auth |
| File Storage | Local filesystem + optional S3 | Firmware binaries |
| Caching | In-memory + optional Redis | Device status, frequent lookups |
| Logging | Serilog | Structured logging |
| API Docs | Swagger / OpenAPI | Auto-generated API documentation |
| Containerization | Docker + Docker Compose | Reproducible deployment |

## 8.2 Project Structure

```
console/
├── src/
│   ├── Tau.Acuvim.Console/              # Main Web API project
│   │   ├── Program.cs
│   │   ├── appsettings.json
│   │   ├── Controllers/                  # API endpoints (if using controllers)
│   │   ├── Endpoints/                    # Minimal API endpoint groups
│   │   │   ├── DeviceEndpoints.cs
│   │   │   ├── TelemetryEndpoints.cs
│   │   │   ├── FirmwareEndpoints.cs
│   │   │   ├── ConfigEndpoints.cs
│   │   │   ├── AlertEndpoints.cs
│   │   │   └── AuthEndpoints.cs
│   │   ├── Services/
│   │   │   ├── MqttBridgeService.cs      # MQTT client (background service)
│   │   │   ├── DeviceService.cs
│   │   │   ├── TelemetryService.cs
│   │   │   ├── FirmwareService.cs
│   │   │   ├── CommandService.cs
│   │   │   ├── HealthMonitorService.cs
│   │   │   └── AlertService.cs
│   │   ├── Models/
│   │   │   ├── Device.cs
│   │   │   ├── TelemetryRecord.cs
│   │   │   ├── FirmwareVersion.cs
│   │   │   ├── DeviceConfig.cs
│   │   │   ├── Alert.cs
│   │   │   ├── Command.cs
│   │   │   └── User.cs
│   │   ├── Data/
│   │   │   ├── AppDbContext.cs
│   │   │   └── Migrations/
│   │   ├── DTOs/                          # Request/response models
│   │   ├── Middleware/
│   │   │   └── ApiKeyMiddleware.cs
│   │   └── Hubs/
│   │       └── DeviceHub.cs              # SignalR for real-time UI updates
│   │
│   └── Tau.Acuvim.Console.Tests/         # Unit and integration tests
│
├── docker-compose.yml
├── Dockerfile
└── README.md
```

## 8.3 Database Schema

### Entity Relationship

```
Devices ──< TelemetryRecords
Devices ──< Alerts
Devices ──< Commands
Devices ──< DeviceConfigs (1:1 current, history)
FirmwareVersions (standalone)
Users (standalone)
```

### Tables

#### `devices`

```sql
CREATE TABLE devices (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id       VARCHAR(32) UNIQUE NOT NULL,    -- e.g., "ACV-AABBCCDDEEFF"
    name            VARCHAR(128),                    -- Friendly name
    description     TEXT,
    mac_address     VARCHAR(17),
    imei            VARCHAR(15),
    iccid           VARCHAR(20),
    hardware        VARCHAR(64),                     -- "TTGO T-Call v1.4"
    firmware_version VARCHAR(16),
    capabilities    JSONB,                           -- ["wifi","gsm","sd","modbus"]
    status          VARCHAR(16) DEFAULT 'offline',   -- online, offline, degraded
    last_heartbeat  TIMESTAMPTZ,
    last_telemetry  TIMESTAMPTZ,
    connection_type VARCHAR(8),                      -- wifi, gsm
    signal_strength INTEGER,
    ip_address      VARCHAR(45),
    boot_count      INTEGER DEFAULT 0,
    group_id        UUID REFERENCES device_groups(id),
    tags            JSONB DEFAULT '[]',
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    updated_at      TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_devices_device_id ON devices(device_id);
CREATE INDEX idx_devices_status ON devices(status);
```

#### `telemetry_records`

```sql
CREATE TABLE telemetry_records (
    id          BIGSERIAL PRIMARY KEY,
    device_id   VARCHAR(32) NOT NULL REFERENCES devices(device_id),
    timestamp   TIMESTAMPTZ NOT NULL,
    data        JSONB NOT NULL,                      -- Full telemetry payload
    source      VARCHAR(8) DEFAULT 'live',           -- live, sd (buffered)
    connection  VARCHAR(8),                          -- wifi, gsm
    received_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_telemetry_device_ts ON telemetry_records(device_id, timestamp DESC);
CREATE INDEX idx_telemetry_timestamp ON telemetry_records(timestamp DESC);

-- Partitioning by month (for large-scale deployments)
-- Consider TimescaleDB extension for automatic time-series management
```

#### `alerts`

```sql
CREATE TABLE alerts (
    id          BIGSERIAL PRIMARY KEY,
    device_id   VARCHAR(32) NOT NULL REFERENCES devices(device_id),
    alert_type  VARCHAR(32) NOT NULL,                -- overvoltage, modbus_loss, etc.
    severity    VARCHAR(16) NOT NULL,                -- info, warning, critical
    message     TEXT NOT NULL,
    value       DOUBLE PRECISION,
    threshold   DOUBLE PRECISION,
    metadata    JSONB,                               -- Additional context
    acknowledged BOOLEAN DEFAULT FALSE,
    acknowledged_by VARCHAR(128),
    acknowledged_at TIMESTAMPTZ,
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    resolved_at TIMESTAMPTZ
);

CREATE INDEX idx_alerts_device ON alerts(device_id, created_at DESC);
CREATE INDEX idx_alerts_unresolved ON alerts(device_id) WHERE resolved_at IS NULL;
```

#### `commands`

```sql
CREATE TABLE commands (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    device_id   VARCHAR(32) NOT NULL REFERENCES devices(device_id),
    request_id  VARCHAR(64) UNIQUE NOT NULL,
    command     VARCHAR(32) NOT NULL,
    params      JSONB,
    status      VARCHAR(16) DEFAULT 'pending',       -- pending, sent, ack, success, error, timeout
    response    JSONB,
    created_by  VARCHAR(128),
    created_at  TIMESTAMPTZ DEFAULT NOW(),
    sent_at     TIMESTAMPTZ,
    completed_at TIMESTAMPTZ
);

CREATE INDEX idx_commands_device ON commands(device_id, created_at DESC);
CREATE INDEX idx_commands_pending ON commands(status) WHERE status IN ('pending', 'sent');
```

#### `firmware_versions`

```sql
CREATE TABLE firmware_versions (
    id           UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    version      VARCHAR(16) UNIQUE NOT NULL,         -- "1.2.0"
    filename     VARCHAR(128) NOT NULL,
    file_path    VARCHAR(256) NOT NULL,
    file_size    INTEGER NOT NULL,
    checksum     VARCHAR(128) NOT NULL,               -- SHA256
    release_notes TEXT,
    hardware     VARCHAR(64),                          -- Target hardware (null = all)
    mandatory    BOOLEAN DEFAULT FALSE,
    active       BOOLEAN DEFAULT TRUE,                 -- Available for deployment
    uploaded_by  VARCHAR(128),
    created_at   TIMESTAMPTZ DEFAULT NOW()
);
```

#### `device_groups`

```sql
CREATE TABLE device_groups (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name        VARCHAR(64) UNIQUE NOT NULL,
    description TEXT,
    created_at  TIMESTAMPTZ DEFAULT NOW()
);
```

## 8.4 MQTT Bridge Service

### `MqttBridgeService.cs`

A background service that maintains an MQTT connection to the broker, subscribes to all device topics, processes incoming messages, and publishes commands.

```csharp
public class MqttBridgeService : BackgroundService
{
    // Subscribe to:
    //   acuvim/+/telemetry     -> TelemetryService.Ingest()
    //   acuvim/+/heartbeat     -> HealthMonitorService.ProcessHeartbeat()
    //   acuvim/+/resp          -> CommandService.ProcessResponse()
    //   acuvim/+/status        -> DeviceService.UpdateStatus()
    //   acuvim/+/alerts        -> AlertService.ProcessAlert()
    //   devices/register       -> DeviceService.Register()

    // Publish to:
    //   acuvim/{device_id}/cmd -> CommandService.Send()
}
```

### Topic Routing

```csharp
private async Task HandleMessage(MqttApplicationMessage msg)
{
    var topic = msg.Topic;
    var payload = Encoding.UTF8.GetString(msg.PayloadSegment);

    // Parse topic: acuvim/{device_id}/{type}
    var parts = topic.Split('/');

    if (topic == "devices/register")
    {
        await _deviceService.Register(payload);
        return;
    }

    if (parts.Length >= 3)
    {
        var deviceId = parts[1];
        var type = parts[2];

        switch (type)
        {
            case "telemetry":
                await _telemetryService.Ingest(deviceId, payload);
                break;
            case "heartbeat":
                await _healthMonitor.ProcessHeartbeat(deviceId, payload);
                break;
            case "resp":
                await _commandService.ProcessResponse(deviceId, payload);
                break;
            case "status":
                await _deviceService.UpdateConnectionStatus(deviceId, payload);
                break;
            case "alerts":
                await _alertService.ProcessAlert(deviceId, payload);
                break;
        }
    }
}
```

## 8.5 REST API Endpoints

### Devices

```
GET    /api/devices                    List all devices (with filtering, pagination)
GET    /api/devices/{deviceId}         Get device details
PUT    /api/devices/{deviceId}         Update device (name, description, group, tags)
DELETE /api/devices/{deviceId}         Remove device from console

GET    /api/devices/{deviceId}/status  Get current device status
POST   /api/devices/{deviceId}/command Send command to device
GET    /api/devices/{deviceId}/commands Command history
```

### Telemetry

```
GET    /api/devices/{deviceId}/telemetry        Latest telemetry
GET    /api/devices/{deviceId}/telemetry/history Historical data (with time range, aggregation)
GET    /api/devices/{deviceId}/telemetry/export  Export as CSV
```

### Firmware

```
GET    /api/firmware                   List all firmware versions
POST   /api/firmware                   Upload new firmware version
GET    /api/firmware/{version}         Get firmware details
DELETE /api/firmware/{version}         Delete firmware version
GET    /api/firmware/download/{version} Download firmware binary (used by devices)
GET    /api/firmware/check             Check for update (used by devices)
POST   /api/firmware/deploy            Deploy firmware to device(s)
```

### Alerts

```
GET    /api/alerts                     List alerts (with filtering)
GET    /api/devices/{deviceId}/alerts  Device-specific alerts
PUT    /api/alerts/{id}/acknowledge    Acknowledge alert
```

### Configuration

```
GET    /api/devices/{deviceId}/config           Get device configuration
POST   /api/devices/{deviceId}/config           Push configuration to device
GET    /api/devices/{deviceId}/config/wifi/scan  Trigger WiFi scan on device
```

### Groups

```
GET    /api/groups                     List device groups
POST   /api/groups                     Create group
PUT    /api/groups/{id}                Update group
DELETE /api/groups/{id}                Delete group
POST   /api/groups/{id}/command        Send command to all devices in group
POST   /api/groups/{id}/deploy         Deploy firmware to group
```

### Auth

```
POST   /api/auth/login                 Login, get JWT token
POST   /api/auth/register              Register new user (admin only)
POST   /api/auth/refresh               Refresh JWT token
GET    /api/auth/me                    Get current user info
```

### Dashboard

```
GET    /api/dashboard/summary          Fleet summary (total, online, offline, alerts)
GET    /api/dashboard/stats            Aggregate statistics
```

## 8.6 Command Service Flow

### Sending a Command

```
1. Frontend calls POST /api/devices/{deviceId}/command
   Body: { "command": "wifi_scan" }
2. CommandService generates request_id, saves to DB (status: pending)
3. CommandService publishes to MQTT: acuvim/{deviceId}/cmd
4. Device receives command, processes, publishes to acuvim/{deviceId}/resp
5. MqttBridge receives response
6. CommandService matches request_id, updates DB (status: success, response: {...})
7. SignalR notifies frontend of command completion
8. If no response in 60 seconds: mark as timeout
```

### Command Timeout

```csharp
public class CommandTimeoutService : BackgroundService
{
    // Every 30 seconds, check for commands where:
    //   status == 'sent' AND sent_at < (now - 60 seconds)
    // Mark as 'timeout'
}
```

## 8.7 Health Monitor Service

### `HealthMonitorService.cs`

Background service that monitors device health based on heartbeats:

```csharp
public class HealthMonitorService : BackgroundService
{
    // Every 30 seconds:
    // - Check all devices where last_heartbeat < (now - 3 * heartbeat_interval)
    //   -> Mark as "degraded"
    // - Check all devices where last_heartbeat < (now - 5 * heartbeat_interval)
    //   -> Mark as "offline"
    // - Optionally send notification (email, webhook) for offline devices

    public async Task ProcessHeartbeat(string deviceId, string payload)
    {
        // Parse heartbeat JSON
        // Update device: status = "online", last_heartbeat, firmware_version,
        //                connection_type, signal_strength, ip_address, boot_count
        // Update health metrics
        // Check for anomalies (high error rate, low memory, brownout resets)
        // Notify frontend via SignalR
    }
}
```

## 8.8 Firmware Service

### Upload Flow

```csharp
public async Task<FirmwareVersion> Upload(IFormFile file, string version,
    string releaseNotes, bool mandatory)
{
    // 1. Validate file is a valid ESP32 binary (check magic bytes)
    // 2. Calculate SHA256 checksum
    // 3. Save to /firmware/{version}/firmware.bin
    // 4. Create DB record
    // 5. Return firmware version info
}
```

### Deployment Flow

```csharp
public async Task Deploy(string version, List<string> deviceIds)
{
    var firmware = await GetVersion(version);

    foreach (var deviceId in deviceIds)
    {
        await _commandService.Send(deviceId, "ota_update", new {
            url = $"{_baseUrl}/api/firmware/download/{version}",
            version = firmware.Version,
            checksum = firmware.Checksum
        });
    }
}
```

### Device Check Endpoint

```csharp
// GET /api/firmware/check?device_id=X&current_version=Y&hardware=Z
public async Task<FirmwareCheckResponse> Check(string deviceId,
    string currentVersion, string hardware)
{
    var latest = await _db.FirmwareVersions
        .Where(f => f.Active)
        .Where(f => f.Hardware == null || f.Hardware == hardware)
        .OrderByDescending(f => f.CreatedAt)
        .FirstOrDefaultAsync();

    if (latest != null && IsNewer(latest.Version, currentVersion))
    {
        return new FirmwareCheckResponse
        {
            UpdateAvailable = true,
            Version = latest.Version,
            Url = $"{_baseUrl}/api/firmware/download/{latest.Version}",
            Size = latest.FileSize,
            Checksum = latest.Checksum,
            ReleaseNotes = latest.ReleaseNotes,
            Mandatory = latest.Mandatory
        };
    }

    return new FirmwareCheckResponse { UpdateAvailable = false };
}
```

## 8.9 Real-Time Updates (SignalR)

### `DeviceHub.cs`

SignalR hub for pushing real-time updates to the frontend:

```csharp
public class DeviceHub : Hub
{
    // Methods the server calls on clients:
    // - DeviceStatusChanged(deviceId, status)
    // - TelemetryReceived(deviceId, data)
    // - HeartbeatReceived(deviceId, data)
    // - AlertCreated(deviceId, alert)
    // - CommandCompleted(deviceId, requestId, response)
    // - DeviceRegistered(device)
}
```

The frontend subscribes to these events for live dashboard updates without polling.

## 8.10 Authentication

### JWT-Based Auth

- Users register/login to get a JWT token
- Token included in `Authorization: Bearer <token>` header
- Token expiry: 24 hours
- Refresh token: 7 days
- Roles: `Admin`, `Operator`, `Viewer`

### Device API Key

Devices authenticate firmware downloads using their device_id as an API key or via MQTT credentials. The firmware check/download endpoints require either a valid JWT or a valid device_id.

## 8.11 Docker Deployment

### `docker-compose.yml`

```yaml
version: '3.8'

services:
  console:
    build: .
    ports:
      - "5000:5000"
    environment:
      - ConnectionStrings__DefaultConnection=Host=db;Database=acuvim;Username=acuvim;Password=secret
      - Mqtt__Broker=mqtt
      - Mqtt__Port=1883
    depends_on:
      - db
      - mqtt
    volumes:
      - firmware-data:/app/firmware

  db:
    image: postgres:16
    environment:
      - POSTGRES_DB=acuvim
      - POSTGRES_USER=acuvim
      - POSTGRES_PASSWORD=secret
    volumes:
      - postgres-data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  mqtt:
    image: eclipse-mosquitto:2
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - mosquitto-data:/mosquitto/data
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf

volumes:
  postgres-data:
  mosquitto-data:
  firmware-data:
```

## 8.12 Configuration

### `appsettings.json`

```json
{
  "ConnectionStrings": {
    "DefaultConnection": "Host=localhost;Database=acuvim;Username=acuvim;Password=secret"
  },
  "Mqtt": {
    "Broker": "localhost",
    "Port": 1883,
    "Username": "console",
    "Password": "",
    "TopicPrefix": "acuvim",
    "ClientId": "acuvim-console"
  },
  "Firmware": {
    "StoragePath": "./firmware",
    "BaseUrl": "https://console.example.com"
  },
  "Jwt": {
    "Secret": "your-secret-key-min-32-chars-long-here",
    "Issuer": "Tau.Acuvim.Console",
    "ExpiryHours": 24
  },
  "HealthMonitor": {
    "CheckIntervalSeconds": 30,
    "DegradedThresholdMultiplier": 3,
    "OfflineThresholdMultiplier": 5
  },
  "Telemetry": {
    "RetentionDays": 365,
    "CleanupIntervalHours": 24
  }
}
```

## 8.13 Testing & Validation

| Test | Method | Pass Criteria |
|------|--------|---------------|
| API starts | `dotnet run` | Swagger UI at /swagger |
| DB migration | `dotnet ef database update` | Tables created |
| MQTT connection | Start broker, run console | MQTT connected, subscribed |
| Device registration | Publish registration via MQTT | Device appears in DB |
| Telemetry ingest | Publish telemetry via MQTT | Record saved, queryable via API |
| Heartbeat processing | Publish heartbeat | Device status updated |
| Health monitor | Stop device heartbeats | Device marked degraded then offline |
| Send command | POST /api/devices/{id}/command | Command published to MQTT, response received |
| Firmware upload | POST /api/firmware with binary | File saved, DB record created |
| Firmware check | GET /api/firmware/check | Correct update response |
| Firmware download | GET /api/firmware/download/{v} | Binary downloaded |
| Deploy firmware | POST /api/firmware/deploy | OTA command sent to devices |
| Auth login | POST /api/auth/login | JWT token returned |
| Auth protected | GET /api/devices without token | 401 Unauthorized |
| SignalR | Connect from browser | Real-time events received |
| Docker | `docker-compose up` | All services start, communicate |
| Alert processing | Publish alert via MQTT | Alert saved, queryable |

## 8.14 Phase 8 Completion Criteria

- [ ] .NET 10 Web API project builds and runs
- [ ] PostgreSQL schema migrated, all tables created
- [ ] MQTT bridge subscribes to all device topics
- [ ] Device registration works (MQTT -> DB)
- [ ] Telemetry ingestion works (MQTT -> DB -> API)
- [ ] Heartbeat processing updates device status
- [ ] Health monitor marks devices degraded/offline
- [ ] Command send/receive flow complete
- [ ] Firmware upload, check, and download working
- [ ] Firmware deployment (push OTA via MQTT)
- [ ] Alert processing and querying
- [ ] JWT authentication on all endpoints
- [ ] SignalR real-time updates
- [ ] Docker Compose deployment working
- [ ] API documented via Swagger/OpenAPI

---

**Previous Phase:** [Phase 7 — Heartbeat, Health & Registration](acuvim-spec-07.md)
**Next Phase:** [Phase 9 — Console Application Frontend](acuvim-spec-09.md)