Tau.Acuvim/docs/acuvim-spec-05.md

# Phase 5: SD Card Offline Buffering

## Objective

Add SD card storage as a last-resort data buffer when both WiFi and GSM are unavailable. Telemetry data is written to the SD card in a replay-friendly format and automatically drained to MQTT when connectivity is restored.

## Prerequisites

- Phase 4 complete (transport failover working)
- MicroSD card (FAT32 formatted, up to 32GB)
- SD card module or breakout (SPI interface)

## Deliverables

1. SPI-based SD card driver with hot-plug detection
2. Offline telemetry buffer (newline-delimited JSON files)
3. Automatic queue drain on connectivity restore
4. File rotation and cleanup (prevent SD from filling up)
5. SD card status in web UI and telemetry

---

## 5.1 SD Card Hardware

### SPI Pin Mapping (from Phase 1)

```
SD_CS    = GPIO 15     # Chip Select
SD_MOSI  = GPIO 2      # Master Out Slave In
SD_MISO  = GPIO 14     # Master In Slave Out
SD_CLK   = GPIO 12     # Clock
```

**Important:** These pins share the HSPI bus. The SD card SPI runs independently of the internal flash (which uses VSPI). Ensure SPI bus is not conflicting with any other peripheral.

### SPI Initialization

```cpp
#include <SPI.h>
#include <SD.h>

SPIClass hspi(HSPI);
hspi.begin(SD_CLK, SD_MISO, SD_MOSI, SD_CS);
SD.begin(SD_CS, hspi);
```

## 5.2 SD Card Manager

### `sd_manager.h` / `sd_manager.cpp`

```cpp
class SdManager {
public:
    bool begin();                              // Initialize SD card
    bool isAvailable();                        // Card present and mounted
    bool isHealthy();                          // Card mounted and writable

    // Buffering
    bool bufferTelemetry(const AcuvimData& data);  // Write one record
    uint32_t getQueuedCount();                      // Records waiting to send
    bool hasQueuedData();                           // Any records to drain

    // Drain (replay to MQTT)
    bool drainNext(String& payload);           // Get next record for sending
    void confirmDrained(const String& filename, uint32_t position);
    void drainAll(MqttClient& mqtt);           // Drain entire queue

    // Maintenance
    uint64_t getTotalSpace();                  // Total SD capacity
    uint64_t getUsedSpace();                   // Used space
    uint64_t getFreeSpace();                   // Available space
    uint32_t getFileCount();                   // Number of buffer files
    bool cleanup(uint32_t maxAgeDays = 30);    // Delete old files
    bool format();                             // Format SD card (destructive)

private:
    bool mounted;
    String currentFileName;
    uint32_t recordsInCurrentFile;

    String generateFileName();
    bool rotateFile();
    bool shouldRotate();
};
```

## 5.3 File Format and Structure

### Directory Structure

```
SD Card Root/
├── /telemetry/
│   ├── 2026-05-16_001.jsonl      # Newline-delimited JSON
│   ├── 2026-05-16_002.jsonl
│   └── 2026-05-17_001.jsonl
├── /drain/                        # Files currently being drained
│   └── (moved here during drain, deleted after)
└── /logs/                         # Optional diagnostic logs
    └── boot.log
```

### File Naming Convention

- Format: `YYYY-MM-DD_NNN.jsonl`
- `NNN`: Sequential file number within the day (001, 002, ...)
- `.jsonl`: JSON Lines format (one JSON object per line)

### Record Format

Each line is a complete, self-contained telemetry JSON (same format as MQTT payload from Phase 2):

```json
{"ts":1716000000,"dev":"ACV-AABBCCDDEEFF","v":{"a":230.1,"b":231.4,"c":229.8},"i":{"a":15.2,"b":14.8,"c":15.5},"p":{"total":10.5},"f":50.01,"e":{"imp_act":12345.6},"src":"sd"}
```

- Added `"src":"sd"` field to indicate this was buffered data (vs. real-time)
- Each line is independently parseable (no array wrapping)
- File can be read line by line without loading entire file into memory

### File Rotation

Rotate to a new file when:
- Current file exceeds 500KB (~2,000-3,000 records depending on content)
- Date changes (new day = new file)
- Current file has 5,000 records (hard limit)

## 5.4 Write Strategy

### Buffering Flow

```
Acuvim poll cycle:
  ├── Transport connected?
  │   ├── YES: Publish to MQTT directly
  │   │        └── Also check: hasQueuedData()?
  │   │            └── YES: drain queued records in background
  │   └── NO: Buffer to SD card
  │        └── SD available?
  │            ├── YES: Write to current .jsonl file
  │            └── NO: Data is lost (log warning)
  └── Continue polling
```

### Write Implementation

```cpp
bool SdManager::bufferTelemetry(const AcuvimData& data) {
    if (!mounted) return false;

    if (shouldRotate()) {
        rotateFile();
    }

    File file = SD.open(currentFileName, FILE_APPEND);
    if (!file) return false;

    // Serialize AcuvimData to compact JSON (same format as MQTT)
    StaticJsonDocument<512> doc;
    // ... populate doc from data ...
    doc["src"] = "sd";

    String line;
    serializeJson(doc, line);
    file.println(line);
    file.close();

    recordsInCurrentFile++;
    return true;
}
```

**Design notes:**
- Open file, write, close immediately (crash-safe, no data loss on power failure)
- Do NOT keep files open between writes (SD card can be removed)
- `FILE_APPEND` mode ensures data is added to end of file
- Each write is atomic at the line level

## 5.5 Drain Strategy

### Drain Flow

When connectivity is restored, queued records are replayed to MQTT:

```
drainAll() logic:
  1. List files in /telemetry/ sorted by name (oldest first)
  2. For each file:
     a. Move file to /drain/ directory (prevents double-read)
     b. Open file, read line by line
     c. For each line:
        - Parse JSON
        - Publish to MQTT telemetry topic
        - Throttle: max 10 records/second (avoid MQTT flood)
     d. After all lines published: delete file from /drain/
     e. If MQTT disconnects mid-drain: stop, leave remaining files
  3. Log drain summary (records sent, time taken)
```

### Drain Throttling

- Max drain rate: 10 records per second
- Yield between publishes to allow main loop tasks (Modbus polling, MQTT keepalive)
- Drain runs as a background task in the main loop, not a blocking operation
- During drain, live telemetry continues to publish normally (live data takes priority)

### Drain Integration in Main Loop

```cpp
void loop() {
    mqtt.loop();
    transport.loop();

    // Normal telemetry polling
    if (pollTimer.elapsed()) {
        AcuvimData data;
        if (acuvim.readAll(data)) {
            if (transport.isConnected() && mqtt.isConnected()) {
                mqtt.publishTelemetry(data);
            } else if (sdManager.isAvailable()) {
                sdManager.bufferTelemetry(data);
            }
        }
    }

    // Background drain (non-blocking, processes a few records per loop)
    if (transport.isConnected() && mqtt.isConnected() && sdManager.hasQueuedData()) {
        sdManager.drainBatch(mqtt, 5);  // Drain up to 5 records per loop iteration
    }
}
```

## 5.6 Storage Management

### Capacity Planning

| Poll Interval | Record Size | Records/Hour | MB/Day | Days on 4GB |
|---------------|-------------|--------------|--------|-------------|
| 5 seconds     | ~250 bytes  | 720          | ~4.3   | ~900        |
| 10 seconds    | ~250 bytes  | 360          | ~2.2   | ~1,800      |
| 30 seconds    | ~250 bytes  | 120          | ~0.7   | ~5,500      |

At 5-second polling, a 4GB SD card holds approximately 2.5 years of data. Storage is not a concern.

### Cleanup Policy

- Default: delete files older than 30 days
- Configurable via NVS (`sd_retention_days`)
- If SD card is >90% full: delete oldest files regardless of age
- Cleanup runs once per hour

### Error Handling

- SD card removed: set `mounted = false`, log warning, data lost until reinserted
- SD card full: delete oldest file, retry write
- Corrupt file during drain: skip file, move to `/errors/`, continue with next
- SD write failure: retry once, then skip and log

## 5.7 Hot-Plug Detection

Check SD card presence periodically (every 30 seconds):

```cpp
void SdManager::checkCard() {
    if (mounted && !SD.exists("/")) {
        // Card was removed
        mounted = false;
        Serial.println("SD card removed");
    } else if (!mounted) {
        // Try to remount
        if (SD.begin(SD_CS, hspi)) {
            mounted = true;
            ensureDirectories();
            Serial.println("SD card inserted");
        }
    }
}
```

## 5.8 Devices Without SD Card

Similar to GSM (Phase 4), handle gracefully:

- On boot, attempt SD card initialization
- If no SD card: set `sd_available = false`
- No error logs for expected absence
- Telemetry flow: transport only (WiFi/GSM), data lost if no transport
- Web UI shows "No SD card" status

## 5.9 Web UI Updates

### Status Page Addition

```
┌──────────────────────────────────────┐
│ Storage                              │
│ ● SD Card: 4.0 GB (0.1% used)       │
│   Queued: 0 records                  │
│   Files: 3                           │
│   Retention: 30 days                 │
│                                      │
│ [Drain Now]  [Cleanup]               │
└──────────────────────────────────────┘
```

### API Endpoints

```
GET /api/sd/status
Response:
{
  "available": true,
  "total_bytes": 4294967296,
  "used_bytes": 4194304,
  "free_bytes": 4290772992,
  "queued_records": 0,
  "file_count": 3,
  "retention_days": 30
}

POST /api/sd/drain
Response:
{
  "success": true,
  "message": "Drain started. 0 records queued."
}

POST /api/sd/cleanup
Response:
{
  "success": true,
  "deleted_files": 2,
  "freed_bytes": 1048576
}
```

## 5.10 Testing & Validation

| Test | Method | Pass Criteria |
|------|--------|---------------|
| SD card init | Insert formatted SD card | Mounted, directories created |
| Buffer write | Disable WiFi and GSM | Records written to .jsonl file |
| File format | Read .jsonl on PC | Valid JSON per line, parseable |
| File rotation | Write >500KB | New file created with incremented number |
| Date rotation | Cross midnight while buffering | New date-prefixed file |
| Drain on reconnect | Restore WiFi after buffering | Records published to MQTT in order |
| Drain throttle | Buffer 1000 records, reconnect | Drain rate <= 10/sec, main loop responsive |
| Live + drain | Reconnect during polling | Live telemetry published + drain in background |
| No SD card | Boot without SD card | Graceful, no errors, WiFi/GSM only |
| SD card removal | Remove SD during operation | Detected within 30s, no crash |
| SD card reinsert | Reinsert after removal | Remounted, buffering resumes |
| Cleanup | Create files > 30 days old | Old files deleted |
| SD full | Fill SD card | Oldest file deleted, write continues |
| Power loss during write | Kill power mid-write | At most last line lost, file intact |

## 5.11 Phase 5 Completion Criteria

- [ ] SD card initializes and creates directory structure
- [ ] Telemetry buffered to SD when no transport available
- [ ] JSONL format verified (parseable per line)
- [ ] File rotation works (size and date based)
- [ ] Automatic drain on connectivity restore
- [ ] Drain is non-blocking and throttled
- [ ] Live telemetry continues during drain
- [ ] Hot-plug detection (removal and reinsertion)
- [ ] Graceful handling when no SD card present
- [ ] Cleanup of old files works
- [ ] Storage stats visible in web UI
- [ ] SD status included in device status API

---

**Previous Phase:** [Phase 4 — GSM & Transport Failover](acuvim-spec-04.md)
**Next Phase:** [Phase 6 — OTA Firmware Updates](acuvim-spec-06.md)