giteaadmin/Tau.Acuvim
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6 Commits 1 Branch 0 Tags 905 KiB
a92b4277ae
Commit Graph

4 Commits

Author SHA1 Message Date
Diseri Pearson
a92b4277ae Phase 14: Push + ingest pipeline (end-to-end fleet aggregation) 
Customer-stack measurements now flow to the Admin-stack central DB via
HTTPS POST, with firmware buffer-and-replay back-fills handled correctly.

Client side (push)
- monitoring.PowerMeasurements gains ReceivedAt (default NOW()) +
  index. Push selects WHERE ReceivedAt > LastCursor, so back-dated
  rows from offline-buffer replays are picked up automatically.
- app.FleetPushState table holds per-resource cursors + backoff state.
- FleetPushClient: HttpClient wrapper, X-Customer-Token header,
  X-Batch-Type, X-Push-Cursor. 413 returns retry-after halving signal.
- FleetPushService: BackgroundService loop. Per tick: sites (full set),
  devices (full set), measurements (cursor-driven up to 3 batches).
  Exponential backoff per resource on failure (1m → 30m cap).
  Honors 429 Retry-After. Only registered when RunMode=Client AND
  FleetIngest__Enabled=true.

Admin side (ingest)
- /api/fleet/ingest: anonymous, X-Customer-Token authed against
  fleet.Customers via SHA-256 indexed lookup. 401 on bad token; 400
  on bad batch type.
- FleetIngestService dispatches by X-Batch-Type:
  sites/devices → upsert by (CustomerId, Id) with ON CONFLICT UPDATE
  measurements → bulk INSERT ON CONFLICT (Time, CustomerId, DeviceId)
                 DO NOTHING (idempotent under re-delivery).
- Updates fleet.Customers.FirstSeenAt/LastSeenAt on each successful batch.
- Writes fleet.IngestEvents audit row per batch (accepted, rejected,
  bytes, client cursor, time-spread, error).
- FleetTimescaleBootstrapper runs after MigrateAsync in Admin mode:
  CREATE EXTENSION timescaledb, create_hypertable on fleet.PowerMeasurements,
  chunk interval 7 days, compression with segmentby=(CustomerId,DeviceId)
  + compress_orderby "Time" DESC, compression policy 7 days, hourly_per_device
  continuous aggregate (realtime, materialized_only=false, 30-day start_offset
  so back-fills get materialized on next refresh tick).

Wiring
- docker-compose.yml threads Application__RunMode + FleetIngest__* from
  .env (defaults safely off) so a single dev host can run two stacks.
- .env.example documents the new vars under their own section.

Tests
- FleetIngestValidationTests (2 new). 53/53 passing.

Verified end-to-end on the dev host
- Client (portal-dev_portal, RunMode=Client, FleetIngest__Enabled=true)
  pushes to Admin (portal-admin-test, RunMode=Admin, separate admin_fleet DB)
  via container DNS.
- Customer registered on Admin (DEV0001), token captured, dropped into
  Client .env, Client restarted, push service started on schedule.
- Ingested measurements (including a 2026-04-01 back-dated sample
  simulating firmware replay) all land in fleet.PowerMeasurements with
  the correct CustomerId.
- Customer.FirstSeenAt/LastSeenAt update, IngestEvents records every
  batch (sites + devices per tick, measurements when cursor advances).
- Hypertable confirmed via timescaledb_information.hypertables;
  hourly_per_device CA confirmed via timescaledb_information.continuous_aggregates.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 10:17:58 +02:00
Diseri Pearson
2c618b776b Phase 13: RunMode flag + AdminDbContext + Customers registry 
Adds the plumbing for the fleet-aggregation feature without moving any
data yet. Same portal binary now supports two modes selected via
Application:RunMode (Client | Admin).

Backend
- New AdminDbContext (identity + branding shared via SharedSchemaConfiguration
  helper + fleet schema). AppDbContext keeps existing identity + branding +
  monitoring + rates; renamed implicitly the "Client" context. Only one is
  registered with DI per RunMode.
- IWhiteLabelStore interface implemented by both contexts so BrandingService
  works in either mode.
- Fleet entities: Customer, FleetSite, FleetDevice, FleetPowerMeasurement,
  IngestEvent (all in the new fleet schema). Migration in Migrations/Admin/.
- CustomerService: 32-byte random token, SHA-256 hash stored, plaintext
  shown once on create + rotate. Token lookup is a single O(log N) indexed
  query.
- RunModeGuards: refuses Admin without conn string; refuses Client+push
  without URL/token; refuses cross-DB pointing (Client at admin_fleet DB
  with fleet.Customers, or Admin at customer DB with monitoring.PowerMeasurements).
- Endpoint maps now branch on RunMode:
  Client → sites/measurements/rates/admin-sites/admin-rates
  Admin  → admin/customers
  Shared → auth, users, branding, grafana, admin-config, app/info, health
- /api/app/info (anonymous) returns {runMode, applicationName, version} so
  the SPA can drive nav without re-fetching auth state.

Frontend
- AppInfoProvider + useAppInfo hook fetch /api/app/info once on load.
- AdminCustomersPage with create / edit / rotate-token / delete.
- TokenShownOnceModal: shows token once, copy-to-clipboard, "I've stored
  it" confirmation gate before closing.
- AppLayout nav swaps Sites <-> Customers based on RunMode and shows a
  FLEET ADMIN tag in the header when in Admin mode.

Tests
- 11 new tests: CustomerTokenTests (5) + RunModeGuardsTests (6).
- 51/51 passing locally.

Verified
- dotnet build + dotnet test clean (zero errors, one EF1002 warning
  suppressed in Phase 11 already).
- Client mode docker rebuild: no regressions, /api/app/info returns
  Client, login works, /api/sites/ works.
- Admin mode spun up on port 8090 against a fresh admin_fleet DB:
  /api/app/info returns Admin, customer ABC0001 registered, 64-char
  token returned, list shows the row.
- Cross-DB guard: Client run against admin_fleet refuses with explicit
  "is pointed at a database that contains fleet.Customers" error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 10:09:41 +02:00
Diseri Pearson
880525b306 Add Fleet ingest design doc (portal/docs/FLEET-DESIGN.md) 
Locked design for Admin / cross-customer aggregation feature.
Implementation lands in phases 13-15.

Key decisions captured:
- Same portal binary, RunMode=Client|Admin config flag.
- Two DbContext classes (ClientDbContext + AdminDbContext) to keep
  schemas cleanly separated and migrations sane.
- Fleet ingest is opt-in (FleetIngest__Enabled=false works exactly
  as today, no data leaves customer stack).
- Push by ReceivedAt, not Time, so firmware offline-buffer replays
  are picked up automatically.
- Per-tick batch cap so a back-fill wave from one customer doesn't
  starve other customers' pushes.
- SHA-256 token hash (not bcrypt) for the high-throughput ingest
  endpoint; tokens shown once on Admin Customers page.
- Realtime continuous aggregates with wide start_offset so late
  back-fills materialize on the next refresh tick.
- No retention policy. TimescaleDB compression on chunks older than
  7 days handles long-term storage cost.
- Open seams (tariff sync, RLS, GDPR delete, dual-token rotation,
  sharding) documented with v2 extension paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 09:55:07 +02:00
Diseri Pearson
e17921a122 Add portal: customer-facing white-labeled monitoring stack 
New top-level portal/ project, peer to console/ and firmware/. Delivers a
.NET 10 + React 18 + TimescaleDB + Grafana stack, one container set per
customer behind Traefik. Built in 12 phases per FrontEndPrompt spec; no
changes to existing console or firmware.

Backend (src/Tau.Acuvim.Portal/):
- .NET 10 minimal API, Serilog, ASP.NET Identity (cookie auth, lockout).
- Single AppDbContext with identity / app / monitoring schemas.
- MigrateAsync + TimescaleBootstrapper (idempotent hypertable creation)
  + IdentityBootstrapper (seeded admin + branding) on startup.
- Pure CostCalculator + DB-backed RateService for tariffs (effective-dated,
  TOU periods, VAT, fixed charges, per-municipality timezone).
- BrandingService with logo upload to mounted volume.
- Time-series ingest + bucketed query services (time_bucket aggregates,
  ON CONFLICT for idempotent re-delivery).
- ConfigOverviewService with redaction-by-construction (passwords never in
  payload).
- DataProtection keys persisted to /data/keys volume for cookie survival
  across container restarts.

Frontend (frontend/):
- React 18 + TypeScript + Vite + Ant Design 5 + TanStack Query.
- BrandingProvider + ThemedRoot for live re-themed white-labelling.
- RequireAuth / RequireRole guards.
- Pages: Login, Dashboard, Dashboards (embedded Grafana), Sites (admin),
  Settings tabs (Branding / Rates / Users / Grafana / App config).

Infra:
- Dev (docker-compose.yml) and prod (docker-compose.prod.yml) compose
  files. Three services per customer; Traefik subdomain + same-origin
  /grafana path-prefix routing wired with labels.
- Grafana 11 with provisioned timescaledb datasource (uid pinned) and
  starter power-overview.json dashboard with device template variable.
- Compose project name documented as lowercase (Compose v2 requirement).

Tests (tests/Tau.Acuvim.Portal.Tests/):
- xUnit, 40 tests. Covers CostCalculator (period match, TZ, overlap,
  VAT, fixed), ConnectionStringResolver (all 4 precedence branches incl.
  Production refusal), TariffValidator, DayOfWeekFlag.
- All passing locally against .NET 10.

Docs:
- README.md (onboarding + 11 spec sections), OPERATIONS.md (per-customer
  provisioning, secret rotation, backup, troubleshooting), TESTING.md
  (manual integration scenarios, frontend test scaffolding recipe).

Production safety guards:
- Refuses to start if Authentication:DefaultAdminPassword is unchanged
  default in Production.
- Refuses to start if Database:AutoProvisionLocalTimescaleDb=true in
  Production.
- Prod Grafana ships with anonymous off and auth mode unset (three
  options documented in README Security) so iframe refuses to load
  until a deliberate prod auth choice is made.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-18 09:30:30 +02:00