feat(db): add lightweight migration system

Introduce server-migrations.ts with a named-migration runner that tracks
applied migrations in a _migrations table. runMigrations(db) is called at
startup before routes, so additive schema changes (ALTER TABLE, new settings)
are applied once and skipped on subsequent restarts.

Update ARCHITECTURE.md: five inline edits + new §4.4 documenting the convention.
This commit is contained in:
Brückner
2026-06-10 16:30:44 +02:00
parent c3931e7f36
commit 5c7ad3140a
3 changed files with 71 additions and 5 deletions

View File

@ -115,7 +115,7 @@
| Component | Technology | Purpose |
|-----------|------------|---------|
| Database | SQLite via `better-sqlite3` | Single file `ghostgrid.db`, WAL journal mode, synchronous queries |
| Schema | Idempotent `CREATE TABLE IF NOT EXISTS` | 8 tables defined in full and created on boot (fresh-install model, no migrations) |
| Schema | Idempotent `CREATE TABLE IF NOT EXISTS` | Baseline 8 tables created on boot via `server-db.ts`; additive changes to live DBs go in `server-migrations.ts` (see §4.4) |
| Settings store | key/value `settings` table | Runtime config for all integrations, seeded with `INSERT OR IGNORE` |
---
@ -195,7 +195,7 @@ Networking (optional, managed in-app)
## 4. Database Schema Design
SQLite file `ghostgrid.db` in `process.cwd()`, opened with `journal_mode = WAL`. Every table is defined in full and created idempotently on boot (`CREATE TABLE IF NOT EXISTS`) — the app assumes a fresh install, so there is no migration layer.
SQLite file `ghostgrid.db` in `process.cwd()`, opened with `journal_mode = WAL`. Baseline tables are created idempotently on boot (`CREATE TABLE IF NOT EXISTS` in `server-db.ts`). Additive changes to live DBs (new columns, tables, default settings) are applied by the migration runner in `server-migrations.ts`. See §4.4.
### 4.1 Schema (as created in `server-db.ts`)
@ -298,7 +298,7 @@ CREATE TABLE IF NOT EXISTS caddy (
| JSON columns | `labs.deviceIds` and `labs.topology` are JSON strings, parsed in the API layer |
| Booleans | Booking flags (`notified`, `emailSent`, `ansible*Triggered`) are `INTEGER` 0/1, mapped to JS booleans on read |
| Cascade | No FK cascades; referential cleanup is done in code (e.g. deleting a device scrubs it from every lab's `deviceIds`/`topology`) |
| Schema changes | Fresh-install model — edit the `CREATE TABLE` block in `server-db.ts` directly. To add columns to an existing DB, place idempotent `ALTER TABLE … ADD COLUMN` calls (wrapped in try/catch) in `server-db.ts` after the `db.exec()` block — they run on every startup and are no-ops when the column already exists |
| Schema changes | Baseline schema in `server-db.ts` (`CREATE TABLE IF NOT EXISTS`). New columns / tables / default settings for live DBs go in `server-migrations.ts` as an appended migration object. See §4.4. |
### 4.3 Settings (key/value config)
@ -313,6 +313,29 @@ Seeded with `INSERT OR IGNORE` (defaults below). Secret keys are never returned
🔒 = in `SECRET_KEYS`, masked on read, only updated when a non-`__SET__` value is sent.
### 4.4 Migration System
Additive schema changes to live databases are handled by `server-migrations.ts`. The runner executes once at the start of `startServer()`, before routes or background jobs, and is synchronous (`better-sqlite3`).
**How it works:**
1. Creates (idempotently) a `_migrations` table: `id TEXT PRIMARY KEY, applied_at TEXT`.
2. Iterates the `migrations` array in order. For each entry whose `id` is not in `_migrations`, runs `up(db)` inside a transaction and records the id on success.
3. If `up()` throws, the transaction rolls back and the id is not recorded — the migration retries on the next start.
4. Already-applied migrations are skipped forever.
**Rules for adding a migration:**
- Append to the end of the `migrations` array in `server-migrations.ts`. **Never reorder or remove entries.**
- Use a unique, descriptive `id` — format: `NNNN_short_description` (e.g. `0001_bookings_add_color`).
- Only additive DDL: `ALTER TABLE … ADD COLUMN`, `CREATE TABLE IF NOT EXISTS`, `INSERT OR IGNORE INTO settings`.
- New default settings for **live DBs** go here. Day-0 seeds (fresh install) stay in `DEFAULT_SETTINGS` in `server-db.ts`.
- New settings still need allow-listing in `PUT /api/settings` (and `SECRET_KEYS` if secret) — the migration only seeds the value.
**Fresh-install behavior:** `server-db.ts` creates all baseline tables first (at module import). `runMigrations` then applies every migration in the array — safe because all migrations are additive over the baseline.
**`_migrations` is excluded** from `/api/database/import` and `/api/database/info` (not in their explicit table lists) — migration state belongs to the current install, not the imported data.
---
## 5. API Design
@ -784,7 +807,7 @@ GhostGrid/
| CheckMK/Semaphore outage | Integration loops catch errors, log them, and retry next cycle; non-fatal |
| Caddy admin API unreachable | `pushCaddyConfig()` failures are logged as warnings; routes apply when Caddy starts |
| Data loss | Back up `ghostgrid.db` + `-wal`/`-shm`; each instance has its own DB |
| Schema evolution | Edit the `CREATE TABLE` block in `server-db.ts` (fresh-install model). For live DBs, add new columns via idempotent `ALTER TABLE … ADD COLUMN` (try/catch) after the `db.exec()` block. New settings need seed + allow-list (+ `SECRET_KEYS` if secret) |
| Schema evolution | For fresh installs, edit the `CREATE TABLE` block in `server-db.ts`. For live DBs (new columns, tables, default settings), append a migration to `server-migrations.ts` (see §4.4). New settings still need allow-listing in `PUT /api/settings` (+ `SECRET_KEYS` if secret). |
---
@ -852,7 +875,7 @@ Express (server.ts) ──► better-sqlite3 (ghostgrid.db, WAL)
- `labs.deviceIds` / `labs.topology` are JSON strings in SQLite, parsed in the API.
- Booking boolean flags are 0/1 integers in SQLite, mapped on read.
- A new settings key must be: **seeded** in `server-db.ts`, **allow-listed** in `PUT /api/settings`, and (if secret) added to `SECRET_KEYS`.
- Schema changes go into the `CREATE TABLE` block in `server-db.ts` (fresh-install). For adding columns to live DBs, use idempotent `ALTER TABLE … ADD COLUMN` (try/catch) placed after the `db.exec()` block.
- Baseline schema in `server-db.ts`. Additive schema changes (new columns, tables, default settings) go in `server-migrations.ts` as an appended migration object — never edit existing entries (see §4.4).
- The SPA catch-all (`app.get('*')`) + static serving are registered **last** in `startServer()`, after every `/api` route — otherwise GET `/api/*` falls through to `index.html`. All `/api` responses carry `Cache-Control: no-store`.
- One Caddy per container; `POST /load` replaces the whole config. Only the `CADDY_MANAGER=true` instance may push/seed/edit routes — never let the non-manager push.
- All user-facing strings are in **English**.