Audience: SREs, DBAs, platform operators
Last updated: 2026-02-23
Database: PostgreSQL 15+ via Amazon RDS (or self-managed)
Connection pooler: PgBouncer
Table of Contents
- Backup
- Restore
- Manual VACUUM
- Index Maintenance
- Schema Migrations
- Monitoring Queries
- Connection Pool Tuning (PgBouncer)
1. Backup
Automated Backups (RDS)
RDS automated backups are configured with a 7-day retention period. Snapshots are taken daily during the maintenance window (02:00-03:00 UTC).
Manual Backup
Use the make backup target or run the equivalent commands directly.
Makefile target:
Manual execution:
Schema-only backup (for migration reference):
Table-specific backup:
RDS Snapshot
Backup Verification
Run monthly to ensure backups are valid:
2. Restore
Restore from pg_dump Backup
Use the make restore target or run the equivalent commands directly.
Makefile target:
Manual execution:
Restore from RDS Snapshot
Point-in-Time Recovery (RDS)
Post-Restore Checklist
- Verify row counts match expectations.
- Refresh planner statistics on all tables (PostgreSQL ANALYZE).
- Verify application connectivity.
- Run a smoke test against the decision endpoint.
- Update DNS or connection strings if restoring to a new instance.
3. Manual VACUUM
When to VACUUM
- Dead tuple ratio exceeds 20% on any table.
- After large bulk deletes or updates.
- Before
VACUUM FULL if table bloat exceeds 50%.
- Autovacuum is lagging (check
pg_stat_user_tables.last_autovacuum).
Standard VACUUM
Does not lock the table. Safe to run during production hours.
VACUUM FULL
Reclaims disk space by rewriting the table. Acquires an exclusive lock. Schedule during maintenance windows only.
Autovacuum Tuning
Monitor Autovacuum Progress
4. Index Maintenance
Check Index Health
Rebuild Indexes
Use REINDEX CONCURRENTLY to avoid locking the table.
Create Missing Indexes
Common queries that benefit from indexes:
Drop Unused Indexes
5. Schema Migrations
KaireonAI uses Prisma 7. The schema.prisma datasource declares only
provider = "postgresql" (the connection URL lives in prisma.config.ts),
and the client is generated to platform/generated/prisma/ via the
@prisma/adapter-pg driver adapter.
prisma db push is SAFE ONLY on a fresh database. On any populated
database it will drop every table that is not modelled in
schema.prisma — that includes the ds_* customer-schema tables created
at runtime by the Data module and the _flow_src_* / _flow_xform_* /
_flow_branch_* staging tables created per-run by the pipeline runtime.
Running db push against a production database therefore causes data loss.
Never run db push, --accept-data-loss, or --force-reset against a
populated database.
Fresh database (initial setup only)
On a brand-new database that has never held customer schemas or pipeline
runs, db push is the fastest way to create the schema:
The Docker API entrypoint (docker/api-entrypoint.sh) runs
npx prisma db push --skip-generate on every start for the single-database
compose flow. This is idempotent and safe there, but the data-loss warning
above still applies to any deployment that has customer schemas or pipeline
staging tables.
Populated database (all schema evolution)
For any database that holds real data, schema changes are applied through
numbered manual-SQL migration files — prisma/manual-sql/<N>_<name>.sql
— which are the source of truth for schema evolution. Add the matching
schema.prisma model/column in the same change so the generated client
knows about it, then apply the SQL with psql:
Note: prisma/manual-sql/ files must not use CREATE INDEX CONCURRENTLY
or other statements that cannot run inside a transaction, because they are
applied as a single transactional file. Build concurrent indexes as a
separate, manually-run step (see Index Maintenance).
Pre-migration checklist:
- Back up the database (see Backup).
- Review the schema diff carefully.
- Check for data loss warnings.
- Run in staging first.
- Coordinate with the team (announce in
#kaireon-deployments).
Post-migration checklist:
- Refresh planner statistics on affected tables (PostgreSQL ANALYZE).
- Verify application starts without errors.
- Check that all API endpoints return valid responses.
- Monitor error rates for 15 minutes.
Writing a manual-SQL migration
Data migrations and custom DDL (constraints, backfills) also go in a
numbered prisma/manual-sql/<N>_<name>.sql file. Wrap the statements in a
single transaction so a partial failure rolls back cleanly:
Rollback Procedure
6. Monitoring Queries
Dashboard Queries
Run these queries periodically or integrate them into Grafana via the PostgreSQL data source.
Database size and growth:
Active connections:
Slow queries (current):
Table statistics:
Cache hit ratio (should be >99%):
Lock contention:
Replication lag (if read replicas exist):
Alerting Thresholds
7. Connection Pool Tuning (PgBouncer)
Current Configuration
PgBouncer sits between the KaireonAI application and PostgreSQL, multiplexing connections.
Architecture:
Configuration File
Located in the PgBouncer ConfigMap: kaireon-pgbouncer-config
Monitoring PgBouncer
Tuning for Common Scenarios
High API traffic (many short queries):
Long-running analytics queries:
Burst traffic handling:
Applying Configuration Changes
Troubleshooting PgBouncer
Clients waiting for connections:
Connection refused errors:
Prepared statement errors in transaction mode:
Automated In-App Maintenance
Some database housekeeping runs automatically inside the application and does
not need a manual DBA action:
- Pipeline staging-table janitor — the pipeline runtime materialises each
run’s source/transform/branch output into per-run
_flow_src_*,
_flow_xform_*, and _flow_branch_* staging tables. The
/api/v1/cron/staging-janitor job (invoked hourly by the in-process
maintenance scheduler when CRON_SECRET is set) drops staging tables older
than FLOW_STAGING_RETENTION_HOURS (default 24). If you disable the
maintenance scheduler (MAINTENANCE_SCHEDULER_ENABLED=false) or run
multi-replica with external cron, wire this route into your scheduler so
staging tables do not accumulate. See the
Worker Mode + Cron Drain runbook.
- Retention / cleanup crons —
/api/v1/cron/cleanup (hourly) and
/api/v1/cron/dsar-purge (daily) enforce data-retention and DSAR erasure
policies. These also run in-process by default when CRON_SECRET is set.
Maintenance Schedule