Skip to main content
Audience: SREs, DBAs, platform operators Last updated: 2026-02-23 Database: PostgreSQL 15+ via Amazon RDS (or self-managed) Connection pooler: PgBouncer

Table of Contents

  1. Backup
  2. Restore
  3. Manual VACUUM
  4. Index Maintenance
  5. Schema Migrations
  6. Monitoring Queries
  7. Connection Pool Tuning (PgBouncer)

1. Backup

Automated Backups (RDS)

RDS automated backups are configured with a 7-day retention period. Snapshots are taken daily during the maintenance window (02:00-03:00 UTC).

Manual Backup

Use the make backup target or run the equivalent commands directly. Makefile target:
Manual execution:
Schema-only backup (for migration reference):
Table-specific backup:

RDS Snapshot

Backup Verification

Run monthly to ensure backups are valid:

2. Restore

Restore from pg_dump Backup

Use the make restore target or run the equivalent commands directly. Makefile target:
Manual execution:

Restore from RDS Snapshot

Point-in-Time Recovery (RDS)

Post-Restore Checklist

  1. Verify row counts match expectations.
  2. Refresh planner statistics on all tables (PostgreSQL ANALYZE).
  3. Verify application connectivity.
  4. Run a smoke test against the decision endpoint.
  5. Update DNS or connection strings if restoring to a new instance.

3. Manual VACUUM

When to VACUUM

  • Dead tuple ratio exceeds 20% on any table.
  • After large bulk deletes or updates.
  • Before VACUUM FULL if table bloat exceeds 50%.
  • Autovacuum is lagging (check pg_stat_user_tables.last_autovacuum).

Standard VACUUM

Does not lock the table. Safe to run during production hours.

VACUUM FULL

Reclaims disk space by rewriting the table. Acquires an exclusive lock. Schedule during maintenance windows only.

Autovacuum Tuning

Monitor Autovacuum Progress


4. Index Maintenance

Check Index Health

Rebuild Indexes

Use REINDEX CONCURRENTLY to avoid locking the table.

Create Missing Indexes

Common queries that benefit from indexes:

Drop Unused Indexes


5. Schema Migrations

KaireonAI uses Prisma 7. The schema.prisma datasource declares only provider = "postgresql" (the connection URL lives in prisma.config.ts), and the client is generated to platform/generated/prisma/ via the @prisma/adapter-pg driver adapter.
prisma db push is SAFE ONLY on a fresh database. On any populated database it will drop every table that is not modelled in schema.prisma — that includes the ds_* customer-schema tables created at runtime by the Data module and the _flow_src_* / _flow_xform_* / _flow_branch_* staging tables created per-run by the pipeline runtime. Running db push against a production database therefore causes data loss. Never run db push, --accept-data-loss, or --force-reset against a populated database.

Fresh database (initial setup only)

On a brand-new database that has never held customer schemas or pipeline runs, db push is the fastest way to create the schema:
The Docker API entrypoint (docker/api-entrypoint.sh) runs npx prisma db push --skip-generate on every start for the single-database compose flow. This is idempotent and safe there, but the data-loss warning above still applies to any deployment that has customer schemas or pipeline staging tables.

Populated database (all schema evolution)

For any database that holds real data, schema changes are applied through numbered manual-SQL migration filesprisma/manual-sql/<N>_<name>.sql — which are the source of truth for schema evolution. Add the matching schema.prisma model/column in the same change so the generated client knows about it, then apply the SQL with psql:
Note: prisma/manual-sql/ files must not use CREATE INDEX CONCURRENTLY or other statements that cannot run inside a transaction, because they are applied as a single transactional file. Build concurrent indexes as a separate, manually-run step (see Index Maintenance).
Pre-migration checklist:
  1. Back up the database (see Backup).
  2. Review the schema diff carefully.
  3. Check for data loss warnings.
  4. Run in staging first.
  5. Coordinate with the team (announce in #kaireon-deployments).
Post-migration checklist:
  1. Refresh planner statistics on affected tables (PostgreSQL ANALYZE).
  2. Verify application starts without errors.
  3. Check that all API endpoints return valid responses.
  4. Monitor error rates for 15 minutes.

Writing a manual-SQL migration

Data migrations and custom DDL (constraints, backfills) also go in a numbered prisma/manual-sql/<N>_<name>.sql file. Wrap the statements in a single transaction so a partial failure rolls back cleanly:

Rollback Procedure


6. Monitoring Queries

Dashboard Queries

Run these queries periodically or integrate them into Grafana via the PostgreSQL data source. Database size and growth:
Active connections:
Slow queries (current):
Table statistics:
Cache hit ratio (should be >99%):
Lock contention:
Replication lag (if read replicas exist):

Alerting Thresholds


7. Connection Pool Tuning (PgBouncer)

Current Configuration

PgBouncer sits between the KaireonAI application and PostgreSQL, multiplexing connections. Architecture:

Configuration File

Located in the PgBouncer ConfigMap: kaireon-pgbouncer-config

Pool Sizing Formula

Monitoring PgBouncer

Tuning for Common Scenarios

High API traffic (many short queries):
Long-running analytics queries:
Burst traffic handling:

Applying Configuration Changes

Troubleshooting PgBouncer

Clients waiting for connections:
Connection refused errors:
Prepared statement errors in transaction mode:

Automated In-App Maintenance

Some database housekeeping runs automatically inside the application and does not need a manual DBA action:
  • Pipeline staging-table janitor — the pipeline runtime materialises each run’s source/transform/branch output into per-run _flow_src_*, _flow_xform_*, and _flow_branch_* staging tables. The /api/v1/cron/staging-janitor job (invoked hourly by the in-process maintenance scheduler when CRON_SECRET is set) drops staging tables older than FLOW_STAGING_RETENTION_HOURS (default 24). If you disable the maintenance scheduler (MAINTENANCE_SCHEDULER_ENABLED=false) or run multi-replica with external cron, wire this route into your scheduler so staging tables do not accumulate. See the Worker Mode + Cron Drain runbook.
  • Retention / cleanup crons/api/v1/cron/cleanup (hourly) and /api/v1/cron/dsar-purge (daily) enforce data-retention and DSAR erasure policies. These also run in-process by default when CRON_SECRET is set.

Maintenance Schedule