Security Hardening

This document provides a comprehensive security checklist and implementation guidance for hardening KaireonAI in production environments.

1. Checklist Summary

Use this checklist to track your hardening progress. Each area links to the detailed section below.

Area	Priority	Section
TLS / HTTPS	P0	Details
Security Headers	P0	Details
Rate Limiting	P0	Details
Secret Management	P0	Details
Network Policies	P1	Details
Pod Security Standards	P1	Details
Database SSL	P0	Details
Audit Log Retention	P1	Details
RBAC	P1	Details
Dependency Scanning	P2	Details
Provenance Signing (cosign)	P0	Provenance signing install guide — install via AWS Secrets Manager (cloud) or local key file (self-host). Without it, every `/api/v1/decisions/:id/provenance` response is `unsigned`.

2. TLS / HTTPS Everywhere

All traffic to and within the KaireonAI platform must be encrypted in transit.

2.1 Ingress TLS Termination

Use an AWS ALB or nginx-ingress controller with TLS certificates managed by cert-manager. nginx-ingress with cert-manager:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kaireon-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.kaireon.example.com
        - api.kaireon.example.com
      secretName: kaireon-tls
  rules:
    - host: app.kaireon.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kaireon-app
                port:
                  number: 3000

AWS ALB Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kaireon-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: app.kaireon.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kaireon-app
                port:
                  number: 3000

2.2 Internal Service TLS

For service-to-service communication within the cluster, consider a service mesh (Istio or Linkerd) for automatic mTLS. At minimum, ensure no sensitive data traverses unencrypted paths.

3. Security Headers

Configure the following HTTP security headers on all responses. These can be set at the ingress level or in the Next.js application middleware.

3.1 Ingress-Level Configuration (nginx)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Strict-Transport-Security: max-age=63072000; includeSubDomains; preload";
      more_set_headers "X-Frame-Options: DENY";
      more_set_headers "X-Content-Type-Options: nosniff";
      more_set_headers "X-XSS-Protection: 0";
      more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
      more_set_headers "Permissions-Policy: camera=(), microphone=(), geolocation=()";

3.2 Content Security Policy

Apply CSP to prevent XSS and data injection attacks. Adjust script-src and style-src directives based on your CDN and third-party integrations.

Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https://api.kaireon.example.com; frame-ancestors 'none'; base-uri 'self'; form-action 'self';

Note: unsafe-inline and unsafe-eval in script-src are necessary for Next.js in production. Use nonce-based CSP if your deployment supports it.

3.3 Header Reference

Header	Value	Purpose
Strict-Transport-Security	max-age=63072000; includeSubDomains; preload	Enforce HTTPS for 2 years
X-Frame-Options	DENY	Prevent clickjacking
X-Content-Type-Options	nosniff	Prevent MIME-type sniffing
X-XSS-Protection	0	Disable legacy XSS filter (use CSP)
Referrer-Policy	strict-origin-when-cross-origin	Limit referrer information leakage
Permissions-Policy	camera=(), microphone=(), geolocation=()	Disable unnecessary browser features
Content-Security-Policy	(see above)	Prevent XSS and injection attacks

4. Rate Limiting

Protect API endpoints from abuse and denial-of-service attacks.

4.1 nginx-ingress Rate Limiting

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "20"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "5"
    nginx.ingress.kubernetes.io/limit-connections: "10"
    nginx.ingress.kubernetes.io/limit-whitelist: "10.0.0.0/8"

4.2 Per-Endpoint Rate Limits

For granular control, implement rate limiting in the application layer using the Redis-backed rate limiter.

Endpoint Pattern	Rate Limit	Window	Notes
`/api/v1/decisions`	100 req/sec	Per IP	Decision scoring
`/api/v1/connectors`	30 req/min	Per user	Connector CRUD
`/api/v1/pipelines/*/run`	5 req/min	Per user	Pipeline execution
All other `/api/v1/*`	60 req/min	Per user	General API

Authentication endpoints (SOC 2 Phase 0, shipped). /api/auth/* is excluded from the Edge middleware’s config.matcher entirely (NextAuth needs the Node runtime), so it enforces its own limit rather than relying on a middleware-level rule: POST /api/auth/callback/credentials (the credentials login endpoint) and the verify action of POST /api/v1/auth/mfa are both rate-limited per IP, 10 attempts / 5 minutes by default, Redis-backed (pool-wide across all instances) — configurable via AUTH_RATE_LIMIT_MAX / AUTH_RATE_LIMIT_WINDOW_MS (see env-vars). The client IP is taken from the rightmost X-Forwarded-For entry (the edge-appended real client IP), not the leftmost client-supplied one, so a spoofed X-Forwarded-For cannot rotate the attacker into a fresh bucket. The Redis-outage posture differs by surface: login fails open (degrades gracefully — a Redis outage will not deny all logins, and the per-account lockout below still limits targeted brute-force during the window; Redis is not required for login to function), while MFA verify fails closed. This is layered on top of, not instead of, the pre-existing per-account lockout (5 failed attempts locks that account for 15 minutes, lib/auth.ts) and the MFA verify endpoint’s own per-user limit (3 attempts / 60s, fail-closed). POST /api/v1/auth/register has its own separate per-IP limit (5/hour) unrelated to AUTH_RATE_LIMIT_MAX.

4.3 Response Headers

Include rate limit information in responses to help clients self-regulate:

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 87
X-RateLimit-Reset: 1708700000
Retry-After: 30

5. Secret Management

5.1 AWS Secrets Manager + External Secrets Operator (ESO)

Never store secrets in Kubernetes manifests, Helm values, or environment variable files in source control. ExternalSecret resource:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kaireon-secrets
  namespace: kaireon
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: kaireon-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: kaireon/production/database
        property: url
    - secretKey: NEXTAUTH_SECRET
      remoteRef:
        key: kaireon/production/auth
        property: nextauth_secret
    - secretKey: JWT_SIGNING_SECRET
      remoteRef:
        key: kaireon/production/auth
        property: jwt_signing_secret
    - secretKey: CONNECTOR_ENCRYPTION_KEY
      remoteRef:
        key: kaireon/production/encryption
        property: connector_key

5.2 Secret Rotation Policy

Secret	Rotation Frequency	Method
DATABASE_URL (password)	90 days	Secrets Manager automatic rotation
NEXTAUTH_SECRET	180 days	Manual rotation with rolling deploy
JWT_SIGNING_SECRET	180 days	Dual-key validation during rotation
CONNECTOR_ENCRYPTION_KEY	365 days	Re-encrypt connectors during rotation
TLS certificates	60 days	cert-manager auto-renewal

5.3 Rotation Procedure

Create the new secret value in AWS Secrets Manager.
Update the External Secrets Operator’s external-secret resource if the remote ref path changed.
Wait for ESO to sync (up to refreshInterval), or force sync: kubectl annotate externalsecret kaireon-secrets force-sync=$(date +%s) --overwrite.
Perform a rolling restart: kubectl rollout restart deployment/kaireon-app -n kaireon.
Verify application health after restart.
Remove the old secret version from Secrets Manager after confirming stability.

6. Network Policies

Restrict pod-to-pod communication to only what is necessary.

6.1 Default Deny All

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: kaireon
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

6.2 Allow Application Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-traffic
  namespace: kaireon
spec:
  podSelector:
    matchLabels:
      app: kaireon-app
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - port: 3000
          protocol: TCP
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgresql
      ports:
        - port: 5432
          protocol: TCP
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - port: 6379
          protocol: TCP
    - to:  # DNS resolution
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    - to:  # External HTTPS (connectors, OIDC providers)
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
          protocol: TCP

7. Pod Security Standards

Apply Kubernetes Pod Security Standards to prevent privilege escalation.

7.1 Namespace Label (Restricted)

kubectl label namespace kaireon \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

7.2 Pod Security Context

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kaireon-app
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: nextjs-cache
              mountPath: /app/.next/cache
      volumes:
        - name: tmp
          emptyDir: {}
        - name: nextjs-cache
          emptyDir: {}

8. Database SSL

8.1 RDS SSL Configuration

Ensure all database connections use SSL by appending sslmode=require (or sslmode=verify-full) to the connection string.

DATABASE_URL=postgresql://$DB_USER:$DB_PASSWORD@kaireon-db.abc123.us-east-1.rds.amazonaws.com:5432/kaireon?sslmode=require&sslrootcert=/app/certs/rds-ca.pem

8.2 Certificate Bundle

Download the RDS CA certificate bundle and mount it into application pods:

curl -o rds-ca.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
kubectl create configmap rds-ca-cert --from-file=rds-ca.pem -n kaireon

8.3 In-Cluster PostgreSQL

For the Startup tier running in-cluster PostgreSQL, enable SSL in the Helm values:

postgresql:
  tls:
    enabled: true
    autoGenerated: true

9. Audit Log Retention

9.1 Application Audit Logs

KaireonAI emits structured audit log entries for all state-changing API operations. These logs include:

Timestamp (ISO 8601)
User identity (user ID, email, IP address)
Action performed (CREATE, UPDATE, DELETE)
Resource type and identifier
Request body hash (not the body itself, for sensitive data protection)
Response status code

9.2 Retention Policy

Log Type	Retention Period	Storage
Application audit logs	1 year	CloudWatch Logs / S3 lifecycle
API access logs	90 days	CloudWatch Logs
Database query logs	30 days	RDS CloudWatch integration
Kubernetes audit logs	90 days	CloudWatch Logs
Security event logs	2 years	S3 Glacier after 90 days

9.3 Log Forwarding

Use Fluent Bit as a DaemonSet to ship logs from pods to CloudWatch Logs or an external SIEM:

[OUTPUT]
    Name              cloudwatch_logs
    Match             kube.kaireon.*
    region            us-east-1
    log_group_name    /kaireon/production/app
    log_stream_prefix pod-
    auto_create_group true

9.4 Tamper Protection

Enable CloudWatch Logs log group data protection for PII detection.
Use S3 Object Lock (compliance mode) for long-term audit log archives.
Enable CloudTrail for AWS API audit trail.

10. RBAC and Least-Privilege Access

10.1 Kubernetes RBAC

Create dedicated ServiceAccounts for each component. Do not use the default ServiceAccount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kaireon-app
  namespace: kaireon
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/kaireon-app-role
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kaireon-app-role
  namespace: kaireon
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["kaireon-secrets"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kaireon-app-binding
  namespace: kaireon
subjects:
  - kind: ServiceAccount
    name: kaireon-app
    namespace: kaireon
roleRef:
  kind: Role
  name: kaireon-app-role
  apiGroup: rbac.authorization.k8s.io

10.2 IAM Roles for Service Accounts (IRSA)

Map Kubernetes ServiceAccounts to IAM roles with minimal permissions:

ServiceAccount	IAM Role	Permissions
kaireon-app	kaireon-app-role	Secrets Manager read, S3 read/write (uploads)
kaireon-worker	kaireon-worker-role	S3 read/write, SQS send/receive
external-secrets	kaireon-eso-role	Secrets Manager read

10.3 Database Access Control

Create separate database roles for the application and workers with the minimum required privileges.
The application role should have SELECT, INSERT, UPDATE, DELETE on application tables.
The worker role should have SELECT, INSERT, UPDATE on pipeline-related tables only.
No role should have DROP, CREATE, or ALTER privileges in production. Schema migrations run as a separate CI/CD step with a privileged role.

11. Additional Hardening

11.1 Dependency and Static Analysis Scanning

Run npm audit in CI pipelines. Fail builds on critical or high severity vulnerabilities. (KaireonAI’s own CI does this, plus a blocking Trivy filesystem scan and weekly Dependabot update PRs.)
Run SAST on every pull request. KaireonAI’s CI runs Semgrep (OSS mode, blocking on ERROR-severity findings) across the platform, Shopify, website, and ML worker code.
Use Snyk or Trivy for container image scanning before pushing to ECR.
Enable Amazon ECR image scanning on push.
All shipped KaireonAI container images run as a non-root user (nextjs in the platform image, worker in the queue worker, appuser in the ML worker).

11.2 Image Provenance

Use specific image tags, never latest.
Sign container images with cosign and verify signatures in admission controllers.
Pull images only from trusted registries (ECR, Docker Hub official images).

11.3 Runtime Protection

Deploy Falco for runtime anomaly detection (unexpected process execution, file access).
Enable AWS GuardDuty for EKS runtime threat detection.
Set readOnlyRootFilesystem: true on all containers (writable paths via emptyDir mounts).

12. API Key Authentication

KaireonAI supports API key authentication as an alternative to JWT Bearer tokens.

How It Works

API keys are passed via the X-API-Key header
Keys are stored as SHA-256 hashes in the api_keys database table (never in plaintext)
Each key has: name, hashedKey, prefix (first 12 chars of the raw key, for display), tenantId, scopes (per-key scope array; empty = data-plane-only — recommend/respond; ["control-plane"] grants management access), expiresAt, lastUsedAt, revokedAt
Keys use the krn_ prefix for easy identification in logs

Production Checklist

Rotate API keys every 90 days
Set expiresAt on all keys — never issue non-expiring keys in production
Monitor kaireon_http_request_duration_seconds filtered by API key usage patterns
Revoke keys immediately when team members leave (set revokedAt)
Use separate keys per integration (never share keys across services)

Key Management API

POST   /api/v1/api-keys             — Create new key (returns plaintext once)
GET    /api/v1/api-keys             — List keys (hashed, no plaintext)
DELETE /api/v1/api-keys?id=<keyId>  — Revoke a key (id passed as query param)

13. Connector Credential Encryption

All connector authentication configurations (authConfig field) are encrypted at rest using AES-256-GCM.

Configuration

Set the CONNECTOR_ENCRYPTION_KEY environment variable to a 32-byte hex string:

# Generate a key
openssl rand -hex 32

# Set in environment or Helm values
CONNECTOR_ENCRYPTION_KEY=<64-char-hex-string>

How It Works

On POST /api/v1/connectors (create) and PUT /api/v1/connectors (update — id is passed in the JSON body), the authConfig JSON is encrypted before storage
On GET /api/v1/connectors, credentials are decrypted for display (masked in list view, full in detail for admins)
Each encrypted value includes a random 12-byte IV and 16-byte auth tag
If CONNECTOR_ENCRYPTION_KEY is not set, credentials are stored in plaintext (development only)

Production Checklist

Set CONNECTOR_ENCRYPTION_KEY in all environments
Store the key in AWS Secrets Manager (see Section 5)
Rotate the key annually — re-encrypt all connectors after rotation
Never log or expose the encryption key in application logs

​1. Checklist Summary

​2. TLS / HTTPS Everywhere

​2.1 Ingress TLS Termination

​2.2 Internal Service TLS

​3. Security Headers

​3.1 Ingress-Level Configuration (nginx)

​3.2 Content Security Policy

​3.3 Header Reference

​4. Rate Limiting

​4.1 nginx-ingress Rate Limiting

​4.2 Per-Endpoint Rate Limits

​4.3 Response Headers

​5. Secret Management

​5.1 AWS Secrets Manager + External Secrets Operator (ESO)

​5.2 Secret Rotation Policy

​5.3 Rotation Procedure

​6. Network Policies

​6.1 Default Deny All

​6.2 Allow Application Traffic

​7. Pod Security Standards

​7.1 Namespace Label (Restricted)

​7.2 Pod Security Context

​8. Database SSL

​8.1 RDS SSL Configuration

​8.2 Certificate Bundle

​8.3 In-Cluster PostgreSQL

​9. Audit Log Retention

​9.1 Application Audit Logs

​9.2 Retention Policy

​9.3 Log Forwarding

​9.4 Tamper Protection

​10. RBAC and Least-Privilege Access

​10.1 Kubernetes RBAC

​10.2 IAM Roles for Service Accounts (IRSA)

​10.3 Database Access Control

​11. Additional Hardening

​11.1 Dependency and Static Analysis Scanning

​11.2 Image Provenance

​11.3 Runtime Protection

​12. API Key Authentication

​How It Works

​Production Checklist

​Key Management API

​13. Connector Credential Encryption

​Configuration

​How It Works

​Production Checklist

1. Checklist Summary

2. TLS / HTTPS Everywhere

2.1 Ingress TLS Termination

2.2 Internal Service TLS

3. Security Headers

3.1 Ingress-Level Configuration (nginx)

3.2 Content Security Policy

3.3 Header Reference

4. Rate Limiting

4.1 nginx-ingress Rate Limiting

4.2 Per-Endpoint Rate Limits

4.3 Response Headers

5. Secret Management

5.1 AWS Secrets Manager + External Secrets Operator (ESO)

5.2 Secret Rotation Policy

5.3 Rotation Procedure

6. Network Policies

6.1 Default Deny All

6.2 Allow Application Traffic

7. Pod Security Standards

7.1 Namespace Label (Restricted)

7.2 Pod Security Context

8. Database SSL

8.1 RDS SSL Configuration

8.2 Certificate Bundle

8.3 In-Cluster PostgreSQL

9. Audit Log Retention

9.1 Application Audit Logs

9.2 Retention Policy

9.3 Log Forwarding

9.4 Tamper Protection

10. RBAC and Least-Privilege Access

10.1 Kubernetes RBAC

10.2 IAM Roles for Service Accounts (IRSA)

10.3 Database Access Control

11. Additional Hardening

11.1 Dependency and Static Analysis Scanning

11.2 Image Provenance

11.3 Runtime Protection

12. API Key Authentication

How It Works

Production Checklist

Key Management API

13. Connector Credential Encryption

Configuration

How It Works

Production Checklist