> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Security Hardening

> Production security checklist and hardening guide for KaireonAI

This document provides a comprehensive security checklist and implementation guidance for hardening KaireonAI in production environments.

***

## 1. Checklist Summary

Use this checklist to track your hardening progress. Each area links to the detailed section below.

| Area                        | Priority | Section                                                                                                                                                                                                                                                         |
| --------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| TLS / HTTPS                 | P0       | [Details](#2-tls--https-everywhere)                                                                                                                                                                                                                             |
| Security Headers            | P0       | [Details](#3-security-headers)                                                                                                                                                                                                                                  |
| Rate Limiting               | P0       | [Details](#4-rate-limiting)                                                                                                                                                                                                                                     |
| Secret Management           | P0       | [Details](#5-secret-management)                                                                                                                                                                                                                                 |
| Network Policies            | P1       | [Details](#6-network-policies)                                                                                                                                                                                                                                  |
| Pod Security Standards      | P1       | [Details](#7-pod-security-standards)                                                                                                                                                                                                                            |
| Database SSL                | P0       | [Details](#8-database-ssl)                                                                                                                                                                                                                                      |
| Audit Log Retention         | P1       | [Details](#9-audit-log-retention)                                                                                                                                                                                                                               |
| RBAC                        | P1       | [Details](#10-rbac)                                                                                                                                                                                                                                             |
| Dependency Scanning         | P2       | [Details](#11-dependency-scanning)                                                                                                                                                                                                                              |
| Provenance Signing (cosign) | P0       | [Provenance signing install guide](../../governance-security/provenance-cosign#installing-cosign-signing) — install via AWS Secrets Manager (cloud) or local key file (self-host). Without it, every `/api/v1/decisions/:id/provenance` response is `unsigned`. |

***

## 2. TLS / HTTPS Everywhere

All traffic to and within the KaireonAI platform must be encrypted in transit.

### 2.1 Ingress TLS Termination

Use an AWS ALB or nginx-ingress controller with TLS certificates managed by cert-manager.

**nginx-ingress with cert-manager:**

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kaireon-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - app.kaireon.example.com
        - api.kaireon.example.com
      secretName: kaireon-tls
  rules:
    - host: app.kaireon.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kaireon-app
                port:
                  number: 3000
```

**AWS ALB Ingress:**

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kaireon-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ACCOUNT:certificate/CERT-ID
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  rules:
    - host: app.kaireon.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kaireon-app
                port:
                  number: 3000
```

### 2.2 Internal Service TLS

For service-to-service communication within the cluster, consider a service mesh (Istio or Linkerd) for automatic mTLS. At minimum, ensure no sensitive data traverses unencrypted paths.

***

## 3. Security Headers

Configure the following HTTP security headers on all responses. These can be set at the ingress level or in the Next.js application middleware.

### 3.1 Ingress-Level Configuration (nginx)

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "Strict-Transport-Security: max-age=63072000; includeSubDomains; preload";
      more_set_headers "X-Frame-Options: DENY";
      more_set_headers "X-Content-Type-Options: nosniff";
      more_set_headers "X-XSS-Protection: 0";
      more_set_headers "Referrer-Policy: strict-origin-when-cross-origin";
      more_set_headers "Permissions-Policy: camera=(), microphone=(), geolocation=()";
```

### 3.2 Content Security Policy

Apply CSP to prevent XSS and data injection attacks. Adjust `script-src` and `style-src` directives based on your CDN and third-party integrations.

```
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https://api.kaireon.example.com; frame-ancestors 'none'; base-uri 'self'; form-action 'self';
```

**Note:** `unsafe-inline` and `unsafe-eval` in `script-src` are necessary for Next.js in production. Use nonce-based CSP if your deployment supports it.

### 3.3 Header Reference

| Header                    | Value                                        | Purpose                              |
| ------------------------- | -------------------------------------------- | ------------------------------------ |
| Strict-Transport-Security | max-age=63072000; includeSubDomains; preload | Enforce HTTPS for 2 years            |
| X-Frame-Options           | DENY                                         | Prevent clickjacking                 |
| X-Content-Type-Options    | nosniff                                      | Prevent MIME-type sniffing           |
| X-XSS-Protection          | 0                                            | Disable legacy XSS filter (use CSP)  |
| Referrer-Policy           | strict-origin-when-cross-origin              | Limit referrer information leakage   |
| Permissions-Policy        | camera=(), microphone=(), geolocation=()     | Disable unnecessary browser features |
| Content-Security-Policy   | (see above)                                  | Prevent XSS and injection attacks    |

***

## 4. Rate Limiting

Protect API endpoints from abuse and denial-of-service attacks.

### 4.1 nginx-ingress Rate Limiting

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "20"
    nginx.ingress.kubernetes.io/limit-burst-multiplier: "5"
    nginx.ingress.kubernetes.io/limit-connections: "10"
    nginx.ingress.kubernetes.io/limit-whitelist: "10.0.0.0/8"
```

### 4.2 Per-Endpoint Rate Limits

For granular control, implement rate limiting in the application layer using the Redis-backed rate limiter.

| Endpoint Pattern          | Rate Limit  | Window   | Notes              |
| ------------------------- | ----------- | -------- | ------------------ |
| `/api/v1/decisions`       | 100 req/sec | Per IP   | Decision scoring   |
| `/api/v1/connectors`      | 30 req/min  | Per user | Connector CRUD     |
| `/api/v1/pipelines/*/run` | 5 req/min   | Per user | Pipeline execution |
| All other `/api/v1/*`     | 60 req/min  | Per user | General API        |

**Authentication endpoints (SOC 2 Phase 0, shipped).** `/api/auth/*` is
excluded from the Edge middleware's `config.matcher` entirely (NextAuth
needs the Node runtime), so it enforces its own limit rather than relying
on a middleware-level rule: `POST /api/auth/callback/credentials` (the
credentials login endpoint) and the `verify` action of
`POST /api/v1/auth/mfa` are both rate-limited **per IP, 10 attempts / 5
minutes by default**, **Redis-backed** (pool-wide across all instances) —
configurable via `AUTH_RATE_LIMIT_MAX` / `AUTH_RATE_LIMIT_WINDOW_MS` (see
[env-vars](/self-host/configure/env-vars#auth-and-mfa)). The client IP is
taken from the **rightmost** `X-Forwarded-For` entry (the edge-appended
real client IP), not the leftmost client-supplied one, so a spoofed
`X-Forwarded-For` cannot rotate the attacker into a fresh bucket. The
Redis-outage posture differs by surface: **login fails open** (degrades
gracefully — a Redis outage will not deny all logins, and the per-account
lockout below still limits targeted brute-force during the window; Redis is
**not** required for login to function), while **MFA verify fails closed**.
This is layered on top of, not instead of, the pre-existing per-account
lockout (5 failed attempts locks that account for 15 minutes, `lib/auth.ts`)
and the MFA verify endpoint's own per-user limit (3 attempts / 60s,
fail-closed). `POST /api/v1/auth/register` has its own separate per-IP limit
(5/hour) unrelated to `AUTH_RATE_LIMIT_MAX`.

### 4.3 Response Headers

Include rate limit information in responses to help clients self-regulate:

```
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 87
X-RateLimit-Reset: 1708700000
Retry-After: 30
```

***

## 5. Secret Management

### 5.1 AWS Secrets Manager + External Secrets Operator (ESO)

Never store secrets in Kubernetes manifests, Helm values, or environment variable files in source control.

**ExternalSecret resource:**

```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: kaireon-secrets
  namespace: kaireon
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: kaireon-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: kaireon/production/database
        property: url
    - secretKey: NEXTAUTH_SECRET
      remoteRef:
        key: kaireon/production/auth
        property: nextauth_secret
    - secretKey: JWT_SIGNING_SECRET
      remoteRef:
        key: kaireon/production/auth
        property: jwt_signing_secret
    - secretKey: CONNECTOR_ENCRYPTION_KEY
      remoteRef:
        key: kaireon/production/encryption
        property: connector_key
```

### 5.2 Secret Rotation Policy

| Secret                     | Rotation Frequency | Method                                |
| -------------------------- | ------------------ | ------------------------------------- |
| DATABASE\_URL (password)   | 90 days            | Secrets Manager automatic rotation    |
| NEXTAUTH\_SECRET           | 180 days           | Manual rotation with rolling deploy   |
| JWT\_SIGNING\_SECRET       | 180 days           | Dual-key validation during rotation   |
| CONNECTOR\_ENCRYPTION\_KEY | 365 days           | Re-encrypt connectors during rotation |
| TLS certificates           | 60 days            | cert-manager auto-renewal             |

### 5.3 Rotation Procedure

1. Create the new secret value in AWS Secrets Manager.
2. Update the External Secrets Operator's external-secret resource if the remote ref path changed.
3. Wait for ESO to sync (up to `refreshInterval`), or force sync: `kubectl annotate externalsecret kaireon-secrets force-sync=$(date +%s) --overwrite`.
4. Perform a rolling restart: `kubectl rollout restart deployment/kaireon-app -n kaireon`.
5. Verify application health after restart.
6. Remove the old secret version from Secrets Manager after confirming stability.

***

## 6. Network Policies

Restrict pod-to-pod communication to only what is necessary.

### 6.1 Default Deny All

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: kaireon
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
```

### 6.2 Allow Application Traffic

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-app-traffic
  namespace: kaireon
spec:
  podSelector:
    matchLabels:
      app: kaireon-app
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: ingress-nginx
      ports:
        - port: 3000
          protocol: TCP
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: postgresql
      ports:
        - port: 5432
          protocol: TCP
    - to:
        - podSelector:
            matchLabels:
              app: redis
      ports:
        - port: 6379
          protocol: TCP
    - to:  # DNS resolution
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    - to:  # External HTTPS (connectors, OIDC providers)
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - port: 443
          protocol: TCP
```

***

## 7. Pod Security Standards

Apply Kubernetes Pod Security Standards to prevent privilege escalation.

### 7.1 Namespace Label (Restricted)

```bash theme={null}
kubectl label namespace kaireon \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted
```

### 7.2 Pod Security Context

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kaireon-app
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: nextjs-cache
              mountPath: /app/.next/cache
      volumes:
        - name: tmp
          emptyDir: {}
        - name: nextjs-cache
          emptyDir: {}
```

***

## 8. Database SSL

### 8.1 RDS SSL Configuration

Ensure all database connections use SSL by appending `sslmode=require` (or `sslmode=verify-full`) to the connection string.

```
DATABASE_URL=postgresql://$DB_USER:$DB_PASSWORD@kaireon-db.abc123.us-east-1.rds.amazonaws.com:5432/kaireon?sslmode=require&sslrootcert=/app/certs/rds-ca.pem
```

### 8.2 Certificate Bundle

Download the RDS CA certificate bundle and mount it into application pods:

```bash theme={null}
curl -o rds-ca.pem https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
kubectl create configmap rds-ca-cert --from-file=rds-ca.pem -n kaireon
```

### 8.3 In-Cluster PostgreSQL

For the Startup tier running in-cluster PostgreSQL, enable SSL in the Helm values:

```yaml theme={null}
postgresql:
  tls:
    enabled: true
    autoGenerated: true
```

***

## 9. Audit Log Retention

### 9.1 Application Audit Logs

KaireonAI emits structured audit log entries for all state-changing API operations. These logs include:

* Timestamp (ISO 8601)
* User identity (user ID, email, IP address)
* Action performed (CREATE, UPDATE, DELETE)
* Resource type and identifier
* Request body hash (not the body itself, for sensitive data protection)
* Response status code

### 9.2 Retention Policy

| Log Type               | Retention Period | Storage                        |
| ---------------------- | ---------------- | ------------------------------ |
| Application audit logs | 1 year           | CloudWatch Logs / S3 lifecycle |
| API access logs        | 90 days          | CloudWatch Logs                |
| Database query logs    | 30 days          | RDS CloudWatch integration     |
| Kubernetes audit logs  | 90 days          | CloudWatch Logs                |
| Security event logs    | 2 years          | S3 Glacier after 90 days       |

### 9.3 Log Forwarding

Use Fluent Bit as a DaemonSet to ship logs from pods to CloudWatch Logs or an external SIEM:

```yaml theme={null}
[OUTPUT]
    Name              cloudwatch_logs
    Match             kube.kaireon.*
    region            us-east-1
    log_group_name    /kaireon/production/app
    log_stream_prefix pod-
    auto_create_group true
```

### 9.4 Tamper Protection

* Enable CloudWatch Logs log group data protection for PII detection.
* Use S3 Object Lock (compliance mode) for long-term audit log archives.
* Enable CloudTrail for AWS API audit trail.

***

## 10. RBAC and Least-Privilege Access

### 10.1 Kubernetes RBAC

Create dedicated ServiceAccounts for each component. Do not use the `default` ServiceAccount.

```yaml theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kaireon-app
  namespace: kaireon
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT:role/kaireon-app-role
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kaireon-app-role
  namespace: kaireon
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames: ["kaireon-secrets"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kaireon-app-binding
  namespace: kaireon
subjects:
  - kind: ServiceAccount
    name: kaireon-app
    namespace: kaireon
roleRef:
  kind: Role
  name: kaireon-app-role
  apiGroup: rbac.authorization.k8s.io
```

### 10.2 IAM Roles for Service Accounts (IRSA)

Map Kubernetes ServiceAccounts to IAM roles with minimal permissions:

| ServiceAccount   | IAM Role            | Permissions                                   |
| ---------------- | ------------------- | --------------------------------------------- |
| kaireon-app      | kaireon-app-role    | Secrets Manager read, S3 read/write (uploads) |
| kaireon-worker   | kaireon-worker-role | S3 read/write, SQS send/receive               |
| external-secrets | kaireon-eso-role    | Secrets Manager read                          |

### 10.3 Database Access Control

* Create separate database roles for the application and workers with the minimum required privileges.
* The application role should have SELECT, INSERT, UPDATE, DELETE on application tables.
* The worker role should have SELECT, INSERT, UPDATE on pipeline-related tables only.
* No role should have DROP, CREATE, or ALTER privileges in production. Schema migrations run as a separate CI/CD step with a privileged role.

***

## 11. Additional Hardening

### 11.1 Dependency and Static Analysis Scanning

* Run `npm audit` in CI pipelines. Fail builds on critical or high severity vulnerabilities. (KaireonAI's own CI does this, plus a blocking Trivy filesystem scan and weekly Dependabot update PRs.)
* Run SAST on every pull request. KaireonAI's CI runs Semgrep (OSS mode, blocking on ERROR-severity findings) across the platform, Shopify, website, and ML worker code.
* Use Snyk or Trivy for container image scanning before pushing to ECR.
* Enable Amazon ECR image scanning on push.
* All shipped KaireonAI container images run as a non-root user (`nextjs` in the platform image, `worker` in the queue worker, `appuser` in the ML worker).

### 11.2 Image Provenance

* Use specific image tags, never `latest`.
* Sign container images with cosign and verify signatures in admission controllers.
* Pull images only from trusted registries (ECR, Docker Hub official images).

### 11.3 Runtime Protection

* Deploy Falco for runtime anomaly detection (unexpected process execution, file access).
* Enable AWS GuardDuty for EKS runtime threat detection.
* Set `readOnlyRootFilesystem: true` on all containers (writable paths via emptyDir mounts).

***

## 12. API Key Authentication

KaireonAI supports API key authentication as an alternative to JWT Bearer tokens.

### How It Works

* API keys are passed via the `X-API-Key` header
* Keys are stored as SHA-256 hashes in the `api_keys` database table (never in plaintext)
* Each key has: `name`, `hashedKey`, `prefix` (first 12 chars of the raw key, for display), `tenantId`, `scopes` (per-key scope array; empty = data-plane-only — recommend/respond; `["control-plane"]` grants management access), `expiresAt`, `lastUsedAt`, `revokedAt`
* Keys use the `krn_` prefix for easy identification in logs

### Production Checklist

* [ ] Rotate API keys every 90 days
* [ ] Set `expiresAt` on all keys — never issue non-expiring keys in production
* [ ] Monitor `kaireon_http_request_duration_seconds` filtered by API key usage patterns
* [ ] Revoke keys immediately when team members leave (set `revokedAt`)
* [ ] Use separate keys per integration (never share keys across services)

### Key Management API

```
POST   /api/v1/api-keys             — Create new key (returns plaintext once)
GET    /api/v1/api-keys             — List keys (hashed, no plaintext)
DELETE /api/v1/api-keys?id=<keyId>  — Revoke a key (id passed as query param)
```

***

## 13. Connector Credential Encryption

All connector authentication configurations (`authConfig` field) are encrypted at rest using AES-256-GCM.

### Configuration

Set the `CONNECTOR_ENCRYPTION_KEY` environment variable to a 32-byte hex string:

```bash theme={null}
# Generate a key
openssl rand -hex 32

# Set in environment or Helm values
CONNECTOR_ENCRYPTION_KEY=<64-char-hex-string>
```

### How It Works

* On `POST /api/v1/connectors` (create) and `PUT /api/v1/connectors` (update — `id` is passed in the JSON body), the `authConfig` JSON is encrypted before storage
* On `GET /api/v1/connectors`, credentials are decrypted for display (masked in list view, full in detail for admins)
* Each encrypted value includes a random 12-byte IV and 16-byte auth tag
* If `CONNECTOR_ENCRYPTION_KEY` is not set, credentials are stored in plaintext (development only)

### Production Checklist

* [ ] Set `CONNECTOR_ENCRYPTION_KEY` in all environments
* [ ] Store the key in AWS Secrets Manager (see Section 5)
* [ ] Rotate the key annually — re-encrypt all connectors after rotation
* [ ] Never log or expose the encryption key in application logs