> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kaireonai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Self-Hosted LLM Setup

> Run KaireonAI's AI features with your own local or self-hosted language model — Ollama, vLLM, LM Studio, or any OpenAI-compatible API.

## Overview

KaireonAI's AI features (chat assistant, insights, content intelligence, rule builder) can run against any OpenAI-compatible LLM endpoint. This guide covers self-hosted options for environments where external API calls are not permitted.

## Quick Start with Ollama

[Ollama](https://ollama.com) is the fastest way to run a local LLM. It runs on Mac, Linux, and Windows.

### 1. Install Ollama

```bash theme={null}
# macOS / Linux
curl -fsSL https://ollama.com/install.sh | sh

# macOS (Homebrew)
brew install ollama
```

### 2. Pull a Model

```bash theme={null}
# Recommended for tool calling (AI assistant features)
ollama pull qwen2.5:7b       # 4.7GB, fast, good at tool calling
ollama pull llama3.1          # 4.7GB, general purpose

# For better quality (needs 16GB+ RAM)
ollama pull qwen2.5:14b      # 9GB, excellent quality
ollama pull llama3.1:70b      # 40GB, near-GPT-4 quality (needs 64GB RAM)
```

### 3. Start Ollama

```bash theme={null}
ollama serve
```

Ollama runs on `http://localhost:11434` by default.

### 4. Configure in KaireonAI

Navigate to **Settings > AI Configuration** and set:

| Setting  | Value                               |
| -------- | ----------------------------------- |
| Provider | `ollama`                            |
| Model    | `qwen2.5:7b` (or your chosen model) |
| Base URL | `http://localhost:11434`            |
| API Key  | (leave empty for Ollama)            |

Or via API:

```bash theme={null}
# Set AI provider to Ollama. The PUT body takes `settings` as an ARRAY of
# { key, value } pairs (an object map is rejected with 400). Requires an
# admin session/API key; in multi-tenant mode only the platform-owner tenant
# may write platform settings.
curl -X PUT http://localhost:3000/api/v1/platform-settings \
  -H "Content-Type: application/json" \
  -H "X-Requested-With: XMLHttpRequest" \
  -H "X-API-Key: <admin-api-key>" \
  -d '{
    "category": "ai",
    "settings": [
      { "key": "ai_provider", "value": "ollama" },
      { "key": "ai_model", "value": "qwen2.5:7b" },
      { "key": "ai_base_url", "value": "http://localhost:11434" },
      { "key": "ai_api_key", "value": "", "encrypted": true }
    ]
  }'
```

## Other Self-Hosted Options

### vLLM (GPU Server)

Best for production deployments with GPU instances.

```bash theme={null}
# On a GPU instance (e.g., AWS g5.xlarge)
pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000
```

Configure in KaireonAI:

* Provider: `openai` (vLLM is OpenAI-compatible)
* Base URL: `http://your-gpu-server:8000/v1`
* Model: `meta-llama/Llama-3.1-8B-Instruct`

### LM Studio (Desktop)

Download from [lmstudio.ai](https://lmstudio.ai), load a model, and start the local server.

Configure in KaireonAI:

* Provider: `lm_studio`
* Base URL: `http://localhost:1234/v1`
* Model: (auto-detected)

### HuggingFace Text Generation Inference

```bash theme={null}
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct
```

Configure: Provider `openai`, Base URL `http://localhost:8080/v1`

## Supported Providers

| Provider           | Tool Calling            | Streaming | Local | Cloud |
| ------------------ | ----------------------- | --------- | ----- | ----- |
| Google (Gemini)    | Yes                     | Yes       | No    | Yes   |
| OpenAI (GPT)       | Yes                     | Yes       | No    | Yes   |
| Anthropic (Claude) | Yes                     | Yes       | No    | Yes   |
| Ollama             | Yes (qwen2.5, llama3.1) | Yes       | Yes   | No    |
| LM Studio          | Partial                 | Yes       | Yes   | No    |
| vLLM               | Yes                     | Yes       | Yes   | Yes   |
| AWS Bedrock        | Yes                     | Yes       | No    | Yes   |

## Bring Your Own Key (BYOK)

On the KaireonAI Playground (`playground.kaireonai.com`), each registered user can configure their own LLM provider:

1. Go to **Settings > AI Configuration**
2. Select your preferred provider
3. Enter your API key (encrypted at rest, never shared)
4. Your key is scoped to your tenant only

<Info>
  API keys are encrypted using AES-256 before storage. They are never returned in API responses — only `****` masking is shown. Keys can be rotated at any time without affecting other tenants.
</Info>

## Model Recommendations

| Use Case                 | Model            | RAM Required | Notes                      |
| ------------------------ | ---------------- | ------------ | -------------------------- |
| Dev/testing              | qwen2.5:7b       | 8GB          | Fast, good tool calling    |
| Demo                     | llama3.1         | 8GB          | Good general quality       |
| Production (self-hosted) | qwen2.5:14b      | 16GB         | Best quality/speed balance |
| Enterprise               | llama3.1:70b     | 64GB         | Near-cloud quality         |
| Cloud (no infra)         | gemini-2.5-flash | N/A          | Free tier: 20 req/min      |

## Docker Deployment with Ollama

For Docker-based deployments, add Ollama as a sidecar:

```yaml theme={null}
# docker-compose.yml
services:
  kaireon-api:
    image: 422500312304.dkr.ecr.us-east-1.amazonaws.com/kaireon-api:latest
    environment:
      - AI_PROVIDER=ollama
      - AI_MODEL=qwen2.5:7b
      - AI_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama
    volumes:
      - ollama_data:/root/.ollama
    # Pull model on first start:
    # docker exec ollama ollama pull qwen2.5:7b

volumes:
  ollama_data:
```