LiteLLM Integration

Overview

Integrate LiteLLM with Cloudidr LLM Ops to automatically track API usage and costs. LiteLLM sends webhook callbacks to our system after each API request, allowing you to monitor usage, costs, and organize by department, team, and agent.

How it works:LiteLLM → Webhook Callback → CloudIDR LLM Ops → DashboardConfigure LiteLLM to send webhooks to our endpoint, add tracking headers to your API requests, and we’ll automatically track all usage and costs.

Your Tracking Token

Keep this token secure!Your tracking token is required for all LiteLLM webhook callbacks. Don’t share it publicly or commit it to version control.

trk_fXOn-A1V8VrCxXyJ1WuMX-KlXn-e84mr

Use environment variables to store your tracking token securely.

Configuration Steps

Update LiteLLM config.yaml

Add webhook callback configuration to your LiteLLM config file

Add Tracking Headers

Include required and optional headers in your API requests

Restart & Test

Restart LiteLLM and verify tracking in your dashboard

Step 1: Update LiteLLM config.yaml

Choose the configuration that matches your setup:

With Database (Recommended)
Without Database (Optional)

Configuration with Database

Use this if you want to keep LiteLLM’s database features (user management, spend tracking, etc.)

# LiteLLM Proxy Configuration - With Database
model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-4
    litellm_params:
      model: claude-sonnet-4
      api_key: os.environ/ANTHROPIC_API_KEY

# Generic API Callback - Point to our webhook endpoint
callback_settings:
  llm_ops:
    callback_type: generic_api
    endpoint: https://api.llm-ops.cloudidr.com/api/litellm/callback
    headers:
      Content-Type: application/json
    event_types:
      - llm_api_success
      - llm_api_failure

litellm_settings:
  callbacks: ["llm_ops"]
  # Use your existing LiteLLM database (PostgreSQL, SQLite, etc.)
  database_url: os.environ/DATABASE_URL  # e.g., "postgresql://user:pass@host:5432/litellm"

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Recommended for most users - This preserves all LiteLLM features while adding LLM Ops tracking.

Configuration without Database

Use this only if you don’t need LiteLLM’s database features and want to disable them.

# LiteLLM Proxy Configuration - Without Database
# ⚠️  Note: This disables LiteLLM's built-in features like user management and spend tracking
# Only use this if you're managing everything through LLM Ops

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/OPENAI_API_KEY

# Generic API Callback - Point to our webhook endpoint
callback_settings:
  llm_ops:
    callback_type: generic_api
    endpoint: https://api.llm-ops.cloudidr.com/api/litellm/callback
    headers:
      Content-Type: application/json
    event_types:
      - llm_api_success
      - llm_api_failure

litellm_settings:
  callbacks: ["llm_ops"]
  database_url: ""  # Empty string disables database

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  disable_database_checks: true  # Required when database_url is empty

Important: This disables LiteLLM’s user management and spend tracking features. Only use if you’re managing everything through LLM Ops.

Recommendation: If you’re already using LiteLLM with a database, use the “With Database” configuration. Only use “Without Database” if you don’t need LiteLLM’s built-in features.

Step 2: Add Tracking Headers to Your Requests

Include tracking headers in your API requests to LiteLLM.

Required Header

Header	Description	Example
`X-Cloudidr-Token`	Required - Your tracking token	`trk_fXOn-A1V8VrCxXyJ1WuMX...`

Optional Metadata Headers

Header	Description	Example
`X-Department`	Organize costs by department	`engineering`, `sales`, `marketing`
`X-Team`	Organize costs by team	`backend`, `frontend`, `ml`
`X-Agent`	Organize costs by agent/application	`chatbot`, `summarizer`, `analyzer`

Optional Metadata: These headers are optional. If omitted, requests will still be tracked, but won’t be organized by department/team/agent in your dashboard.

Code Examples

Python
JavaScript
cURL

import openai
from openai import OpenAI

client = OpenAI(
    api_key="your-openai-key",
    base_url="http://localhost:4000/v1",  # Your LiteLLM proxy URL local host or your production url
)

# Make request with tracking headers
# X-Cloudidr-Token is required; X-Department, X-Team, X-Agent are optional
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={
        "X-Cloudidr-Token": "trk_fXOn-A1V8VrCxXyJ1WuMX-KlXn-e84mr",  # Required
        # Optional metadata headers:
        "X-Department": "engineering",  # Optional
        "X-Team": "backend",            # Optional
        "X-Agent": "chatbot"            # Optional
    }
)

print(response.choices[0].message.content)

Use extra_headers parameter to pass custom headers to LiteLLM.

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'your-openai-key',
  baseURL: 'http://localhost:4000/v1',  // Your LiteLLM proxy URL
  defaultHeaders: {
    'X-Cloudidr-Token': 'trk_fXOn-A1V8VrCxXyJ1WuMX-KlXn-e84mr',  // Required
    // Optional metadata headers:
    'X-Department': 'engineering',  // Optional
    'X-Team': 'backend',            // Optional
    'X-Agent': 'chatbot'            // Optional
  }
});

const response = await client.chat.completions.create({
  model: 'gpt-4',
  messages: [{ role: 'user', content: 'Hello!' }]
});

console.log(response.choices[0].message.content);

Use defaultHeaders to set headers once for all requests.

curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer your-master-key" \
  -H "Content-Type: application/json" \
  -H "X-Cloudidr-Token: trk_fXOn-A1V8VrCxXyJ1WuMX-KlXn-e84mr" \
  -H "X-Department: engineering" \
  -H "X-Team: backend" \
  -H "X-Agent: chatbot" \
  -d '{
    "model": "gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "Hello!"
      }
    ]
  }'

Note: X-Cloudidr-Token is required. X-Department, X-Team, X-Agent are optional.

Step 3: Restart LiteLLM and Test

Restart your LiteLLM proxy and make a test request. Check your dashboard to verify the request was tracked.

# Restart LiteLLM with updated config
litellm --config config.yaml --port 4000

# Make a test request
curl -X POST http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer your-master-key" \
  -H "Content-Type: application/json" \
  -H "X-Cloudidr-Token: trk_fXOn-A1V8VrCxXyJ1WuMX-KlXn-e84mr" \
  -d '{
    "model": "gpt-4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Success! If everything is configured correctly, you should see the request appear in your LLM Ops Dashboard within a few seconds.

Troubleshooting

Webhooks not being received?

Common issues:

✅ Verify callback_settings is correctly formatted in config.yaml
✅ Check that headers field is included (even if just Content-Type: application/json)
✅ Ensure the endpoint URL is accessible from your LiteLLM server
✅ Check LiteLLM logs for callback errors (enable set_verbose: true)

Test webhook connectivity:

curl -X POST https://api.llm-ops.cloudidr.com/api/litellm/callback \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

Requests not showing in dashboard?

Common issues:

✅ Verify X-Cloudidr-Token header is included in your requests
✅ Check that the tracking token is active (not revoked)
✅ Ensure the token belongs to your organization
✅ Check API server logs for [LITELLM] messages

Debug tip: Enable verbose logging in LiteLLM:

litellm_settings:
  set_verbose: true

Metadata (Department/Team/Agent) not showing?

Remember:

These headers are optional - requests will still be tracked without them
Verify headers are passed correctly: X-Department, X-Team, X-Agent
Check that headers are passed via extra_headers (Python) or defaultHeaders (JavaScript)
For cURL, include headers directly: -H "X-Department: engineering"

Note: We currently extract metadata from custom headers. LiteLLM’s built-in x-litellm-tags header is not automatically mapped to our metadata fields.

Database configuration issues?

With Database:

Keep your existing database_url configuration
Don’t add disable_database_checks

Without Database:

Set database_url: "" (empty string)
Must add disable_database_checks: true

Error: “Database checks failed”

You forgot disable_database_checks: true when using empty database_url

What Gets Tracked

LLM Ops automatically captures from LiteLLM webhooks: ✅ Token usage - Input, output, and total tokens
✅ Cost - Real-time cost calculation
✅ Latency - Request duration
✅ Model - Which model was used
✅ Metadata - Department, team, agent (from headers)
✅ Errors - Failed requests and error types
✅ Source - Marked as litellm in the database

What We DON’T Track:

❌ Customer API keys
❌ Request content (prompts)
❌ Response content (completions)

We only track metadata needed for cost analytics.

Important Notes

Required vs Optional Headers

Required: X-Cloudidr-Token header must be included in all requests
Optional: X-Department, X-Team, X-Agent headers are optional metadata for organizing costs

LiteLLM Proxy Mode

LiteLLM proxy mode requires callback_settings with generic_api type
Direct URLs don’t work - must use webhook callbacks
The headers field is required in callback_settings, even if just Content-Type: application/json

Database Configuration

If you’re using LiteLLM’s database, keep your existing database_url configuration
Only set database_url: "" and disable_database_checks: true if you don’t need LiteLLM’s database features

Security Best Practices

For production, use environment variables for API keys and master keys
Never commit tokens or keys to version control
Rotate tracking tokens regularly
Use HTTPS for production deployments

LiteLLM Tags

We currently extract metadata from custom headers (X-Department, etc.).LiteLLM’s built-in x-litellm-tags header is not automatically mapped to our metadata fields.Use the X-Department, X-Team, X-Agent headers for proper organization.

View Your Data

After making requests, view your costs in the LLM Ops Dashboard:

Agent Explorer - See costs by agent/application
Department Breakdown - Compare department spending
Team Analysis - Track team-level costs
Model Comparison - Compare costs across models routed through LiteLLM
Time Series - Track spending over time
Source Filter - Filter by source (all LiteLLM requests marked as litellm)

LiteLLM Features

LiteLLM provides powerful features that work seamlessly with LLM Ops tracking:

Multi-Provider Support

Route to OpenAI, Anthropic, Google, and 100+ providers

Load Balancing

Distribute requests across multiple API keys

Fallback Handling

Automatic failover when providers are down

Rate Limiting

Control costs with built-in rate limits

All features work with LLM Ops cost tracking - no configuration needed!

Need Help?

Email Support

Discord Community

Join our Discord for quick help

LiteLLM Docs

Official LiteLLM documentation

View Dashboard

Check your tracked requests

Next Steps

Configure LiteLLM

Add webhook callback to your config.yaml

Add Headers

Include X-Cloudidr-Token in your requests

Monitor Costs

View usage and costs in your dashboard

Optimize

Use insights to reduce API costs

Email Support

Get Started

Flex Compute

LLM Ops

Overview

Your Tracking Token

Configuration Steps

Step 1: Update LiteLLM config.yaml

Configuration with Database

Configuration without Database

Step 2: Add Tracking Headers to Your Requests

Required Header

Optional Metadata Headers

Code Examples

Step 3: Restart LiteLLM and Test

Troubleshooting

What Gets Tracked

Important Notes

View Your Data

LiteLLM Features

Multi-Provider Support

Load Balancing

Fallback Handling

Rate Limiting

Need Help?

Discord Community

LiteLLM Docs

View Dashboard

Next Steps

Get Started

Flex Compute

LLM Ops

​Overview

​Your Tracking Token

​Configuration Steps

​Step 1: Update LiteLLM config.yaml

​Configuration with Database

​Configuration without Database

​Step 2: Add Tracking Headers to Your Requests

​Required Header

​Optional Metadata Headers

​Code Examples

​Step 3: Restart LiteLLM and Test

​Troubleshooting

​What Gets Tracked

​Important Notes

​View Your Data

​LiteLLM Features

Multi-Provider Support

Load Balancing

Fallback Handling

Rate Limiting

​Need Help?

Email Support

Discord Community

LiteLLM Docs

View Dashboard

​Next Steps

Overview

Your Tracking Token

Configuration Steps

Step 1: Update LiteLLM config.yaml

Configuration with Database

Configuration without Database

Step 2: Add Tracking Headers to Your Requests

Required Header

Optional Metadata Headers

Code Examples

Step 3: Restart LiteLLM and Test

Troubleshooting

What Gets Tracked

Important Notes

View Your Data

LiteLLM Features

Need Help?

Next Steps