Skip to main content

AI Provider Configuration

How to configure AI providers in orchestr8 for secure, enterprise-grade AI workloads.

orchestr8 Provider Security​

orchestr8 enhances AI provider integration with:

  • External Secrets Management: API keys stored in AWS Secrets Manager/HashiCorp Vault
  • Network Policies: Controlled egress to AI provider APIs
  • RBAC Integration: Fine-grained access control per workload
  • Audit Logging: Complete audit trail of AI API usage
  • Cost Tracking: Per-tenant usage monitoring and billing

Quick Setup​

1. Store Provider Secrets​

Store AI provider API keys in your external secret management system:

# AWS Secrets Manager example
aws secretsmanager put-secret-value \
  --secret-id /orchestr8/llama-stack/api-keys \
  --secret-string '{
    "openai_api_key": "sk-...",
    "anthropic_api_key": "sk-ant-..."
  }'

2. Verify Secret Sync​

orchestr8 automatically syncs these secrets to Kubernetes:

# Check if secrets are available
kubectl get secret llama-stack-api-keys -n llama-stack

# Test provider connectivity
o8 llama providers --provider openai

3. Deploy AI Workloads​

With providers configured, create and deploy AI workloads:

# Create a RAG application
o8 llama init my-rag-app --template rag --provider openai

# Deploy it
o8 llama deploy --env dev

# Monitor status
o8 llama status

orchestr8 Security Features​

Network Isolation​

  • AI workloads can only access authorized provider APIs
  • Network policies enforce default-deny with explicit allow rules
  • All provider communication is logged and auditable

Multi-Tenant Cost Tracking​

  • Per-namespace cost allocation and budgeting
  • Usage dashboards in Grafana
  • Automated budget alerts and limits

Compliance Controls​

  • All AI API calls are logged for audit compliance
  • Data residency controls for regulated workloads
  • Automated compliance reporting

Management Commands​

# List all configured providers
o8 llama providers

# Test provider connectivity
o8 llama providers --provider openai

# Monitor AI workload status
o8 llama status

# View provider usage logs
o8 llama logs --follow

Troubleshooting​

Check Provider Configuration​

# Verify secrets are synced
kubectl get secret llama-stack-api-keys -n llama-stack

# Test provider connectivity
o8 llama providers --provider openai

Common Issues​

  • Authentication Errors: Check API key validity in secret management system
  • Network Connectivity: Verify network policies allow egress to provider APIs
  • Rate Limiting: Monitor usage and implement caching strategies

Next Steps​