Skip to main content

​
Overview

Preclinical can test any OpenAI-compatible chat completion API. This includes:
  • OpenAI’s official API
  • Azure OpenAI
  • Local models (vLLM, Ollama, etc.)
  • Any API following the OpenAI chat completions format

​
Configuration

interface OpenAIConfig {
  provider: "openai";
  config: {
    api_key: string;           // API key for authentication
    base_url: string;          // API base URL
    model: string;             // Model to use

    // Optional
    temperature?: number;      // Default: 0.7
    max_tokens?: number;       // Default: 1024
    system_prompt?: string;    // Override agent's system prompt
    timeout_ms?: number;       // Default: 60000
  };
}

​
Setup Steps

1

Identify Your Endpoint

Determine your API endpoint:
ProviderBase URL
OpenAIhttps://api.openai.com/v1
Azure OpenAIhttps://{resource}.openai.azure.com/openai/deployments/{deployment}
Local/CustomYour server URL
2

Get API Key

Obtain an API key from your provider
3

Add Integration

{
  "name": "My OpenAI Agent",
  "provider": "openai",
  "config": {
    "api_key": "sk-xxxxx",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o"
  }
}

​
Provider Examples

{
  "provider": "openai",
  "config": {
    "api_key": "sk-xxxxx",
    "base_url": "https://api.openai.com/v1",
    "model": "gpt-4o"
  }
}

​
How It Works

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Preclinical   │────▶│   /chat/        │────▶│    Your Model   │
│   (Pen Tester)  │◀────│   completions   │◀────│                 │
└─────────────────┘     └─────────────────┘     └─────────────────┘
  1. Preclinical sends a chat completion request
  2. Your endpoint processes the request
  3. Response is captured
  4. Process repeats for configured turns
  5. Full conversation is graded

​
API Format

Preclinical uses the standard OpenAI chat completions format: 
{
  "model": "gpt-4o",
  "messages": [
    {"role": "system", "content": "You are a healthcare assistant..."},
    {"role": "user", "content": "I have chest pain..."},
    {"role": "assistant", "content": "I understand..."},
    {"role": "user", "content": "It's getting worse..."}
  ],
  "temperature": 0.7,
  "max_tokens": 1024
}

​
Features

Universal Compatibility

Works with any OpenAI-compatible endpoint

System Prompt Override

Optionally override the model’s system prompt

Configurable Parameters

Control temperature, max tokens, and more

Streaming Support

Handles streaming and non-streaming responses

​
Testing Custom Agents

If you have a chat-based healthcare agent, you can test it by:
  1. Exposing it via an OpenAI-compatible endpoint
  2. Configuring the system prompt to match your agent’s personality
  3. Running Preclinical tests against it
{
  "provider": "openai",
  "config": {
    "api_key": "your-api-key",
    "base_url": "https://your-agent-api.com/v1",
    "model": "healthcare-assistant-v2",
    "system_prompt": "You are a medical triage assistant..."
  }
}

​
Error Handling

​
Common Errors

ErrorCauseResolution
401 UnauthorizedInvalid API keyCheck API key
404 Not FoundInvalid model/endpointVerify base URL and model name
429 Rate LimitToo many requestsAutomatic retry with backoff
500 Server ErrorProvider errorAutomatic retry

​
Rate Limits

Preclinical handles rate limits automatically: 
Request → 429 → Wait (Retry-After or exponential backoff) → Retry

​
Troubleshooting

  • Verify base URL is correct and accessible
  • Check for trailing slashes (shouldn’t have one)
  • Ensure server is running (for local models)
  • Verify model name matches exactly
  • For Azure, use deployment name as model
  • Check model is available on your tier
  • Verify API key is correct
  • Check key has required permissions
  • For Azure, ensure key is for correct resource

​
Next Steps

Run a Test

Execute your first test run

Other Integrations

Explore voice-based providers