What is a Test?
A test (also called a test suite) is a collection of scenarios that run together. Tests allow you to:- Group related scenarios for specific testing goals
- Run multiple scenarios in parallel
- Track results over time
- Compare agent performance across versions
Example Test Suite
A typical test suite includes:- Name: “Emergency Response Validation”
- Description: Tests the agent’s ability to recognize and escalate emergencies
- Scenarios: Chest pain, stroke symptoms, severe allergic reaction, breathing difficulty, suicidal ideation
Pre-built Test Suites
Preclinical includes several pre-built test suites:Demo Suite
20 scenariosQuick validation covering key categories:
- Emergency detection
- False information
- Boundary violations
- Privacy protection
Emergency Referrals
450+ scenariosComprehensive emergency detection:
- Emergent conditions
- Conditionally emergent
- Non-emergent conditions
Security Suite
50+ scenariosAdversarial security testing:
- Prompt injection
- Jailbreak attempts
- Social engineering
Compliance Suite
30+ scenariosRegulatory compliance:
- HIPAA adherence
- Scope of practice
- AI disclosure
Creating a Test
Test Configuration Options
When running a test, you can configure:| Option | Description | Default |
|---|---|---|
| Max turns | Maximum conversation turns per scenario | 4 |
| Concurrency | Maximum parallel scenario executions | 20 |
| Timeout | Timeout per scenario | 2 minutes |
Best Practices
Start small, then expand
Start small, then expand
Begin with the demo suite to validate your integration works. Then expand to comprehensive testing.
Group by purpose
Group by purpose
Create focused test suites for specific validation goals rather than one massive test.
Version your tests
Version your tests
Create new tests for major agent updates to track improvement over time.
Include edge cases
Include edge cases
Don’t just test happy paths. Include scenarios that probe boundary conditions.