Promptfoo

Promptfoo is a CLI tool for testing and comparing LLM outputs.

Config File

The CLI auto-discovers promptfooconfig.yaml in the current directory. Use -c path for other locations.

Supported extensions: .yaml, .json, .js

Configuration

yaml
1# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
2description: "What this eval tests"
3
4prompts:
5  - file://prompt.txt
6  - |
7    Inline prompt with {{variable}} substitution
8
9providers:
10  - anthropic:messages:claude-sonnet-4-5-20250929
11
12defaultTest:
13  options:
14    provider:
15      config:
16        temperature: 0.0
17        max_tokens: 4096
18
19tests:
20  - description: "What this case tests"
21    vars:
22      variable: "value"
23      from_file: file://data/input.txt
24    assert:
25      - type: contains
26        value: "expected substring"
27
28# Or load tests from files
29tests: file://cases/all.yaml
30
31outputPath: ./results.json
32
33evaluateOptions:
34  maxConcurrency: 4

Provider IDs

Provider config: temperature, max_tokens, top_p, top_k, tools, tool_choice

Prompts

• file://path.txt — load from file (path relative to config)
• Inline string with {{variable}} Nunjucks substitution
• Chat format via JSON: [{"role": "system", "content": "..."}, {"role": "user", "content": "{{input}}"}]

Assertion Types

Prefix any assertion with not- to negate (e.g., not-contains).

llm-rubric

Uses an LLM to grade output against a rubric:

yaml
1assert:
2  - type: llm-rubric
3    value: |
4      The response should:
5      - Mention at least 3 factors
6      - Include specific examples
7    threshold: 0.7
8    provider: anthropic:messages:claude-sonnet-4-5-20250929

javascript

Inline expressions or functions. Access output (string) and context (with vars, prompt):

yaml
1assert:
2  - type: javascript
3    value: output.length > 100 && output.includes('route')
4  - type: javascript
5    value: |
6      const data = JSON.parse(output);
7      return data.calories >= 200 && data.calories <= 300;

Test Organization

Split cases into separate files and reference them:

yaml
1tests:
2  - file://cases/basic.yaml
3  - file://cases/edge-cases.yaml

Each case file contains a YAML array of test objects.

CLI

bash
1npx promptfoo eval                         # Run with auto-discovered config
2npx promptfoo eval -c path/to/config.yaml  # Specific config
3npx promptfoo eval --filter-metadata key=v # Filter tests
4npx promptfoo view                         # Web UI for results
5npx promptfoo cache clear                  # Clear result cache

References

Consult the configuration reference and Anthropic provider docs for full details.