Skip to content

Prompts, tests, and outputs

Configure how promptfoo evaluates your LLM applications.

:::tip Detailed Documentation For comprehensive guides, see the dedicated pages:

Quick Start

promptfooconfig.yaml
# Define your prompts
prompts:
  - 'Translate to {{language}}: {{text}}'

# Configure test cases
tests:
  - vars:
      language: French
      text: Hello world
    assert:
      - type: contains
        value: Bonjour
# Run evaluation
# promptfoo eval

Core Concepts

📝 Prompts

Define what you send to your LLMs - from simple strings to complex conversations.

Common patterns **Text prompts** 
prompts:
  - 'Summarize this: {{content}}'
  - file://prompts/customer_service.txt
**Chat conversations** 
prompts:
  - file://prompts/chat.json
**Dynamic prompts** 
prompts:
  - file://generate_prompt.js
  - file://create_prompt.py

Learn more about prompts →

🧪 Test Cases

Configure evaluation scenarios with variables and assertions.

Common patterns **Inline tests** 
tests:
  - vars:
      question: "What's 2+2?"
    assert:
      - type: equals
        value: '4'
**CSV test data** 
tests: file://test_cases.csv
**HuggingFace datasets** 
tests: huggingface://datasets/rajpurkar/squad
**Azure Blob Storage** 
tests: az://myaccount/evals/tests.json
**Dynamic generation** 
tests: file://generate_tests.js

Learn more about test cases →

📊 Output Formats

Save and analyze your evaluation results.

Available formats 
# Visual report
promptfoo eval --output results.html

# Data analysis
promptfoo eval --output results.json

# Spreadsheet
promptfoo eval --output results.csv

Learn more about outputs →

Complete Example

Here's a real-world example that combines multiple features:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Customer service chatbot evaluation

prompts:
  # Simple text prompt
  - 'You are a helpful customer service agent. {{query}}'

  # Chat conversation format
  - file://prompts/chat_conversation.json

  # Dynamic prompt with logic
  - file://prompts/generate_prompt.js

providers:
  - openai:gpt-5-mini
  - anthropic:claude-haiku-4-5

tests:
  # Inline test cases
  - vars:
      query: 'I need to return a product'
    assert:
      - type: contains
        value: 'return policy'
      - type: llm-rubric
        value: 'Response is helpful and professional'

  # Load more tests from CSV
  - file://test_scenarios.csv

# Save results
outputPath: evaluations/customer_service_results.html

Quick Reference

Supported File Formats

Format Prompts Tests Use Case
.txt ✅ ❌ Simple text prompts
.json ✅ ✅ Chat conversations, structured data
.yaml ✅ ✅ Complex configurations
.csv ✅ ✅ Bulk data, multiple variants
.js/.ts ✅ ✅ Dynamic generation with logic
.py ✅ ✅ Python-based generation
.md ✅ ❌ Markdown-formatted prompts
.j2 ✅ ❌ Jinja2 templates
HuggingFace datasets ❌ ✅ Import from existing datasets

Variable Syntax

Variables use Nunjucks templating:

# Basic substitution
prompt: 'Hello {{name}}'

# Filters
prompt: 'URGENT: {{message | upper}}'

# Conditionals
prompt: '{% if premium %}Premium support: {% endif %}{{query}}'

File References

All file paths are relative to the config file:

# Single file
prompts:
  - file://prompts/main.txt

# Multiple files with glob
tests:
  - file://tests/*.yaml

# Specific function
prompts:
  - file://generate.js:createPrompt

Wildcards like path/to/prompts/**/*.py:func_name are also supported.

Next Steps