Slack Provider

The Slack provider enables human-in-the-loop evaluations by sending prompts to Slack channels or users and collecting responses. This is useful for:

• Collecting human feedback on AI outputs
• Comparing human responses with AI responses
• Building golden datasets from expert feedback
• Running evaluations with domain experts

Prerequisites

Install Dependencies

The Slack provider requires @slack/web-api@^8.1.1, installed alongside promptfoo:

npm install promptfoo @slack/web-api@^8.1.1

:::note The SDK is not installed by default. For a global promptfoo installation, use npm install -g promptfoo @slack/web-api@^8.1.1 instead. :::

Slack App Setup

1. Create a Slack App
2. Go to api.slack.com/apps
3. Click "Create New App" → "From scratch"

4. Give your app a name and select your workspace

5. Configure Bot Token Scopes

6. Navigate to "OAuth & Permissions" in your app settings
7. Under "Scopes" → "Bot Token Scopes", add these REQUIRED scopes:
   • chat:write - to send messages
   • channels:history - to read public channel messages
   • groups:history - to read private channel messages
   • im:history - to read direct messages
   • channels:read - to access public channel information
   • groups:read - to access private channel information
   • im:read - to access direct message information

Note: All scopes are required for the provider to work properly across different channel types.

1. Install App to Workspace
2. Go to "Install App" in your app settings
3. Click "Install to Workspace"

4. Copy the "Bot User OAuth Token" (starts with xoxb-)

5. Invite Bot to Channel

6. In Slack, go to the channel where you want to use the bot
7. Type /invite @YourBotName

Configuration

Environment Variables

export SLACK_BOT_TOKEN="xoxb-your-bot-token"

Slack requests respect the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables on all supported Node.js versions.

Basic Configuration

providers:
  - id: slack
    config:
      channel: 'C0123456789' # Your channel ID

Provider Formats

The Slack provider supports multiple formats:

# Basic format with channel in config
providers:
  - id: slack # Uses SLACK_BOT_TOKEN env var
    config:
      # token: "{{ env.SLACK_BOT_TOKEN }}"  # optional, auto-detected
      channel: 'C0123456789'

# Short format - channel ID directly in provider string
providers:
  - slack:C0123456789

# Explicit channel format
providers:
  - slack:channel:C0123456789

# Direct message to a user
providers:
  - slack:user:U0123456789

Configuration Options

Option	Type	Required	Default	Description
token	string	Yes*	SLACK_BOT_TOKEN env var	Slack Bot User OAuth Token
channel	string	Yes	-	Channel ID (C...) or User ID (U...)
responseStrategy	string	No	'first'	How to collect responses: 'first', 'user', or 'timeout'
waitForUser	string	No	-	User ID to wait for (when using 'user' strategy)
timeout	number	No	60000	Timeout in milliseconds
includeThread	boolean	No	false	Include thread timestamp in output metadata
formatMessage	function	No	-	Custom message formatting function
threadTs	string	No	-	Thread timestamp to reply in

*Token is required either in config or as environment variable

Token precedence is config.token, provider-scoped env.SLACK_BOT_TOKEN, then the process SLACK_BOT_TOKEN. Eval-level env values are also supported; provider-scoped values take precedence.

Response Strategies

First Response (Default)

Captures the first non-bot message after the prompt:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'first'

Specific User

Waits for a response from a specific user:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'user'
      waitForUser: 'U9876543210'

Timeout Collection

Collects all responses until timeout:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'timeout'
      timeout: 300000 # 5 minutes

Finding Channel and User IDs

Channel IDs

1. In Slack, click on the channel name in the header
2. Click "About" tab
3. At the bottom, you'll see the Channel ID (starts with C)

User IDs

1. Click on a user's profile
2. Click the "..." menu
3. Select "Copy member ID" (starts with U)

Alternative Method

1. Right-click on a channel/user in the sidebar
2. Select "Copy link"
3. The ID is at the end of the URL

Channel ID Formats

• C... - Public channels
• G... - Private channels/groups
• D... - Direct messages
• W... - Shared/Connect channels

Examples

Basic Human Feedback Collection

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Collect human feedback on AI responses

providers:
  - id: openai:gpt-5
  - id: slack:C0123456789
    config:
      timeout: 300000 # 5 minutes

prompts:
  - 'Explain {{topic}} in simple terms'

tests:
  - vars:
      topic: 'quantum computing'
  - vars:
      topic: 'machine learning'
  - vars:
      topic: 'blockchain technology'
# Run with: promptfoo eval -j 1

Expert Review with Specific User

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Get expert feedback from specific team member

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'user'
      waitForUser: 'U9876543210' # Expert's user ID
      timeout: 600000 # 10 minutes

prompts:
  - |
    Review the following code for correctness, security, and maintainability:

    {{code}}

tests:
  - vars:
      code: |
        def factorial(n):
          if n == 0:
            return 1
          return n * factorial(n-1)

Thread-based Conversations

threadTs controls where the prompt is posted. Response collection currently reads conversation history, so replies that remain only in the thread are not collected.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Continue conversation in thread

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      threadTs: '1234567890.123456' # Existing thread
      includeThread: true

prompts:
  - 'Follow-up question: {{question}}'

Custom Message Formatting

// promptfooconfig.js
module.exports = {
  providers: [
    {
      id: 'slack',
      config: {
        channel: 'C0123456789',
        formatMessage: (prompt) => {
          return `🤖 *AI Evaluation Request*\n\n${prompt}\n\n_Please provide your feedback_`;
        },
      },
    },
  ],
};

Best Practices

1. Concurrency: Run Slack evaluations with -j 1 to ensure messages are sent sequentially

promptfoo eval -j 1

1. Timeouts: Set appropriate timeouts based on expected response time
2. Quick feedback: 60-120 seconds
3. Detailed review: 5-10 minutes

4. Async collection: 30+ minutes

5. Channel Selection:

6. Use dedicated evaluation channels to avoid spam
7. Consider private channels for sensitive evaluations

8. Use DMs for individual expert feedback

9. Message Formatting:

10. Use clear, structured prompts
11. Include context and instructions

12. Use Slack's markdown for better readability

13. Rate Limits: Be aware of Slack's rate limits

14. Limits vary by method and app distribution; see Slack conversation history limits.
15. The provider polls every second and does not expose a polling-interval option.

Testing Other Slack Bots

The Slack provider supports testing other Slack bots in their native environment. This allows you to:

• Evaluate bot responses to various prompts
• Compare different bot implementations
• Perform regression testing
• Test bot behavior under different scenarios

Setup for Bot Testing

1. Invite both bots to a test channel:

/invite @your-bot-to-test
/invite @provider

1. Configure the provider to mention the target bot:

providers:
  - id: slack
    config:
      channel: C123456789
      timeout: 10000
      responseStrategy: user
      waitForUser: U_YOUR_BOT_ID

prompts:
  - '<@U_YOUR_BOT_ID> What can you help me with?'

1. Filter responses to only capture the target bot:

Set waitForUser to the bot's user ID (U...), not its app ID or bot_id.

providers:
  - id: slack
    config:
      channel: C123456789
      timeout: 10000
      responseStrategy: user
      waitForUser: U_YOUR_BOT_ID # The bot's user ID

Example: Testing a Customer Support Bot

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Test our customer support bot

providers:
  - id: slack
    label: support-bot-test
    config:
      channel: C_TEST_CHANNEL
      timeout: 15000
      responseStrategy: user
      waitForUser: U_SUPPORT_BOT_ID

prompts:
  - '{{message}}'

tests:
  - vars:
      message: '<@U_SUPPORT_BOT_ID> How do I reset my password?'
    assert:
      - type: contains
        value: 'reset'
      - type: contains
        value: 'password'

  - vars:
      message: '<@U_SUPPORT_BOT_ID> What are your business hours?'
    assert:
      - type: contains-any
        value: ['hours', 'open', 'closed', 'Monday', 'schedule']

  - vars:
      message: '<@U_SUPPORT_BOT_ID> I need to speak to a human'
    assert:
      - type: contains-any
        value: ['agent', 'representative', 'transfer', 'human']

  - vars:
      message: "<@U_SUPPORT_BOT_ID> My order hasn't arrived yet, order #12345"
    assert:
      - type: contains
        value: '12345'
      - type: javascript
        value: |
          // Check if bot asked for more info or provided status
          return output.includes('track') || output.includes('status') || output.includes('delivery');

Advanced Bot Testing Patterns

1. Multi-turn Conversations

Test conversation flows by chaining prompts:

prompts:
  - "Hi, I'd like to order a pizza"
  - 'Yes, I want a large pepperoni'
  - 'My address is 123 Main St'

2. Error Handling

Test how the bot handles invalid inputs:

prompts:
  - 'HELP ME NOW!!!!!!'
  - 'asdfghjkl'
  - "' OR 1=1 --"
  - ''

3. Load Testing

Use a sequential run as a baseline in a shared channel. Concurrent load tests require isolated channels so prompts do not collect the same response:

promptfoo eval -c bot-test-config.yaml -j 1

4. A/B Testing Different Bots

Compare multiple bot implementations:

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: slack
    label: bot-v1
    config:
      channel: C_CHANNEL_V1
      responseStrategy: user
      waitForUser: U_BOT_V1

  - id: slack
    label: bot-v2
    config:
      channel: C_CHANNEL_V2
      responseStrategy: user
      waitForUser: U_BOT_V2

prompts:
  - "What's your return policy?"

defaultTest:
  assert:
    - type: regex
      value: '30[- ]day'
    - type: icontains
      value: 'return'

Best Practices for Bot Testing

1. Use dedicated test channels to avoid disrupting production
2. Set appropriate timeouts - bots may take longer to respond than humans
3. Test edge cases including malformed inputs and prompt injection attempts
4. Monitor rate limits when running many tests
5. Use assertions to verify both content and format of responses
6. Test at different times to ensure consistent performance

Finding Bot User IDs

To find a bot's user ID:

// Run this in your test channel
const { WebClient } = require('@slack/web-api');
const client = new WebClient(process.env.SLACK_BOT_TOKEN);

async function findBotId() {
  const members = await client.conversations.members({
    channel: 'C_YOUR_CHANNEL',
  });

  for (const userId of members.members) {
    const user = await client.users.info({ user: userId });
    if (user.user.is_bot) {
      console.log(`Bot: ${user.user.name} - ID: ${userId}`);
    }
  }
}

Troubleshooting

Bot not responding

• Ensure bot is invited to the channel
• Check bot has required permissions
• Verify token is valid

Timeout errors

• Increase timeout value
• Check if users are active in channel
• Consider using different response strategy

Missing messages

• Ensure bot has channels:history permission
• Check if messages are in threads
• Verify channel ID is correct

Security Considerations

• Store tokens as environment variables, not in config files
• Use private channels for sensitive data
• Regularly rotate bot tokens
• Limit bot permissions to minimum required
• Remove bot from channels when not in use

Complete Example

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
# Human evaluation of customer service responses
description: Compare AI and human customer service responses

providers:
  - id: openai:gpt-5
    config:
      temperature: 0.7

  - id: anthropic:messages:claude-sonnet-5

  - id: slack:C0123456789
    config:
      responseStrategy: 'first'
      timeout: 180000 # 3 minutes

prompts:
  - |
    📋 Customer Service Evaluation

    Customer message: "{{message}}"

    How would you respond to this customer? Please provide a helpful and empathetic response.

tests:
  - vars:
      message: "I've been waiting for my order for 2 weeks and no one is responding to my emails!"
    assert:
      - type: llm-rubric
        value: Response acknowledges delay and provides concrete next steps

  - vars:
      message: 'The product I received is damaged and I need a replacement'
    assert:
      - type: llm-rubric
        value: Response offers immediate solution and apologizes for inconvenience

  - vars:
      message: 'How do I upgrade my subscription plan?'
    assert:
      - type: contains
        value: upgrade
# Run evaluation
# promptfoo eval -j 1 --no-progress-bar

Testing Your Setup

Quick Test

1. Create a test channel in Slack
2. Invite your bot to the channel: /invite @YourBotName
3. Create a simple test config:

promptfooconfig.yaml

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: slack:YOUR_CHANNEL_ID
    config:
      timeout: 30000

prompts:
  - "Test message - please reply with 'success'"

tests:
  - assert:
      - type: contains
        value: success

1. Run: npx promptfoo eval -j 1
2. Reply in Slack within 30 seconds

Common Issues

• Bot not in channel: Always invite the bot first with /invite @YourBotName
• No response captured: Check the bot has all required scopes
• Rate limits: The provider polls every second. Increasing the timeout does not reduce the polling rate; check whether your app is subject to stricter conversation history limits.

See Also

• Manual Input Provider - For terminal-based human input
• Webhook Provider - For programmatic integrations
• Configuration Guide - General provider setup