Skip to content

Using the node package

Installation

promptfoo is available as a node package on npm:

npm install promptfoo

:::note

promptfoo requires Node.js 22.22.0 or newer. Node.js 24 LTS is recommended. See the runtime support guide.

:::

For deeper programmatic usage, see the Node API reference, examples, and quick reference.

Usage

Use promptfoo as a library in your project by importing the evaluate function and other utilities:

import promptfoo from 'promptfoo';

const evalRecord = await promptfoo.evaluate(testSuite, options);
const results = await evalRecord.toEvaluateSummary();

The evaluate function takes the following parameters:

The evaluate function returns an Eval record. Call toEvaluateSummary() on that record to get an EvaluateSummary object.

Each evaluate() call uses its own env overrides; an omitted or empty env uses process settings without inheriting a previous evaluation. Set testSuite.basePath to resolve local references from another directory. Relative values resolve from the working directory; saved config retains the absolute directory. SDK test-file and generator inputs remain source references in saved configs, so replay requires those sources.

Provider functions

A ProviderFunction is a Javascript function that implements an LLM API call. It takes a prompt string and a context. It returns the LLM response or an error. See ProviderFunction type.

You can load providers using the loadApiProvider function:

import { loadApiProvider } from 'promptfoo';

// Load a provider with default options
const provider = await loadApiProvider('openai:o3-mini');

// Load a provider with custom options
const providerWithOptions = await loadApiProvider('azure:chat:test', {
  options: {
    apiHost: 'test-host',
    apiKey: 'test-key',
  },
});

During loadApiProvider and loadApiProviders, an omitted or undefined env inherits the active environment. An object replaces it, including {}. For loadApiProvider('file://provider.yaml', { env }), precedence from lowest to highest is inherited environment, file env, explicit caller env, then provider options.env. Providers that read environment variables later use the environment active at that call.

loadApiProviders('file://provider.yaml', { env }) expands the file into provider options, so the file's env overrides caller env. File references supplied as provider objects, such as loadApiProviders([{ id: 'file://provider.yaml' }], { env }), follow the direct-loader precedence instead. These distinctions preserve existing loader behavior.

Environment files loaded by reusable Node/MCP eval calls stay within that call. The values remain available to provider configuration templates and subprocesses, subject to each provider's environment inheritance controls, without changing the host's process.env. Custom JavaScript providers should use templated provider config for values supplied by those files.

Pass environment files with the invocation's envPath (CLI: --env-file) when values are needed during config or provider loading. Config-defined commandLineOptions.envPath retains its existing timing: it loads after config resolution.

Assertion functions

An Assertion can take an AssertionValueFunction as its value. The function receives:

  • output: the LLM output string
  • context: execution context, including prompt, vars, test, logProbs, config, provider, providerResponse, and optional trace data for debugging
Type definition 
type AssertionValueFunction = (
  output: string,
  context: AssertionValueFunctionContext,
) => AssertionValueFunctionResult | Promise<AssertionValueFunctionResult>;

interface AssertionValueFunctionContext {
prompt: string | undefined;
vars: Record<string, unknown>;
test: AtomicTestCase;
logProbs: number[] | undefined;
config?: Record<string, any>;
provider: ApiProvider | undefined;
providerResponse: ProviderResponse | undefined;
trace?: TraceData;
}

type AssertionValueFunctionResult = boolean | number | GradingResult;

interface GradingResult {
// Whether the test passed or failed
pass: boolean;

// Test score, typically between 0 and 1
score: number;

// Plain text reason for the result
reason: string;

// Map of labeled metrics to values
namedScores?: Record<string, number> | null;

// Weighted denominator for namedScores when assertion weights are used
namedScoreWeights?: Record<string, number> | null;

// Record of tokens usage for this assertion
tokensUsed?: Partial<{
total: number;
prompt: number;
completion: number;
cached?: number;
}>;

// Additional matcher/provider metadata
metadata?: Record<string, unknown>;

// List of results for each component of the assertion
componentResults?: GradingResult[] | null;

// The assertion that was evaluated
assertion?: Assertion;
}

````
</details>

For more info on different assertion types, see [assertions & metrics](/docs/configuration/expected-outputs/).

### Transform functions

When using the node package, you can pass JavaScript functions directly as `transform`, `transformVars`, or `contextTransform` values — instead of string expressions or `file://` references.

This enables better IDE support, type checking, and debugging:

```ts
import promptfoo from 'promptfoo';

const evalRecord = await promptfoo.evaluate({
  prompts: ['What tools did you use to answer: {{question}}'],
  providers: ['openai:gpt-5-mini'],
  tests: [
    {
      vars: { question: 'What is 2+2?' },
      options: {
        // Transform the output before assertions
        transform: (output, context) => {
          return output.toUpperCase();
        },
      },
      assert: [
        {
          type: 'contains',
          value: 'calculator',
          // Transform just for this assertion
          transform: (output, context) => {
            const tools = context.metadata?.toolCalls ?? [];
            return tools.map((t) => t.name).join(', ');
          },
        },
      ],
    },
  ],
});
const results = await evalRecord.toEvaluateSummary();
Transform functions receive: - `output`: the LLM output (string or object) - `context`: an object containing `vars`, `prompt`, and optionally `metadata` from the provider response :::note Function transforms are not serializable. If you use `writeLatestResults: true`, function transforms will not be persisted in the stored config. Use string expressions or `file://` references if you need results to be fully reproducible from the stored eval. ::: For more on transforms, see [Transforming Outputs](/docs/configuration/guide#transforming-outputs). ## Example `promptfoo` exports an `evaluate` function that you can use to run prompt evaluations. 
import promptfoo from 'promptfoo';

const evalRecord = await promptfoo.evaluate(
  {
    prompts: ['Rephrase this in French: {{body}}', 'Rephrase this like a pirate: {{body}}'],
    providers: ['openai:gpt-5-mini'],
    tests: [
      {
        vars: {
          body: 'Hello world',
        },
      },
      {
        vars: {
          body: "I'm hungry",
        },
      },
    ],
    writeLatestResults: true, // write results to disk so they can be viewed in web viewer
  },
  {
    maxConcurrency: 2,
  },
);
const results = await evalRecord.toEvaluateSummary();

console.log(results);
````

This code imports the `promptfoo` library, defines the evaluation options, and then calls the `evaluate` function with these options.

You can also supply functions as `prompts`, `providers`, or `asserts`:

```js
import promptfoo from 'promptfoo';

(async () => {
  const evalRecord = await promptfoo.evaluate({
    prompts: [
      'Rephrase this in French: {{body}}',
      (vars) => {
        return `Rephrase this like a pirate: ${vars.body}`;
      },
    ],
    providers: [
      'openai:gpt-5-mini',
      (prompt, context) => {
        // Call LLM here...
        console.log(`Prompt: ${prompt}, vars: ${JSON.stringify(context.vars)}`);
        return {
          output: '<LLM output>',
        };
      },
    ],
    tests: [
      {
        vars: {
          body: 'Hello world',
        },
      },
      {
        vars: {
          body: "I'm hungry",
        },
        assert: [
          {
            type: 'javascript',
            value: (output) => {
              const pass = output.includes("J'ai faim");
              return {
                pass,
                score: pass ? 1.0 : 0.0,
                reason: pass ? 'Output contained substring' : 'Output did not contain substring',
              };
            },
          },
        ],
      },
    ],
  });
  const results = await evalRecord.toEvaluateSummary();
  console.log('RESULTS:');
  console.log(results);
})();
There's a full example on Github [here](https://github.com/promptfoo/promptfoo/tree/main/examples/config-node-package). Here's the example output in JSON format: 
{
  "version": 3,
  "timestamp": "2026-05-02T12:43:10.000Z",
  "prompts": [
    {
      "raw": "Rephrase this in French: {{body}}",
      "label": "Rephrase this in French: {{body}}"
    }
  ],
  "results": [
    {
      "prompt": {
        "raw": "Rephrase this in French: Hello world",
        "label": "Rephrase this in French: {{body}}"
      },
      "vars": {
        "body": "Hello world"
      },
      "response": {
        "output": "Bonjour le monde",
        "tokenUsage": {
          "total": 19,
          "prompt": 16,
          "completion": 3
        }
      },
      "success": true,
      "score": 1
    }
  ],
  "stats": {
    "successes": 1,
    "failures": 0,
    "errors": 0,
    "tokenUsage": {
      "total": 19,
      "prompt": 16,
      "completion": 3
    }
  }
}
## Sharing Results To get a shareable URL, set `sharing: true` along with `writeLatestResults: true`: 
const evalRecord = await promptfoo.evaluate({
  prompts: ['Your prompt here'],
  providers: ['openai:gpt-5-mini'],
  tests: [{ vars: { input: 'test' } }],
  writeLatestResults: true,
  sharing: true,
});
const results = await evalRecord.toEvaluateSummary();

console.log(results.shareableUrl); // https://app.promptfoo.dev/eval/abc123
Requires a [Promptfoo Cloud](/docs/enterprise) account or self-hosted server. For self-hosted, pass `sharing: { apiBaseUrl, appBaseUrl }` instead of `true`.