PR
Dev teams / BYOK · CLI · API / SDK · Free / OSS

Promptfoo

Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming

What you stop vibecoding

Replaces custom scripts and internal glue code for evals, tracing & guardrails workflows.

Dossier overview

Promptfoo is a free / oss framework in Evals, Tracing & Guardrails. Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming.

Quickstart

npx promptfoo@latest init

What people say on X

Live feed on X ↗

We pull unfiltered builder mentions (praise and complaints) from X on our daily sync. Inspect the live unfiltered stream directly on X right now:

Read latest "Promptfoo" mentions on X ↗

Best use cases

  • Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming
  • Replace custom evals & tracing glue code with a verified framework

Field notes

Strengths

  • Open-source and inspectable
  • Built for evals, tracing & guardrails
  • Designed specifically for dev teams / byok

Tradeoffs

  • Free/starter tiers have rate or seat limits
  • Self-hosting requires managing your own model keys and runtime

Compare with Evals & Tracing alternatives

ToolBest forRuns inPricingSetup
PromptfooDev teams / BYOKCLI · API / SDKFree / OSSCLI / SDK install
LangfuseDev teams / BYOKAPI / SDKFreemiumCLI / SDK install
BraintrustAgent buildersAPI / SDKFreemiumCLI / SDK install
Arize PhoenixDev teams / BYOKAPI / SDKFree / OSSCLI / SDK install
LangSmithAgent buildersAPI / SDKFreemiumAPI / SDK setup

Top Promptfoo alternatives

All Evals, Tracing & Guardrails
LA

Langfuse

langfuse.com

Open-source LLM engineering platform for agent tracing, prompt management, and evals

Evals & TracingFreemium
BR

Braintrust

braintrust.dev

End-to-end evals, trace logging, and PR regression diffs for production AI agents

Evals & TracingFreemium
AP

Arize Phoenix

phoenix.arize.com

Open-source OpenTelemetry tracing and evaluation suite for LLM and agent trajectories

Evals & TracingFree / OSS