Dev teams / BYOK · CLI · API / SDK · Free / OSS
Promptfoo
Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming
What you stop vibecoding
Replaces custom scripts and internal glue code for evals, tracing & guardrails workflows.
Dossier overview
Promptfoo is a free / oss framework in Evals, Tracing & Guardrails. Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming.
Quickstart
npx promptfoo@latest initWhat people say on X
Live feed on X ↗We pull unfiltered builder mentions (praise and complaints) from X on our daily sync. Inspect the live unfiltered stream directly on X right now:
Read latest "Promptfoo" mentions on X ↗Best use cases
- Open-source CLI for automated LLM evals, agent regression tests, and security red-teaming
- Replace custom evals & tracing glue code with a verified framework
Field notes
Strengths
- Open-source and inspectable
- Built for evals, tracing & guardrails
- Designed specifically for dev teams / byok
Tradeoffs
- Free/starter tiers have rate or seat limits
- Self-hosting requires managing your own model keys and runtime
Compare with Evals & Tracing alternatives
| Tool | Best for | Runs in | Pricing | Setup |
|---|---|---|---|---|
| Promptfoo | Dev teams / BYOK | CLI · API / SDK | Free / OSS | CLI / SDK install |
| Langfuse | Dev teams / BYOK | API / SDK | Freemium | CLI / SDK install |
| Braintrust | Agent builders | API / SDK | Freemium | CLI / SDK install |
| Arize Phoenix | Dev teams / BYOK | API / SDK | Free / OSS | CLI / SDK install |
| LangSmith | Agent builders | API / SDK | Freemium | API / SDK setup |