~/promptry·released August 2026·v1.1.0

Catch prompt regressions before your users do.

Change one import and every model call is logged — cost, tokens, latency. Python eval suites version every prompt and catch drift with statistical tests. Edit prompts live, group calls into cost-attributed traces, set budgets with alerts. One SQLite file on your laptop, no account required.

py $ pip install promptry

js $ npm install promptry-js

Read the docs View on GitHub

MIT license / SQLite storage / Zero telemetry

~/rag-service · promptry

# change one import — every call is logged

from promptry.openai import OpenAI

client = OpenAI()

client.chat.completions.create(model="gpt-4o-mini", messages=[...])

# ↳ cost, tokens, latency, name — recorded

# then version prompts + guard them with a suite

@suite("rag-regression")

def test_quality():

response = my_pipeline("What is photosynthesis?")

assert_semantic(response, "Converts light into chemical energy")

$ promptry run rag-regression --compare prod

PASS test_quality (142ms)

└ semantic(0.891) ok

PASS test_grounding (98ms)

└ grounding(0.942) ok

WARN test_safety (218ms)

└ refusal drifted -0.04 vs baseline

────────────────────────────────

Overall: PASS score: 0.913 prod: 0.921 (-0.008)

→ open dashboard http://localhost:8420

$ promptry prompt diff rag-qa 12 13

rag-qa v12 → v13

- You are a helpful assistant. Answer concisely.

+ You are a helpful assistant. Answer in 2-3 sentences,

+ citing sources with [n] markers where relevant.

impact across last 500 runs:

groundedness ↑ 0.847 → 0.921

avg tokens ↑ 142 → 188

cost / 1k ↑ $0.21 → $0.28

refusal rate ↓ 2.1% → 1.4%

~/capabilities

Keep a prompt honest.

From the first commit to the cost of every call in production.

#Zero-config capture

-Bolt a telemetry wrapper around every call site before you see a single number.

+Swap from openai import OpenAI for from promptry.openai import OpenAI — cost, tokens, and latency, streaming and failures included. Same idea in JS via wrapOpenAI().

#Prompt versioning

-Tweak the prompt, ship it, forget which version is live.

+Every track() call is content-hashed. Diff any two versions.

#Eval suites

-Ad-hoc asserts scattered across notebooks and PR comments.

+Eight assertion types: semantic, schema, judge, tool-use, all in one @suite.

#Regression & drift detection

-Prompt works. Six weeks later, scores have crawled down and nobody knows what broke it.

+Bisect to the run that broke it; a rolling Mann-Whitney U test flags slow drift before you do.

#Live prompt CMS

-Fixing a typo in a prompt means a code change and a redeploy.

+render_prompt() serves dashboard-edited templates live — no redeploy.

#Cache optimization

-Inputs sit at the top of the prompt, quietly busting the prefix cache on every call.

+Reorder inputs to the end, consolidate near-duplicate prompts, flag redundant wording — measured, never auto-applied.

#Cost tracking

-Bill spikes. Which prompt? Which model? Nobody knows.

+Module → prompt → call drill-down, plus daily/monthly budget caps that alert to Slack, PagerDuty, or a webhook on breach.

#Call traces + feedback

-A multi-step agent burns tokens and you can't tell which step; a thumbs-down vanishes into a log file.

+Wrap a run in trace() for a per-step cost waterfall. Opt-in, sampled request/response capture. Ratings ingest from your app via API.

#CI + GitHub Action

-Reviewer eyeballs a prompt diff and hopes for the best.

+One-line GitHub Action fails the build and posts the score delta.

~/flow

Four steps. Plain-text prompt in, regression tests out.

No server, no account, no telemetry. Just a CLI and a SQLite file.

localhost:8420 · promptry dashboard

local

Workspace

Overview

Suites

Runs

Prompts

Cost

Models

Traces

Playground

Recent

rag-regression

summarizer-v2

safety-suite

nameverhashtagage

rag-qav13a3f9c2eprod2h

rag-qav128b2f14a3d

summarizerv5d91a0b7prod5d

safety-refusalv84c1e52a1w

classify-intentv27890fedstaging1w

code-reviewv32345abc2w

grounding-checkv76f1a09eprod2w

rag-regression ● passing

12 tests · semantic + grounding + schema · score 0.913

summarizer-quality ● passing

8 tests · semantic + judge · score 0.872

safety-refusal ▲ drift

6 tests · refusal + not-contains · score 0.844 −0.04σ

classify-intent ● passing

4 tests · matches + schema · score 0.955

grounding-check ● passing

9 tests · grounded + contains · score 0.891

timesuitescoreverdict

10:42rag-regression0.913✓ PASS

09:15summarizer-quality0.872✓ PASS

08:03safety-refusal0.844▲ WARN

07:14classify-intent0.955✓ PASS

06:30rag-regression0.905✓ PASS

05:45summarizer-quality0.870✓ PASS

04:02grounding-check0.891✓ PASS

yesterdaysafety-refusal0.762✗ FAIL

Score

0.913+0.02

Runs

1,284

Pass rate

96.4%-0.8

Cost / 1k

$0.21

Root cause

rag-qa v12 → v13 · refusal template shortened

Drift window

7 day · safety-refusal -0.04σ

~/landscape

Three things you won't get anywhere else.

Local by default, no account, deterministic assertions instead of a metered LLM-judge tax — plus statistical drift tests, hard model-spend caps, and prompt-cache optimization none of the others ship.

Dimension	promptry	Promptfoo	RAGAS	LangSmith	DeepEval
Authoring surface	Python decorators	YAML / JS / TS	Python metrics	SDK + web UI	Python (pytest)
Data location	Local SQLite	Local	Local	Cloud · self-host (Ent.)	Local + cloud push
Account required	No, ever	No	No (OSS)	Yes	No (OSS)
CI cost per run	$0 default · judge opt-in	$0 default · judge opt-in	Mostly judge-call	Per-trace billing	Mostly judge-call
Prompt versioning	Automatic content-hash	Manual + git	·	Prompt Hub	Cloud only
Live prompt editing	Dashboard, no redeploy	·	·	Prompt Hub (cloud)	Cloud only
Statistical drift detection	Mann-Whitney U + p-values	·	·	Dashboards + alerts	Dashboards + alerts
Model-spend budget caps	Hard daily/monthly caps	·	·	Own billing only	·
Prompt cache optimization	Reorder / consolidate / shorten	·	·	·	·
Commercial tier	None planned	Enterprise	Vibrant Labs (YC)	SaaS	Confident AI

~/start

Local SQLite. No Docker. No SaaS. No account.

Install, wrap one prompt, write one assertion. Promptry takes it from there. Deterministic assertions by default — LLM-as-judge is opt-in, for when you actually want to spend the tokens.

Read the docs Examples on GitHub

60-second quickstart

bash

$ pip install promptry

Successfully installed promptry-1.0.4

$ promptry init

created promptry.toml · evals.py · SQLite ready

$ promptry run smoke-test --module evals

PASS test_quality (142ms)

PASS test_safety (218ms)

$ promptry dashboard

→ open http://localhost:8420