Benchable Sign in

Catch the slowdown before it merges

Post any benchmark from CI — go test, hyperfine, k6, Lighthouse — or let your coding agent record it, and get a statistically tested verdict, the span that moved and the commits that did it. Self-hostable, on Postgres, by linesofcode.

Sign in Docs
For agents
llms.txt AGENTS.md This page as Markdown

or hand it to your agent

Free plan is the whole product. Currently v0.1.0.

POST /api/v1/runs → 201captured output
{
  "runId": "fhTNzgoqngNdETKAPYLkC",
  "branch": "main",
  "format": "benchable",
  "idempotent": false,
  "regressions": 1,
  "metrics": [
    { "key": "api.latency_ms", "value": 210, "unit": "ms",
      "baseline": 120.6, "delta": 89.4, "deltaPct": 74.13, "verdict": "regressed" }
  ]
}
+74.13%caught before merge

CI fails on a header

Gate merges without writing a parser.

POST /api/v1/runs answers with a verdict per metric and X-Benchable-Regressions. See sending a run.

$ curl -X POST http://localhost:3000/api/v1/runsrun
curl -X POST http://localhost:3000/api/v1/runs \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $GITHUB_RUN_ID" \
  -d '{
    "branch": "main",
    "commitSha": "9f3c1ab",
    "environment": "ci-linux-x64",
    "metadata": { "runner": "github-actions", "node": "24" },
    "metrics": {
      "build.time_ms": 4210,
      "api.latency_ms": { "value": 118.4, "p50": 110, "p95": 180, "p99": 260, "samples": 500 }
    },
    "spans": [
      { "id": "root", "name": "POST /checkout", "startMs": 0, "durationMs": 118.4 },
      { "id": "db", "parentId": "root", "name": "SELECT orders", "startMs": 12, "durationMs": 40 }
    ]
  }'

Typed client, one file

Autocomplete on the payload the server validates.

The benchable SDK has no dependencies — vendor it and call .run(). See SDK.

benchable SDKcaptured output
import { Benchable } from "benchable";

const benchable = new Benchable({ apiKey: process.env.BENCHABLE_KEY! });

const result = await benchable.run({
  branch: "main",
  commitSha: process.env.GITHUB_SHA,
  metrics: { "api.latency_ms": { value: 118.4, p95: 180, p99: 260 } },
});

if (result.regressions > 0) process.exit(1);

Only real changes fail

Fewer false alarms, so nobody mutes the channel.

Send stddev or raw values and the verdict comes from Welch's t-test or Mann–Whitney U, with false-discovery control across the suite. See is the change real?.

312metrics tested
15flagged by raw p-value
2flagged after control
verdict — api.latency_mscaptured output
{ "key": "api.latency_ms", "value": 106, "baseline": 100, "deltaPct": 6,
  "verdict": "neutral", "reason": "not significant (p = 0.287)",
  "significance": { "test": "welch", "p": 0.287, "effectSize": 0.52,
                    "interval": { "low": -5.9, "high": 17.9, "level": 0.95 },
                    "significant": false } }

Noise bands measure themselves

Thresholds that match your hardware, not a guess.

Each metric learns its own stability profile; switch the band to Measured to gate on it. See noise bands.

±9.4%api.latency_ms measured noise
46runs of history
±10.5%cold_start — too unstable to gate
GET /api/v1/metrics — text/plaincaptured output
GET /api/v1/metrics -H 'Accept: text/plain'

key                    name              unit  better  noise band  stability
--------------------  ----------------  ----  ------  ----------  ---------------------------------------
api.latency_ms        Api Latency       ms    lower   ±9.4% auto  noisy, run-to-run noise ±9.4% over 46 runs
bundle.main_kb        Bundle Main       KB    lower   ±5%         stable, run-to-run noise ±0.0% over 40 runs
cold_start.p99_ms     Cold Start P99    ms    lower   ±5%         flaky — too unstable to gate on, ±10.5%

Laptops never race CI

A hardware change stops looking like a regression.

Baselines are searched narrowest-first, and baselineScope says which one matched. See baselines.

baselineScope — narrowest firsttable
baselineScopeMeaning
branch+environmentSame branch, same environment — a like-for-like comparison
branchSame branch, another environment
default-branch+environmentThe default branch, same environment
default-branchThe default branch, another environment

Post the file you have

Adopt it this afternoon. No translation shim.

POST /api/v1/import?format=auto detects the format from the content. See importing.

  • benchable
  • go-bench
  • hyperfine
  • pytest-benchmark
  • google-benchmark
  • criterion
  • vitest-bench
  • k6
  • lighthouse
  • jmh
  • prometheus
  • csv
  • otlp-trace
  • jaeger
  • zipkin
  • chrome-trace
$ go test -bench=. -benchmem ./... > bench.txtrun
go test -bench=. -benchmem ./... > bench.txt

curl -X POST "$BENCHABLE_URL/api/v1/import?branch=main&commitSha=$(git rev-parse HEAD)" \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  -H "Accept: text/plain" \
  --data-binary @bench.txt
POST /api/v1/import — text/plaincaptured output
run WJ6S2WQYUr8NCdyWUSvLH on main (go-bench)

metric                                   value  baseline     change
--------------------------------  ------------  --------  ---------
go.BenchmarkEncode.ns_per_op            1.05µs         —  first run
go.BenchmarkEncode.B_per_op              512 B         —  first run
go.BenchmarkStream.MB_per_s           452 MB/s         —  first run

No regressions.

One regression, one alert

Your channel fires once, not forty times.

Later detections bump occurrences silently; a run back at the old level resolves it. See regressions.

GET /api/v1/regressionscaptured output
GET /api/v1/regressions -H 'Accept: text/plain'

id                     metric              status  kind        branch    now    was   change  seen   since
--------------------  ------------------  ------  ----------  ------  -----  -----  -------  ----  ----------
kf3ZcUxhw3I3EKIcBKJI  api.latency_ms      open    regression  main    155ms  100ms   +55.0%    3x  2026-05-02
llT2cPh0mA26KPed4cQz  bundle.main_kb      open    budget      main    310KB  240KB   +29.2%    7x  2026-04-28

Find the query that moved

From "checkout is slow" to the exact query.

Attach spans, or POST an OTLP, Jaeger, Zipkin or Chrome trace — the run page renders every trace as a filterable waterfall, and GET /api/v1/compare ranks both sides by each span's own time. See which span moved.

Flamegraph beside the number? bunx benchable upload --file flame.svg.

GET /api/v1/comparecaptured output
trace 118.0ms → 196.0ms

span                   self    total   change    state
------------------  -------  -------  -------  -------
  SELECT orders     +78.0ms  118.0ms  +195.0%  changed
  INSERT audit_log        —   12.0ms        —    added
POST /checkout      −12.0ms  196.0ms   +66.1%  changed

Ranked by change in the span's own work, which is what attributes a regression.

Hold the absolute line

The bundle cannot grow 1% a day forever.

budgetMax and budgetMin fail a run on its value, whatever the baseline says. See budgets.

Verdict on the pull request? bunx benchable report --run last --marker pr-42.

budgets — text digestcaptured output
metric                 value  baseline     change            budget
--------------------  ------  --------  ---------  ----------------
budget.bundle_kb      251 KB    240 KB     · 4.58%       over by 0%
api.latency_ms         118ms     120ms     · −1.67%      32% left

Catch the slow creep

Hand the range to git log. The cause is inside.

GET /api/v1/metrics/{key}/changepoints finds each level shift and its commit range. See change points.

changepoints — search.throughput_opscaptured output
search.throughput_ops on main — 40 runs analysed

when            level  change          from            to      commit range       p
----------  ---------  ------  ------------  ------------  ----------------  ------
2026-09-10  regressed  -6.30%  23.67k ops/s  22.18k ops/s  aaaaaaa..aaa0aaa  <0.001
2026-09-14  regressed  -5.63%  22.32k ops/s  21.07k ops/s  aaaf1da..a2aafab   0.002

The cause of each change is in the commits inside its range.

Your agent reads it too

Let Claude or Cursor triage the regression queue.

An MCP server at /api/mcp answers in text, not JSON. See agents.

MCP client configcaptured output
{
  "mcpServers": {
    "benchable": {
      "url": "https://benchable.example.com/api/mcp",
      "headers": { "Authorization": "Bearer bmk_..." }
    }
  }
}

Agents benchmark, you get charts

Ask whether it got slower. Get a tested answer and a link.

npx skills add TimMikeladze/benchable teaches Claude Code, Codex and OpenCode to benchable record before and after a change, online or offline. See agent skill.

$ metric value baseline change verdict testrun
metric                  value  baseline    change   verdict  test
----------------------  -----  --------  --------  --------  --------------------
hyperfine.parse.mean_ms  9.0ms    14.6ms  ▼ -38.4%  improved  mann-whitney p<0.001

view: https://benchable.sh/p/web/runs/r_8c1…

What holds, what is a guess

3 hold

  • Evidence only ever downgrades a verdict — what gated CI yesterday still gates today.
  • Ingest is idempotent: a retried step gets the original run back.
  • API keys are stored as SHA-256 hashes. Artifacts are never public.

3 are judgements

  • The ±5% default band is a guess — switch the metric to Measured.
  • False-discovery control runs at q = 0.05, the field default.
  • Stability needs six runs of history before it is reported.

4 are not here yet

  • AI diagnosis needs AI_GATEWAY_API_KEY; without it you get 503 ai_unavailable.
  • No scheduler — CI owns the cadence, Benchable ingests.
  • Cross-environment baselines are labelled, not corrected.
  • The benchable package is not on npm yet — vendor the SDK file.

Priced by how big it gets

The free plan is the entire product — every ingest format, statistical verdicts, noise bands, budgets, traces, regressions, AI diagnosis, artifacts, webhooks and the MCP server. What it caps is volume. Pro removes every cap, per member. Enterprise runs inside your own perimeter.

Free

$0forever

The whole product, sized for one repository.

  • Every ingest format, verdict and chart
  • AI diagnosis, artifacts, webhooks and share links
  • No card, no trial clock
Sign in
most popular

Pro

$20per member, per month

Everything, uncapped, billed by the people using it.

  • Unlimited projects, runs, members and history
  • Retention you choose, not one we impose
  • Seats follow your team — add someone, we prorate
Sign in

Enterprise

Let's talkannual, invoiced

Run it inside your own perimeter.

  • Self-hosted, in your cloud account
  • VPC peering and private networking
  • SSO, SCIM and an audit log
  • Custom retention, invoicing and support terms
Talk to us
plans — every limit and featuretable
FreeProEnterprise
Projects2UnlimitedUnlimited
Members3UnlimitedUnlimited
Runs this month1,000UnlimitedUnlimited
Retention30 daysUnlimitedUnlimited
AI diagnoses this month20UnlimitedUnlimited
Artifacts per run5UnlimitedUnlimited
Webhooks per project1UnlimitedUnlimited
API keys per project2UnlimitedUnlimited
AI diagnosis, artifacts, webhooks, share links, digests, trace diffs✓✓✓
SAML / OIDC single sign-on——✓
Audit log——✓
Self-hosted deployment——✓
VPC peering——✓
Priority support—✓✓
What counts as a seat?

Everyone in the workspace, including you. Invitations that have not been accepted count against the free plan's member limit — otherwise a three-seat workspace could hand out thirty links and discover the cap one acceptance at a time — but Stripe only bills for people who actually joined.

What happens when I hit a free limit?

The specific action is refused with a sentence saying which limit and what the number is. Ingest answers 402 with the same error envelope as every other API error, so a CI step fails readably instead of retrying forever. Nothing already stored is hidden, and the only thing ever deleted is history past your plan's retention window, which the nightly job trims.

What happens if a payment fails?

Stripe retries on its own dunning schedule. While the subscription is past due the workspace falls back to free limits, and everything is restored the moment it is paid — the subscription and its history are never deleted.

Can I self-host instead?

Yes, and then none of this applies. A deployment with no Stripe keys configured has billing switched off entirely and every workspace is unlimited. Enterprise is for when you want that with support, VPC peering and SSO behind it.

Do you handle sales tax?

Not yet. Stripe Tax is deliberately switched off rather than enabled without the tax registrations behind it, which would collect nothing while looking like it was working. If you need a VAT invoice, talk to us first.

Prices are in US dollars, billed monthly. Seats are counted from your workspace's member list — adding someone mid-month is prorated, removing them credits the difference.

Get your first verdict

Sign up, create a project, copy its key from Settings, and post a run. The first POST creates every metric in it.

  1. 01Create a project
  2. 02Copy the API key
  3. 03Post a run from CI
Sign in See pricing

Rather run it yourself? One Postgres URL and Bun.

$ bun installrun
bun install
cp .env.example .env     # set DATABASE_URL and BETTER_AUTH_SECRET
bun run db:migrate
bun run db:seed          # optional: a demo workspace with 54 runs across 3 branches
bun run dev              # picks the first free port at or after 3000
$ bunx benchable importrun
export BENCHABLE_URL=https://benchable.example.com
export BENCHABLE_KEY=bmk_...

bunx benchable import  --file bench.txt --branch main --commit "$(git rev-parse HEAD)"
bunx benchable submit  --metrics metrics.json --branch main
bunx benchable summary
bunx benchable report --run last --marker pr-42   # Markdown for a PR comment
bunx benchable regressions --status live          # the regression queue
bunx benchable upload --run last --file flame.svg --kind flamegraph
bunx benchable artifacts --run last
bunx benchable download --run last --name flame.svg
bunx benchable formats