The requirement held.
The assertion ran and the output satisfied it.
Exit code 0.
v0.1.1 · offline, provider-independent
Catch AI output regressions before they ship.
m2aitestkit never calls a model. Point it at what your app produced and it tells you, per assertion, whether the output passed, failed or could not be evaluated reliably, and what changed since your last known-good run.
npm install --save-dev m2aitestkit- pass
is-jsonstructural - fail
has-fields(invoiceNumber,vendor,total)pass → failmissing required field(s): vendor
- fail
field-type(total:number)pass → failfield total has type string, expected number
- pass
forbid(no-markdown)rule
Real output from runSuite and diffReports. No model called, no key needed.
0runtime dependencies
Node ≥20ESM only, ships its own types
3outcomes: pass, fail, unevaluable
0 · 1 · 2 · 3ok, failed, bad input, unevaluable
Three outcomes, not two
A check that couldn’t run is not a pass.
Most test tools collapse “couldn’t tell” into green. m2aitestkit keeps it separate and fails CI on it by default.
The requirement was broken.
The assertion ran and the output violated it. The message names the rule, never the offending text.
Exit code 1. pass → fail blocks the build.
Nobody can say.
Malformed JSON, a crashing custom check, a dead evaluator, or a stale or missing recording. Never silently a pass.
Exit code 3, unless you opt out.
What you get
Tests that don’t hit a paid, non-deterministic API on every run.
Committed recordings
Redacted, key-sorted, timestamp-free JSON that diffs cleanly in code review. Raw inputs are never stored, only a hash.
Assertion-level baseline diff
pass → fail and pass → unevaluable are regressions. Removed cases and assertions are always listed, so a suite can't be weakened silently.
Honest labels
Every assertion is exact, structural, rule, heuristic, semantic or custom, so lexical overlap never passes for correctness.
Structured-output checks
Valid JSON, JSON Schema or any Standard Schema validator, required fields, types, ranges and tool-call rules.
Private by default
No network, no telemetry. Emails, tokens, keys and card-like numbers are redacted before anything is evaluated or saved.
Built for CI
Deterministic JSON and JUnit reports, and exit codes 0, 1, 2 and 3. Two runs of the same recording are byte-identical.
Workflow
Record once. Replay forever. Diff on every change.
One workflow, aimed at one problem: outputs your code depends on drifting quietly when a prompt, model or retrieval step changes.
Record
Run your app once and commit the redacted recording. This is the only step that needs a model.
- 1
m2aitestkit record .
Replay
CI re-evaluates the committed outputs offline, so parsing and assertion changes are caught without a key.
- 2
m2aitestkit run .
Diff
Each assertion is compared with its baseline status. Regressions block the build.
- 3
m2aitestkit run . --update-baseline
A green replay says your assertions and parsing still hold on the recorded output. To detect model drift, re-record or run --live. Read the limitations →
How it works
From a suite file, from a test runner, from CI.
- 1
Write a suite
Your app runs behind
run(), with any provider. Assertions are plain functions.tsimport { defineSuite, fieldConstraint, forbid, hasFields, isJson, toolCalls } from "m2aitestkit"; declare function supportBot(input: { question: string }): Promise<string>; export default defineSuite({ name: "support-bot", run: (input) => supportBot(input as { question: string }), // your code, any provider cases: [ { id: "refund-request", input: { question: "Can I get a refund?" }, metadata: { model: "gpt-x", promptVersion: "v3" }, assertions: [ isJson(), hasFields(["answer", "refundAmount"]), fieldConstraint("refundAmount", { min: 0, max: 100 }), forbid({ "competitor-mention": /acme/i }), toolCalls({ allow: ["lookup_order"], forbid: ["issue_refund"] }), ], }, ], }); - 2
Record, then run
Commit
.m2aitestkit. Later runs replay it offline and diff against the baseline.shnpx m2aitestkit record . # call run() once, write a redacted recording npx m2aitestkit run . --update-baseline # evaluate it and save the baseline git add .m2aitestkit # commit both npx m2aitestkit run . # later: offline, no API key - 3
Gate the pull request
No API key in CI. A JUnit file feeds your usual test reporting.
yaml- run: npm ci - run: npx m2aitestkit run . --format junit --output m2aitestkit.xmltsimport { assertOutput, hasFields, isJson } from "m2aitestkit"; declare function summarize(doc: string): Promise<string>; // Works under node:test, vitest, jest or mocha: it only throws. await assertOutput(await summarize("…"), [isJson(), hasFields(["summary"])]);
Playground
Change the output. Watch the gate react.
This runs the m2aitestkit source from this repository in your browser: the same assertions, the same replay and the same baseline diff as the CLI. Nothing is mocked and nothing is uploaded.
Blocked2 pass · 2 fail · 0 unevaluable · 2 regressions
is-jsonstructuralbaselinepassnowpasshas-fields(invoiceNumber,vendor,total)structuralregressedbaselinepassnowfailfield-type(total:number)structuralregressedbaselinepassnowfailforbid(no-markdown)rulebaselinepassnowpass
The new output is still valid JSON, but vendor is gone and total became a string. Two assertions that passed in the baseline now fail, so CI blocks. How the outcomes work →
The code behind this panel
ts
import { diffReports, formatPretty, runSuite, suiteFromSpec, type AssertionSpec, type Diff, type RunReport } from "m2aitestkit";
export type Input = {
/** What was sent to the model. Only a hash of it is ever stored. */
input: string;
/** The known-good output you recorded and committed. */
baseline: string;
/** What the application produces now, after a prompt, model or retrieval change. */
current: string;
assertions: AssertionSpec[];
};
export type Evaluation = { baseline: RunReport; current: RunReport; diff: Diff; text: string };
/** The same functions the CLI uses: replay each output against the assertions, then diff the two runs. */
export async function evaluate({ input, baseline, current, assertions }: Input): Promise<Evaluation> {
const replay = async (output: string): Promise<RunReport> => {
const suite = suiteFromSpec({ name: "demo", cases: [{ id: "reply", input, output, assertions }] });
return (await runSuite(suite, { mode: "replay" })).report;
};
const before = await replay(baseline);
const after = await replay(current);
const diff = diffReports(before, after);
return { baseline: before, current: after, diff, text: formatPretty(after, diff) };
}Assertion kinds
Every check says how much weight it can bear.
Reports label each assertion, so a heuristic is never read as proof.
exactDeep equality against a stored expectation.structuralJSON shape, schema, fields, types and ranges.ruleForbidden content, length caps, tool-call constraints.heuristicDeterministic but approximate: refusal phrasing, lexical similarity.semanticJudged by an evaluator you supply. Flagged when it uses an external service.customYour own function. A throw means unevaluable, never fail.
Documentation
Everything is written down.
Getting StartedInstall m2aitestkit, record a known-good run and gate CI on it.Core ConceptsThree outcomes, assertion kinds, recordings and baselines.AssertionsEvery built-in check, what it proves and what it cannot.Recordings & BaselinesCommit outputs, replay offline and diff at assertion level.CLI & CICommands, flags, report formats and exit codes.Programmatic APISuites, evaluators, assertOutput and the JSON report.Privacy & RedactionWhat is stored, what is redacted and what never leaves the process.LimitationsWhat a green run does and does not tell you.ComparisonHow it relates to promptfoo, vitest-evals and Evalite.
More from M2Toolkit