Big Mac Framework run & the quarterly soft audit
Audit a whole account against the complete Big Mac Framework (233 atomic tests), in one local HTML report. Automated auditors fill what they can; the soft audit lets you fill the rest by hand, once a quarter, and the report shows exactly how the account is doing and how much is automated.
1What it is
The Big Mac Framework run (bigmac-run) is an aggregator. It does not judge anything itself — it syncs the framework from Airtable, collects results from capability auditors, fills the full 233-test grid, and renders one local HTML report. Any test no auditor has covered yet shows In progress.
Two skills, one report. sprint-audit is the first automated auditor (the Scrum Management capability, from Fireflies). The soft-audit is the manual auditor — a friendly local form where you fill everything the automation does not, once per quarter. Both feed the same run.
2How the run works
Pull Framework v2 from Airtable (233 atomic tests, every column) and normalize it to framework_full.json — the per-quarter snapshot.
Each live auditor writes a <capability>.results.json into results/. sprint-audit → Scrum Management (read-only); the soft audit → everything you fill by hand.
Index every result by Airtable record id, fill each test, and default the rest to In progress. Map levels to Passed / Failed / N/A.
One tabbed HTML report: a dashboard, the capability-auditor roadmap, and the full Airtable grid paginated by category.
The run is pure and deterministic — given the same framework snapshot and the same results files, it renders the same report. New auditors plug in by dropping a results file; the run never changes.
3The report & dashboard
The report is one HTML file with three tabs.
Dashboard home
Two focuses. Big Mac Framework results — the whole-framework pass rate (Passed of all 233), with per-category bars: how the account is doing. Automation coverage — % automated, plus manual and in-progress shares: how much runs by itself.
Capability auditors
Every capability mapped to its owning skill, live or planned, with its evidence source and how many tests it covers. The automation roadmap, inside the deliverable.
Audit grid
The full Airtable replica — every test, every column — paginated by category (Engineering / Product Management / QA), each with its own filtered summary.
How results are counted
| State | Means | Counts toward |
|---|---|---|
Passed / Failed / N/A | A result from an auditor (auto or manual) | Pass rate & "assessed" |
In progress | No auditor has covered this test yet | Neither — it is the backlog |
| Automated | An automated auditor produced the result | % automated only |
| Manual | You filled it in the soft audit | "Assessed" & pass rate, not % automated |
Level mapping: exemplary/solid → Passed · partial/weak/absent → Failed · not_applicable/not_evaluable → N/A.
4Capability auditors
One skill audits one capability and emits one file. That file is the only coupling to the run, so a new auditor never touches the run's code.
The contract results/<capability>.results.json
A list of results[], each keyed by the Airtable criterion_id with a test_result (Passed / Failed / N/A), and a top-level method of auto or manual. The run indexes by id and fills the grid.
Live vs planned registry.yaml
tools/bigmac-run/registry.yaml maps all 25 capabilities to their owning skill and status. Live today: Scrum Management (sprint-audit) plus Communication & Collaboration, Stakeholder Management, Continuous Improvement and Customer & Stakeholder Feedback (capability-audit). The engineering/QA technical capabilities have no central data source and stay In progress until a connector-backed auditor exists.
capability-audit is the sprint-audit model generalized: a per-capability spec (capability_specs.yaml) names the Notion DBs + Fireflies to consult, the model judges each test with a 0..1 confidence and verbatim evidence, and only judgments at or above the bar (default 0.75) are emitted. Below-bar tests are held - they stay In progress for the soft audit, so automation fills only what the data clearly supports.
Adding a capability auditor: give it a capability_specs.yaml entry (data sources + guidance) and flip status: live in registry.yaml - the run picks it up automatically. Full contract: plugins/em-bigmac-framework/skills/bigmac-run/reference/capability-results-contract.md.
5The soft audit (quarterly)
The soft audit is run once per quarter. Each quarter is a self-contained workspace under your report folder — Q1…Q4 — holding its own framework snapshot, the form, every dated answer export, and the report.
<report_folder>\Q2\
_run-data\framework_full.json ← framework snapshot for the quarter
_run-data\results\ ← auto + soft-audit results
answers\soft_audit_answers_Q2_<date>[_N].json ← every export (dated history)
bigmac_soft_audit.html ← the form
bigmac_framework_run.html ← the reportThree flows
Start a quarter
Say start Q2 soft audit. The skill snapshots the framework, runs the live auto auditors into the quarter, and renders the form. Open it, and fill each not-yet-automated test Pass / Fail / N/A with an optional note. Auto-scored tests are hidden by default and pre-filled; flip Show auto-scored to review or override them.
Continue
Your answers auto-save in the browser as you go. When you click Export answers, a dated JSON downloads (soft_audit_answers_Q2_<date>.json). Say continue soft audit and the skill ingests that export into the quarter folder and re-renders the form seeded with it — so you resume even on a different browser or machine.
Process
Say process Q2 soft audit. The skill folds your latest answers into a method: manual results file and regenerates the report. Overrides of an auto result are applied and tagged; multiple runs in a quarter never overwrite — each export is dated.
Override. If you disagree with an automated result, change it in the form. The override is tagged, moves that test from Automated to Manual, and wins in the report — it is now your judgment, not the machine's.
Attestation. Export and Copy require a sworn statement: a modal where you, named, declare the answers are true and reflect the actual reality of the account being audited, gated by a confirmation checkbox. This is a formal audit — you own the accuracy.
6Setup
One manifest block per account. Every path resolves from your EM manifest — the first of ~/.claude/config/em-manifest.json or the in-repo .claude/config/em-manifest.json that exists.
Add a bigmac_run block
"bigmac_run": {
"report_folder": "C:\\Users\\<you>\\Desktop\\<Account>\\BigMac Framework",
"data_folder": "C:\\Users\\<you>\\Desktop\\<Account>\\BigMac Framework\\_run-data",
"renderer_path": "<repo>/tools/bigmac-run",
"framework": { "base_id": "<airtable base id>", "table_id": "<framework v2 table id>" },
"sources": { "sprint_audit_engagement_dir": "C:\\...\\Sprint Audits\\_audit-data\\<account>\\<engagement>" }
}| Key | Meaning |
|---|---|
report_folder | Root of the quarter folders (Q1…Q4) and reports |
data_folder | Default working data for an ad-hoc (non-quarter) run |
renderer_path | Absolute path to tools/bigmac-run in your checkout |
framework | Airtable base + Framework v2 table to sync |
sources.sprint_audit_engagement_dir | Where the sprint-audit adapter reads evaluation.json |
Use plugins/em-core/shared/em-manifest.template.json for the exact shape, or run /em-setup.
Install the runner
Python 3.11+ with pyyaml. The renderer reuses the shared report house style in tools/_reporting_style. No secret is required for the soft audit; the framework sync uses the Airtable connector through the skill.
7Running it
Drive everything from a Claude Code session.
| You say… | What happens |
|---|---|
run the big mac framework for <account> | Syncs the framework, collects live auto results, renders the full report. |
start Q2 soft audit | Creates the Q2 workspace and the form for you to fill. |
continue soft audit | Ingests your latest export and re-renders the form, seeded with your answers. |
process Q2 soft audit | Folds your answers into the report (overrides applied, dated history kept). |
No quarter named? The skill uses the current quarter. Exports land in your Downloads folder (browsers cannot pick a save folder); the skill ingests the newest one into the quarter automatically.
8Troubleshooting
| Symptom | Fix |
|---|---|
"Account has no bigmac_run block" | Add the block (section 6) or run /em-setup. |
| Continue/process says no answers found | Fill the form and click Export answers first — the export must exist in Downloads or the quarter folder. |
| Many tests skipped on process | Usually ids from an older framework snapshot — re-sync the quarter and re-export. |
| Most tests show In progress | Expected — only Scrum Management is automated today. Fill the rest in the soft audit, or build more capability auditors. |
| The form looks empty after reopening | It resumes from this browser's autosave; on a new browser, run continue to re-seed from your latest export. |
Where things live
tools/bigmac-run/ · renderer, soft-audit form, adapters, registry, quarter paths
skills/bigmac-run/reference/capability-results-contract.md · the auditor contract
plugins/em-core/shared/config-resolution.md · how the manifest resolves