Sprint Audit — the engineering manager's guide

1What the sprint audit is

The sprint audit evaluates a team's Agile ceremonies — sprint planning, backlog refinement, daily standup, demo/review and retrospective — against the Qualitara Big Mac Best Practices Framework (the 47-criterion Scrum Management capability), and renders a set of HTML reports for the team, the client and Qualitara leadership.

Its job is to turn "how is this team running its sprints?" from an EM opinion into a repeatable, defensible, evidence-backed score. The same transcripts scored twice give the same numbers, because the model never invents a score — it only judges each criterion, and deterministic Python does every calculation.

2How it works — the pipeline

An audit runs as six stages. Three are deterministic Python (the orchestrator's prepare / score / finalize subcommands); the model's work slots in between them. You never run these by hand — the skill drives them — but knowing the shape tells you exactly where to look when something needs attention.

3The framework & how scoring works

There is exactly one framework: the 47 atomic criteria of the Big Mac Scrum Management capability, sourced from Airtable and shipped committed in framework/. Every criterion carries source: bigmac. (Earlier versions blended in a separate agile layer; that was removed in v0.4.0.)

Two governance mechanisms you control per engagement

4First-run setup — do this once

Four one-time steps get a fresh machine ready. Every per-account path resolves from your EM manifest — there is no project-dir environment variable to set, and nothing here is hardcoded in the runner.

pip install -e ".[dev]"

cp config/config.example.yaml config/config.yaml

fireflies_api_key: "..."    # fetches ceremony transcripts that aren't already on disk

"sprint_audit": {
  "engagements": ["<engagement to audit>"],
  "report_folder": "C:\\Users\\<you>\\Desktop\\<Account>\\Sprint Audits",
  "data_folder":   "C:\\Users\\<you>\\Desktop\\<Account>\\Sprint Audits\\_audit-data",
  "renderer_path": "<repo>/tools/sprint-audit"
}

<data_folder>\
  <account-slug>\
    <engagement-slug>\
      engagement.json          ← engagement config (you write)
      history.json             ← auto-appended by finalize — do not hand-edit
      sprints\
        sprint_01\
          manifest.json        ← per-sprint config (you write)
          transcripts\         ← Fireflies-pulled .txt + .meta.json
          audit\               ← orchestrator + model write here
          evaluation.json      ← the reports' source of truth

5Running an audit

You drive everything through the em-bigmac-framework skill from a Claude Code session. The most common forms:

Under the hood the skill resolves RENDERER, AUDITS_DIR and REPORTS_DIR from your manifest, then runs each orchestrator command from $RENDERER with PYTHONPATH=src:

# Prepare  (fetch transcripts, analyze, write per-ceremony summaries)
PYTHONPATH=src python -m sprint_audit.orchestrator prepare \
  --account "<account>" --engagement "<engagement>" --audits-dir "$AUDITS_DIR"

# Score    (validate narratives, deterministic scorer)
PYTHONPATH=src python -m sprint_audit.orchestrator score "$SPRINT_DIR"

# Finalize (assemble, validate, append history, render every report flat)
PYTHONPATH=src python -m sprint_audit.orchestrator finalize "$SPRINT_DIR" \
  --reports-dir "$REPORTS_DIR" --flat-reports

PDFs are on demand only — never part of a normal run. When you actually need them: python -m sprint_audit.cli.export_pdf --reports-dir <sprint folder> (needs headless Chrome or Edge).

6The reports you get

One finalize renders these per sprint, into <report_folder>\sprint_NN\:

Durable state stays under data_folder: the per-ceremony audit/*.json, the sprint evaluation.json, and the engagement history.json (auto-appended — never hand-edit it).

7Configuration reference

engagement.json — one per engagement

{
  "account": "Acme Robotics",
  "account_slug": "acme-robotics",
  "engagement": "Digital Service",
  "engagement_slug": "digital-service",
  "fireflies_title_hints": ["ACR |", "Digital Service |"],
  "team_size": 7,
  "notes": "anything you want to remember",

  "team_roster": {
    "members": [
      {"name": "Marcus Cole", "role": "Developer",
       "aliases": ["Cole, Marcus", "marcus.cole@client.com"]}
    ]
  },

  "exceptions": [
    {
      "criterion_id": "reclaYUZIYFMBXwNW",
      "criterion_text": "Each sprint has a clearly defined, achievable sprint goal",
      "source": "bigmac",
      "tier": "compensating_control",
      "scope": "engagement", "scope_value": null,
      "reason": "Team runs three products; a single sprint goal is structurally infeasible.",
      "compensating_control": "Per-product weekly OKRs reviewed at planning.",
      "approved_by": "EM + PO (named)", "approved_on": "2026-04-01", "review_on": "2026-10-01",
      "applied_since_sprint": 1, "added_at": "2026-04-01T09:00:00-06:00"
    }
  ]
}

manifest.json — one per sprint

{
  "sprint_name": "Sprint 02 26.2.2",
  "sprint_number": 2,
  "team_name": "Digital Service - Acme Robotics",
  "start_date": "2026-04-08",
  "end_date": "2026-04-21",
  "ceremonies": [
    {"type": "sprint_planning",    "fireflies_title": "FW Sprint Planning", "fireflies_id": "01KNMA..."},
    {"type": "backlog_refinement", "fireflies_title": "ACR | Backlog Refinement", "date": "2026-04-16"},
    {"type": "demo",               "fireflies_title": "ACR | Sprint Review",  "date": "2026-04-22"},
    {"type": "retrospective",      "fireflies_title": "ACR | Retrospective",  "date": "2026-04-23"},
    {"type": "daily_standup",      "fireflies_title": "Digital Service | Daily Standup"}
  ]
}

Ceremony matching, in priority order: (1) fireflies_id — exact, no ambiguity; (2) fireflies_title + date — fuzzy title within a few days; (3) daily_standup is special-cased — every matching transcript between start_date and end_date is bundled into one stitched transcript. Omit a ceremony entirely when it wasn't held; the renderer shows "not held" everywhere it matters. Valid type values: sprint_planning, backlog_refinement, daily_standup, demo, retrospective.

outcomes — delivery results (optional, in the sprint manifest)

Add an outcomes block to a sprint's manifest.json to light up the Predictability panel and the sprint-goal-met maturity gate. These are board facts, not transcript evidence — so the audit measures outcomes, not just ceremony mechanics.

"outcomes": {
  "committed_points": 40,
  "completed_points": 32,
  "scope_added_points": 8,
  "sprint_goal_met": true,
  "cycle_time_days_median": 3.5,
  "board_url": "https://your-board/sprint-42"
}

All fields optional. committed+completed → commitment reliability; scope_added → scope churn; sprint_goal_met → a missed goal blocks Foundation maturity for that sprint. When the block is absent the panel and the gate are simply omitted — no penalty.

8Troubleshooting