Define the unit of accepted work and its quality threshold before the pilot. Record every attempt, including rejected, blocked, abandoned, and pending work. The research companion guide includes an editable CSV log, measurement rules, and a fictional worked example.
Task / intended use / reviewer:
Who measures, and whether they are independent of the people who built or sponsor the workflow:
Baseline period and how the manual condition was observed (same people, same period, same task mix):
Minimum tasks and duration agreed before the pilot, and the reason:
Acceptance criteria:
Sample selection / comparison method / known biases:
| Task / attempt / configuration IDs | Method | Preparation minutes | Production minutes | Supervision minutes | Review minutes | Correction minutes | Recovery minutes | Attempt state | Elapsed minutes | Service cost / currency / status |
|---|---|---|---|---|---|---|---|---|---|---|
| Manual | ||||||||||
| Assisted |
Human effort = preparation + production/prompting + supervision + review + correction + recovery person-minutes across all attempts. Count each person's interval once. Record unattended machine wait separately.
Accepted tasks = unique task IDs within each condition with a verified accepted result. A retry can yield one accepted task, never two. Distinguish this from accepted-attempt rate, whose denominator includes every attempt in that condition.
Task elapsed time runs from the first attempt's start to final verification, including gaps. It is not the sum of attempt durations. Record the final artifact and verification evidence.
Cost per accepted task includes all attempt costs and any stated labor/setup allocation. Unknown cost makes the total incomplete; zero accepted tasks makes the ratio undefined. Numeric blanks mean unmeasured, not zero.
Operating view
Setup / evaluation effort:
Maintenance and support estimate / period:
Newly attempted work, tracked separately:
Quality findings and unresolved exceptions:
| Defect type (wrong fact, missing item, wrong scope, formatting, policy breach, other) | Count | Caught by review or after acceptance | Consequence | Fix or accepted |
|---|---|---|---|---|
Feedback from people who received or depended on the output:
Risk events during the pilot (data placed where it should not be, action taken without approval, provider outage) and their handling:
Cost per accepted task trend across the pilot, and what drove the change:
Follow-up date / completed or missing / delayed defects:
Retained-understanding exercise / result or not observed:
Recommendation: expand / redesign / gather more evidence / stop
Approval / permitted scope / approver / date:
Execution status / independent verification / evidence:
Supported scope / limitations / owner / next review:
Common failure modes
- Only successful attempts are logged, so the accepted-attempt rate looks better than it was.
- Review time is counted for the pilot but not for the manual baseline.
- Enthusiastic early users are compared with the whole team's ordinary output.
- Savings are projected from a week of pilot data and never re-measured.
This scorecard measures effort, acceptance, and cost for one task. It does not establish that the workflow is safe to release; use the AI workflow release review for that.