The value of an AI workflow should be visible in the work the firm accepts and uses. That means looking at the time required to produce it, the time required to review it, and the effort spent correcting exceptions. I would establish that picture before expanding a pilot. It gives the team a way to distinguish a promising experience from a result that is worth repeating.
Source
The task mix is part of the result
METR’s May 2026 analysis of task substitution asks how changing work choices affect productivity measurement. That matters when a pilot makes new tasks possible alongside speeding up familiar ones.
I would track those effects separately. Compare like-for-like accepted work where possible, then evaluate the usefulness of newly attempted work on its own terms. The proposal is a scorecard that makes the change in work visible before it becomes a headline percentage.
Proposal
Define the unit of useful work
Choose a repeatable task and the conditions under which its result is accepted. For a hypothetical document-summary pilot, that could mean a summary with correct material facts, usable source links, and no unresolved issue that prevents its intended use.
Measure the whole path: preparation, generation, review, correction, and delivery. Keep human effort separate from elapsed time. A workflow can finish sooner while consuming more reviewer attention, or free human time without reducing the wait for a final result.
Record work that fails or is abandoned. Excluding difficult cases makes the average look better while hiding the burden that operators will encounter after rollout.
Source
Compare equivalent work
Build a representative sample and record differences in task difficulty, reviewer experience, and source quality. Where practical, compare similar tasks with and without assistance. Avoid having the same person immediately repeat a task whose answer they now know without accounting for that learning effect.
Research should inform the measurement without supplying a borrowed return-on-investment claim. DORA’s 2025 work frames AI as an amplifier of organizational strengths and weaknesses. That is a useful reason to examine the surrounding workflow rather than assume the tool alone determines the result.
METR’s February 2026 update discusses selection and measurement difficulties in estimating developer productivity effects. Its setting differs from an investment-firm research workflow, but the caution is relevant: who uses the tool, which tasks they choose, and what gets measured can change the conclusion.
Count review and rework
For an illustrative calculation, suppose a manual task requires 30 minutes of human effort. An assisted version takes 5 minutes to prepare, 10 minutes to review, and 8 minutes to correct. The observed saving is 7 minutes per accepted task before allocating maintenance and other operating costs. These numbers are an example, not a study result.
Describe this figure
A horizontal stacked bar chart of the piece's illustrative calculation, in minutes of human effort per accepted task, on a scale from 0 to 30. The manual task takes 30 minutes. The assisted version takes 23: 5 minutes to prepare, 10 to review, and 8 to correct. The remaining 7 minutes, outlined at the end of the assisted bar, are the observed saving per accepted task before maintenance and other operating costs are allocated. The numbers are an example, not a study result.
If the assisted workflow also causes some outputs to be rejected, report that rate and the recovery effort. A faster draft can still be an expensive route to an accepted result.
Include direct service costs, setup effort, evaluation, and ongoing support over a defined period. Keep uncertain estimates as ranges rather than converting them into precise-looking totals.
Notice when the work changes
AI may lead people to attempt tasks they previously skipped. METR’s discussion of task substitution explains why a changing task mix complicates simple speed comparisons. Track newly attempted work separately and ask whether its outputs are actually useful.
End the pilot with a decision: expand the tested scope, redesign a weak step, continue gathering evidence, or stop. Record the conditions behind that choice.
I want the result to explain where the workflow helps, what it costs to supervise, and where it remains unreliable. That is enough to make a sensible next decision without claiming a universal productivity effect.
Proposal
What I would test next
I would end the pilot with one decision for the tested workflow: expand, redesign, gather more evidence, or stop. The result should include rejected tasks and review effort, so the choice reflects operating value rather than enthusiasm alone.