Use this before a language model is given a defined step in the firm's investment process, and again whenever the model, the prompt, or the firm's criteria change. Fill it from the firm's own past decisions, not from a demonstration. The method follows Csaszar, Ketkar, and Kim (Strategy Science 9, no. 4, 2024, pages 322 to 345; figures from the authors' arXiv version of August 2024); the comparison is the firm's own evidence.
Firm / scope (funds, stages, the step the model will take):
Prepared by / date:
Owner (who can explain any score the model gives):
Model and service used, and its terms for retention and training on inputs:
Placement
- [ ] The step is named in one sentence (for example, "scores every inbound plan against the first-look criteria" or "writes a devil's advocate note on each memo before the partner meeting").
- [ ] The model does not decline or advance a company on its own. A named person does, and can see the score.
- [ ] Each score is the average of at least three runs, and the record shows the spread.
- [ ] The criteria the model scores against are the firm's own, written down, and the same ones partners use.
- [ ] Founders' materials go only to the approved service, and the firm has decided whether founders are told.
Step the model takes:
Runs per score: Criteria document and version:
Blind comparison
Run on 30 to 50 companies the firm has already decided on. Score each with the model as it would be used, without showing it the decision.
Companies / period:
Correlation between the model's average score and the partners' first-look view:
| Company | Model score (average) | Spread across runs | Partners' decision | Outcome, if known | Largest disagreement? |
|---|---|---|---|---|---|
- [ ] The ten largest disagreements are reviewed by a partner, and the reason for each is recorded.
- [ ] The review notes whether disagreements cluster on novel companies, a sector, or a criterion.
- [ ] Criteria where the model and partners agree least are marked for partner attention, not model scoring.
Where the model and partners disagree most (criteria, sectors, kinds of company):
Devil's advocate pass
- [ ] Tried on at least five past memos, and a partner judged whether each note raised something the meeting missed.
Notes judged useful / tried: /
Decision: adopt for the named step / adopt the devil's advocate pass only / run the comparison again later / do not adopt
Approved by / date:
Next review (at least annually, and after any change of model, prompt, or criteria):
Common failure modes
- One run is treated as a score, and the next run gives a different answer.
- The model is scored against its own rubric rather than the firm's, so agreement means nothing.
- A low score quietly removes a company from the queue.
- The comparison is skipped because the summaries read well.
- Decks go into a personal account or an unapproved service.
This review decides where a model sits in the investment process and how its output is used. It does not assess the service's security or contract terms; use the vendor review and acceptance brief for that.