# Model-in-the-decision review

Use this before a language model is given a defined step in the firm's investment process, and again whenever the model, the prompt, or the firm's criteria change. Fill it from the firm's own past decisions, not from a demonstration. The method follows Csaszar, Ketkar, and Kim (*Strategy Science* 9, no. 4, 2024, pages 322 to 345; figures from the authors' arXiv version of August 2024); the comparison is the firm's own evidence.

**Firm / scope (funds, stages, the step the model will take):** ____________________

**Prepared by / date:** ____________________

**Owner (who can explain any score the model gives):** ____________________

**Model and service used, and its terms for retention and training on inputs:** ____________________

## Placement

- [ ] The step is named in one sentence (for example, "scores every inbound plan against the first-look criteria" or "writes a devil's advocate note on each memo before the partner meeting").
- [ ] The model does not decline or advance a company on its own. A named person does, and can see the score.
- [ ] Each score is the average of at least three runs, and the record shows the spread.
- [ ] The criteria the model scores against are the firm's own, written down, and the same ones partners use.
- [ ] Founders' materials go only to the approved service, and the firm has decided whether founders are told.

**Step the model takes:** ____________________

**Runs per score:** ______ **Criteria document and version:** ____________________

## Blind comparison

Run on 30 to 50 companies the firm has already decided on. Score each with the model as it would be used, without showing it the decision.

**Companies / period:** ____________________

**Correlation between the model's average score and the partners' first-look view:** ______

| Company | Model score (average) | Spread across runs | Partners' decision | Outcome, if known | Largest disagreement? |
| --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |
|  |  |  |  |  |  |

- [ ] The ten largest disagreements are reviewed by a partner, and the reason for each is recorded.
- [ ] The review notes whether disagreements cluster on novel companies, a sector, or a criterion.
- [ ] Criteria where the model and partners agree least are marked for partner attention, not model scoring.

**Where the model and partners disagree most (criteria, sectors, kinds of company):** ____________________

## Devil's advocate pass

- [ ] Tried on at least five past memos, and a partner judged whether each note raised something the meeting missed.

**Notes judged useful / tried:** ______ / ______

**Decision:** adopt for the named step / adopt the devil's advocate pass only / run the comparison again later / do not adopt

**Approved by / date:** ____________________

**Next review (at least annually, and after any change of model, prompt, or criteria):** ____________________

## Common failure modes

- One run is treated as a score, and the next run gives a different answer.
- The model is scored against its own rubric rather than the firm's, so agreement means nothing.
- A low score quietly removes a company from the queue.
- The comparison is skipped because the summaries read well.
- Decks go into a personal account or an unapproved service.

This review decides where a model sits in the investment process and how its output is used. It does not assess the service's security or contract terms; use the vendor review and acceptance brief for that.