Prediction Evaluation

Is this prediction worth tracking?

Not all predictions deserve attention. This checklist separates signal from noise by scoring prediction quality before you invest time tracking it.

The SMART-BF Checklist

Six dimensions, scored 0-2 each. Total: 0-12 points.

Dimension	Question	0	1	2
Specific	Is it precise and unambiguous?	Vague ("AI will change things")	Somewhat specific	Precise ("GDP-Val >90% by Dec 2026")
Measurable	Can we objectively verify resolution?	No clear verification	Partially measurable	Binary yes/no with clear criteria
Actionable	Does it enable positioning decisions?	Entertainment only	Indirect implications	Direct action if true/false
Resolution	Is there a clear time horizon?	No timeframe	Vague ("soon", "eventually")	Specific date or trigger event
Testable	What would prove it wrong?	Unfalsifiable	Weak falsification criteria	Clear falsifying conditions
Base rate	Is there historical precedent?	No analogous history	Weak analogies	Strong base rate available
Factored	Does it depend on other predictions?	Many hidden dependencies	Some dependencies acknowledged	Independent or dependencies explicit

Scoring Guide

Score	Quality	Action
10-12	Excellent	Track actively, assign conviction, position
7-9	Good	Track, but note quality gaps
4-6	Marginal	Improve specificity before tracking
0-3	Poor	Don't track — reframe or discard

Quality → Conviction Mapping

High quality prediction ≠ high conviction prediction.

• Quality = how well-formed is the prediction itself?
• Conviction = how likely do you think it is to occur?

A prediction can score 12/12 on quality ("Bitcoin hits $200K by Dec 31, 2025") while you have low conviction (1/5) it will happen.

Quality Score	Eligible Conviction Range
10-12	Full range (0-5)
7-9	Cap at 4/5 (quality uncertainty)
4-6	Cap at 3/5 (prediction unclear)
0-3	Don't assign conviction

Worked Example

Prediction: "AI solves at least one Clay Millennium Prize math problem in 2026"

Dimension	Score	Reasoning
Specific	2	Clear outcome (one of 7 named problems)
Measurable	2	Clay Institute verification process exists
Actionable	1	Indirect positioning implications
Resolution	2	"In 2026" = by Dec 31, 2026
Testable	2	No solution announced = falsified
Base rate	1	No prior AI math proof at this level
Factored	1	Depends on AI capability trajectory

Total: 11/12 — Excellent quality, worth tracking.

Conviction assignment: 3/5 (uncertain on timeline, confident on direction)

Common Quality Failures

Vague predictions (low Specificity)

• "AI will transform business" → Better: "50% of Fortune 500 will have AI-native divisions by 2027"
• "Crypto will go mainstream" → Better: "US spot Bitcoin ETFs exceed $100B AUM by Dec 2026"

Unfalsifiable predictions (low Testability)

• "We're in the early innings of AI" → Better: "Frontier Math Tier 4 exceeds 40% by Dec 2026"
• "The future belongs to builders" → Better: "Single-founder billion-dollar startup emerges by 2027"

Missing base rates (low Base rate)

• "AGI by 2027" → Add: "Based on GPT-2→GPT-4 capability doubling timeline of ~2 years"
• "10x efficiency gains" → Add: "Manufacturing automation precedent: 8-12x over 20 years"

Hidden dependencies (low Factored)

• "Level-5 autonomy deployed in 2026" → Add: "Depends on: regulatory approval, liability framework, OEM adoption"

The Inversion Test

Before scoring, ask: What would make this prediction worse?

If the answer includes:

• "Be more specific" → Specificity problem
• "Define success" → Measurability problem
• "Pick a date" → Resolution problem
• "Acknowledge what could prove it wrong" → Testability problem

Using This Checklist

1. Before adding to prediction database: Score quality first
2. When reviewing others' predictions: Apply checklist before forming conviction
3. When your conviction changes: Check if quality score also changed (new information → reframe prediction)

Context

• Prediction Process — The five questions for every prediction
• Superforecasting — Build the discipline
• Probability — Size bets correctly
• Prediction Database — Track what matters