Expert knowledge for AI forecasting — superforecasting principles, signal taxonomy, confidence calibration, reasoning chains, and accuracy tracking
73%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Low
Low-risk findings worth noting
Fix and improve this skill with Tessl
tessl review fix ./crates/openfang-hands/bundled/predictor/SKILL.mdBased on research by Philip Tetlock and the Good Judgment Project:
| Type | Description | Weight | Example |
|---|---|---|---|
| Leading indicator | Predicts future movement | High | Job postings surge → company expanding |
| Lagging indicator | Confirms past movement | Medium | Quarterly earnings → business health |
| Base rate | Historical frequency | High | "80% of startups fail within 5 years" |
| Expert opinion | Informed prediction | Medium | Analyst forecast, CEO statement |
| Data point | Factual measurement | High | Revenue figure, user count, benchmark |
| Anomaly | Deviation from pattern | High | Unusual trading volume, sudden hiring freeze |
| Structural change | Systemic shift | Very High | New regulation, technology breakthrough |
| Sentiment shift | Collective mood change | Medium | Media tone change, social media trend |
STRONG signal (high predictive value):
- Multiple independent sources confirm
- Quantitative data (not just opinions)
- Leading indicator with historical track record
- Structural change with clear causal mechanism
MODERATE signal (some predictive value):
- Single authoritative source
- Expert opinion from domain specialist
- Historical pattern that may or may not repeat
- Lagging indicator (confirms direction)
WEAK signal (limited predictive value):
- Social media buzz without substance
- Single anecdote or case study
- Rumor or unconfirmed report
- Opinion from non-specialist95% — Almost certain (would bet 19:1)
90% — Very likely (would bet 9:1)
80% — Likely (would bet 4:1)
70% — Probable (would bet 7:3)
60% — Slightly more likely than not
50% — Toss-up (genuine uncertainty)
40% — Slightly less likely than not
30% — Unlikely (but plausible)
20% — Very unlikely (but possible)
10% — Extremely unlikely
5% — Almost impossible (but not zero)The gold standard for measuring prediction accuracy:
Brier Score = (predicted_probability - actual_outcome)^2
actual_outcome = 1 if prediction came true, 0 if not
Perfect score: 0.0 (you're always right with perfect confidence)
Coin flip: 0.25 (saying 50% on everything)
Terrible: 1.0 (100% confident, always wrong)
Good forecaster: < 0.15
Average forecaster: 0.20-0.30
Bad forecaster: > 0.35| Source Type | Examples | Use For |
|---|---|---|
| Product roadmaps | GitHub issues, release notes, blog posts | Feature predictions |
| Adoption data | Stack Overflow surveys, NPM downloads, DB-Engines | Technology trends |
| Funding data | Crunchbase, PitchBook, TechCrunch | Startup success/failure |
| Patent filings | Google Patents, USPTO | Innovation direction |
| Job postings | LinkedIn, Indeed, Levels.fyi | Technology demand |
| Benchmark data | TechEmpower, MLPerf, Geekbench | Performance trends |
| Source Type | Examples | Use For |
|---|---|---|
| Economic data | FRED, BLS, Census | Macro trends |
| Earnings | SEC filings, earnings calls | Company performance |
| Analyst reports | Bloomberg, Reuters, S&P | Market consensus |
| Central bank | Fed minutes, ECB statements | Interest rates, policy |
| Commodity data | EIA, OPEC reports | Energy/commodity prices |
| Sentiment | VIX, put/call ratio, AAII survey | Market mood |
| Source Type | Examples | Use For |
|---|---|---|
| Official sources | Government statements, UN reports | Policy direction |
| Think tanks | RAND, Brookings, Chatham House | Analysis |
| Election data | Polls, voter registration, 538 | Election outcomes |
| Trade data | WTO, customs data, trade balances | Trade policy |
| Military data | SIPRI, defense budgets, deployments | Conflict risk |
| Diplomatic signals | Ambassador recalls, sanctions, treaties | Relations |
| Source Type | Examples | Use For |
|---|---|---|
| Scientific data | IPCC, NASA, NOAA | Climate trends |
| Energy data | IEA, EIA, IRENA | Energy transition |
| Policy data | COP agreements, national plans | Regulation |
| Corporate data | CDP disclosures, sustainability reports | Corporate action |
| Technology data | BloombergNEF, patent filings | Clean tech trends |
| Investment data | Green bond issuance, ESG flows | Capital allocation |
PREDICTION: [Specific, falsifiable claim]
1. REFERENCE CLASS (Outside View)
Base rate: [What % of similar events occur?]
Reference examples: [3-5 historical analogues]
2. SPECIFIC EVIDENCE (Inside View)
Signals FOR (+):
a. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
b. [Signal] — strength: [strong/moderate/weak] — adjustment: +X%
Signals AGAINST (-):
a. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
b. [Signal] — strength: [strong/moderate/weak] — adjustment: -X%
3. SYNTHESIS
Starting probability (base rate): X%
Net adjustment: +/-Y%
Final probability: Z%
4. KEY ASSUMPTIONS
- [Assumption 1]: If wrong, probability shifts to [W%]
- [Assumption 2]: If wrong, probability shifts to [V%]
5. RESOLUTION
Date: [When can this be resolved?]
Criteria: [Exactly how to determine if correct]
Data source: [Where to check the outcome]{
"id": "pred_001",
"created": "2025-01-15",
"prediction": "OpenAI will release GPT-5 before July 2025",
"confidence": 0.65,
"domain": "tech",
"time_horizon": "2025-07-01",
"reasoning_chain": "...",
"key_signals": ["leaked roadmap", "compute scaling", "hiring patterns"],
"status": "active|resolved|expired",
"resolution": {
"date": "2025-06-30",
"outcome": true,
"evidence": "Released June 15, 2025",
"brier_score": 0.1225
},
"updates": [
{"date": "2025-03-01", "new_confidence": 0.75, "reason": "New evidence: leaked demo"}
]
}ACCURACY DASHBOARD
==================
Total predictions: N
Resolved predictions: N (N correct, N incorrect, N partial)
Active predictions: N
Expired (unresolvable):N
Overall accuracy: X%
Brier score: 0.XX
Calibration:
Predicted 90%+ → Actual: X% (N predictions)
Predicted 70-89% → Actual: X% (N predictions)
Predicted 50-69% → Actual: X% (N predictions)
Predicted 30-49% → Actual: X% (N predictions)
Predicted <30% → Actual: X% (N predictions)
Strengths: [domains/types where you perform well]
Weaknesses: [domains/types where you perform poorly]Before finalizing any prediction, check for these biases:
Anchoring: Am I fixated on the first number I encountered?
Availability bias: Am I overweighting recent or memorable events?
Confirmation bias: Am I only looking for evidence that supports my prediction?
Narrative bias: Am I choosing a prediction because it makes a good story?
Overconfidence: Am I too sure?
Scope insensitivity: Am I treating very different scales the same?
Recency bias: Am I extrapolating recent trends too far?
Status quo bias: Am I defaulting to "nothing will change"?
When enabled, for each consensus prediction:
RightNow-AI/openfang
acf2587
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.