How do I know whether a Brier score is good?

Compare it with a baseline on the same events. In the five-forecast example, the forecaster scores 0.228. Always forecasting the base rate of 60% scores 0.24, so the skill score is 1 - 0.228 / 0.24 = 0.05: a 5% improvement on simply knowing how often events happen.

Which baseline should I use?

The simplest forecast a user could have made without skill. For binary events, the base rate over the same sample is standard; always 50% is a weaker baseline that makes skill look larger. In the example, the skill score against 50% is 0.088, against the base rate only 0.05.

Can a market price be the baseline?

Yes, and for traders it is the relevant one. A forecaster with a positive skill score against the market price is adding information the market lacked; one with a negative score would have done better simply copying the market.

How can a forecaster beat the base rate but lose to the market?

Easily. If the market priced the same five events at 0.7, 0.5, 0.4, 0.8 and 0.3, its Brier score would be 0.206. The forecaster's 0.228 then gives a skill score of about -0.11 against the market, even though it is +0.05 against the base rate.

How many events does a skill score need?

Many more than five. In the example, if the fifth event had not happened, the forecaster's Brier score would fall to 0.108 and the base rate would become 40%, scoring 0.24: the skill score would jump from 0.05 to 0.55 on one outcome. Small samples make skill scores noisy.

Is the Brier skill score related to other skill measures?

In spirit. Like an information ratio for a fund manager, it measures performance relative to a benchmark rather than in absolute terms, and the answer depends on choosing a benchmark the user could actually have had.

Brier Skill Score

Prediction Markets & Event Contracts

Definition

The Brier skill score (BSS) measures how much better a set of forecasts is than a reference forecast on the same events: BSS = 1 - (Brier score of the forecasts / Brier score of the reference). A score of 1 is perfect, 0 means no better than the reference, and a negative score means worse. Common references are always forecasting the base rate (climatology) or always forecasting 50%.

Why it matters

Skill scores turn a raw accuracy number into an answer to the real question: does this forecast add anything? The same forecasts can look skilled against a coin flip, modestly skilled against the base rate and unskilled against the market price, so the baseline is the claim. Ask which baseline was used and prefer the base rate or, for anyone trading, the market price at the time of the forecast. Ask how many events stand behind the number, since one outcome can swing a small-sample score widely. And treat small positive skill on a small sample as weak evidence.

Common misconceptions

•A skill score depends on the baseline, and a weak baseline flatters, because always forecasting 50% is easier to beat than the base rate on the same events.
•Positive skill against climatology is not skill against the market, because market prices already include information beyond how often events happen.
•Small samples make skill scores noisy, because one changed outcome in five events can move the skill score from 0.05 to 0.55.
•A skill score of zero is not a failure to forecast, because it means matching the reference exactly, which may already be a good forecast.
•Positive skill does not guarantee trading profit, because a trader also pays fees and spreads that a skill score does not count.

Technical details

Worked example (same five events as the Brier score page)

Forecaster's Brier score: 0.228.

Base-rate reference (always 60%): 0.24. BSS = 1 - 0.228/0.24 = 0.05.

50% reference: 0.25. BSS = 1 - 0.228/0.25 = 0.088.

Same forecasts, different baselines, different conclusions.

Worked example: the market as baseline (illustrative)

Market prices on the same events: 0.7, 0.5, 0.4, 0.8, 0.3; outcomes 1, 0, 0, 1, 1.

Squared errors 0.09, 0.25, 0.16, 0.04, 0.49; sum 1.03; market Brier score 0.206.

Forecaster against the market: 1 - 0.228/0.206 = about -0.11.

Sensitivity: if the fifth event had resolved 0, the forecaster would score 0.108 and the 40% base rate 0.24, a skill score of 0.55.

What AltStreet's reviews and filings show

AltStreet's Kalshi study (July 23, 2026) archived 15,685 markets and 15,272 settled outcomes; 1,060 had a price captured while the contract was live, 218 had a quote within three hours of close, and 195, spread across nine series, also had a spread of 10 cents or less; any skill score on those contracts rests on 195 events, not 15,272.

Kalshi study: one series showed z = -3.91 across 109 contracts and decayed to z = +0.28 with more data and filters, the kind of instability that also affects skill scores on small samples.

Kalshi review (August 2026): 'no systematically exploitable edge after fees,' which in skill-score terms means no persistent positive skill against the market price net of costs was found.

Kalshi study: an independent working paper on more than 300,000 Kalshi contracts reports a favorite-longshot pattern that a 195-contract test lacks the power to detect.

Choosing a baseline

Always 50%: the weakest reference; useful only to show a forecaster beats a coin flip.

Base rate (climatology): how often the event type happened in the same sample; the standard reference for binary events.

Market price at the same time: the relevant reference for anyone trading, since it is the price that could have been paid.

State the baseline, the event set and the number of events with every skill score.

How investors apply the term

Investors usually apply this term by tying the definition back to the controlling documents and then asking which cash flows, rights, restrictions, or valuation assumptions it changes. A useful review looks at the term alongside fees, leverage, liquidity, reporting frequency, and manager discretion rather than treating it as a standalone concept.

The practical diligence test is simple: identify who benefits if the assumption is wrong, who has control when conditions change, and what evidence the investor receives after closing. Terms that cannot be reconciled to documents, cash movements, and monitoring data should be treated as marketing shorthand until proven otherwise.

Frequently asked questions

How do I know whether a Brier score is good?: Compare it with a baseline on the same events. In the five-forecast example, the forecaster scores 0.228. Always forecasting the base rate of 60% scores 0.24, so the skill score is 1 - 0.228 / 0.24 = 0.05: a 5% improvement on simply knowing how often events happen.
Which baseline should I use?: The simplest forecast a user could have made without skill. For binary events, the base rate over the same sample is standard; always 50% is a weaker baseline that makes skill look larger. In the example, the skill score against 50% is 0.088, against the base rate only 0.05.
Can a market price be the baseline?: Yes, and for traders it is the relevant one. A forecaster with a positive skill score against the market price is adding information the market lacked; one with a negative score would have done better simply copying the market.
How can a forecaster beat the base rate but lose to the market?: Easily. If the market priced the same five events at 0.7, 0.5, 0.4, 0.8 and 0.3, its Brier score would be 0.206. The forecaster's 0.228 then gives a skill score of about -0.11 against the market, even though it is +0.05 against the base rate.
How many events does a skill score need?: Many more than five. In the example, if the fifth event had not happened, the forecaster's Brier score would fall to 0.108 and the base rate would become 40%, scoring 0.24: the skill score would jump from 0.05 to 0.55 on one outcome. Small samples make skill scores noisy.
Is the Brier skill score related to other skill measures?: In spirit. Like an information ratio for a fund manager, it measures performance relative to a benchmark rather than in absolute terms, and the answer depends on choosing a benchmark the user could actually have had.