# Why is it possible to calibrate your subjective probabilities?

Humans tend to be overconfident in their predictions; when most people say that there's a 95% chance that something will happen, they're usually wrong far more than 5% of the time. Whereas what ought to happen is that for all $x$, if a person makes a lot of predictions of the form "There is an $x\%$ chance that P will occur," then about $x\%$ of them should come true. If a person's predictions does satisfy this property, then their subjective probabilities are said to be well-calibrated.

Now there may be no one who is perfectly well-calibrated, but some people are better than others; see the book "Superforecasting" for information on people who are extremely well-calibrated. But calibration isn't something fixed from birth; you can train yourself to be better-calibrated by playing a game where you assign a probabilities to a bunch of things and then you're scored by how well-calibrated your subjective probabilities are (using e.g. the Brier score). See this web site and this website.

But my question is, why is it possible to become well-calibrated using such a game? Let $A$ be the set of possible events $f$ be a function from $A$ to $\mathbb R$ that gives a numerical assessment of how confident a person feels about a given event. Then what playing the game is designed to do is to help you find an increasing function $g$ from $\mathbb R$ to $[0,1]$ such that $g \circ f$ is well-calibrated.

But in order for such a $g$ to exist, it needs to be the case that for all $x,y \in \mathbb R$, if $x>y$ then the percentage of outcomes in $\{P \in A: f(P) =x\}$ which are true needs to be greater than the percentage of outcomes in $\{P \in A: f(P) =y\}$ which are true. In other words, it needs to already be true that you're right more often about statements that you're more confident in than statements you're less confident in. But why should this be the case? Are there any decision-theoretic assumptions or rationality axioms that imply this is the case? Has it been empirically observed that humans are more often right about what they're more confident in?

It would certainly be a useful evolutionary skill if humans' internal confidence levels reflected external reality, but I've never heard of anyone discuss this before.

• To answer your question: I don't see why it would it would be the other way round. It doesn't make sense for people to be confident about things that are incorrect and unconfident about things that are correct, unless they are misinformed. – ahorn Nov 11 '16 at 8:43
• @ahorn Well, "misinformed" is exactly what this whole topic is about. We as humans have incomplete and potentially faulty information, and based on that we make subjective probability judgements about what statements are true or false. Look at the games I linked to above; they involve trivia questions which you may or may not know the answer to, and then you choose what you think is the most likely answer along with how confident you are that you're right. – Keshav Srinivasan Nov 11 '16 at 9:48
• @ahorn "It doesn't make sense for people to be confident about things that are incorrect and unconfident about things that are correct, unless they are misinformed." Note that even a perfectly well-calibrated person will be incorrect 10% of the time on things they're 90% confident in. So it makes perfect sense to be incorrect about something you're highly confident in, and to be correct about things you're not that confident. – Keshav Srinivasan Nov 11 '16 at 10:02
• I think it's a fairly weak assumption to make that people are generally more confident about things they know about. This is certainly an assumption the model is making. If we consider the case where this isn't true then it must be the case that: A person is uncertain about the things he does know, and certain about the things he does not? I think this is reasonable only given a highly misinformed person and if are set of events was an extremely narrowly focused set of events. However I think that this set of events to calibrate to suggests it refers to all possible events. – Lee Sin Nov 11 '16 at 17:51
• @LeeSin "I think it's a fairly weak assumption to make that people are generally more confident about things they know about." The Dunning-Kruger works show that that's a largely unjustifiable assumption. People generally are uncertain about the things they know best, and certain about the things they know little about. – EnergyNumbers Nov 12 '16 at 10:58