9
$\begingroup$

Humans tend to be overconfident in their predictions; when most people say that there's a 95% chance that something will happen, they're usually wrong far more than 5% of the time. Whereas what ought to happen is that for all $x$, if a person makes a lot of predictions of the form "There is an $x\%$ chance that P will occur," then about $x\%$ of them should come true. If a person's predictions does satisfy this property, then their subjective probabilities are said to be well-calibrated.

Now there may be no one who is perfectly well-calibrated, but some people are better than others; see the book "Superforecasting" for information on people who are extremely well-calibrated. But calibration isn't something fixed from birth; you can train yourself to be better-calibrated by playing a game where you assign a probabilities to a bunch of things and then you're scored by how well-calibrated your subjective probabilities are (using e.g. the Brier score). See this web site and this website.

But my question is, why is it possible to become well-calibrated using such a game? Let $A$ be the set of possible events $f$ be a function from $A$ to $\mathbb R$ that gives a numerical assessment of how confident a person feels about a given event. Then what playing the game is designed to do is to help you find an increasing function $g$ from $\mathbb R$ to $[0,1]$ such that $g \circ f$ is well-calibrated.

But in order for such a $g$ to exist, it needs to be the case that for all $x,y \in \mathbb R$, if $x>y$ then the percentage of outcomes in $\{P \in A: f(P) =x\}$ which are true needs to be greater than the percentage of outcomes in $\{P \in A: f(P) =y\}$ which are true. In other words, it needs to already be true that you're right more often about statements that you're more confident in than statements you're less confident in. But why should this be the case? Are there any decision-theoretic assumptions or rationality axioms that imply this is the case? Has it been empirically observed that humans are more often right about what they're more confident in?

It would certainly be a useful evolutionary skill if humans' internal confidence levels reflected external reality, but I've never heard of anyone discuss this before.

Improve this question
$\endgroup$
5
  • $\begingroup$ To answer your question: I don't see why it would it would be the other way round. It doesn't make sense for people to be confident about things that are incorrect and unconfident about things that are correct, unless they are misinformed. $\endgroup$
    – ahorn
    Commented Nov 11, 2016 at 8:43
  • $\begingroup$ @ahorn Well, "misinformed" is exactly what this whole topic is about. We as humans have incomplete and potentially faulty information, and based on that we make subjective probability judgements about what statements are true or false. Look at the games I linked to above; they involve trivia questions which you may or may not know the answer to, and then you choose what you think is the most likely answer along with how confident you are that you're right. $\endgroup$
    – Keshav Srinivasan
    Commented Nov 11, 2016 at 9:48
  • $\begingroup$ @ahorn "It doesn't make sense for people to be confident about things that are incorrect and unconfident about things that are correct, unless they are misinformed." Note that even a perfectly well-calibrated person will be incorrect 10% of the time on things they're 90% confident in. So it makes perfect sense to be incorrect about something you're highly confident in, and to be correct about things you're not that confident. $\endgroup$
    – Keshav Srinivasan
    Commented Nov 11, 2016 at 10:02
  • $\begingroup$ I think it's a fairly weak assumption to make that people are generally more confident about things they know about. This is certainly an assumption the model is making. If we consider the case where this isn't true then it must be the case that: A person is uncertain about the things he does know, and certain about the things he does not? I think this is reasonable only given a highly misinformed person and if are set of events was an extremely narrowly focused set of events. However I think that this set of events to calibrate to suggests it refers to all possible events. $\endgroup$
    – Lee Sin
    Commented Nov 11, 2016 at 17:51
  • 2
    $\begingroup$ @LeeSin "I think it's a fairly weak assumption to make that people are generally more confident about things they know about." The Dunning-Kruger works show that that's a largely unjustifiable assumption. People generally are uncertain about the things they know best, and certain about the things they know little about. $\endgroup$
    – 410 gone
    Commented Nov 12, 2016 at 10:58

1 Answer 1

Reset to default
1
$\begingroup$

Empirically yes, it seems confidence has a positive correlation with correctness. One reference is Bordalo et al. 2018 Beliefs about gender, see Fig.1 for example. The actual probability of a correct answer is different from own or other people's estimate of the probability of a correct answer, but if the estimated probability is higher, then the actual probability is higher.

As to the axioms or theoretical assumptions - you can just assume that people's estimates of their correctness are increasing in the correctness. Or assume Bayesian updating about own correctness, in which case past correct guesses would increase the guesser's confidence in own correctness. The number of past correct guesses in turn is positively correlated with the guesser's actual correctness (by the Law of large numbers).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.