I am looking for quantitative measures for the value of a forecast. A forecast is more valuable to me if

  1. it tells me things that are not already implied by the conventional knowledge todate, and
  2. the future will show this forecast to be accurate.

For example, suppose an analyst forecasts that NVDA will drop from over \$800 to below \$500 by 9/30/2024. Here is how I see the value of that forecast:

  1. How much of a surprise is this forecast given the current futures market for NVDA?
  2. On 9/30/2024, we will know how right or wrong this forecast was.

Obviously, #2 above can only be measured in the future with 20/20 hindsight. Why would I look for a way to measure the value of a forecast that can only be computed in the future? Reputation. Track record. If the 5-year track record for this analyst were to show a steady stream of highly valuable forecasts, ... well ... where do I sign up? BTW, I am fascinated by the book "Superforecasting: The Art and Science of Prediction" by Tetlock and Gardner, although I felt that they focus more on #2 above than on #1 above. The bios for venture capitalists on their websites often hint at their prowess in #1. And politicians ("My position remains unchanged ..."). Track records for financial analysts should be a lot more objective and easier to compute. But, I have not found any measures.

I am tempted to define my own measure by concocting something that attempts to combine the above two. But, I don't want to reinvent the wheel, especially since I am not an expert in statistics or economics.

Are there quantitative measures for the value of a forecast that combine both (1) how informative (surprising?) the forecast was at the time it was made and (2) how right or wrong the future proved it to be?

To keep things simple, let's assume that the forecast is for a single variable (e.g., stock price for NVDA at close on 9/30/2024), and that "conventional knowledge" on any given day is a probability distribution for that single variable. Then, one way to measure #1 above is some kind of statistical distance between this point forecast and the present value of the distribution. By 9/30/2024, the distribution will converge to a single point -- the closing price. Then we compute #2 above as the same statistical distance. Now, how would I combine the two statistical distances? How about d1 / d2? Aaargh! Someone must have done this before.

One reason I am leaning towards statistical distance is because it works even before the date of the forecast. For example, market sentiments about NVDA may swing wildly between now and 9/30/2024. As it does so, the above way of measuring the value of the forecast kind of makes sense.

  • $\begingroup$ So you are trying to assess the value a forecast had at date X using information available at the latter date Y? How do you defend against brute force hacking, i.e. me making a prediction for all possible lotto numbers? Ex post you would see one of these as being extremely valuable, but ex post you have no way of knowing which one. $\endgroup$
    – Giskard
    Commented Apr 10 at 7:57
  • $\begingroup$ @Giskard, was the last ex post meant as ex ante? $\endgroup$ Commented Apr 10 at 8:47
  • $\begingroup$ @RichardHardy Indeed. Sadly, it is too late to edit. $\endgroup$
    – Giskard
    Commented Apr 10 at 9:28
  • $\begingroup$ I don't think you've chosen the right forum. Please post this on Cross-Validated (Statistics) and perhaps even on Data-Science if you can formulate your question in a way that suits the DS forum. $\endgroup$ Commented Apr 11 at 10:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.