When treating a relative, normalized utility function as a pmf, what is the interpretation of Shannon entropy or Shannon information?

Question

Suppose $\Omega$ is a set of mutually exclusive outcomes of a discrete random variable and $f$ is a utility function where $0 < f(\omega) \leq 1$, $\sum_\Omega f(\omega) = 1$, etc.

When $f$ is uniformly distributed over $\Omega$ and $f$ is a probability mass function, the Shannon entropy $H(\Omega) = \sum_{\Omega}f(\omega)log\frac{1}{f(\omega)}$ is maximized ($=log|\Omega|)$, and when one element in $\Omega$ has all of $f$'s mass, the Shannon entropy is minimized ($0$, in fact). This corresponds to intuitions about surprisal (or uncertainty reduction) and outcomes and uncertainty (or expected surprisal) and random variables:

• When $f$ is uniformly distributed, uncertainty is maximized, and the more outcomes there are for mass to be uniformly distributed over, the more uncertain we are.
• When $f$ has all its mass concentrated in one outcome, we have no uncertainty.
• When we assign an outcome a probability of $1$, we gain no information (are "unsurprised") when we actually observe it.
• When we assign an outcome a probability closer and closer to $0$, the observation of it actually occurring becomes more and more informative ("surprising").

(This all says nothing about the much more concrete -- but less epistemic -- coding interpretation of Shannon information/entropy, of course.)

However, when $f$ has the interpretation of a utility function, is there a sensical interpretation of $log\frac{1}{f(\omega)}$ or $\sum f(\omega)log\frac{1}{f(\omega)}$? It seems to me that there might be:

• if $f$ as a PMF represents a uniform distribution over $\Omega$, then $f$ as a utility function corresponds to indifference over the outcomes that could not be greater*
• a utility function where one outcome has all of the utility and the rest have none (as skewed of a utility as there could be) corresponds to very strong relative preferences -- a lack of indifference.

Is there a reference expanding on this? Have I missed something about the limitations on comparing probability mass functions and normalized, relative utilities over discrete random variables?

*I am aware of indifference curves and do not see how they might be relevant to my question for a variety of reasons, starting with my focus on a categorical sample space and with the fact that I am not interested in 'indifference' per se, but rather how to interpret utilities as probabilities and how to interpret functionals on probabilities when the (discrete) 'probability distribution' in question actually or (additionally) has the interpretation of a utility function.

Answer 1

Before discussion Shannon's entropy, there is another point that should be discussed: it appears that you have in mind cardinal utility rather than ordinal.

"Normalized" utility functions can be derived of course in both cases. But the concept of "relative preference" can be defined and measured only in the context of cardinal utility.

And the issue does not arise at the two extremes you describe, but in all the possible intermediate cases.

A simple example: assume that there are three "outcomes", $A, B, C$ (say, consumption levels, or three different goods each at some quantity). Your utility function assigned to them the values

$$V(A) = 1, \;\;V(B) = 9,\;\; V(C) = 90$$

Under ordinal utility, this just tells us that

$$A <_{pr} B <_{pr} C$$

Certainly we can normalize these by dividing by $100$ to obtain

$$U_V(A)=0.01, \;\; U_V(B) = 0.09,\;\; U_V(C) =0.9$$ and the ranking of the three outcomes is preserved

But under ordinal utility, we could very well use another utility function that would assign

$$W(A) = 31, \;\;W(B) = 32,\;\; W(C) = 37$$

and obtain

$$U_W(A)=0.31, \;\; U_W(B) = 0.32,\;\; U_W(C) =0.37$$

The ranking is the same so the two utility functions $V$ and $W$ are equivalent under ordinal utility.

But in what you are describing, the $W$ utility function represents different relative preferences than $V$ and so it is not the same utility function. But this is meaningful only under cardinal utility, where quantitative comparisons between utility numbers are assumed to have meaning.

Are you familiar with the issues surrounding cardinal utility?

Answer 2

After the exchange with the OP in my other answer, let's work a bit with his approach.

We have a discrete random variable $X$ with finite support, $X = \{x_1,...,x_k\}$, and probability mass function (PMF), $\Pr(X=x_i)=p_i, i=1,...,k$

The values in the support of $X$ are also inputs in a real-valued cardinal utility function, $u(x_i) > 0\; \forall i$. We then consider the normalized utility function

$$w(X): w(x_i) = \frac {u(x_i)}{\sum_{i=1}^ku(x_i)},\;\;i=1,...,k \tag{1}$$

and we are told that

$$w(x_i) = p_i \tag{2}$$

Note that we do not just make the observation that a normalized non-negative discrete function of finite domain, satisfies the properties of a probability mass function in general -we specifically assume that $w(x_i)$ has the functional form of the PMF of the random variable whose values $w(x_i)$ takes as inputs.

Since $w(x_i)$ is a measurable function of a random variable, it, too, is a random variable. So we can meaningfully consider things like its expected value. Using the Law of the Unconscious Statistician we have

$$E[w(X)] = \sum_{i=1}^kp_iw(x_i) = \sum_{i=1}^kp_i^2 \tag{3}$$

This is a convex function, and if we try to extremize it over the $p_i$'s under the constraint $\sum_{i=1}^kp_i=1$ we easily obtain

$$\text{argmin} E[w(X)] = \mathbf p^*: p_1=p_2=...=p_k=1/k \tag {4}$$

and we have obtained a general result:

The normalized utility function as defined above has minimum expected value iff the distribution of $X$ is Uniform.

Obviously in such a case $w(X)$ will be a constant function, a degenerate random variable with $E[w(X)]=1/k$ and zero variance.

Let's turn to Shannon's Entropy which is the focus of the OP. To be calculated, Shannon's Entropy needs the probability mass function of the random variable... so we should find the PMF of the random variable $w(X)$...

But it is my impression that this is not what the OP has in mind. Rather, it views Shannon's Entropy as a metric that has some desirable algebraic properties and perhaps can measure compactly in a meaningful way something of interest.

This has been done before in Economics, specifically in Industrial Organization, were Indices of Market Concentration ("degree of competition/monopolistic structure of a market") have been constructed. I note two that look particularly relevant here.

A) The Herfindahl Index, has as its arguments the market shares of the $n$ companies operating in a market, $s_i$, so they sum to unity by construction. Its unscaled version is

$$H = \sum_{i=1}^n s_i^2$$

which is an expression that has the exact same structure with the expected value of $w(X)$ derived above.

B) The Entropy Index $$R_e = -\sum_{i=1}^n s_i\ln s_i$$ which has the exact mathematical form with Shannon's Entropy.

Encaoua, D., & Jacquemin, A. (1980). Degree of monopoly, indices of concentration and threat of entry. International Economic Review, 87-105., provide an axiomatic derivation of "allowable" concentration indices, i.e. they define the properties that such an index must possess. Since their approach is abstract, I believe it may be useful to what the OP wishes to explore and attach meaning to.

Answer 3

It seems the utility function is not only cardinal here, but even defined on a ratio scale. Consider two outcomes with utilities 1/4 and 3/4. Clearly, we can apply the affine transformation: $v=v*2-0.5$ in which case the utilities become 0 and 1. However, we have now changed the entropy from a strictly positive value to zero!

Thus, you would need to first provide a meaningful ratio scale to your utility. One way to do this is to give an interpretation to the natural 0 utility level. Without this specification the entropy is meaningless.