Is there any more information about why von Neumann had this attitude?
There must be, but I have never seen it.
Or can we infer a reasonable answer?
Here is one possibility based on hearsay among the Game Theory community.
Von Neumann thought that there was a sharp distinction between zero-sum games and other games.
- In zero-sum games, there was no hope of cooperation. Therefore, it makes sense for players to engage in selfish, individualistic decision making. It is natural to assume that players choose what to do independently and with their best interests in mind. The resulting solution concept for such games is the concept of minimax equilibrium, which coincides with the concept of Nash equilibrium.
- In other games, there is usually at least some room for mutually beneficial cooperation. It makes sense for rational people to talk to each other and try to find ways to cooperate. Hence, it is not evident that players must choose what to do independently and with only their best interests in mind. The way I've heard the story is that, for such games, Von Neumann favored the approach of Cooperative Game Theory in which decisions could be made collectively by groups of people.
I will add a further note. In the 1950s, there was no empirical evidence to support either hypothesis. In retrospect, we do see in practice that people often cooperate when they face prisoners' dilemmas. Also, the most widely cited field evidence supporting equilibrium behavior is from competitive settings that are close to zero-sum games (penalty kicks, tennis serves).