I'd like to understand how this infinite recursion (for lack of a better term...) of knowledge figures mathematically into say, equilibrium concepts.
It figures mathematically, for example, in the simplification of strategy sets through elimination of strictly-dominated alternatives. Others can correct me, but the point isn't to write down a mathematical statement for the infinitely-recursive assumptions; the point is to force you to think about the fact that in order to solve game theory problems, we really must make some heavy assumptions. It is not enough to assume that players are rational, because a rational player with no knowledge about her opponent's state of mind cannot safely rule out the possibility that her opponent will choose sub-optimal strategies. This is especially important for games where the a priori strategy set is infinite, and the solution concept requires arguing heuristically for the elimination of large swaths of available strategies (e.g., the "intelligent brute force" approach).
If you allow for imperfect information about preferences (e.g., Bayesian games), it may seem that you are getting free of the recursion since you are explicitly allowing the states of nature to include ones in which one or more players gets a higher payoff from apparently mutually destructive play. A common classroom example looks at the Cuban Missile Crisis through this lens.
But this doesn't quite get you off the hook for assuming the recursion - it is about uncertainty of preferences, not uncertainty about whether players act in accordance with those preferences. In order for MAD to present a credible threat, firing the missiles must give the dictator greater utility along a path of play that arises with non-zero probability. And you need the infinitely-recursive assumption in order to assess that probability.
What breaks down if at some level, the 'knowledge of knowing' doesn't hold?
Basically, everything. To be less glib, you lose the rigorous connection between preferences and action, because you cannot restrict players to act in accordance with them. Obviously, in real life we don't run this recursion through our heads as we play games - but in real life, it's also an open question whether any given agent really knows their own preferences, and so in a chess match, we also cannot say for sure that our opponent's next move won't be to knock all the pieces from the board.
