I’ve seen definitions of “the core” that rely on some specific assumptions, such as that the game is one of a transfer of resources, or that there is “transferability of utility” (presumably intending to capture the notion of money being the source of this).
But what is the most general definition of the core? Can we talk about the core of any arbitrary game, or does the concept fundamentally only work for specific types of games?
We can talk of the core for any arbitrary game. To explain how let me compare it directly to how we define non-cooperative games. In a standard non-cooperative game we define Players, $I$, action sets for each player, $A_i$ for each $i\in I$ and payoff functions that map any profile of actions to payoffs $u_i: A\rightarrow \mathbb{R}$, where $A=\prod_{i\in I} A_i$. Finally the solution concept, for example, Nash equilibrium, correlated equilibrium or rationalizability is a condition on these objects that we think are a good prediction of what will happen. For instance, Nash equilibrium predicts that every player is best responding to a correct belief about what other players are doing.
The Core is another solution concept, however, the language developed so far is not sufficient, because the idea of the core is that it predicts that no group of people (or coalition) will have a profitable deviation. It is clear that now the unit of analysis are groups of people, so the "agents" in this setup are the elements of $P(I)$, the power set of $I$, which contains all possible groups of people, so you will need to define the action sets for each of these groups. A natural way could be that if $C\in P(I)$ then $A_C=\prod_{i\in I\cap C} A_i$, i.e. the coalition $C$ can choose the actions of all its players. It is clearly not the only possibility, maybe you want to allow for players to have some autonomy within the coalition, but let's say you define action sets that way.
You will also have to define payoff functions for each possible coalition. Again a natural way could be to define $u_C=\sum_{i\in iI\cap C} u_i$, i.e. the sum of utilities of players in the coalition. Note that your assumption on this coalition utility will necessarily come with some assumptions of how you substitute between players' utilities within the coalition.
Finally, you will have to define which coalitions are possible to co-exist. For example, can players belong to multiple coalitions or not? Again a natural way would be to rule out this possibility so that in equilibrium each player must belong to only one coalition, i.e. $C\cap C'=\emptyset$. This language will allow you to define that the core will be a situation where all players belong to a coalition, each coalition chooses an action for each of their players and there is no subset of players that will rather separate from their current coalition to form another one and choose a different profile of actions.
Hopefully, I have made clear that I needed to make several assumptions just to be able to define the core. Especially if I want to have some counterpart with the original non-cooperative game I started with. This is cumbersome.
I think this is why, to some extent, the way some of the literature developed was by not subordinating the definition of core (intrinsically a cooperative solution concept) to the terminology of a non-cooperative game. One way to do it is by directly defining a payoff function for each player from being part of a coalition $V_i(C)$ for each player where $C\in P(I)$ (this notation has implicit that being in a coalition can probably imply different actions on your part). The core is then a partition of players (each player belongs to a coalition, say $i\in C_i^*$) such that there is no coalition $C'\in P(I)$ such that for all $i\in C'$, you have $V_i(C')\geq V_i(C_i^*)$ and with strict inequality for some $i\in C'$.
Clearly, this last formulation has fewer assumptions than my first formulation. However, it is also clear that it does have assumptions. The main ones (at least for me) is that suffices to know what coalition you belong to figuree out your payoff (i.e. no externalities between coalitions). And the other important assumption is that players can only belong to one coalition. This is restrictive in many situations, for example, politicians that belong to multiple interest groups, etc.
Depending on what you are willing to assume about the nature of cooperation and deviations from cooperation, you will get a different-looking definition of core, but the general principle is the same. The core is the prediction that captures that no subgroup of players has incentives to coordinate and improve their condition. So the concept is not bound to specific kinds of games, but I think that there is no consensus on how to apply it universally to all kinds of games.
