One of the key (and often criticized) assumptions of Game Theory is the rationality of the players/agents and common knowledge thereof. That is the agents will make the best decision for themselves given the information available.
This assumption basically says cognitive costs are negligable. E.g. I can easily tell wether it is optimal for me to buy luggage insurance. In reality I would probably need to look up statistics and make some basic calculations.
Agent-based modelling frequently uses heuristic behavioral rules instead of optimization. In some circumstances this rules are not optimal, but very close. A nice example is the gaze heuristic:
The gaze heuristic is used by humans and animals for catching flying objects. It entails the fixation of one's gaze to the object and adjustment of the running speed so that the angle of the gaze remains constant while approaching the object (see the three decision rules in the table above). Empirical evidence shows that experienced ball-catchers use the gaze heuristic and similar heuristics, as do dogs when trying to catch Frisbees.
Similarly in the Solow model one assumes that consumers save a constant portion of their income, without any utility calculations.
By simulating models with different behavioral rules one may find a model which is a good approximation of reality or one may find a simple rule which would make near optimal decisions.
While ABM is more flexible, that flexibility is also a problem: There is a myriad of heuristics to chose from, and different heuristics frequently lead to wildly different outcomes.