Mechanism design is a field in economics and game theory that takes an engineering approach to designing economic mechanisms or incentives, toward desired objectives, in strategic settings, where players act rationally.
In mechanism design, a principal (or mechanism designer) designs the "rules" of a Bayesian game to implement some goal, but takes as given an economic environment. In standard settings, this given environment consists of a given number of agents, payoff functions conditional on types and outcomes, a type space, and beliefs about type profiles. The principal commits to a set of rules that determine what each player can do (and in which order) and what happens for each action profile, i.e., she sets an action space and an outcome function mapping each action profile into an outcome (such as "who gets the good and what is the price", or "is the public good provided or not and who has to contribute what"). Together with the economic environment, a Bayesian game is induced. Typically, the goal of the principal is to either maximize welfare or profit.
In such settings, often (but not always) a revelation principle holds, which essentially implies that whatever the principal can achieve through some arbitrary set of rules can also be achieved through an incentive-compatible direct mechanism: If a social choice function${}^1$ can be implemented${}^2$ by an arbitrary mechanism, then the same function can be implemented by an incentive-compatible and direct mechanism with the same outcome. Here is an accessible sketch of the proof. As a result, the principal can focus on direct mechanisms in which participants have incentives to report their types truthfully, and she then maximizes her goal function constrained by incentive compatibility and possible other contraints such as individual rationality (a participation constraint).
Other than the revelation principle, notable results are the revenue equivalence theorem, the Gibbard–Satterthwaite theorem, and the Myerson–Satterthwaite theorem. Seminal contributions include Vickrey–Clarke–Groves mechanisms and Myerson's optimal auction. Mechanism design is closely connected to auction design, while the design of matching markets (often without transfers) often runs under the broader label "market design".
${}^1$ A social-choice function maps type profiles $\theta$ into outcomes $x$, $f: \Theta \rightarrow X$.
${}^2$ A mechanism implements a social choice function, if the (game induced by the) mechanism has an equilibrium outcome that corresponds to the outcome of the social choice function.
