Most prominently, it was Bob Wilson who criticized the role of common priors in game theory. Starting from the "Wilson doctrine", some work in mechanism was done in that direction. Note that this research endeavor is not hopeless: in a popular auction format, the second-price auction, it is a (weakly) dominant strategy to report your true value. That is, your belief about others' types or strategies does not matter.
Chung and Ely (ReStud 2007) have a paper on this topic:
Robert Wilson criticizes applied game theory's reliance on
common-knowledge assumptions. In reaction to Wilson's critique, the
recent literature of mechanism design has adopted the goal of finding
detail-free mechanisms in order to eliminate this reliance. In
practice this has meant restricting attention to simple mechanisms
such as dominant-strategy mechanisms. However, there has been little
theoretical foundation for this approach. In particular it is not
clear the search for an optimal mechanism that does not rely on
common-knowledge assumption would lead to simpler mechanisms rather
than more complicated ones. This paper tries to fill the void. In the
context of an expected revenue maximizing auctioneer, we investigate
some foundations for using simple, dominant-strategy auctions.
Another popular approach is by Bergemann and Morris (2005) (several papers).
However, I guess this is not so much what you have in mind, but they have to be mentioned here.
The mechanism design literature assumes too much common knowledge of the environment among the players and planner. We relax this assumption by studying mechanism design on richer type spaces.
We ask when ex post implementation is equivalent to interim (or Bayesian) implementation for all possible type spaces. The equivalence holds in the case of separable environments; examples of separable environments arise (1) when the planner is implementing a social choice function (not correspondence) and (2) in a quasilinear environment with no restrictions on transfers. The equivalence fails in general, including in some quasilinear environments with budget balance.
In private value environments, ex post implementation is equivalent to dominant strategies implementation. The private value versions of our results offer new insights into the relationship between dominant strategy implementation and Bayesian implementation.
Usually in mechanism design, there is private information to be extracted and you need to take a stand on how to model it. That could be Bayesian or not, but the classical Nash equilibrium concept is suited for full information settings. The "prior-free" mechanism design literature pushed by computer scientists is concerned with worst-case analysis.
