Efficiency can be a very "objective" concept because it is conditional on whatever "non-economic" environment production will take place. Consider the "production possibilities frontier" of a society, that says "given the resources we have, we can produce $100$ of X only, or $70$ of Y only, or some combination of the two, in a year". As long as we are on the production possibility frontier, we are efficient.
Right. How do we measure how much we can produce? It looks like a totally objective "engineering" exercise (however complicated). "We have available $1000$ labor hours and with them we can produce $100$ units of $X$, it is simple as that".
Hmmm... and why do we have $1000$ labor hours available, and not, say, $2000$? How did we determine that? We determined it by considering perhaps some "average" number of hours worked per person times the working population of the society. Ok, and how did this "average" number of labor hours per working person cοme about? In practice it is strongly affected by the institutional environment in the specific economy. But his institutional environment is taken as given in determining whether we are efficient or not.
So this concept of efficiency is conditional on technology and socio-economic institutions (market structure is included here).
But can the socio-economic institutions be assessed as to whether they cause inefficiency themselves? They certainly can, but we just moved one step higher than the previous concept of efficiency. And here things get cloudy, because the socio-economic institutions are never free of ideology, values and the like.
So to say "the structure causes inefficiency" has the possible rebound "given your values which is not my values". In theory we could even quantitatively assess that -but we would have to perform real-world experiments, something like "let's introduce slavery, and see whether in a long term perspective we produce more or not". Ok. Are you willing to play slave in this scenario (that will last many-many years), so that the world will benefit from the objective knowledge we will gain?