As well as the other answer, I would add two further points.
Firstly, smaller engines with lighter components tend to entail faster moving parts internally, or higher fuelling loads which exert larger forces over smaller component areas, and so must generally work "harder". This not only impairs the long-term reliability and durability of the engine, but in some cases, requires the engine to be driven much faster and in lower gears in order to achieve the desired acceleration or power output.
The other point is that the necessary power output is determined not just by engine design or driver choice, but by driving norms and the standards of road engineering, and therefore an inadequately-sized engine may spend most of its time being driven at high revs and low gears, well outside optimal parameters even for fuel consumption, let alone optimal parameters for reliability.
I noticed this once when given an underpowered hire car. It was so underpowered that it routinely had to be driven to the redline, with the accelerator constantly to the floor. The effect was that fuel consumption soared beyond normal levels, for a responsiveness that was ultimately still inadequate.
Other drivers will also retribute attempts by drivers to penny-pinch on fuel by driving small-engined cars inconsiderately and unresponsively, by driving aggressively, by overtaking, and by otherwise causing the mental (or accident risk) burdens on the tardy driver to increase.