You are correct. Trade deficits are not unfair, although it is hard to define unfair in an economic sense. No one is forced to trade and trade represents a consensual agreement.
The strongest economic argument for considering trade deficits unfair is so-called dumping, which is a form of unfair competition. That is when firms sell goods abroad at below a normal price in order to drive out the local competition in the long run. As the foreign businesses would make losses doing so, it only works if the foreign governments subsidize those firms. That is why subsidies leading to unfair competition are banned according to the World Trade Organization rules. In the case of the steel tariffs imposed by the U.S., dumping by China was used as an argument, although there was no evidence of China conducting steel dumping and China isn't even one of the main exporters of steel to the U.S.
Another reason is that a trade deficit means that foreign firms are doing better than domestic ones, thereby threatening the jobs of domestic workers. However, for this to happen consumers must be voluntarily buying those foreign products. If foreign firms offer a much better and/or cheaper product, then a trade deficit is great for consumers, but bad for domestic workers who produce those goods. In this case, trade still represents an overall net increase in welfare, but there are winners and losers. If the winners (consumers) would compensate the workers who lost their jobs, everything would be fine. If they do not it could be considered unfair, although it is not the foreign country being unfair in this case.
From my experience, there may also be illogical reasons that people consider trade unfair, which I discuss below.
When a country runs a trade deficit that means that it is consuming more than it produces. The country consumes at least as much as it produced and then a bit from what other countries produced too. When that happens, a country is essentially borrowing money from the rest of the world, because consuming more than you have is like spending more money than you have. That is what people mean when they say the U.S. owes a lot of money to China. In this sense many people who take out a loan with a bank to buy something think it is unfair that they have to pay it back. That could be one way in which deficits are considered unfair.
Another way in which deficits are considered unfair is by comparing a country to a business, which is wrong and leads to wrong policy. A business wants to maximize profits, so it would like to sell more goods than it buys. The business fails otherwise. In this sense, a business seeing other business succeeding where it fails might think it is being treated unfairly. As discussed, however, a trade surplus just means that you are lending money to the rest of the world. That may be good for some time, but only if you eventually get it back, i.e. eventually run a deficit. So for a business, perpetually making profits is a good thing. For a country, a perpetual surplus forever is not welfare enhancing.
