The fundamental reason, beyond any details about shifting interest rates, is that lenders are in the business of lending money. If you pay off the loan early, then they're doing less business. If you pay off part of the loan, you're paying less interest, which means they have to go out and find another source of income to replace you.
Extreme case, suppose you pay off the full loan the very first day, and they don't charge any fees, it's as if they never made a loan at all. They could have stayed in bed.
The thought of getting out of bed for nothing makes them so angry (I exaggerate) that they may well charge fees for early repayments and/or setting up the loan in the first place, to ensure that (aside from being paid for their capital) they still get paid for their time. You can think of early repayment fees as making up for any part of the cost of their time that they rolled into the interest rate, although that's not all there is to it. Addressing your specific points:
The lender gets their principal back ahead of schedule
So if they can find another customer, they can get back to where they started before you made the early payment. That's not an advantage, that's an opportunity to break even.
the amount at risk in the event of a default is reduced [and] the risk of default is reduced
If they wanted to avoid risk then they could not lend money to you in the first place. They made the loan because they think the interest rate in return for the risk is a good deal. You're ending something they thought was a good deal for them.
Lenders do like people who've demonstrated that they can pay off loans, but mainly because it proves that they might be able to take bigger loans in future and generate more income for the lender. What lenders really want, above all else, is the interest[*]. And who they like, most of all, is people who borrow money from them and make the payments. If you pay off early, you're not that person any more.
So, why do they even let you make early payments? Firstly because in many cases the law says so, and secondly because they have to offer customers enough flexibility that they'll actually take the deal.
This is all in general, of course. There might be specific circumstances where a lender does have a liquidity problem and does want to solve it (in part) by taking as many early repayments as they can. But that's akin to a retailer solving a short-term problem by closing stores: if the solution to your problem is "do less business" then things are pretty bad.
Speaking of the market in general rather than retail lending in particular, there are loans that can't be repaid early under the terms that banks might offer for personal loans. For example, a government acting as a borrower can buy back its bonds on the market, but the terms of the bond don't (usually?) allow it to just "pay off the loan early" by requiring the bold holders (lenders) to give them back in return for the bond's face value. They have to pay the market price.
[*] For a while they used to like above all else the ability to trade and use as capital reserve, obscure derivatives of the loans, for prices and ratings that amounted to a wild-ass-guess and therefore in many cases were way too high ;-) That became suddenly less respectable in 2008.