You anticipate the answer when you ask:
This question can be easily answered if there were any way in which
new money can leave the central bank without being paid back. Are
there such transactions I don't know about?
Indeed, there is always a way that money leaves the central bank without being paid back: the central bank does something with its net interest earnings, usually sending them to the central government, which treats them as part of its revenue, and is therefore able to get by on a little less tax revenue than it would otherwise need. For instance, in 2013 the Federal Reserve remitted $78 billion in profits to the US Treasury.
Let's incorporate this observation into an augmented version of your example. Suppose that the central bank keeps the supply of money at \$10, with a corresponding loan of \$10 to banks, and the annual nominal interest rate is 10%. Suppose that the central government spends \$5 per year and runs a balanced budget.
Each year, banks pay \$1 in interest to the central bank. This is profit and is sent to the central government, which then only needs to raise \$4 in taxes to pay for its \$5 in spending. The net effect of this is to put \$1 in the hands of the public, which ultimately finds its way to paying for the \$1 in interest that banks owed to the central bank. (After all, these banks are presumably lending out the money and collecting interest from the public themselves.)
One can make the example much more intricate, but the key point is that we don't get an exponentially growing debt owed to the central bank by the rest of the economy - because the central bank sends its profits to the government, and then they're recycled into the rest of the economy.
(As user4385 points out in a comment, the situation is similar for any kind of debt. Suppose, for instance, that Connecticut is a net creditor to the rest of the country - where does the rest of the country get the money to pay interest to Connecticut? The answer is that Connecticut eventually spends its interest earnings on goods and services from the rest of the country - funds flow in both directions.)