It could be a sustainable economic policy. It depends.
Assuming your government doesn't have a big budget surplus, it will have to replace revenue lost from tariffs -- i.e. raise other taxes. Most taxes are distortionary, i.e. discourage productive economic behaviour. Especially for less developed countries and countries with weaker international transport links (including islands), tariffs tend to be cheaper and easier to collect than a lot of other taxes. They also have an implicit progressive bias towards taxing luxuries, and can be given a more explicit one quite easily. So the main and most obvious reason not to abolish tariffs is the difficulty of finding something better to replace them with.
As you say, there are benefits too: the inherent benefits of greater trade volumes, and the competitive benefit of cheaper imported inputs.
When you ask your final question, this is really about the game theory of tariff reductions. I'd suggest modelling the situation like this:
Your country gains a small amount from abolishing its own tariffs (this is arguable but assume it for now). It gains a lot from your next door neighbour abolishing theirs, because that would help your export industries (this is not controversial). You don't know whether your neighbour would gain or lose from abolishing its own tariffs, but you know they would benefit from you ditching yours.
What would you do in this situation? Your main objective is to get your neighbour to get rid of their tariffs, and you know that their main objective is to persuade you to get rid of yours.
You could unilaterally abolish your tariffs, but there goes all your bargaining power. You have no idea whether your neighbour will want to reciprocate.
A better strategy is to offer to abolish your tariffs in exchange for your neighbour doing the same. That's the history of bilateral trade agreements. In a world with lots of countries, you might find this ends up being a lot of work, and wish there was some way of simplifying things. That's GATT and the WTO.