I saw youtube videos like how geography doomed Africa and heard how Japan's prosperity is an anomaly due to its isolation and they got me interested.
A good starting point which should be accessible to people at all levels is the book "Guns, Germs, and Steel" by Jared Diamond.
I believe this is one of the first and main seminal works in this area.
That being said, it is not settled science that geography (alone) can doom a country's economy.
As mentioned by @BBKing, "Guns, Germs, and Steel" by Jared Diamond is a great book. Another interesting one is "Why Nations Fails" by Daron Acemoglu and James A. Robinson.
They argue that geography is insufficient in explaining the political and economic success or failure of states. They support their thesis by comparing country case studies.
They identify countries that are similar in many of the above-mentioned factors, but because of different political and institutional choices become more or less prosperous. The most incisive example is Korea, which was divided into North Korea and South Korea in 1953. Both countries' economies have diverged completely, with South Korea becoming one of the richest countries in Asia while North Korea remains among the poorest. Further examples include the border cities Nogales (Sonora, Mexico) and Nogales (Arizona, USA). By referencing border cities, the authors analyze the impact of the institutional environment on the prosperity of people from the same geographical area and same culture. (source Wikipedia)
Regarding @PulseReborn's comment
I was expecting more of textbook recommendations
I can recommend "Modern Economic Growth" by Acemoglu, a graduate textbook. In particular, Chapter 4 has a section called "Geography". He presents some arguments for and against this explanation.