Countries do not just stop recognizing foreign wealth because there are two kinds of wealth: real assets, and claims on the production of others; the values of both of these types of wealth are not determined by consensus, but rather by the use or pleasure one can derive from them.
To the first type— real assets are stuff like machines and cars and homes and all sorts of things that you can touch and which we value for one reason or another. The answer to your question as pertains this category is easy: you can "not recognize" my car as wealth all day long, but that won't affect its value to me or anyone else one bit.
To the second type— claims on the production of others come primarily in the form of money, but also in the form of non-money debt and equity. I will skip over non-money debt and equity, as they both provide a claim on a future stream of money, so their value hinges on the value of money. Now, rephrasing your question slightly, we arrive at: why is a foreign country's money valuable? Or, more specifically, why is a foreign country's money valuable in a way that is not subject to one's recognition of it?
The answer to this is simple (and a fundamental concept in international economics): a country's money is valuable to the extent that you want to buy that country's stuff (or buy claims to their production, i.e., debt and equity). This is why countries with economies based around commodity exports see their currencies fluctuate with the value of those commodities (so-called "commodity currencies")— because when demand for those countries' production increases, the values of their currencies increase. So— one can go all day not recognizing the value of another country's money, so long as one doesn't want to buy that country's stuff.
I will confess to not being terribly familiar with the situation you describe in Tanzania, but I would suggest simply that the locals who choose to sell their fish rather than eat them are likely to have a much better understanding of the implications of doing so than you or I. I would suppose that they're selling their fish because they receive something else in exchange (money), which they turn into something of greater value to them than the fish they sold. Let's say that what they receive in return is gasoline, for use in boat motors (which are essential to catching fish), generators that power lights and pumps to provide clean water, and inefficient and dirty cooking devices (that are nonetheless better than the alternative, which would be attempting to scavenge firewood). Are they better off with fewer fish and more gasoline? Yes, otherwise they wouldn't make that trade. Would they be better off if they didn't sell most of the fish and get gasoline in return, or would they be more likely to "build up important things"? No, almost certainly not— otherwise they would not sell their fish. Unless the Tanzanians you describe are being forced to trade, it's a safe bet that they're trading because it is beneficial to them. They're poor, yes— but that doesn't mean they're stupid.