I want to provide a different perspective to those already presented, which is that of culture. As you say, gold has almost no inherent value (at least it had little in the past, before the invention of electronics). However, cultural norms and customs give value to gold.
An interesting example comes from 500's year old book Utopia, by Thomas More, which describes an "ideal" (some might say totalitarian) society, without scarcity, poverty, conflict, et cetera. This society disregards gold (and silver, diamonds, etc), partly by associating gold with slavery.
The context of the quotes below is that of a traveler who spent some time in Utopia, and describes such society to the reader. The quotes in extenso (emphasis mine):
It is certain that all things appear incredible to us in proportion as they differ from known customs; but one who can judge aright will not wonder to find that, since their constitution differs so much from ours, their value of gold and silver should be measured by a very different standard; for since they have no use for money among themselves, but keep it as a provision against events which seldom happen, and between which there are generally long intervening intervals, they value it no farther than it deserves—that is, in proportion to its use. So that it is plain they must prefer iron either to gold or silver, for men can no more live without iron than without fire or water; but Nature has marked out no use for the other metals so essential as not easily to be dispensed with. The folly of men has enhanced the value of gold and silver because of their scarcity; whereas, on the contrary, it is their opinion that Nature, as an indulgent parent, has freely given us all the best things in great abundance, such as water and earth, but has laid up and hid from us the things that are vain and useless.
“If these metals were laid up in any tower in the kingdom it would raise a jealousy of the Prince and Senate, and give birth to that foolish mistrust into which the people are apt to fall — a jealousy of their intending to sacrifice the interest of the public to their own private advantage. If they should work it into vessels, or any sort of plate, they fear that the people might grow too fond of it, and so be unwilling to let the plate be run down, if a war made it necessary, to employ it in paying their soldiers. To prevent all these inconveniences they have fallen upon an expedient which, as it agrees with their other policy, so is it very different from ours [European kingdoms], and will scarce gain belief among us who value gold so much, and lay it up so carefully. They eat and drink out of vessels of earth or glass, which make an agreeable appearance, though formed of brittle materials; while they make their chamber-pots and close-stools of gold and silver, and that not only in their public halls but in their private houses. Of the same metals they likewise make chains and fetters for their slaves, to some of which, as a badge of infamy, they hang an earring of gold, and make others wear a chain or a coronet of the same metal; and thus they take care by all possible means to render gold and silver of no esteem; and from hence it is that while other nations part with their gold and silver as unwillingly as if one tore out their bowels, those of Utopia would look on their giving in all they possess of those metals (when there were any use for them) but as the parting with a trifle, or as we would esteem the loss of a penny! They find pearls on their coasts, and diamonds and carbuncles on their rocks; they do not look after them, but, if they find them by chance, they polish them, and with them they adorn their children, who are delighted with them, and glory in them during their childhood; but when they grow to years, and see that none but children use such baubles, they of their own accord, without being bid by their parents, lay them aside, and would be as much ashamed to use them afterwards as children among us, when they come to years, are of their puppets and other toys.
What this utopian example presents is an alternative cultural setting, one where gold is associated with deplorable things. In this setting gold would not act as a safe haven. Naturally, More is simplifying reality. For example, since other nations outside utopia value gold, holding gold reserves would actually act as accumulating a mean of payment, for example for imports. Still, I think this is a thought-provoking example of how the value society puts on gold might ultimately just be a cultural phenomenon.