A good answer to my question is contained in Alexander Tabarrok (2008). The hot-toy problem:
A firm wants to build capacity to serve the expected flow demand rather than the unexpected peak demand. Thus, short supply is not difficult to explain. What is difficult to explain is why manufacturers do not exploit the ex post high demand by raising the price of the hot toy,thereby eliminating the shortage
The answer to the problem is later postulated as:
The loss in profit is smaller the larger the shortage because a large shortage is a sign, ceteris paribus, of an elastic demand curve. When the demand curve is elastic a small increase in price is enough to decrease the quantity demanded.
Additionally Tabarrok postulates that for the specific example of toys it could be related to return policies, though this wouldn't apply to things like concert or Disneyland tickets:
Retailers often extensively advertise their return policies such as “If you are not satisfied, bring it back for a full refund.” If the hot toy increases in price whenever there is a seasonal shortage, however, retailers open themselves upto be gamed by consumers. Consumers who purchase at the high price will be motivated to return the product only to buy it again at the lower post-shortage price.
This subject also comes up in Becker, Gary S. (1991) “A Note on Restaurant Pricing and Other Examples of Social Influences on Price.”:
Price increases will be discouraged if consumers believe that they are
unfair (see Kahneman, Knetsch, and Thaler 1986). This may sometimes
help explain why prices do not rise to take advantage of temporary
shortfalls in supply, but it is not plausible when rationing is more
permanent. A series of gradual price increases could eliminate the gap
in figure 1 without causing serious complaints about unfair pricing.
In this note I provide a different explanation, which assumes that a
consumer's demand for some goods depends on the demands by other
consumers. The motivation for this approach is the recognition that
restaurant eating, watching a game or play, attending a concert, or
talking about books are all social activities in which people consume
a product or service together and partly in public
Suppose that the pleasure from a good is greater when many people want to consume it, perhaps because a person does not wish to be out of step with what is popular or because confidence in the quality of the food, writing, or performance is greater when a restau- rant, book, or theater is more popular. This attitude is consistent with Groucho Marx's principle that he would not join any club that would accept him
And concludes with the following anecdote:
This may explain why customers who have trouble getting into the popular Palo Alto restaurant mentioned at the beginning of this note do not switch to the nearby unpopular one. When I suggest doing this to my wife, she usually answers that she prefers the amenities at the popular restaurant. But the main difference in "amenities" is that one restaurant is crowded and has queues, while the other one is partially empty and provides immediate seating!
So to answer my specific questions:
Mass events such as Burning Man, the Olympics, and the World Cup sell out tickets very quickly which forces fans to resort to scalping or using bots if they want to secure a spot
If going to Burning Man became a matter of shelling out $5k for a ticket, it would quickly stop being "cool" and the appeal would go away.
Theme parks like Disneyland are constantly overcrowded and require patrons to stand in lines for hours, rather than increasing ticket prices to reduce wait times to a minimum
Queues in Disneyland are a feature, not a bug. If tickets started costing $500/person, the park would start looking half empty in photos and it wouldn't look "cool" in photos anymore, even though the actual rides experience would be far better. I've also been told by parents that waiting in queues is a huge chunk of the experience for kids.
I'm assuming Disneyland could introduce VIP tickets of some sort that would let one skip all the queues (as done by Six Flags) but this is probably not done to avoid the backlash.
Concerts of popular artists sell out very quickly which creates a massive scalping market
People want to go see these artists precisely because the tickets sell out very quickly, its just part of the experience. Sharing the fact that you snagged a front row seat on a Paul McCartney concert could be just as valuable as actually going there.
Limited editions of clothing suffer from automated bots buying up and reselling almost all of their stock in a matter of seconds
The bots are a "feature", not an actual problem. People complain about them but at the end of the day they still want to buy these items, ensuring that limited editions always sell out and help advertise non-limited items from their store.
Popular restaurants sometimes require making a reservation several days in advance or don't take reservations and have everyone wait in line for a long time
Discussed in detail by Gary Becker in his paper linked above.
Daycares in Canada and the US frequently have waiting lists of several years which forces parents to sign up for one as soon as they conceive
The waiting list symbolizes a high quality of care and ensures a constant supply of customers. Not having one would cause suspicion on behalf of the parents and potential gaps in the number of clients as kids quickly grow up.