Why do we treat demographic variables as being exogenous?

I I would like to ask about demographic variables. In general, we usually include those variables as independent variables when we run regressions such as wage equation or test score equation. What I wonder is that why we take them as exogenous in general? Intuitively, I understand that those are something given to agents when they need to make a decision, so in that regard it makes sense that we say they are exogenous. However, mathematically E[Xe]=0 is the definition of exogoneity, and so I guess demographic variables can be still correlated with unobserved error terms. For example, if we consider the following equation Y_i = X_i*b_i + e_i where Y_i is test score, X_i are demographic covariates (race, gender, age ...). Then, isn't it possible those variables are correlated with innate abilities or whatever in e_i ? But, I do not find any convincing answers why (nearly) every people take those as exogenous....

You are completely correct that mathematically endogeneity is defined $$\text{cov(X,e)}\neq0$$, which is true not only when it’s correlated with omitted variable but also with measurement error or reverse causation/simultaneity.