Question: I want to clarify my understanding of the basics of OLS regression in matrix form.

Let's assume we have 2 different independent variables $x_1$ and $x_2$.

Our 'model' will be the plane that lives in $\mathbb{R^3}$ that minimises the sum of squared distances between each point on the plane corresponding to observations of our pair of independent variable points $x_{1i}$ & $x_{2i}$ and the corresponding point $y_i$. These individual distance are our estimated residuals call them $\bar μ_i$

Vector form: In vector form we instead have single vectors in $n$ dimensional space, whereby our number of observations defines $n$. $y$ and each $x_i$ variable form $n$ dimensional vectors in this space.

The span of these $x_i$ vectors will form a hyperplane $X$ in this $n$ dimensional space. Now our vector $\hat y$ which represents our model. This is the orthogonal projection of $y$ onto the plane $X$, and the coefficients that define the linear combination of $x_i$ vectors that give $\hat y$ are the weights of our $β_ι$ parameters in our model.

What's bothering me is that in this lecture series the author states that.

In OLS regression we are trying to get as close to this dependent variable vector $y$ as we can, given that we don't have a vector or a space $X$ which is as highly dimensional [as $y$].

What's bothering me here is that, each of the vectors $x_i$ are as "highly dimensional" as $y$ because they live in the same $n$ dimensional space, but i appreciate that $X$ does not span all of $\mathbb{R^n}$ and so we can not get y with just a linear combination of our different $x_i$. Is this interpretation correct?

Question: If we had the "true model" then this distance between the true plane call it $\bar X$ and $y$ would be our error term vector $u$ and i assume that $u$ would itself be the linear combination or many other unaccounted for independent variables?

Question: When i try and visualise this, naturally all i come up with is a plane in $\mathbb{R^3}$ and then $y$ is outside of this plane. This visual analogy implies that only one $x$ variable is missing i.e. we just need a 3rd linearly independent vector. But surely for any dimension of space we chose. Our $y$ is just one vector away from $X$ as we can always point wise define an extra vector $y - lin(X)$ and indeed if we take $\hat y$ to be our vector $\in X$ then $y - \hat y$ is just $\bar u$ our residual vector.

So i'm unclear how to think about this missing extra dimensionality the author mentions. I suppose in reality this $\bar u$ could be decomposed into many more LI vectors of uncounted for independent variables. But I would appreciate clarification.

  • $\begingroup$ How does differ from your other question that has already attracted some useful comments? $\endgroup$
    – Dave
    Commented Jun 19, 2023 at 23:03
  • $\begingroup$ Hi Dave thanks for commenting. Yes they are almost identical. The reason for the duel post is that. 1) I first posted here as it's econometrics, but then thought maybe it's more appropriate for statistics. 2) I have posted in more than one stack forum before and got different answers depending on the community. Economists vs statasticians vs mathmeticans sometimes provide different answers focusses etc. So it's quite helpful getting both. i appreciate this is maybe problematic for the "knowledge base" gaol of stack, as duplicates are undesirable, but from a user perspective it is helpful! $\endgroup$
    – CormJack
    Commented Jun 21, 2023 at 8:34
  • $\begingroup$ @Dave If you have any time i'd really appreciate your thoughts on this question. Similar topic, but different focus. economics.stackexchange.com/questions/55700/… $\endgroup$
    – CormJack
    Commented Jun 21, 2023 at 8:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.