I seem to be having trouble in comprehending what it means for a time series to be covariance stationary. Specifically, with the third condition that for any $t,h$ the $cov(x_t,x_{t+h})$ only depends on $h$ and not $t$.
Would anyone have any examples of how a time series might be covariance stationary or any examples of non-covariance stationary time series?
Let $\epsilon_1, \epsilon_2, ... ,\epsilon_t,...$ denote i.i.d. stochastic variables, each having a standard deviation of $\sigma$.
Let $$ x_t = \sum_{i = 1}^t \epsilon_i. $$ Then $$ cov(x_t,x_{t+h}) = cov\left( \sum_{i = 1}^t \epsilon_i, \sum_{i = 1}^{t+h} \epsilon_i \right). $$ Because of the i.i.d. property $$ cov\left( \sum_{i = 1}^t \epsilon_i, \sum_{i = 1}^{t+h} \epsilon_i \right) = \sum_{i = 1}^t cov\left(\epsilon_i,\epsilon_i\right) + \sum_{i = 1}^t\sum_{j \neq i} cov\left(\epsilon_i,\epsilon_j\right) $$ and furthermore because of independence $$ \sum_{i = 1}^t cov\left(\epsilon_i,\epsilon_i\right) + \sum_{i = 1}^t\sum_{j \neq i} cov\left(\epsilon_i,\epsilon_j\right) = \sum_{i = 1}^t \sigma + \sum_{i = 1}^t\sum_{j \neq i} 0 = t \cdot \sigma. $$ Now we have shown that for this time series $$ cov(x_t,x_{t+h}) = t \cdot \sigma $$ depends on $t$ hence it is not covariance stationary.
For a covariance stationary time series, let $$ z_t = \epsilon_{t-1} + \epsilon_{t}. $$ By performing the same analyis, you can conlude that for $t > 1$, $$ cov(z_t,z_{t+h}) = \left\{ \begin{array}{ll} \sigma & \text{ if } h = 1 \\ 0 & \text{ if } h > 1. \end{array} \right. $$ Since this depends on $h$, but not on $t$, the time series $z_t$ will be covariance stationary (after $t = 1$).
@ M Damon: If you're just starting with time series ( actually, even if you're not ) , think it's best to forget about Wold Decomp and ARIMA models and all that jibber jabber. I re-read your question and I think the best way to think of covariance stationary is the following.
You're sitting in the audience and there's a stage that you look at. The stage is such that the beginning of the curtain to the end of the curtain is of length h. You also have the ability to see the h numbers ( picture each number taking up a unit length of space ) that come out ( some stochastic process causes them to arrive on stage ) in the h second interval so that they perfectly fit on stage from the left side of the stage all the way to the right.
So, at time $t = 1, X_1$ arrives on the stage. Then, the $X_2$ arrives on stage at $t = 2$, to the right of $X_1$ ( from your vantage point ). So, after $h$ seconds, you have $X_1, \ldots X_h$ on stage and ordered from left to right.
You are sitting there at time $t=h$ ( picture time stopping for a moment ) and since the observations are from some stochastic process, there is a covariance structure between those h elements on stage. Whatever it is, doesn't matter. Maybe it's $\theta^i$ where $i$ is the lag between two $X_t$ say $X_t$ and $X_{t-i}$.
Next, time starts again and you close your eyes for another $h$ seconds so that a new set of $h$ elements arrive on stage and the old ones leave. Now you're at time $2h$ and you open your eyes. So, it's time $t = 2h$, but the covariance structure between the new $h$ elements on stage does not change from what it was earlier. So, the specific elements themselves don't matter when it comes to the covariance of any two elements. All one needs to know is the distance between any two elements in order to know what the covariance ( or correlation ) of them is. 2h plays no role. All that matters for knowing the covariance of any two elements on stage is their distance. This is the practical meaning of the term "covariance stationary".
Note that stage could have been bigger or smaller. $h$ was just random number picked.
This assumption is made and is useful because ( informally speaking ) it kind of allows one to estimate model parameters in an unbiased way with only one set of time series observations. If this assumption was not made, then it would be kind of like having a sample size of $n=1$ because the stochastic realization is one element. I hope that helps.
