The Linear algebra part should not worry you, it is just basic linear algebra plus familiarizing yourself a bit with differentiation of vectors and matrices (and for some chapters, visualizing the Kronecker product). Hayashi goes some length into writing out explicitly large matrices, which is virtually with no-precedent (see eg. pages 267, or 288), something that eventually facilitates understanding -as long as you don't back out from a matrix half a page large, but you really look at it.
What you should really go slow over upon is Hayashi's notation, which is non-standard in many places. So don't expect to understand through familiar notation. Be prepared for that and "read" carefully every symbol and its role. Many people's discomfort with this book is due to the notation, but they don't realize it.
As for the Statistics part: In Econometrics, since we deal almost always with non-experimental, observational data, a large part of Statistics (originating usually in Biostatistics) do not really enter (our) picture. For example, apart from proving that under certain conditions the OLS estimator is best linear unbiased, you won't see much regarding the issues of minimum variance estimation or sufficient statistics. We don't even discuss the Exponential Family of distributions and the concept of Generalized Linear Models. Also, in statistical Hypothesis testing the Fisherian spirit prevails (oozing through the Neyman-Pearson apparatus), meaning, among other things, that issues like the "power of the test" are not so often considered.
Moreover "frequentist Econometrics", which is what Hayashi presents, are considerably lighter than Bayesian Econometrics in terms of the Algebra of Random Variables and probability theory.
What really matters is a good familiarity with basic Asymptotic Theory and Stochastic Processes (for the Time Series part). Although Hayashi in most cases just states results, providing references for the proofs, nevertheless it would be good to feel comfortable with asymptotic results and properties, because in most cases, it will be all your estimators will possess. For this part, I will once more advocate A. Spanos' book, Probability Theory and Statistical Inference - Econometric Modeling with Observational Data. It is a valuable book, Hayashi or not Hayashi. It is Statistics with the Econometrics graduate student in mind.
In case you want some more specific Time Series material, Hamilton's Time Series Analysis remains a standard reference.
Finally, regarding Unit-root Econometrics and Co-Integration, the subject is inherently advanced, and I think Hayashi has done a good job in collecting and presenting the very basics.