I am trying to determine if machine learning models provide better predictions of stock market returns when a proxy for investor sentiment is included as factor in the explanatory data.

I am using google search volume index as my proxy for investor sentiment, as such I have a limited time range that can be used only back to the beginning of 2004 when google has available data. As such I want to use panel data across multiple countries, currently the US, UK and Canada to give myself more data points to test my models predictive accuracy. I have not been able to find anything anywhere that explains how to use panel data in a machine learning problem with time series data.

Is it possible to use panel data for time series data in machine learning, and if it is can you point me towards some resources that I can use to assist me?

I am currently using an LSTM, Random Forest and ANN models.

  • $\begingroup$ Have you tried encoding the groups one-hot? $\endgroup$ Apr 24, 2022 at 3:39
  • $\begingroup$ @RegressForward, thanks so much for your answer. I am quite new to this, I have spent some time looking at encoding the groups as one-hot but I am not sure how you thought it might help. $\endgroup$
    – Andrew
    Apr 25, 2022 at 6:40

1 Answer 1


This is a question that crosses all over the place, each of these techniques are different. Here are some very loose guidelines.

As a baseline, recall that in econometrics you may have performed exercises and discovered how the results allow regular regression to recover the estimates for more complex regressions. In particular, I want to remind that fixed-effect regression in the linear regression case is equivalent to encoding each of the variables one hot (ie. adding dummies for each variable). The same result occurs if you demean by group before regressing. So: FE, demeaning, and one-hot are essentially the same. This tends to fall apart if you are using nonlinear functional forms (you cannot just force in dummies to nonlinear models and expect correct recovery of coefficients) - so you are right to be cautious.

In ML I am not aware of many specialized first-differencing or fixed-effect estimation techniques (like in Stata's regHDFE) that takes raw data and performs specific tasks exclusively to deal with the notion of panel data. (I believe LSTM networks do contain some elements that will complicate the process, however, so I will be mute about them.)

Since the scripts you are working with do not preprocess the data, the choices you have are typically to:

  1. Preprocess the data in a way that matches your objectives.
  2. Encode the data one-hot (ie add dummies/fixed effects).
  3. Encode the data as a vector/list.

Point 1) Critically, nothing stops you from preprocessing the data during the normalization step in a way that demeans by individual group (a la fixed-effects) or finds differences between periods (a la first-differencing). You can train the ML technique on this preprocessed data and it will help you interpret the results in a way you're familiar with. Ultimately, regHDFE and similar programs simply does this preprocessing for you. You can and should preprocess your data such that they are normalized - and these are permissible ways to perform that normalization.

Points 2/3) In RF you might find it most common to encode the individuals one-hot and the time is typically passed as a 0..T list. The difference is because of the relation between time and individuals. If you encode the groups as a list of individuals 1...I, they will get clustered together inappropriately, individuals 2 and 3 will often be in the same leaf though, presumably, their labeling as individuals 2 and 3 is typically random. Time can be passed as a list since time periods 2 & 3 actually are related. You are not prohibited from passing time as a large set of one-hot variables. The difference is loosely akin to your choice of including time trends vs individual time-based FE ... though ML techniques are infinitely more flexible in how that "time trend" can be manifested and it seems to lose information by entirely isolating each period. NN's can also parse the individuals as fixed-effects and the time as a sequence (or as more FE).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.