5
$\begingroup$

I'm currently writing my master's thesis where I'm applying a multinomial logit regression with interactions effect.

The model I'm using is $$ P(y=j) = exp(xb_j)/(1+∑exp(xb_h)) $$ where $$ xb_j =x_1b_{ij} + x_2b_{2j} + x_1x_2b_{3j} + XB + \epsilon $$ $x_1$ and $x_2$ are dummy variables ($x_1$ is coded 0/1 for man/woman).

I use marginal (or incremental) effects to report my results, since I'm interested in the effect of a unit change on the probability of af given outcome of my y-variable.

My question is regarding my results, where I get an insignificant marginal effect of the interaction term, which I guess indicates that there is no difference between how men and women are affected by a unit change in $x_2$. But when I look separately at men and women, I get that the marginal effect of $x_2$ is significant for men ($\frac { \partial P(y=j) } {\partial x_2 }$ when $x_1=0$) and insignificant for women ($\frac { \partial P(y=j) } {\partial x_2 }$ when $x_1=1$). (I use three different significance levels (0.1, 0.05 and 0.01))

How do I interpret an insignificant effect from the interaction, but significant in a subgroup (men)?

Improve this question
$\endgroup$
3
  • $\begingroup$ To be clear: you are saying that you interact gender with some other thing. Let us, for clarity, assume this other thing is effort and that your dependent variable is wage. In these terms, you are saying that the effort itself is significant in wages, but the interaction of women and effort is insignificant while the interaction of men and effort is significant? $\endgroup$
    – 123
    Commented Feb 28, 2018 at 14:44
  • $\begingroup$ For simplicity, I guess you could say that. Although I have only interacted the variable with woman and I derive the average marginal effect since it's a non-linear model. When I derive the average effect of a change in $x_2$ it is significant (both women and men). When I derive the average marginale effect of a change in $x_2$ for women ($x_1=1$) it's insignificant and when I derive the average marginal effect of men ($x_1=0$) it's significant. Then when I derive the interaction effect, which is the difference between men and women it's insignificant. $\endgroup$
    – user2012
    Commented Mar 1, 2018 at 9:08
  • $\begingroup$ The interaction effect is derived as (in incremental terms): $$ \frac{\Delta P(y=j)}{\Delta x_{1}x_{2}}=\left[\varLambda\left(b_{1}+b_{2}+b_{3}+XB+\epsilon\right)-\varLambda\left(b_{2}+XB+\epsilon\right)\right]-\left[\varLambda\left(b_{1}+XB+\epsilon\right)-\varLambda\left(XB+\epsilon\right)\right]$$ where $$ \varLambda\left(x^{'}\beta\right)=P(y=j)=exp(xb_{j})/(1+\sum exp(xb_{h})) $$ The above is from the article »Testing hypotheses about interaction terms in nonlinear models« from Greene(2010) $\endgroup$
    – user2012
    Commented Mar 1, 2018 at 9:17

1 Answer 1

Reset to default
1
$\begingroup$

It is not 100% clear what procedure you used to do the subgroup tests, so I will cover a few bases. Below, I suggest exploratory tests to better understand the results. The logic in this response is the same as you would follow in a linear model.

If you are taking the results from the single model and doing post hoc tests of the coefficents, be sure that you are interpreting the model correctly. Because men are the implicit reference category, the constant represents the men's average (ceteris paribus) and the constant plus b1 is the women's average. Similarly, b2 is men's "slope" with respect to x2, while b2 + b3 is women's "slope" with respect to x2. So, to test whether x2 matters at all for women, you would have to test b2 AND b3 together.

That said, it sounds to me like, after running the full model, you ran the model separately for men and women (minus the sex-related terms) to investigate further. Here's a few things to investigate:

  1. How different are the other parameters for men and women in the separate models from the main model?

    • Are there similar point estimates?
    • Are the same variables significant?
    • Do the models have comparable model fit?

  2. What kind of variation do you have in your data? You can do descriptive and exploratory analyses to understand this.

    • Present a descriptive table of the frequencies by sex, x2, and j. If every "cell" created in this way had exactly the same count, you would be rather unlikely to get a disparity between the subgroup models and the full model (unless the data generating processes for men and women are entirely different, and/or there is heteroskedasticity by sex).
      • That is, my hunch is that you have "micronumerosity" for some of the women in certain outcomes, and thus you cannot get a small enough standard error to distinguish the interaction term.
      • Another way of looking at this is figuring out the realm of common support for your model.
    • Run descriptive statistics (mean, SD) on the other variables separately for men and women and look for discrepancies. If there are substantial differences, it could mean something other than sex is driving the difference, and/or that collinearity between that other variable and sex is complicating the estimation.
      • If the covariates are normally distributed, you could run a series of t-tests in addition to eyeballing the means. (If you wanted to be more formal about this and take into account multiple hypothesis testing, you could do various kinds of balance checks, essentially using the x variables to predict sex.)
    • Graphs of the area of common support or distributions of other variables conditional on sex can help show what you find.

To write up the results, I'd suggest presenting a table with the coefficients for the full model without the interaction, the full model with the interaction (showing the interaction is not significant), and the models by gender.

For the discussion section, you can mention that you looked into the role of the interaction term. You can show your results about micronumerosity and covariate balance by sex (whether or not they seem to explain the problem). All of these are exploratory analyses, and so they can't stand on their own as definitive explanations. But they help you address what should be done in future analyses (e.g. that a larger N overall is needed or that oversampling people in small cells would help, or finding other ways to increase the precision of the model).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much for your answer. I only run one model with an interaction variable. Since I'm interested in the marginal effects, which are not the coefficients in an multinomial logit model (as it is in an OLS), I calculate marginal effects which are dependent on all other variables. I then perform t-tests on the marginal effects to see whether they are significant or not. I report the marginal effects (not the coefficients) with their standard errors. $\endgroup$
    – user2012
    Commented Mar 5, 2018 at 8:57
  • $\begingroup$ The marginal effect of a change in x2 for men is: $ \left.\frac{\Delta P(y=j)}{\Delta x_{2}}\right|_{x_{1=0}}=\varLambda\left(\beta_{2}+XB\right)-\varLambda\left(XB\right) $ The marginal effect of a change in x2 for women is: $ \left.\frac{\Delta P(y=j)}{\Delta x_{2}}\right|_{x_{1=1}}=\varLambda\left(\beta_{1}+\beta_{2}+\beta_{3}+XB\right)-\varLambda\left(\beta_{1}+XB\right) $ $\endgroup$
    – user2012
    Commented Mar 5, 2018 at 8:57
  • $\begingroup$ Regarding answer 2) There is around 400 men and 400 women, and I did a few Pearsons chi^2 test to see whether the variables where independent from sex. Regarding x2, More women than men had x2=1, whereas more men had x2=0. But regarding "micronumerosity" is that when I have a variable x3, where only four women have x3=1? $\endgroup$
    – user2012
    Commented Mar 5, 2018 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.