(Before applying serial correlation tests to the residuals, you want to visually inspect the residuals for whiteness---look at the sample ACF and PACF. Serial correlation test statistics are often some kind of transformations of the sample ACF.)
...in AR(p) model..The error term must be i.i.d...
That statement is not correct. The i.i.d. condition is not required for ARIMA models, and also more strict than the null hypothesis of a large sample serial correlation test. In large sample, typically you would not expect to be able to distinguish between i.i.d. and covariance-stationary white noise processes.
The text book says Breusch-Godfrey's LM test is designed to test
autoregression in the error term model such as $Y_t = \beta_1 + \beta_2 X_t + u_t...$...I wonder if we can use BG's LM test to test serial
correlation of the error term in AR(p) model.
Yes. The asymptotic distribution under the Breusch-Godfrey null hypothesis obtains under the condition $E[u_t X_t] = 0$. Under this condition and stationarity, the residuals $\hat{u}_t$ approximate $u_t$ in large sample and the F-statistic of the auxiliary regression has the usual $\chi^2$-distribution.
With lagged dependent variable---e.g. $X_t = Y_{t-1}$, and the model is AR(1)
$$
Y_t = \beta_1 + \beta_2 Y_{t-1} + u_t,
$$
under the null that $u_t$ has no serial correlation, then $E[u_t Y_{t-1}] = 0$. The point here is that the time series model is correctly specified under the null.
