Consider the following two models where E(epsilon) = 0 and V ar(epsilon) = 2*sigma*I:
A : y =X1*beta1 + epsilon
B : y =X1*beta1 +X2*beta2 + epsilon
Show that R^2 of A < R^2 of B. Here R2 is defined as the multiple R-squared in the linear regression model. What
does this imply for the usage of multiple R-squared in selecting among models of different dimensions?
SSE of Model B is always larger than or equal to that of Model A(use proof of contradiction), which is: R2 of B >= R2 of A. This means if we add more variables, R2 tend to be higher. So if we use R2 to select model, we tend to choose an overfitting one. We can use Adjusted R2 instead.