{"id":964,"date":"2016-04-21T22:43:35","date_gmt":"2016-04-21T22:43:35","guid":{"rendered":"https:\/\/courses.lumenlearning.com\/introstats1xmaster\/?post_type=chapter&p=964"},"modified":"2017-07-20T19:48:57","modified_gmt":"2017-07-20T19:48:57","slug":"model-selection","status":"web-only","type":"chapter","link":"https:\/\/courses.lumenlearning.com\/suny-suffolk-introstats1\/chapter\/model-selection\/","title":{"raw":"Model Selection","rendered":"Model Selection"},"content":{"raw":"
\r\n\r\nThe best model is not always the most complicated. Sometimes including variables that\u00a0are not evidently important can actually reduce the accuracy of predictions. In this section we discuss model selection strategies, which will help us eliminate variables from the model\u00a0that are found to be less important.\r\n\r\nIn practice, the model that includes all available explanatory variables is often referred\u00a0to as the full model<\/strong>. The full model may not be the best model, and if it isn't, we want\u00a0to identify a smaller model that is preferable.\r\n

Identifying variables in the model that may not be helpful<\/h2>\r\nAdjusted R<\/em>2<\/sup><\/em>\u00a0describes the strength of a model fit, and it is a useful tool for evaluating\u00a0which predictors are adding value to the model, where adding value<\/em> means they are (likely)\u00a0improving the accuracy in predicting future outcomes.\r\n\r\nLet's consider two models, which are shown in Tables 1\u00a0and 2. The first table\u00a0summarizes the full model since it includes all predictors, while the second does not include\u00a0the duration variable.\r\n\r\n<\/div>\r\ndf<\/em> = 136\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 1.\u00a0The fit for the full regression model, including the adjusted R<\/em>2<\/sup><\/em>.<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
<\/td>\r\nEstimate<\/td>\r\nStd. Error<\/td>\r\nt value<\/td>\r\nPr( ><\/em>|t|)<\/td>\r\n<\/tr>\r\n
(Intercept)<\/td>\r\n36.2110<\/td>\r\n1.5140<\/td>\r\n23.92<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
cond_new<\/td>\r\n5.1306<\/td>\r\n1.0511<\/td>\r\n4.88<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
stock_photo<\/td>\r\n1.0803<\/td>\r\n1.0568<\/td>\r\n1.02<\/td>\r\n0.3085<\/td>\r\n<\/tr>\r\n
duration<\/td>\r\n\u20130.0268<\/td>\r\n0.1904<\/td>\r\n\u20130.14<\/td>\r\n0.8882<\/td>\r\n<\/tr>\r\n
wheels<\/td>\r\n7.2852<\/td>\r\n0.5547<\/td>\r\n13.13<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
R<\/em>2<\/sup>adj<\/em><\/sub><\/em><\/em>\u00a0= 0.<\/em>7108<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n
<\/div>\r\n
\u00a0\u00a0\u00a0\u00a0\u00a0df<\/em> = 137\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 2. The fit for the regression model for predictors cond_new,\u00a0stock_photo, and wheels.<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
<\/td>\r\nEstimate<\/td>\r\nStd. Error<\/td>\r\nt value<\/td>\r\nPr( ><\/em>|t|)<\/td>\r\n<\/tr>\r\n
(Intercept)<\/td>\r\n36.0483<\/td>\r\n0.9745<\/td>\r\n36.99<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
cond_new<\/td>\r\n5.1763<\/td>\r\n0.9961<\/td>\r\n5.20<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
stock_photo<\/td>\r\n1.1177<\/td>\r\n1.0192<\/td>\r\n1.10<\/td>\r\n0.2747<\/td>\r\n<\/tr>\r\n
wheels<\/td>\r\n7.2984<\/td>\r\n0.5448<\/td>\r\n13.40<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
R<\/em>2<\/sup>adj<\/em><\/sub><\/em><\/em>\u00a0= 0.<\/em>7128<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n \r\n
\r\n

Example<\/h3>\r\nWhich of the two models is better?\r\n\r\nSolution:\r\n\r\nWe compare the adjusted R2<\/sup><\/em>\u00a0of each model to determine which to choose. Since the\u00a0first model has an R2<\/sup>adj<\/sub><\/em>\u00a0smaller than the R2<\/sup>adj<\/sub><\/em>\u00a0of the second model, we prefer the\u00a0second model to the first.\r\n\r\nWill the model without duration be better than the model with duration? We cannot\u00a0know for sure, but based on the adjusted R2<\/sup><\/em>, this is our best assessment.\r\n\r\n<\/div>\r\n

Two model selection strategies<\/h2>\r\nTwo common strategies for adding or removing variables in a multiple regression model are\u00a0called backward elimination<\/em> and forward selection<\/em>. These techniques are often referred to as stepwise<\/strong> model selection strategies, because they add or delete one variable at a time\u00a0as they \"step\" through the candidate predictors.\r\n\r\nBackward elimination<\/strong> starts with the model that includes all potential predictor\u00a0variables. Variables are eliminated one-at-a-time from the model until we cannot improve the adjusted R<\/em>2<\/sup><\/em>. The strategy within each elimination step is to eliminate the variable that\u00a0leads to the largest improvement in adjusted R<\/em>2<\/sup><\/em>.\r\n

<\/p>\r\n\r\n

\r\n

Example<\/h3>\r\nResults corresponding to the full model<\/em> for the mario kart data are shown in Table 8.6. How should we proceed under the backward elimination\u00a0strategy?\r\n\r\nSolution:\r\n\r\nOur baseline adjusted R2<\/sup><\/em>\u00a0from the full model is\u00a0R<\/em>2<\/sup>adj<\/em><\/sub> = 0.<\/em>7108, and we need to\u00a0determine whether dropping a predictor will improve the adjusted R2<\/sup><\/em>. To check, we\r\n\r\nfit four models that each drop a different predictor, and we record the adjusted R2<\/sup><\/em>\r\n\r\nfrom each:\r\n\r\n[latex]\\begin{array}\\text{Exclude . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.6626\\hfill&R^2_{adj}=0.7107\\hfill&R^2_{adj}=0.7128\\hfill&R^2_{adj}=0.3487\\end{array}[\/latex]\r\n\r\nThe third model without duration has the highest adjusted R2<\/sup><\/em>\u00a0of 0.7128, so we\u00a0compare it to the adjusted R2<\/sup><\/em>\u00a0for the full model. Because eliminating duration\u00a0leads to a model with a higher adjusted R2<\/sup><\/em>, we drop duration from the model.\u00a0Since we eliminated a predictor from the model in the first step, we see whether\u00a0we should eliminate any additional predictors. Our baseline adjusted R2<\/sup><\/em>\u00a0is now\u00a0R2<\/sup>adj<\/sub><\/em>\u00a0= 0.<\/em>7128. We now fit three new models, which consider eliminating each of the\u00a0three remaining predictors:\r\n\r\n[latex]\\begin{array}\\text{Exclude duration and . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.6587\\hfill&R^2_{adj}=0.7124\\hfill&R^2_{adj}=0.3414\\end{array}[\/latex]\r\n\r\nNone of these models lead to an improvement in adjusted R2<\/sup><\/em>, so we do not eliminate\u00a0any of the remaining predictors. That is, after backward elimination, we are left with the model that keeps cond new, stock photos, and wheels, which we can summarize\u00a0using the coefficients from Table 2:\r\n\r\n[latex]\\displaystyle\\hat{y}=b_0+b_1x_1+b_2x_2+b_4x_4[\/latex]\r\n\r\n[latex]\\displaystyle\\widehat{\\text{price}}=36.05+5.18\\times\\text{cond_new}+1.12\\times\\text{stock_photo}+7.30\\times\\text{wheels}[\/latex]\r\n\r\n<\/div>\r\nThe forward selection<\/strong> strategy is the reverse of the backward elimination technique.\u00a0Instead of eliminating variables one-at-a-time, we add variables one-at-a-time until we\u00a0cannot find any variables that improve the model (as measured by adjusted R<\/em>2<\/sup><\/em>).\r\n\r\n \r\n
\r\n

Example<\/h3>\r\nConstruct a model for the mario kart data set using the forward\u00a0selection strategy.\r\n\r\nSolution:\r\n\r\nWe start with the model that includes no variables. Then we fit each of the possible\u00a0models with just one variable. That is, we fit the model including just cond_new, then the model including just stock photo, then a model with just duration, and a\u00a0model with just wheels. Each of the four models provides an adjusted R2<\/sup><\/em>\u00a0value:\r\n\r\n[latex]\\begin{array}\\text{Add . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.3459\\hfill&R^2_{adj}=0.0332\\hfill&R^2_{adj}=0.1338\\hfill&R^2_{adj}=0.6390\\end{array}[\/latex]\r\n\r\nIn this first step, we compare the adjusted R2<\/sup><\/em>\u00a0against a baseline model that has\u00a0no predictors. The no-predictors model always has\u00a0R<\/em>2<\/sup>adj<\/em><\/sub>\u00a0= 0. The model with one\u00a0predictor that has the largest adjusted R2<\/sup><\/em>\u00a0is the model with the wheels predictor,\u00a0and because this adjusted R2<\/sup><\/em>\u00a0is larger than the adjusted R2<\/sup><\/em>\u00a0from the model with no\u00a0predictors (R<\/em>2<\/sup>adj<\/em><\/sub>\u00a0= 0), we will add this variable to our model.\r\n\r\nWe repeat the process again, this time considering 2-predictor models where one of\u00a0the predictors is wheels and with a new baseline of R2<\/sup>adj<\/sub><\/em>\u00a0= 0.<\/em>6390:\r\n\r\n[latex]\\begin{array}\\text{Add wheels and . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\\\\\text{ }\\hfill&R^2_{adj}=0.7124\\hfill&R^2_{adj}=0.6587\\hfill&R^2_{adj}=0.6528\\end{array}[\/latex]\r\n\r\nThe best predictor in this stage, cond new, has a higher adjusted R2<\/sup><\/em>\u00a0(0.7124) than\u00a0the baseline (0.6390), so we also add cond_new to the model.\r\n\r\nSince we have again added a variable to the model, we continue and see whether it\u00a0would be beneficial to add a third variable:\r\n\r\n[latex]\\begin{array}\\text{Add wheels, cond_new, and . . .}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\\\\\text{ }\\hfill&R^2_{adj}=0.7128\\hfill&R^2_{adj}=0.7107\\end{array}[\/latex]\r\n\r\nThe model adding stock photo improved adjusted R2<\/sup><\/em>\u00a0(0.7124 to 0.7128), so we add\u00a0stock_photo to the model.\r\n\r\nBecause we have again added a predictor, we check whether adding the last variable,\u00a0duration, will improve adjusted R2<\/sup><\/em>. We compare the adjusted R2<\/sup><\/em>\u00a0for the model with duration and the other three predictors (0.7108) to the model that only considers wheels, cond_new, and stock photo (0.7128). Adding duration does not improve the adjusted R2<\/sup><\/em>, so we do not add it to the model, and we have arrived at the same\u00a0model that we identified from backward elimination.\r\n\r\n<\/div>\r\n
\r\n

Model Selection Strategies<\/h3>\r\nBackward elimination begins with the largest model and eliminates variables one-by-one until we are satisfied that all remaining variables are important to the model. Forward selection starts with no variables included in the model, then it adds in variables according to their importance until no other important variables\u00a0are found.\r\n\r\n<\/div>\r\nThere is no guarantee that backward elimination and forward selection will arrive at\u00a0the same final model. If both techniques are tried and they arrive at different models, we\u00a0choose the model with the larger R<\/em>2<\/sup>adj<\/em><\/sub><\/em>; other tie-break options exist but are beyond the\u00a0scope of this book.\r\n

The p<\/em>-Value Approach, an Alternative to Adjusted R<\/em>2<\/sup><\/h2>\r\nThe p<\/em>-value may be used as an alternative to adjusted R<\/em>2<\/sup>\u00a0for model selection.\r\n\r\nIn backward elimination, we would identify the predictor corresponding to the largest\u00a0p<\/em>-value. If the p<\/em>-value is above the significance level, usually \u03b1<\/em> = 0.<\/em>05, then we would drop that variable, refit the model, and repeat the process. If the largest p<\/em>-value is less than \u03b1<\/em> = 0.<\/em>05, then we would not eliminate any predictors and the current model would be our\u00a0best-fitting model.\r\n\r\nIn forward selection with p<\/em>-values, we reverse the process. We begin with a model\u00a0that has no predictors, then we fit a model for each possible predictor, identifying the model where the corresponding predictor's p<\/em>-value is smallest. If that p<\/em>-value is smaller than\u00a0\u03b1<\/em>\u00a0= 0.<\/em>05, we add it to the model and repeat the process, considering whether to add more variables one-at-a-time. When none of the remaining predictors can be added to the model and have a p<\/em>-value less than 0.05, then we stop adding variables and the current\u00a0model would be our best-fitting model.\r\n\r\n \r\n
\r\n

Try It<\/h3>\r\nExamine Table 2, which considers the model including the cond_new, stock_photo, and wheels predictors. If we were using the p<\/em>-value approach with backward elimination and we were considering this model, which of these three variables would be up for elimination? Would we drop that variable,\u00a0or would we keep it in the model?\r\n\r\nSolution:\r\n\r\nThe stock photo predictor is up for elimination since it has the largest p<\/em>-value. Additionally, since that p<\/em>-value is larger than 0.05, we would in fact eliminate stock photo from the model.\r\n\r\n<\/div>\r\nWhile the adjusted R<\/em>2<\/sup><\/em><\/em>\u00a0and p<\/em>-value approaches are similar, they sometimes lead to\u00a0different models, with the adjusted R<\/em>2<\/sup><\/em><\/em>\u00a0approach tending to include more predictors in the final model. For example, if we had used the p<\/em>-value approach with the auction data, we\u00a0would not have included the stock photo predictor in the final model.\r\n
\r\n

When to use the adjusted R<\/em>2<\/sup>\u00a0and when to use the p<\/em>-value approach<\/h3>\r\nWhen the sole goal is to improve prediction accuracy, use adjusted R<\/em>2<\/sup>. This is\u00a0commonly the case in machine learning applications.\r\n\r\nWhen we care about understanding which variables are statistically significant predictors of the response, or if there is interest in producing a simpler model at the potential cost of a little prediction accuracy, then the p<\/em>-value approach is preferred.\r\n\r\n<\/div>\r\nRegardless of whether you use adjusted R<\/em>2<\/sup>\u00a0or the p<\/em>-value approach, or if you use\u00a0the backward elimination of forward selection strategy, our job is not done after variable\u00a0selection. We must still verify the model conditions are reasonable.","rendered":"
\n

The best model is not always the most complicated. Sometimes including variables that\u00a0are not evidently important can actually reduce the accuracy of predictions. In this section we discuss model selection strategies, which will help us eliminate variables from the model\u00a0that are found to be less important.<\/p>\n

In practice, the model that includes all available explanatory variables is often referred\u00a0to as the full model<\/strong>. The full model may not be the best model, and if it isn’t, we want\u00a0to identify a smaller model that is preferable.<\/p>\n

Identifying variables in the model that may not be helpful<\/h2>\n

Adjusted R<\/em>2<\/sup><\/em>\u00a0describes the strength of a model fit, and it is a useful tool for evaluating\u00a0which predictors are adding value to the model, where adding value<\/em> means they are (likely)\u00a0improving the accuracy in predicting future outcomes.<\/p>\n

Let’s consider two models, which are shown in Tables 1\u00a0and 2. The first table\u00a0summarizes the full model since it includes all predictors, while the second does not include\u00a0the duration variable.<\/p>\n<\/div>\n

df<\/em> = 136<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
Table 1.\u00a0The fit for the full regression model, including the adjusted R<\/em>2<\/sup><\/em>.<\/th>\n<\/tr>\n<\/thead>\n
<\/td>\nEstimate<\/td>\nStd. Error<\/td>\nt value<\/td>\nPr( ><\/em>|t|)<\/td>\n<\/tr>\n
(Intercept)<\/td>\n36.2110<\/td>\n1.5140<\/td>\n23.92<\/td>\n0.0000<\/td>\n<\/tr>\n
cond_new<\/td>\n5.1306<\/td>\n1.0511<\/td>\n4.88<\/td>\n0.0000<\/td>\n<\/tr>\n
stock_photo<\/td>\n1.0803<\/td>\n1.0568<\/td>\n1.02<\/td>\n0.3085<\/td>\n<\/tr>\n
duration<\/td>\n\u20130.0268<\/td>\n0.1904<\/td>\n\u20130.14<\/td>\n0.8882<\/td>\n<\/tr>\n
wheels<\/td>\n7.2852<\/td>\n0.5547<\/td>\n13.13<\/td>\n0.0000<\/td>\n<\/tr>\n
R<\/em>2<\/sup>adj<\/em><\/sub><\/em><\/em>\u00a0= 0.<\/em>7108<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/div>\n
\u00a0\u00a0\u00a0\u00a0\u00a0df<\/em> = 137<\/p>\n\n\n\n\n\n\n\n\n\n\n
Table 2. The fit for the regression model for predictors cond_new,\u00a0stock_photo, and wheels.<\/th>\n<\/tr>\n<\/thead>\n
<\/td>\nEstimate<\/td>\nStd. Error<\/td>\nt value<\/td>\nPr( ><\/em>|t|)<\/td>\n<\/tr>\n
(Intercept)<\/td>\n36.0483<\/td>\n0.9745<\/td>\n36.99<\/td>\n0.0000<\/td>\n<\/tr>\n
cond_new<\/td>\n5.1763<\/td>\n0.9961<\/td>\n5.20<\/td>\n0.0000<\/td>\n<\/tr>\n
stock_photo<\/td>\n1.1177<\/td>\n1.0192<\/td>\n1.10<\/td>\n0.2747<\/td>\n<\/tr>\n
wheels<\/td>\n7.2984<\/td>\n0.5448<\/td>\n13.40<\/td>\n0.0000<\/td>\n<\/tr>\n
R<\/em>2<\/sup>adj<\/em><\/sub><\/em><\/em>\u00a0= 0.<\/em>7128<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

 <\/p>\n

\n

Example<\/h3>\n

Which of the two models is better?<\/p>\n

Solution:<\/p>\n

We compare the adjusted R2<\/sup><\/em>\u00a0of each model to determine which to choose. Since the\u00a0first model has an R2<\/sup>adj<\/sub><\/em>\u00a0smaller than the R2<\/sup>adj<\/sub><\/em>\u00a0of the second model, we prefer the\u00a0second model to the first.<\/p>\n

Will the model without duration be better than the model with duration? We cannot\u00a0know for sure, but based on the adjusted R2<\/sup><\/em>, this is our best assessment.<\/p>\n<\/div>\n

Two model selection strategies<\/h2>\n

Two common strategies for adding or removing variables in a multiple regression model are\u00a0called backward elimination<\/em> and forward selection<\/em>. These techniques are often referred to as stepwise<\/strong> model selection strategies, because they add or delete one variable at a time\u00a0as they “step” through the candidate predictors.<\/p>\n

Backward elimination<\/strong> starts with the model that includes all potential predictor\u00a0variables. Variables are eliminated one-at-a-time from the model until we cannot improve the adjusted R<\/em>2<\/sup><\/em>. The strategy within each elimination step is to eliminate the variable that\u00a0leads to the largest improvement in adjusted R<\/em>2<\/sup><\/em>.<\/p>\n

\n

\n

Example<\/h3>\n

Results corresponding to the full model<\/em> for the mario kart data are shown in Table 8.6. How should we proceed under the backward elimination\u00a0strategy?<\/p>\n

Solution:<\/p>\n

Our baseline adjusted R2<\/sup><\/em>\u00a0from the full model is\u00a0R<\/em>2<\/sup>adj<\/em><\/sub> = 0.<\/em>7108, and we need to\u00a0determine whether dropping a predictor will improve the adjusted R2<\/sup><\/em>. To check, we<\/p>\n

fit four models that each drop a different predictor, and we record the adjusted R2<\/sup><\/em><\/p>\n

from each:<\/p>\n

[latex]\\begin{array}\\text{Exclude . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.6626\\hfill&R^2_{adj}=0.7107\\hfill&R^2_{adj}=0.7128\\hfill&R^2_{adj}=0.3487\\end{array}[\/latex]<\/p>\n

The third model without duration has the highest adjusted R2<\/sup><\/em>\u00a0of 0.7128, so we\u00a0compare it to the adjusted R2<\/sup><\/em>\u00a0for the full model. Because eliminating duration\u00a0leads to a model with a higher adjusted R2<\/sup><\/em>, we drop duration from the model.\u00a0Since we eliminated a predictor from the model in the first step, we see whether\u00a0we should eliminate any additional predictors. Our baseline adjusted R2<\/sup><\/em>\u00a0is now\u00a0R2<\/sup>adj<\/sub><\/em>\u00a0= 0.<\/em>7128. We now fit three new models, which consider eliminating each of the\u00a0three remaining predictors:<\/p>\n

[latex]\\begin{array}\\text{Exclude duration and . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.6587\\hfill&R^2_{adj}=0.7124\\hfill&R^2_{adj}=0.3414\\end{array}[\/latex]<\/p>\n

None of these models lead to an improvement in adjusted R2<\/sup><\/em>, so we do not eliminate\u00a0any of the remaining predictors. That is, after backward elimination, we are left with the model that keeps cond new, stock photos, and wheels, which we can summarize\u00a0using the coefficients from Table 2:<\/p>\n

[latex]\\displaystyle\\hat{y}=b_0+b_1x_1+b_2x_2+b_4x_4[\/latex]<\/p>\n

[latex]\\displaystyle\\widehat{\\text{price}}=36.05+5.18\\times\\text{cond_new}+1.12\\times\\text{stock_photo}+7.30\\times\\text{wheels}[\/latex]<\/p>\n<\/div>\n

The forward selection<\/strong> strategy is the reverse of the backward elimination technique.\u00a0Instead of eliminating variables one-at-a-time, we add variables one-at-a-time until we\u00a0cannot find any variables that improve the model (as measured by adjusted R<\/em>2<\/sup><\/em>).<\/p>\n

 <\/p>\n

\n

Example<\/h3>\n

Construct a model for the mario kart data set using the forward\u00a0selection strategy.<\/p>\n

Solution:<\/p>\n

We start with the model that includes no variables. Then we fit each of the possible\u00a0models with just one variable. That is, we fit the model including just cond_new, then the model including just stock photo, then a model with just duration, and a\u00a0model with just wheels. Each of the four models provides an adjusted R2<\/sup><\/em>\u00a0value:<\/p>\n

[latex]\\begin{array}\\text{Add . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\hfill&\\text{wheels}\\\\\\text{ }\\hfill&R^2_{adj}=0.3459\\hfill&R^2_{adj}=0.0332\\hfill&R^2_{adj}=0.1338\\hfill&R^2_{adj}=0.6390\\end{array}[\/latex]<\/p>\n

In this first step, we compare the adjusted R2<\/sup><\/em>\u00a0against a baseline model that has\u00a0no predictors. The no-predictors model always has\u00a0R<\/em>2<\/sup>adj<\/em><\/sub>\u00a0= 0. The model with one\u00a0predictor that has the largest adjusted R2<\/sup><\/em>\u00a0is the model with the wheels predictor,\u00a0and because this adjusted R2<\/sup><\/em>\u00a0is larger than the adjusted R2<\/sup><\/em>\u00a0from the model with no\u00a0predictors (R<\/em>2<\/sup>adj<\/em><\/sub>\u00a0= 0), we will add this variable to our model.<\/p>\n

We repeat the process again, this time considering 2-predictor models where one of\u00a0the predictors is wheels and with a new baseline of R2<\/sup>adj<\/sub><\/em>\u00a0= 0.<\/em>6390:<\/p>\n

[latex]\\begin{array}\\text{Add wheels and . . .}\\hfill&\\text{cond_new}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\\\\\text{ }\\hfill&R^2_{adj}=0.7124\\hfill&R^2_{adj}=0.6587\\hfill&R^2_{adj}=0.6528\\end{array}[\/latex]<\/p>\n

The best predictor in this stage, cond new, has a higher adjusted R2<\/sup><\/em>\u00a0(0.7124) than\u00a0the baseline (0.6390), so we also add cond_new to the model.<\/p>\n

Since we have again added a variable to the model, we continue and see whether it\u00a0would be beneficial to add a third variable:<\/p>\n

[latex]\\begin{array}\\text{Add wheels, cond_new, and . . .}\\hfill&\\text{stock_photo}\\hfill&\\text{duration}\\\\\\text{ }\\hfill&R^2_{adj}=0.7128\\hfill&R^2_{adj}=0.7107\\end{array}[\/latex]<\/p>\n

The model adding stock photo improved adjusted R2<\/sup><\/em>\u00a0(0.7124 to 0.7128), so we add\u00a0stock_photo to the model.<\/p>\n

Because we have again added a predictor, we check whether adding the last variable,\u00a0duration, will improve adjusted R2<\/sup><\/em>. We compare the adjusted R2<\/sup><\/em>\u00a0for the model with duration and the other three predictors (0.7108) to the model that only considers wheels, cond_new, and stock photo (0.7128). Adding duration does not improve the adjusted R2<\/sup><\/em>, so we do not add it to the model, and we have arrived at the same\u00a0model that we identified from backward elimination.<\/p>\n<\/div>\n

\n

Model Selection Strategies<\/h3>\n

Backward elimination begins with the largest model and eliminates variables one-by-one until we are satisfied that all remaining variables are important to the model. Forward selection starts with no variables included in the model, then it adds in variables according to their importance until no other important variables\u00a0are found.<\/p>\n<\/div>\n

There is no guarantee that backward elimination and forward selection will arrive at\u00a0the same final model. If both techniques are tried and they arrive at different models, we\u00a0choose the model with the larger R<\/em>2<\/sup>adj<\/em><\/sub><\/em>; other tie-break options exist but are beyond the\u00a0scope of this book.<\/p>\n

The p<\/em>-Value Approach, an Alternative to Adjusted R<\/em>2<\/sup><\/h2>\n

The p<\/em>-value may be used as an alternative to adjusted R<\/em>2<\/sup>\u00a0for model selection.<\/p>\n

In backward elimination, we would identify the predictor corresponding to the largest\u00a0p<\/em>-value. If the p<\/em>-value is above the significance level, usually \u03b1<\/em> = 0.<\/em>05, then we would drop that variable, refit the model, and repeat the process. If the largest p<\/em>-value is less than \u03b1<\/em> = 0.<\/em>05, then we would not eliminate any predictors and the current model would be our\u00a0best-fitting model.<\/p>\n

In forward selection with p<\/em>-values, we reverse the process. We begin with a model\u00a0that has no predictors, then we fit a model for each possible predictor, identifying the model where the corresponding predictor’s p<\/em>-value is smallest. If that p<\/em>-value is smaller than\u00a0\u03b1<\/em>\u00a0= 0.<\/em>05, we add it to the model and repeat the process, considering whether to add more variables one-at-a-time. When none of the remaining predictors can be added to the model and have a p<\/em>-value less than 0.05, then we stop adding variables and the current\u00a0model would be our best-fitting model.<\/p>\n

 <\/p>\n

\n

Try It<\/h3>\n

Examine Table 2, which considers the model including the cond_new, stock_photo, and wheels predictors. If we were using the p<\/em>-value approach with backward elimination and we were considering this model, which of these three variables would be up for elimination? Would we drop that variable,\u00a0or would we keep it in the model?<\/p>\n

Solution:<\/p>\n

The stock photo predictor is up for elimination since it has the largest p<\/em>-value. Additionally, since that p<\/em>-value is larger than 0.05, we would in fact eliminate stock photo from the model.<\/p>\n<\/div>\n

While the adjusted R<\/em>2<\/sup><\/em><\/em>\u00a0and p<\/em>-value approaches are similar, they sometimes lead to\u00a0different models, with the adjusted R<\/em>2<\/sup><\/em><\/em>\u00a0approach tending to include more predictors in the final model. For example, if we had used the p<\/em>-value approach with the auction data, we\u00a0would not have included the stock photo predictor in the final model.<\/p>\n

\n

When to use the adjusted R<\/em>2<\/sup>\u00a0and when to use the p<\/em>-value approach<\/h3>\n

When the sole goal is to improve prediction accuracy, use adjusted R<\/em>2<\/sup>. This is\u00a0commonly the case in machine learning applications.<\/p>\n

When we care about understanding which variables are statistically significant predictors of the response, or if there is interest in producing a simpler model at the potential cost of a little prediction accuracy, then the p<\/em>-value approach is preferred.<\/p>\n<\/div>\n

Regardless of whether you use adjusted R<\/em>2<\/sup>\u00a0or the p<\/em>-value approach, or if you use\u00a0the backward elimination of forward selection strategy, our job is not done after variable\u00a0selection. We must still verify the model conditions are reasonable.<\/p>\n\n\t\t\t

\n\t\t\t

Candela Citations<\/h3>\n\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
CC licensed content, Shared previously<\/div>