{"id":973,"date":"2016-04-21T22:43:35","date_gmt":"2016-04-21T22:43:35","guid":{"rendered":"https:\/\/courses.lumenlearning.com\/introstats1xmaster\/?post_type=chapter&p=973"},"modified":"2017-07-20T20:23:49","modified_gmt":"2017-07-20T20:23:49","slug":"introduction-to-logistic-regression","status":"publish","type":"chapter","link":"https:\/\/courses.lumenlearning.com\/ntcc-introstats1\/chapter\/introduction-to-logistic-regression\/","title":{"raw":"Introduction to Logistic Regression","rendered":"Introduction to Logistic Regression"},"content":{"raw":"In this section we introduce logistic regression<\/strong> as a tool for building models when there is\u00a0a categorical response variable with two levels. Logistic regression is a type of generalized <\/strong>linear model<\/strong> (GLM) for response variables where regular multiple regression does not work very well. In particular, the response variable in these settings often takes a form\u00a0where residuals look completely different from the normal distribution.\r\n\r\nGLMs can be thought of as a two-stage modeling approach. We first model the\u00a0response variable using a probability distribution, such as the binomial or Poisson distribution. Second, we model the parameter of the distribution using a collection of predictors\u00a0and a special form of multiple regression.\r\n\r\nIn this page\u00a0we will data about emails. These emails were\u00a0collected from a single email account, and we will work on developing a basic spam filter using these data. The response variable, spam, has been encoded to take value 0 when a message is not spam and 1 when it is spam. Our task will be to build an appropriate model that classifies messages as spam or not spam using email characteristics coded as predictor variables. While this model will not be the same as those used in large-scale spam filters,\u00a0it shares many of the same features.\r\n

Email Data<\/h2>\r\nThere are several\u00a0variables available that might be useful for classifying spam. Descriptions of these variables are presented in Table 1. The spam variable will be the outcome, and the other 10 variables will be the model predictors. While we have limited the predictors used in this section to be categorical variables (where many are represented as indicator variables), numerical predictors may also be used in logistic\u00a0regression.[footnote]Recall that if outliers are present in predictor variables, the corresponding observations may be especially influential on the resulting model. This is the motivation for omitting the numerical variables, such as the number of characters and line breaks in emails. These variables exhibit extreme skew. We could resolve this issue by transforming these variables (e.g. using a log-transformation), but we will omit this further investigation for brevity.[\/footnote]\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 1. Descriptions for 11 variables in the email data set. Notice that all of the variables are indicator variables, which take the value 1 if the specified characteristic is present and 0 otherwise.<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
Variable<\/th>\r\nDescription<\/th>\r\n<\/tr>\r\n
spam<\/td>\r\nSpecifies whether the message was spam<\/td>\r\n<\/tr>\r\n
to_multiple<\/td>\r\nAn indicator variable for if more than one person was listed in the To field of the email.<\/td>\r\n<\/tr>\r\n
cc<\/td>\r\nAn indicator for if someone was CCed on the email<\/td>\r\n<\/tr>\r\n
attach<\/td>\r\nAn indicator for if there was an attachment, such as a document or image<\/td>\r\n<\/tr>\r\n
dollar<\/td>\r\nAn indicator for if the word \u201cdollar\u201d or dollar symbol ($) appeared in the email.<\/td>\r\n<\/tr>\r\n
winner<\/td>\r\nAn indicator for if the word \u201cwinner\u201d appeared in the email message<\/td>\r\n<\/tr>\r\n
inherit<\/td>\r\nAn indicator for if the word \u201cinherit\u201d (or a variation, like \u201cinheritance\u201d) appeared in the email.<\/td>\r\n<\/tr>\r\n
password<\/td>\r\nAn indicator for if the word \u201cpassword\u201d was present in the email.<\/td>\r\n<\/tr>\r\n
format<\/td>\r\nIndicates if the email contained special formatting, such as bolding, tables, or links<\/td>\r\n<\/tr>\r\n
re_subj<\/td>\r\nIndicates whether \u201cRe:\u201d was included at the start of the email subject.<\/td>\r\n<\/tr>\r\n
exclaim_subj<\/td>\r\nIndicates whether any exclamation point was included in the email subject<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n

Modeling the Probability of an Event<\/h2>\r\n
\r\n

TIP: Notation for a Logistic Regression Model<\/h3>\r\nThe outcome variable for a GLM is denoted by Yi<\/sub><\/em>, where the index i<\/em> is used to\u00a0represent observation i<\/em>. In the email application, Yi<\/sub><\/em> will be used to represent\u00a0whether email i<\/em> is spam (Yi<\/sub><\/em> = 1) or not (Yi<\/sub><\/em>\u00a0= 0).\r\n\r\nThe predictor variables are represented as follows: x<\/em>1, i<\/em><\/sub> is the value of variable 1 for\u00a0observation i<\/em>, x<\/em>2, i<\/em><\/sub> is the value of variable 2 for observation i<\/em>, and so on.\r\n\r\n<\/div>\r\nLogistic regression is a generalized linear model where the outcome is a two-level\u00a0categorical variable. The outcome, Yi<\/sub><\/em><\/em>, takes the value 1 (in our application, this represents a spam message) with probability pi<\/sub><\/em>\u00a0and the value 0 with probability 1 \u2212 pi<\/sub><\/em>. It is the\u00a0probability pi<\/sub><\/em>\u00a0that we model in relation to the predictor variables.\r\n\r\nThe logistic regression model relates the probability an email is spam (pi<\/sub><\/em>) to the\u00a0predictors x<\/em><\/em>1, i<\/em><\/sub>, x<\/em><\/em>2, i<\/em><\/sub>\u00a0..., x<\/em>k, i<\/em><\/sub><\/em>\u00a0through a framework much like that of multiple regression:\r\n

[latex]\\text{transformation}\\left(p_i\\right)=\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}[\/latex]<\/p>\r\nWe want to choose a transformation in the equation above\u00a0that makes practical and mathematical sense. For example, we want a transformation that makes the range of possibilities on the left hand side of the equation above\u00a0equal to the range of possibilities for the right hand side; if there was no transformation for this equation, the left hand side could only take values between 0 and 1, but the right hand side could take values outside of this range. A\u00a0common transformation for\u00a0<\/i>pi<\/sub><\/em>\u00a0is the logit transformation<\/strong>, which may be written as\r\n

[latex]\\displaystyle\\text{logit}\\left(p_i\\right)=\\log_{e}\\left(\\frac{p_i}{1-p_i}\\right)[\/latex]<\/p>\r\nThe logit transformation is shown in Figure 1. Below, we rewrite the transformation equation\u00a0using\u00a0the logit transformation of\u00a0<\/i>pi<\/sub><\/em>:\r\n

[latex]\\displaystyle\\log_e\\left(\\frac{p_i}{1-p_i}\\right)=\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}[\/latex]<\/p>\r\n\r\n\r\n[caption id=\"attachment_1468\" align=\"aligncenter\" width=\"768\"]\"A Figure 1. Values of pi<\/sub><\/i> against values of logit(pi<\/sub><\/em>).[\/caption]\r\n\r\nIn our spam example, there are 10 predictor variables, so k<\/em> = 10. This model isn't very\u00a0intuitive, but it still has some resemblance to multiple regression, and we can fit this model using software. In fact, once we look at results from software, it will start to feel like we're\u00a0back in multiple regression, even if the interpretation of the coefficients is more complex.\r\n\r\n \r\n

\r\n

Example<\/h3>\r\nHere we create a spam filter with a single predictor: to_multiple.\r\n\r\nThis variable indicates whether more than one email address was listed in the To<\/em> field of the email. The following logistic regression model was fit using statistical software:\r\n\r\n[latex]\\displaystyle\\log\\left(\\frac{p_i}{1-p_i}\\right)=-2.12-1.81\\times\\text{to_multiple}[\/latex]\r\n\r\nIf an email is randomly selected and it has just one address in the To<\/em> field, what is\u00a0the probability it is spam? What if more than one address is listed in the To<\/em> field?\r\n\r\nSolution:\r\n\r\nIf there is only one email in the To<\/em> field, then to_multiple takes value 0 and the\u00a0right side of the model equation equals \u22122.12. Solving for\u00a0pi<\/sub><\/em>:\r\n\r\n[latex]\\displaystyle\\frac{e^{-2.12}}{1+e^{-2.12}}=0.11[\/latex]\r\n\r\nJust as\u00a0we labeled a fitted value of\u00a0yi<\/sub><\/em>\u00a0with a \"hat\" in single-variable and multiple regression,\u00a0we will do the same for this probability: [latex]\\hat{p}_i=0.11[\/latex].\r\n\r\nIf there is more than one address listed in the To<\/em> field, then the right side of the model\u00a0equation is \u22122.<\/em>12 \u2212\u00a01.<\/em>81 \u00d7\u00a01 = \u22123.<\/em>93, which corresponds to a probability [latex]\\hat{p}_i=0.02[\/latex].\r\n\r\nNotice that we could examine \u22122.12 and \u22123.93 in Figure 1 to estimate the probability\u00a0before formally calculating the value.\r\n\r\n<\/div>\r\nTo convert from values on the regression-scale (e.g. \u22122.12 and \u22123.93 in Example 1),\u00a0use the following formula, which is the result of solving for p<\/em>i<\/em> in the regression model:\r\n

[latex]\\displaystyle{p}_i=\\frac{e^{\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}}}{1 + e^{\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}}}[\/latex]<\/p>\r\nAs with most applied data problems, we substitute the point estimates for the parameters\u00a0(the\u00a0\u03b2i<\/sub><\/em>) so that we may make use of this formula. In Example 1, the probabilities were\u00a0calculated as\r\n

[latex]\\displaystyle\\begin{array}\\text{ }\\frac{e^{-2.12}}{1+e^{-2.12}}=0.11\\hfill&\\text{ }\\hfill&\\frac{e^{-2.12-1.81}}{1+e^{-2.12-1.81}}=0.02\\end{array}[\/latex]<\/p>\r\nWhile the information about whether the email is addressed to multiple people is a helpful start in classifying email as spam or not, the probabilities of 11% and 2% are not dramatically different, and neither provides very strong evidence about which particular email messages are spam. To get more precise estimates, we'll need to include many more\u00a0variables in the model.\r\n\r\nWe used statistical software to fit the logistic regression model with all ten predictors\u00a0described in Table 1. Like multiple regression, the result may be presented in a summary table, which is shown in Table 2. The structure of this table is almost identical to that of multiple regression; the only notable difference is that the p<\/em>-values are calculated using\u00a0the normal distribution rather than the t<\/em>-distribution.\r\n\r\n \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 2. Summary table for the full logistic regression model for the\u00a0spam filter example<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
<\/td>\r\nEstimate<\/td>\r\nStd. Error<\/td>\r\nz value<\/td>\r\nPr ( ><\/em> |z|)<\/td>\r\n<\/tr>\r\n
(Intercept)<\/td>\r\n\u20130.8362<\/td>\r\n0.0962<\/td>\r\n\u20138.69<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
to_multiple<\/td>\r\n\u20132.8836<\/td>\r\n0.3121<\/td>\r\n\u20139.24<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
winner<\/td>\r\n1.7038<\/td>\r\n0.3254<\/td>\r\n5.24<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
format<\/td>\r\n\u20131.5902<\/td>\r\n0.1239<\/td>\r\n\u201312.84<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
re_subj<\/td>\r\n\u20132.9082<\/td>\r\n0.3708<\/td>\r\n\u20137.84<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
exclaim_subj<\/td>\r\n0.1355<\/td>\r\n0.2268<\/td>\r\n0.60<\/td>\r\n0.5503<\/td>\r\n<\/tr>\r\n
cc<\/td>\r\n\u20130.4863<\/td>\r\n0.3054<\/td>\r\n\u20131.59<\/td>\r\n0.1113<\/td>\r\n<\/tr>\r\n
attach<\/td>\r\n0.9790<\/td>\r\n0.2170<\/td>\r\n4.51<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
dollar<\/td>\r\n\u20130.0582<\/td>\r\n0.1589<\/td>\r\n\u20130.37<\/td>\r\n0.7144<\/td>\r\n<\/tr>\r\n
inherit<\/td>\r\n0.2093<\/td>\r\n0.3197<\/td>\r\n0.65<\/td>\r\n0.5127<\/td>\r\n<\/tr>\r\n
password<\/td>\r\n\u20131.4929<\/td>\r\n0.5295<\/td>\r\n\u20132.82<\/td>\r\n0.0048<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nJust like multiple regression, we could trim some variables from the model using the\u00a0p<\/em>-value. Using backward elimination with a p<\/em>-value cutoff of 0.05 (start with the full model and trim the predictors with p<\/em>-values greater than 0.05), we ultimately eliminate the exclaim_subj, dollar, inherit, and cc predictors. The remainder of this section will\u00a0rely on this smaller model, which is summarized in Table 3.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 3. Summary table for the logistic regression model for the spam\u00a0filter, where variable selection has been performed<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
<\/td>\r\nEstimate<\/td>\r\nStd. Error<\/td>\r\nz value<\/td>\r\nPr ( ><\/em> |z|)<\/td>\r\n<\/tr>\r\n
(Intercept)<\/td>\r\n\u20130.8595<\/td>\r\n0.0910<\/td>\r\n\u20139.44<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
to_multiple<\/td>\r\n\u20132.8372<\/td>\r\n0.3092<\/td>\r\n\u20139.18<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
winner<\/td>\r\n1.7370<\/td>\r\n0.3218<\/td>\r\n5.40<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
format<\/td>\r\n\u20131.5569<\/td>\r\n0.1207<\/td>\r\n\u201312.90<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
re_subj<\/td>\r\n\u20133.0482<\/td>\r\n0.3630<\/td>\r\n\u20138.40<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
attach<\/td>\r\n0.8643<\/td>\r\n0.2042<\/td>\r\n4.23<\/td>\r\n0.0000<\/td>\r\n<\/tr>\r\n
password<\/td>\r\n\u20131.4871<\/td>\r\n0.5290<\/td>\r\n\u20132.81<\/td>\r\n0.0049<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n \r\n
\r\n

Try It<\/h3>\r\nExamine the summary of the reduced model in Table 3,\u00a0and in particular, examine the to_multiple row. Is the point estimate the same as\u00a0we found before, \u22121.81, or is it different? Explain why this might be.\r\n\r\nSolution:\r\n\r\nThe new estimate is different: \u22122.87. This new value represents the estimated coefficient when we are also accounting for other variables in the logistic regression model.\r\n\r\n<\/div>\r\n \r\n\r\nPoint estimates will generally change a little\u2014and sometimes a lot\u2014depending on which other variables are included in the model. This is usually due to colinearity in the predictor variables. We previously saw this in the eBay auction example when we compared the coefficient of cond_new in a single-variable model and the corresponding coefficient in the multiple regression model that used three additional variables.\r\n\r\n \r\n
\r\n

Example<\/h3>\r\nSpam filters are built to be automated, meaning a piece of software\u00a0is written to collect information about emails as they arrive, and this information is put in the form of variables. These variables are then put into an algorithm that uses a statistical model, like the one we've fit, to classify the email. Suppose we write software for a spam filter using the reduced model shown in Table 3. If an incoming email has the word \"winner\" in it, will this raise or lower the model's\u00a0calculated probability that the incoming email is spam?\r\n\r\nSolution:\r\n\r\nThe estimated coefficient of winner is positive (1.7370). A positive coefficient estimate in logistic regression, just like in multiple regression, corresponds to a positive association between the predictor and response variables when accounting for the other variables in the model. Since the response variable takes value 1 if an email is spam and 0 otherwise, the positive coefficient indicates that the presence of \"winner\"\u00a0in an email raises the model probability that the message is spam.\r\n\r\n<\/div>\r\n
\r\n

Example<\/h3>\r\nSuppose the same email from Example 2\u00a0was in HTML format,\u00a0meaning the format variable took value 1. Does this characteristic increase or decrease the probability that the email is spam according to the model?\r\n\r\nSolution:\r\n\r\nSince HTML corresponds to a value of 1 in the format variable and the coefficient of\u00a0this variable is negative (\u22121.5569), this would lower the probability estimate returned\u00a0from the model.\r\n\r\n<\/div>\r\n

Practical Decisions in the Email Application<\/h2>\r\nExamples 2\u00a0and 3\u00a0highlight a key feature of logistic and multiple regression. In the\u00a0spam filter example, some email characteristics will push an email's classification in the\u00a0direction of spam while other characteristics will push it in the opposite direction.\r\n\r\nIf we were to implement a spam filter using the model we have fit, then each future\u00a0email we analyze would fall into one of three categories based on the email's characteristics:\r\n
    \r\n \t
  1. The email characteristics generally indicate the email is not spam, and so the resulting\u00a0probability that the email is spam is quite low, say, under 0.05.<\/li>\r\n \t
  2. The characteristics generally indicate the email is spam, and so the resulting probability that the email is spam is quite large, say, over 0.95.<\/li>\r\n \t
  3. The characteristics roughly balance each other out in terms of evidence for and against\u00a0the message being classified as spam. Its probability falls in the remaining range,\u00a0meaning the email cannot be adequately classified as spam or not spam.<\/li>\r\n<\/ol>\r\nIf we were managing an email service, we would have to think about what should be\u00a0done in each of these three instances. In an email application, there are usually just two possibilities: filter the email out from the regular inbox and put it in a \"spambox,\" or let\u00a0the email go to the regular inbox.\r\n\r\n \r\n
    \r\n

    Try It<\/h3>\r\nThe first and second scenarios are intuitive. If the evidence strongly suggests a message is not spam, send it to the inbox. If the evidence strongly\u00a0suggests the message is spam, send it to the spambox. How should we handle emails\u00a0in the third category?\r\n\r\nSolution:\r\n\r\nIn this particular application, we should err on the side of sending more mail to the inbox rather than mistakenly putting good messages in the spambox. So, in summary: emails in the first and last categories go to the regular inbox, and those in the second scenario go to the spambox.\r\n\r\n<\/div>\r\n
    \r\n

    Try It<\/h3>\r\nSuppose we apply the logistic model we have built as a\u00a0spam filter and that 100 messages are placed in the spambox over 3 months. If we\u00a0used the guidelines above for putting messages into the spambox, about how many\u00a0legitimate (non-spam) messages would you expect to find among the 100 messages?\r\n\r\nSolution:\r\n\r\nFirst, note that we proposed a cutoff for the predicted probability of 0.95 for spam. In a worst case scenario, all the messages in the spambox had the minimum probability equal to about 0.95. Thus, we should expect to find about 5 or fewer legitimate messages among the 100 messages placed in the spambox.\r\n\r\n<\/div>\r\nAlmost any classifier will have some error. In the spam filter guidelines above, we\u00a0have decided that it is okay to allow up to 5% of the messages in the spambox to be real messages. If we wanted to make it a little harder to classify messages as spam, we could use a cutoff of 0.99. This would have two effects. Because it raises the standard for what can be classified as spam, it reduces the number of good emails that are classified as spam. However, it will also fail to correctly classify an increased fraction of spam messages. No matter the complexity and the confidence we might have in our model, these practical considerations are absolutely crucial to making a helpful spam filter. Without them, we\u00a0could actually do more harm than good by using our statistical model.\r\n

    Diagnostics for the Email Classifier<\/h2>\r\n
    \r\n

    Logistic Regression Conditions<\/h3>\r\nThere are two key conditions for fitting a logistic regression model:\r\n
      \r\n \t
    1. Each predictor xi<\/sub><\/em> is linearly related to logit(pi<\/sub><\/em>) if all other predictors are\u00a0held constant.<\/li>\r\n \t
    2. Each outcome Yi<\/sub><\/em> is independent of the other outcomes.<\/li>\r\n<\/ol>\r\n<\/div>\r\nThe first condition of the logistic regression model is not easily checked without a\u00a0fairly sizable amount of data. Luckily, we have 3,921 emails in our data set! Let's first visualize these data by plotting the true classification of the emails against the model's fitted probabilities, as shown in Figure 2. The vast majority of emails (spam or not)\u00a0still have fitted probabilities below 0.5.\r\n\r\n[caption id=\"attachment_1470\" align=\"aligncenter\" width=\"779\"]\"Scatter Figure 2: The predicted probability that each of the 3,912 emails is spam is classified by their grouping, spam or not. Noise (small, random vertical shifts) have been added to each point so that points with nearly identical values aren't plotted exactly on top of one another. This makes it possible\u00a0to see more observations.[\/caption]\r\n\r\nThis may at first seem very discouraging: we have fit a logistic model to create a spam\u00a0filter, but no emails have a fitted probability of being spam above 0.75. Don't despair; we will discuss ways to improve the model through the use of better variables\u00a0later.\r\n\r\nWe'd like to assess the quality of our model. For example, we might ask: if we look\u00a0at emails that we modeled as having a 10% chance of being spam, do we find about 10% of them actually are spam? To help us out, we'll borrow an advanced statistical method called natural splines<\/strong> that estimates the local probability over the region 0.00 to 0.75 (the largest predicted probability was 0.73, so we avoid extrapolating). All you need to know about natural splines to understand what we are doing is that they are used to fit\u00a0\ufb02exible lines rather than straight lines.\r\n\r\nThe curve fit using natural splines is shown in Figure 3\u00a0as a solid black line. If\u00a0the logistic model fits well, the curve should closely follow the dashed y<\/em> = x<\/em> line. We have added shading to represent the confidence bound for the curved line to clarify what \ufb02uctuations might plausibly be due to chance. Even with this confidence bound, there are weaknesses in the first model assumption. The solid curve and its confidence bound dips below the dashed line from about 0.1 to 0.3, and then it drifts above the dashed line from about 0.35 to 0.55. These deviations indicate the model relating the parameter to the\u00a0predictors does not closely resemble the true relationship.\r\n\r\nWe could evaluate the second logistic regression model assumption\u2014independence of\u00a0the outcomes\u2014using the model residuals. The residuals for a logistic regression model are calculated the same way as with multiple regression: the observed outcome minus the expected outcome. For logistic regression, the expected value of the outcome is the fitted\u00a0probability for the observation, and the residual may be written as\r\n

      [latex]\\displaystyle{e}_i=Y_i-\\hat{p}_i[\/latex]<\/p>\r\nWe could plot these residuals against a variety of variables or in their order of collection,\u00a0as we did with the residuals in multiple regression. However, since the model will need to be revised to effectively classify spam and you have already seen similar residual plots, we won't investigate the residuals here.\r\n\r\n[caption id=\"attachment_1471\" align=\"aligncenter\" width=\"774\"]\"Figure Figure 3: The solid black line provides the empirical estimate of the probability for observations based on their predicted probabilities (confidence bounds are also shown for this line), which is fit using natural splines. A small amount of noise was added to the observations in the plot to allow\u00a0more observations to be seen.[\/caption]\r\n

      Improving the Set of Variables for a Spam Filter<\/h2>\r\nIf we were building a spam filter for an email service that managed many accounts (e.g.\u00a0Gmail or Hotmail), we would spend much more time thinking about additional variables that could be useful in classifying emails as spam or not. We also would use transformations or other techniques that would help us include strongly skewed numerical variables as\u00a0predictors.\r\n\r\nTake a few minutes to think about additional variables that might be useful in identifying spam. Below is a list of variables we think might be useful:\r\n
        \r\n \t
      1. An indicator variable could be used to represent whether there was prior two-way\u00a0correspondence with a message's sender. For instance, if you sent a message to john@example.com and then John sent you an email, this variable would take value 1 for the email that John sent. If you had never sent John an email, then the variable\u00a0would be set to 0.<\/li>\r\n \t
      2. A second indicator variable could utilize an account's past spam \ufb02agging information.\u00a0The variable could take value 1 if the sender of the message has previously sent\u00a0messages \ufb02agged as spam.<\/li>\r\n \t
      3. A third indicator variable could \ufb02ag emails that contain links included in previous\u00a0spam messages. If such a link is found, then set the variable to 1 for the email.\u00a0Otherwise, set it to 0.<\/li>\r\n<\/ol>\r\nThe variables described above take one of two approaches. Variable (1) is specially designed\u00a0to capitalize on the fact that spam is rarely sent between individuals that have two-way\u00a0communication. Variables (2) and (3) are specially designed to \ufb02ag common spammers or\u00a0spam messages. While we would have to verify using the data that each of the variables is\u00a0effective, these seem like promising ideas.\r\n\r\nTable 4\u00a0shows a contingency table for spam and also for the new variable described\u00a0in (1) above. If we look at the 1,090 emails where there was correspondence with the sender in the preceding 30 days, not one of these message was spam. This suggests variable (1) would be very effective at accurately classifying some messages as not spam. With this single variable, we would be able to send about 28% of messages through to the inbox with\u00a0confidence that almost none are spam.\r\n\r\n \r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
        Table 4.\u00a0A contingency table for spam and a new variable that represents\u00a0whether there had been correspondence with the sender in the preceding\u00a030 days<\/th>\r\n<\/tr>\r\n<\/thead>\r\n
        <\/td>\r\nprior correspondence<\/td>\r\n<\/tr>\r\n
        <\/td>\r\nno<\/td>\r\nyes<\/td>\r\nTotal<\/td>\r\n<\/tr>\r\n
        spam<\/td>\r\n367<\/td>\r\no<\/td>\r\n367<\/td>\r\n<\/tr>\r\n
        not spam<\/td>\r\n2464<\/td>\r\n1090<\/td>\r\n3554<\/td>\r\n<\/tr>\r\n
        Total<\/td>\r\n2831<\/td>\r\n1090<\/td>\r\n3921<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nThe variables described in (2) and (3) would provide an excellent foundation for distinguishing messages coming from known spammers or messages that take a known form of spam. To utilize these variables, we would need to build databases: one holding email addresses of known spammers, and one holding URLs found in known spam messages. Our access to such information is limited, so we cannot implement these two variables in this textbook. However, if we were hired by an email service to build a spam filter, these would\u00a0be important next steps.\r\n\r\nIn addition to finding more and better predictors, we would need to create a customized\u00a0logistic regression model for each email account. This may sound like an intimidating task, but its complexity is not as daunting as it may at first seem. We'll save the details for a\u00a0statistics course where computer programming plays a more central role.\r\n\r\nFor what is the extremely challenging task of classifying spam messages, we have made\u00a0a lot of progress. We have seen that simple email variables, such as the format, inclusion of certain words, and other circumstantial characteristics, provide helpful information for spam classification. Many challenges remain, from better understanding logistic regression to carrying out the necessary computer programming, but completing such a task is very\u00a0nearly within your reach.","rendered":"

        In this section we introduce logistic regression<\/strong> as a tool for building models when there is\u00a0a categorical response variable with two levels. Logistic regression is a type of generalized <\/strong>linear model<\/strong> (GLM) for response variables where regular multiple regression does not work very well. In particular, the response variable in these settings often takes a form\u00a0where residuals look completely different from the normal distribution.<\/p>\n

        GLMs can be thought of as a two-stage modeling approach. We first model the\u00a0response variable using a probability distribution, such as the binomial or Poisson distribution. Second, we model the parameter of the distribution using a collection of predictors\u00a0and a special form of multiple regression.<\/p>\n

        In this page\u00a0we will data about emails. These emails were\u00a0collected from a single email account, and we will work on developing a basic spam filter using these data. The response variable, spam, has been encoded to take value 0 when a message is not spam and 1 when it is spam. Our task will be to build an appropriate model that classifies messages as spam or not spam using email characteristics coded as predictor variables. While this model will not be the same as those used in large-scale spam filters,\u00a0it shares many of the same features.<\/p>\n

        Email Data<\/h2>\n

        There are several\u00a0variables available that might be useful for classifying spam. Descriptions of these variables are presented in Table 1. The spam variable will be the outcome, and the other 10 variables will be the model predictors. While we have limited the predictors used in this section to be categorical variables (where many are represented as indicator variables), numerical predictors may also be used in logistic\u00a0regression.[1]<\/sup><\/a><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Table 1. Descriptions for 11 variables in the email data set. Notice that all of the variables are indicator variables, which take the value 1 if the specified characteristic is present and 0 otherwise.<\/th>\n<\/tr>\n<\/thead>\n
        Variable<\/th>\nDescription<\/th>\n<\/tr>\n
        spam<\/td>\nSpecifies whether the message was spam<\/td>\n<\/tr>\n
        to_multiple<\/td>\nAn indicator variable for if more than one person was listed in the To field of the email.<\/td>\n<\/tr>\n
        cc<\/td>\nAn indicator for if someone was CCed on the email<\/td>\n<\/tr>\n
        attach<\/td>\nAn indicator for if there was an attachment, such as a document or image<\/td>\n<\/tr>\n
        dollar<\/td>\nAn indicator for if the word \u201cdollar\u201d or dollar symbol ($) appeared in the email.<\/td>\n<\/tr>\n
        winner<\/td>\nAn indicator for if the word \u201cwinner\u201d appeared in the email message<\/td>\n<\/tr>\n
        inherit<\/td>\nAn indicator for if the word \u201cinherit\u201d (or a variation, like \u201cinheritance\u201d) appeared in the email.<\/td>\n<\/tr>\n
        password<\/td>\nAn indicator for if the word \u201cpassword\u201d was present in the email.<\/td>\n<\/tr>\n
        format<\/td>\nIndicates if the email contained special formatting, such as bolding, tables, or links<\/td>\n<\/tr>\n
        re_subj<\/td>\nIndicates whether \u201cRe:\u201d was included at the start of the email subject.<\/td>\n<\/tr>\n
        exclaim_subj<\/td>\nIndicates whether any exclamation point was included in the email subject<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Modeling the Probability of an Event<\/h2>\n
        \n

        TIP: Notation for a Logistic Regression Model<\/h3>\n

        The outcome variable for a GLM is denoted by Yi<\/sub><\/em>, where the index i<\/em> is used to\u00a0represent observation i<\/em>. In the email application, Yi<\/sub><\/em> will be used to represent\u00a0whether email i<\/em> is spam (Yi<\/sub><\/em> = 1) or not (Yi<\/sub><\/em>\u00a0= 0).<\/p>\n

        The predictor variables are represented as follows: x<\/em>1, i<\/em><\/sub> is the value of variable 1 for\u00a0observation i<\/em>, x<\/em>2, i<\/em><\/sub> is the value of variable 2 for observation i<\/em>, and so on.<\/p>\n<\/div>\n

        Logistic regression is a generalized linear model where the outcome is a two-level\u00a0categorical variable. The outcome, Yi<\/sub><\/em><\/em>, takes the value 1 (in our application, this represents a spam message) with probability pi<\/sub><\/em>\u00a0and the value 0 with probability 1 \u2212 pi<\/sub><\/em>. It is the\u00a0probability pi<\/sub><\/em>\u00a0that we model in relation to the predictor variables.<\/p>\n

        The logistic regression model relates the probability an email is spam (pi<\/sub><\/em>) to the\u00a0predictors x<\/em><\/em>1, i<\/em><\/sub>, x<\/em><\/em>2, i<\/em><\/sub>\u00a0…, x<\/em>k, i<\/em><\/sub><\/em>\u00a0through a framework much like that of multiple regression:<\/p>\n

        [latex]\\text{transformation}\\left(p_i\\right)=\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}[\/latex]<\/p>\n

        We want to choose a transformation in the equation above\u00a0that makes practical and mathematical sense. For example, we want a transformation that makes the range of possibilities on the left hand side of the equation above\u00a0equal to the range of possibilities for the right hand side; if there was no transformation for this equation, the left hand side could only take values between 0 and 1, but the right hand side could take values outside of this range. A\u00a0common transformation for\u00a0<\/i>pi<\/sub><\/em>\u00a0is the logit transformation<\/strong>, which may be written as<\/p>\n

        [latex]\\displaystyle\\text{logit}\\left(p_i\\right)=\\log_{e}\\left(\\frac{p_i}{1-p_i}\\right)[\/latex]<\/p>\n

        The logit transformation is shown in Figure 1. Below, we rewrite the transformation equation\u00a0using\u00a0the logit transformation of\u00a0<\/i>pi<\/sub><\/em>:<\/p>\n

        [latex]\\displaystyle\\log_e\\left(\\frac{p_i}{1-p_i}\\right)=\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}[\/latex]<\/p>\n

        \"A<\/p>\n

        Figure 1. Values of pi<\/sub><\/i> against values of logit(pi<\/sub><\/em>).<\/p>\n<\/div>\n

        In our spam example, there are 10 predictor variables, so k<\/em> = 10. This model isn’t very\u00a0intuitive, but it still has some resemblance to multiple regression, and we can fit this model using software. In fact, once we look at results from software, it will start to feel like we’re\u00a0back in multiple regression, even if the interpretation of the coefficients is more complex.<\/p>\n

         <\/p>\n

        \n

        Example<\/h3>\n

        Here we create a spam filter with a single predictor: to_multiple.<\/p>\n

        This variable indicates whether more than one email address was listed in the To<\/em> field of the email. The following logistic regression model was fit using statistical software:<\/p>\n

        [latex]\\displaystyle\\log\\left(\\frac{p_i}{1-p_i}\\right)=-2.12-1.81\\times\\text{to_multiple}[\/latex]<\/p>\n

        If an email is randomly selected and it has just one address in the To<\/em> field, what is\u00a0the probability it is spam? What if more than one address is listed in the To<\/em> field?<\/p>\n

        Solution:<\/p>\n

        If there is only one email in the To<\/em> field, then to_multiple takes value 0 and the\u00a0right side of the model equation equals \u22122.12. Solving for\u00a0pi<\/sub><\/em>:<\/p>\n

        [latex]\\displaystyle\\frac{e^{-2.12}}{1+e^{-2.12}}=0.11[\/latex]<\/p>\n

        Just as\u00a0we labeled a fitted value of\u00a0yi<\/sub><\/em>\u00a0with a “hat” in single-variable and multiple regression,\u00a0we will do the same for this probability: [latex]\\hat{p}_i=0.11[\/latex].<\/p>\n

        If there is more than one address listed in the To<\/em> field, then the right side of the model\u00a0equation is \u22122.<\/em>12 \u2212\u00a01.<\/em>81 \u00d7\u00a01 = \u22123.<\/em>93, which corresponds to a probability [latex]\\hat{p}_i=0.02[\/latex].<\/p>\n

        Notice that we could examine \u22122.12 and \u22123.93 in Figure 1 to estimate the probability\u00a0before formally calculating the value.<\/p>\n<\/div>\n

        To convert from values on the regression-scale (e.g. \u22122.12 and \u22123.93 in Example 1),\u00a0use the following formula, which is the result of solving for p<\/em>i<\/em> in the regression model:<\/p>\n

        [latex]\\displaystyle{p}_i=\\frac{e^{\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}}}{1 + e^{\\beta_0+\\beta_1x_{1,i}+\\beta_2x_{2,i}+\\dots\\beta_k{x}_{k,i}}}[\/latex]<\/p>\n

        As with most applied data problems, we substitute the point estimates for the parameters\u00a0(the\u00a0\u03b2i<\/sub><\/em>) so that we may make use of this formula. In Example 1, the probabilities were\u00a0calculated as<\/p>\n

        [latex]\\displaystyle\\begin{array}\\text{ }\\frac{e^{-2.12}}{1+e^{-2.12}}=0.11\\hfill&\\text{ }\\hfill&\\frac{e^{-2.12-1.81}}{1+e^{-2.12-1.81}}=0.02\\end{array}[\/latex]<\/p>\n

        While the information about whether the email is addressed to multiple people is a helpful start in classifying email as spam or not, the probabilities of 11% and 2% are not dramatically different, and neither provides very strong evidence about which particular email messages are spam. To get more precise estimates, we’ll need to include many more\u00a0variables in the model.<\/p>\n

        We used statistical software to fit the logistic regression model with all ten predictors\u00a0described in Table 1. Like multiple regression, the result may be presented in a summary table, which is shown in Table 2. The structure of this table is almost identical to that of multiple regression; the only notable difference is that the p<\/em>-values are calculated using\u00a0the normal distribution rather than the t<\/em>-distribution.<\/p>\n

         <\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
        Table 2. Summary table for the full logistic regression model for the\u00a0spam filter example<\/th>\n<\/tr>\n<\/thead>\n
        <\/td>\nEstimate<\/td>\nStd. Error<\/td>\nz value<\/td>\nPr ( ><\/em> |z|)<\/td>\n<\/tr>\n
        (Intercept)<\/td>\n\u20130.8362<\/td>\n0.0962<\/td>\n\u20138.69<\/td>\n0.0000<\/td>\n<\/tr>\n
        to_multiple<\/td>\n\u20132.8836<\/td>\n0.3121<\/td>\n\u20139.24<\/td>\n0.0000<\/td>\n<\/tr>\n
        winner<\/td>\n1.7038<\/td>\n0.3254<\/td>\n5.24<\/td>\n0.0000<\/td>\n<\/tr>\n
        format<\/td>\n\u20131.5902<\/td>\n0.1239<\/td>\n\u201312.84<\/td>\n0.0000<\/td>\n<\/tr>\n
        re_subj<\/td>\n\u20132.9082<\/td>\n0.3708<\/td>\n\u20137.84<\/td>\n0.0000<\/td>\n<\/tr>\n
        exclaim_subj<\/td>\n0.1355<\/td>\n0.2268<\/td>\n0.60<\/td>\n0.5503<\/td>\n<\/tr>\n
        cc<\/td>\n\u20130.4863<\/td>\n0.3054<\/td>\n\u20131.59<\/td>\n0.1113<\/td>\n<\/tr>\n
        attach<\/td>\n0.9790<\/td>\n0.2170<\/td>\n4.51<\/td>\n0.0000<\/td>\n<\/tr>\n
        dollar<\/td>\n\u20130.0582<\/td>\n0.1589<\/td>\n\u20130.37<\/td>\n0.7144<\/td>\n<\/tr>\n
        inherit<\/td>\n0.2093<\/td>\n0.3197<\/td>\n0.65<\/td>\n0.5127<\/td>\n<\/tr>\n
        password<\/td>\n\u20131.4929<\/td>\n0.5295<\/td>\n\u20132.82<\/td>\n0.0048<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

        Just like multiple regression, we could trim some variables from the model using the\u00a0p<\/em>-value. Using backward elimination with a p<\/em>-value cutoff of 0.05 (start with the full model and trim the predictors with p<\/em>-values greater than 0.05), we ultimately eliminate the exclaim_subj, dollar, inherit, and cc predictors. The remainder of this section will\u00a0rely on this smaller model, which is summarized in Table 3.<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
        Table 3. Summary table for the logistic regression model for the spam\u00a0filter, where variable selection has been performed<\/th>\n<\/tr>\n<\/thead>\n
        <\/td>\nEstimate<\/td>\nStd. Error<\/td>\nz value<\/td>\nPr ( ><\/em> |z|)<\/td>\n<\/tr>\n
        (Intercept)<\/td>\n\u20130.8595<\/td>\n0.0910<\/td>\n\u20139.44<\/td>\n0.0000<\/td>\n<\/tr>\n
        to_multiple<\/td>\n\u20132.8372<\/td>\n0.3092<\/td>\n\u20139.18<\/td>\n0.0000<\/td>\n<\/tr>\n
        winner<\/td>\n1.7370<\/td>\n0.3218<\/td>\n5.40<\/td>\n0.0000<\/td>\n<\/tr>\n
        format<\/td>\n\u20131.5569<\/td>\n0.1207<\/td>\n\u201312.90<\/td>\n0.0000<\/td>\n<\/tr>\n
        re_subj<\/td>\n\u20133.0482<\/td>\n0.3630<\/td>\n\u20138.40<\/td>\n0.0000<\/td>\n<\/tr>\n
        attach<\/td>\n0.8643<\/td>\n0.2042<\/td>\n4.23<\/td>\n0.0000<\/td>\n<\/tr>\n
        password<\/td>\n\u20131.4871<\/td>\n0.5290<\/td>\n\u20132.81<\/td>\n0.0049<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

         <\/p>\n

        \n

        Try It<\/h3>\n

        Examine the summary of the reduced model in Table 3,\u00a0and in particular, examine the to_multiple row. Is the point estimate the same as\u00a0we found before, \u22121.81, or is it different? Explain why this might be.<\/p>\n

        Solution:<\/p>\n

        The new estimate is different: \u22122.87. This new value represents the estimated coefficient when we are also accounting for other variables in the logistic regression model.<\/p>\n<\/div>\n

         <\/p>\n

        Point estimates will generally change a little\u2014and sometimes a lot\u2014depending on which other variables are included in the model. This is usually due to colinearity in the predictor variables. We previously saw this in the eBay auction example when we compared the coefficient of cond_new in a single-variable model and the corresponding coefficient in the multiple regression model that used three additional variables.<\/p>\n

         <\/p>\n

        \n

        Example<\/h3>\n

        Spam filters are built to be automated, meaning a piece of software\u00a0is written to collect information about emails as they arrive, and this information is put in the form of variables. These variables are then put into an algorithm that uses a statistical model, like the one we’ve fit, to classify the email. Suppose we write software for a spam filter using the reduced model shown in Table 3. If an incoming email has the word “winner” in it, will this raise or lower the model’s\u00a0calculated probability that the incoming email is spam?<\/p>\n

        Solution:<\/p>\n

        The estimated coefficient of winner is positive (1.7370). A positive coefficient estimate in logistic regression, just like in multiple regression, corresponds to a positive association between the predictor and response variables when accounting for the other variables in the model. Since the response variable takes value 1 if an email is spam and 0 otherwise, the positive coefficient indicates that the presence of “winner”\u00a0in an email raises the model probability that the message is spam.<\/p>\n<\/div>\n

        \n

        Example<\/h3>\n

        Suppose the same email from Example 2\u00a0was in HTML format,\u00a0meaning the format variable took value 1. Does this characteristic increase or decrease the probability that the email is spam according to the model?<\/p>\n

        Solution:<\/p>\n

        Since HTML corresponds to a value of 1 in the format variable and the coefficient of\u00a0this variable is negative (\u22121.5569), this would lower the probability estimate returned\u00a0from the model.<\/p>\n<\/div>\n

        Practical Decisions in the Email Application<\/h2>\n

        Examples 2\u00a0and 3\u00a0highlight a key feature of logistic and multiple regression. In the\u00a0spam filter example, some email characteristics will push an email’s classification in the\u00a0direction of spam while other characteristics will push it in the opposite direction.<\/p>\n

        If we were to implement a spam filter using the model we have fit, then each future\u00a0email we analyze would fall into one of three categories based on the email’s characteristics:<\/p>\n

          \n
        1. The email characteristics generally indicate the email is not spam, and so the resulting\u00a0probability that the email is spam is quite low, say, under 0.05.<\/li>\n
        2. The characteristics generally indicate the email is spam, and so the resulting probability that the email is spam is quite large, say, over 0.95.<\/li>\n
        3. The characteristics roughly balance each other out in terms of evidence for and against\u00a0the message being classified as spam. Its probability falls in the remaining range,\u00a0meaning the email cannot be adequately classified as spam or not spam.<\/li>\n<\/ol>\n

          If we were managing an email service, we would have to think about what should be\u00a0done in each of these three instances. In an email application, there are usually just two possibilities: filter the email out from the regular inbox and put it in a “spambox,” or let\u00a0the email go to the regular inbox.<\/p>\n

           <\/p>\n

          \n

          Try It<\/h3>\n

          The first and second scenarios are intuitive. If the evidence strongly suggests a message is not spam, send it to the inbox. If the evidence strongly\u00a0suggests the message is spam, send it to the spambox. How should we handle emails\u00a0in the third category?<\/p>\n

          Solution:<\/p>\n

          In this particular application, we should err on the side of sending more mail to the inbox rather than mistakenly putting good messages in the spambox. So, in summary: emails in the first and last categories go to the regular inbox, and those in the second scenario go to the spambox.<\/p>\n<\/div>\n

          \n

          Try It<\/h3>\n

          Suppose we apply the logistic model we have built as a\u00a0spam filter and that 100 messages are placed in the spambox over 3 months. If we\u00a0used the guidelines above for putting messages into the spambox, about how many\u00a0legitimate (non-spam) messages would you expect to find among the 100 messages?<\/p>\n

          Solution:<\/p>\n

          First, note that we proposed a cutoff for the predicted probability of 0.95 for spam. In a worst case scenario, all the messages in the spambox had the minimum probability equal to about 0.95. Thus, we should expect to find about 5 or fewer legitimate messages among the 100 messages placed in the spambox.<\/p>\n<\/div>\n

          Almost any classifier will have some error. In the spam filter guidelines above, we\u00a0have decided that it is okay to allow up to 5% of the messages in the spambox to be real messages. If we wanted to make it a little harder to classify messages as spam, we could use a cutoff of 0.99. This would have two effects. Because it raises the standard for what can be classified as spam, it reduces the number of good emails that are classified as spam. However, it will also fail to correctly classify an increased fraction of spam messages. No matter the complexity and the confidence we might have in our model, these practical considerations are absolutely crucial to making a helpful spam filter. Without them, we\u00a0could actually do more harm than good by using our statistical model.<\/p>\n

          Diagnostics for the Email Classifier<\/h2>\n
          \n

          Logistic Regression Conditions<\/h3>\n

          There are two key conditions for fitting a logistic regression model:<\/p>\n

            \n
          1. Each predictor xi<\/sub><\/em> is linearly related to logit(pi<\/sub><\/em>) if all other predictors are\u00a0held constant.<\/li>\n
          2. Each outcome Yi<\/sub><\/em> is independent of the other outcomes.<\/li>\n<\/ol>\n<\/div>\n

            The first condition of the logistic regression model is not easily checked without a\u00a0fairly sizable amount of data. Luckily, we have 3,921 emails in our data set! Let’s first visualize these data by plotting the true classification of the emails against the model’s fitted probabilities, as shown in Figure 2. The vast majority of emails (spam or not)\u00a0still have fitted probabilities below 0.5.<\/p>\n

            \"Scatter<\/p>\n

            Figure 2: The predicted probability that each of the 3,912 emails is spam is classified by their grouping, spam or not. Noise (small, random vertical shifts) have been added to each point so that points with nearly identical values aren’t plotted exactly on top of one another. This makes it possible\u00a0to see more observations.<\/p>\n<\/div>\n

            This may at first seem very discouraging: we have fit a logistic model to create a spam\u00a0filter, but no emails have a fitted probability of being spam above 0.75. Don’t despair; we will discuss ways to improve the model through the use of better variables\u00a0later.<\/p>\n

            We’d like to assess the quality of our model. For example, we might ask: if we look\u00a0at emails that we modeled as having a 10% chance of being spam, do we find about 10% of them actually are spam? To help us out, we’ll borrow an advanced statistical method called natural splines<\/strong> that estimates the local probability over the region 0.00 to 0.75 (the largest predicted probability was 0.73, so we avoid extrapolating). All you need to know about natural splines to understand what we are doing is that they are used to fit\u00a0\ufb02exible lines rather than straight lines.<\/p>\n

            The curve fit using natural splines is shown in Figure 3\u00a0as a solid black line. If\u00a0the logistic model fits well, the curve should closely follow the dashed y<\/em> = x<\/em> line. We have added shading to represent the confidence bound for the curved line to clarify what \ufb02uctuations might plausibly be due to chance. Even with this confidence bound, there are weaknesses in the first model assumption. The solid curve and its confidence bound dips below the dashed line from about 0.1 to 0.3, and then it drifts above the dashed line from about 0.35 to 0.55. These deviations indicate the model relating the parameter to the\u00a0predictors does not closely resemble the true relationship.<\/p>\n

            We could evaluate the second logistic regression model assumption\u2014independence of\u00a0the outcomes\u2014using the model residuals. The residuals for a logistic regression model are calculated the same way as with multiple regression: the observed outcome minus the expected outcome. For logistic regression, the expected value of the outcome is the fitted\u00a0probability for the observation, and the residual may be written as<\/p>\n

            [latex]\\displaystyle{e}_i=Y_i-\\hat{p}_i[\/latex]<\/p>\n

            We could plot these residuals against a variety of variables or in their order of collection,\u00a0as we did with the residuals in multiple regression. However, since the model will need to be revised to effectively classify spam and you have already seen similar residual plots, we won’t investigate the residuals here.<\/p>\n

            \"Figure<\/p>\n

            Figure 3: The solid black line provides the empirical estimate of the probability for observations based on their predicted probabilities (confidence bounds are also shown for this line), which is fit using natural splines. A small amount of noise was added to the observations in the plot to allow\u00a0more observations to be seen.<\/p>\n<\/div>\n

            Improving the Set of Variables for a Spam Filter<\/h2>\n

            If we were building a spam filter for an email service that managed many accounts (e.g.\u00a0Gmail or Hotmail), we would spend much more time thinking about additional variables that could be useful in classifying emails as spam or not. We also would use transformations or other techniques that would help us include strongly skewed numerical variables as\u00a0predictors.<\/p>\n

            Take a few minutes to think about additional variables that might be useful in identifying spam. Below is a list of variables we think might be useful:<\/p>\n

              \n
            1. An indicator variable could be used to represent whether there was prior two-way\u00a0correspondence with a message’s sender. For instance, if you sent a message to john@example.com and then John sent you an email, this variable would take value 1 for the email that John sent. If you had never sent John an email, then the variable\u00a0would be set to 0.<\/li>\n
            2. A second indicator variable could utilize an account’s past spam \ufb02agging information.\u00a0The variable could take value 1 if the sender of the message has previously sent\u00a0messages \ufb02agged as spam.<\/li>\n
            3. A third indicator variable could \ufb02ag emails that contain links included in previous\u00a0spam messages. If such a link is found, then set the variable to 1 for the email.\u00a0Otherwise, set it to 0.<\/li>\n<\/ol>\n

              The variables described above take one of two approaches. Variable (1) is specially designed\u00a0to capitalize on the fact that spam is rarely sent between individuals that have two-way\u00a0communication. Variables (2) and (3) are specially designed to \ufb02ag common spammers or\u00a0spam messages. While we would have to verify using the data that each of the variables is\u00a0effective, these seem like promising ideas.<\/p>\n

              Table 4\u00a0shows a contingency table for spam and also for the new variable described\u00a0in (1) above. If we look at the 1,090 emails where there was correspondence with the sender in the preceding 30 days, not one of these message was spam. This suggests variable (1) would be very effective at accurately classifying some messages as not spam. With this single variable, we would be able to send about 28% of messages through to the inbox with\u00a0confidence that almost none are spam.<\/p>\n

               <\/p>\n\n\n\n\n\n\n\n\n\n
              Table 4.\u00a0A contingency table for spam and a new variable that represents\u00a0whether there had been correspondence with the sender in the preceding\u00a030 days<\/th>\n<\/tr>\n<\/thead>\n
              <\/td>\nprior correspondence<\/td>\n<\/tr>\n
              <\/td>\nno<\/td>\nyes<\/td>\nTotal<\/td>\n<\/tr>\n
              spam<\/td>\n367<\/td>\no<\/td>\n367<\/td>\n<\/tr>\n
              not spam<\/td>\n2464<\/td>\n1090<\/td>\n3554<\/td>\n<\/tr>\n
              Total<\/td>\n2831<\/td>\n1090<\/td>\n3921<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

              The variables described in (2) and (3) would provide an excellent foundation for distinguishing messages coming from known spammers or messages that take a known form of spam. To utilize these variables, we would need to build databases: one holding email addresses of known spammers, and one holding URLs found in known spam messages. Our access to such information is limited, so we cannot implement these two variables in this textbook. However, if we were hired by an email service to build a spam filter, these would\u00a0be important next steps.<\/p>\n

              In addition to finding more and better predictors, we would need to create a customized\u00a0logistic regression model for each email account. This may sound like an intimidating task, but its complexity is not as daunting as it may at first seem. We’ll save the details for a\u00a0statistics course where computer programming plays a more central role.<\/p>\n

              For what is the extremely challenging task of classifying spam messages, we have made\u00a0a lot of progress. We have seen that simple email variables, such as the format, inclusion of certain words, and other circumstantial characteristics, provide helpful information for spam classification. Many challenges remain, from better understanding logistic regression to carrying out the necessary computer programming, but completing such a task is very\u00a0nearly within your reach.<\/p>\n\n\t\t\t

              \n\t\t\t

              Candela Citations<\/h3>\n\t\t\t\t\t
              \n\t\t\t\t\t\t
              \n\t\t\t\t\t\t\t
              CC licensed content, Shared previously<\/div>