Chapter 12 Other Methods of Data Collection and Analysis

12.1 Focus Groups

Focus groups resemble qualitative interviews in that a researcher may prepare an interview guide in advance and interact with participants by asking them questions. But anyone who has conducted both one-on-one interviews and focus groups knows that each is unique. In an interview, usually one member (the research participant) is most active while the other (the researcher) plays the role of listener, conversation guider, and question asker. Focus groupsPlanned discussions designed to elicit group interaction and gather information about the researcher’s topic of interest., on the other hand, are planned discussions designed to elicit group interaction and “obtain perceptions on a defined area of interest in a permissive, nonthreatening environment” (Krueger & Casey, 2000, p. 5).Krueger, R. A., & Casey, M. A. (2000). Focus groups: A practical guide for applied research (3rd ed.). Thousand Oaks, CA: Sage. In this case, the researcher may play a less active role than in a one-on-one interview. The researcher’s aim is to get participants talking to each other and to observe interactions among participants.

Focus groups are typically more dynamic than interviews. The researcher takes the role of moderator, posing questions or topics for discussion, but then lets the group members discuss the question or topic among themselves. Participants may ask each other follow-up questions, agree or disagree with one another, display body language that tells us something about their feelings about the conversation, or even come up with questions not previously conceived of by the researcher. It is just these sorts of interactions and displays that are of interest to the researcher. A researcher conducting focus groups collects data on more than people’s direct responses to her or his questions; the group interaction is a key focal point. Due to the nature and unpredictability of group interaction, and the fact that focus group researchers generally want to draw out group interaction, focus groups tend to be qualitative rather than quantitative.

There are numerous examples of sociological research using focus group methodology. In their 2008 study, for example, Amy Slater and Marika Tiggemann (2010)Slater, A., & Tiggemann, M. (2010). “Uncool to do sport”: A focus group study of adolescent girls’ reasons for withdrawing from physical activity. Psychology of Sport and Exercise, 11, 619–626. conducted six focus groups with 49 adolescent girls between the ages of 13 and 15 to learn more about girls’ attitudes toward their own and other girls’ participation in sports. In order to get focus group participants to speak with one another rather than with the group facilitator, the study’s interview guide contained just two questions: “Can you tell me some of the reasons that girls stop playing sports or other physical activities?” and “Why do you think girls don’t play as much sport/physical activity as boys?” In another focus group study, Virpi Ylanne and Angie Williams (2009)Ylanne, V., & Williams, A. (2009). Positioning age: Focus group discussions about older people in TV advertising. International Journal of the Sociology of Language, 200, 171–187. held nine focus group sessions with adults of different ages to gauge their perceptions of how older characters are represented in television commercials. Among other considerations, the researchers were interested in discovering how focus group participants position themselves and others in terms of age stereotypes and identities during the group discussion. In both examples, the researchers’ core interest in group interaction could not have been assessed had interviews been conducted on a one-on-one basis; thus the focus group method was the ideal choice in each instance.

The preceding examples come from the work of academics who have used focus groups as their method of data collection. But focus groups have proven quite useful for those outside of academia as well. In fact, this method is especially popular among applied researchers. Market researchers use focus groups to gather information about the products or services they aim to sell. Government officials and political campaign workers use them to learn how members of the public feel about a particular issue or candidate. One of the earliest documented uses of focus groups comes from World War II when researchers used them to assess the effectiveness of troop training materials and of various propaganda efforts (Merton & Kendall, 1946; Morgan, 1997).Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage. Market researchers quickly adopted this method of collecting data to learn about human beliefs and behaviors. Within social science, the use of focus groups did not really take off until the 1980s, when demographers and communication researchers began to appreciate their use in understanding knowledge, attitudes, and communication (Morgan, 1997).Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage.

Focus groups share many of the strengths and weaknesses of one-on-one qualitative interviews. Both methods can yield very detailed, in-depth information; are excellent for studying social processes; and provide researchers with an opportunity not only to hear what participants say but also to observe what they do in terms of their body language. Focus groups offer the added benefit of giving researchers a chance to collect data on human interaction by observing how group participants respond and react to one another. Like one-on-one qualitative interviews, focus groups can also be quite expensive and time-consuming. However, there may be some time savings with focus groups as it takes fewer group events than one-on-one interviews to gather data from the same number of people. Another potential drawback of focus groups, which is not a concern for one-on-one interviews, is that one or two participants might dominate the group, silencing other participants. Careful planning and skillful moderation on the part of the researcher are crucial for avoiding, or at least dealing with, such possibilities. The various strengths and weaknesses of focus group research are summarized in Table 12.1 “Strengths and Weaknesses of Focus Group Research”.

As mentioned, careful planning and skillful moderating are two crucial considerations in the effective use of focus groups as a method of data collection. In some ways, focus groups require more advance planning than other qualitative methods of data collection such as one-on-one interviews, where a researcher may be better able to control the setting and the dialogue, or field research, where “going with the flow” and observing events as they happen in their natural setting is the primary aim and time is less limited. Researchers must take care to form focus groups whose members will want to interact with one another and to control the timing of the event so that participants are not asked nor expected to stay for a longer time than they’ve agreed to participate. The researcher should also be prepared to inform focus group participants of their responsibility to maintain the confidentiality of what is said in the group. But while the researcher can and should encourage all focus group members to maintain confidentiality, she should also clarify to participants that the unique nature of the group setting prevents her from being able to promise that confidentiality will be maintained.

Group size should be determined in part by the topic of the interview and your sense of the likelihood that participants will have much to say without much prompting. If the topic is one about which you think participants feel passionately and will have much to say, I think a group of 3–5 is ideal. Groups larger than that, especially for heated topics, can easily become unmanageable. Some recommend that a group of about 6–10 participants is the ideal size for focus group research (Morgan, 1997);Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage. others recommend that groups should include 3–12 participants (Adler & Clark, 2008).Adler, E. S., & Clark, R. (2008). How it’s done: An invitation to social research (3rd ed.). Belmont, CA: Thomson Wadsworth. The size of the focus group is ultimately your decision as the researcher. When forming groups and deciding how large or small to make them, take into consideration what you know about the topic and participants’ potential interest in, passion for, and feelings about the topic. Also consider your comfort level and experience in conducting focus groups. These factors will help you decide which size is right in your particular case.

It may seem counterintuitive, but in general, it is better to form focus groups consisting of participants who do not know one another than to create groups consisting of friends, relatives, or acquaintances (Agar & MacDonald, 1995).Agar, M., & MacDonald, J. (1995). Focus groups and ethnography. Human Organization, 54, 78–86. The reason for this is that groups who know each other may share some take-for-granted knowledge or assumptions. In sociological research, it is precisely the taken-for-granted that is often of interest; thus the focus group researcher should avoid setting up interactions where participants may be discouraged to question or raise issues that they take for granted. However, groups should not be so heterogeneous that participants will be unlikely to feel comfortable talking with one another.

Focus group researchers must carefully consider the composition of the groups they put together. In his text on conducting focus groups, Morgan suggests that “homogeneity in background and not homogeneity in attitudes” (p. 36) should be the goal, since participants must feel comfortable speaking up but must also have enough differences to facilitate a productive discussion (1997).Morgan, D. L. (1997). Focus groups as qualitative research (2nd ed.). Thousand Oaks, CA: Sage. Whatever composition a researcher designs for her or his focus groups, the important point to keep in mind is that focus group dynamics are shaped by multiple social contexts (Hollander, 2004).Hollander, J. A. (2004). The social context of focus groups. Journal of Contemporary Ethnography, 33, 602–637. Participants’ silences as well as their speech may be shaped by gender, race, class, sexuality, age, or other background characteristics or social dynamics, all of which might be suppressed or exacerbated depending on the composition of the group. Hollander suggests that researchers must pay careful attention to group composition, must be attentive to group dynamics during the focus group discussion, and should triangulate multiple methods of data collection in order to “untangle participants’ responses and their relationship to the social contexts of the focus group” (p. 632).

In addition to the importance of advance planning, focus groups also require skillful moderation. While a researcher certainly doesn’t want to be viewed as a stick-in-the-mud or as overly domineering, it is important to set ground rules for focus groups at the outset of the discussion. Remind participants that you’ve invited them to participate because you want to hear from all of them. Therefore the group should aim to let just one person speak at a time and avoid letting just a couple of participants dominate the conversation. One way to do this is to begin the discussion by asking participants to briefly introduce themselves or to provide a brief response to an opening question. This will help set the tone of having all group members participate. Also ask participants to avoid having side conversations; sharing thoughts about or reactions to what is said in the group is important and should not be limited to only a few group members.

As the focus group gets rolling, the moderator will play a less active role than he does in a one-on-one interview. There may be times when the conversation stagnates or when you, as moderator, wish to guide the conversation in another direction. In these instances, it is important to demonstrate that you’ve been paying attention to what participants have said. Being prepared to interject statements or questions such as “I’d really like to hear more about what Sally and Joe think about what Dominick and Ashley have been saying” or “Several of you have mentioned ____. What do others think about this?” will be important for keeping the conversation going. It can also help redirect the conversation, shift the focus to participants who have been less active in the group, and serve as a cue to those who may be dominating the conversation that it is time to allow others to speak.

In sum, focus groups are a useful method for researchers who wish to gather in-depth information about social processes. Focus groups are similar to one-on-one qualitative interviews in many ways, but they give researchers the opportunity to observe group dynamics that cannot be observed in one-on-one interviews. Historically, focus group research was more commonly used by applied researchers than by academics, though in recent decades social scientists from all domains have discovered the usefulness of focus groups for gaining understanding of social processes and have begun using this method of data collection in their studies.

12.2 Experiments

Experiments are an excellent data collection strategy for those wishing to observe the consequences of very specific actions or stimuli. Most commonly a quantitative research method, experiments are used more often by psychologists than sociologists, but understanding what experiments are and how they are conducted is useful for all social scientists, whether they actually plan to use this methodology or simply aim to understand findings based on experimental designs. An experimentA method of data collection designed to test hypotheses under controlled conditions. is a method of data collection designed to test hypotheses under controlled conditions. Students in my research methods classes often use the term experiment to describe all kinds of empirical research projects, but in social scientific research, the term has a unique meaning and should not be used to describe all research methodologies.

Several kinds of experimental designs exist. In general, designs considered to be “true experiments” contain three key features: independent and dependent variables, pretesting and posttesting, and experimental and control groups. In the classic experimentThe effect of a stimulus is tested by comparing an experimental group to a control group., the effect of a stimulus is tested by comparing two groups: one that is exposed to the stimulus (the experimental groupThe group of participants who receive the stimulus in an experiment.) and another that does not receive the stimulus (the control groupThe group of participants who do not receive the stimulus in an experiment.). In other words, the effects of an independent variable upon a dependent variable are tested. Because the researcher’s interest lies in the effects of an independent variable, she must measure participants on the dependent variable before and after the independent variable (or stimulus) is administered. Thus pretesting and posttesting are both important steps in a classic experiment.

One example of experimental research can be found in Shannon K. McCoy and Brenda Major’s (2003)McCoy, S. K., & Major, B. (2003). Group identification moderates emotional response to perceived prejudice. Personality and Social Psychology Bulletin, 29, 1005–1017. study of people’s perceptions of prejudice. In one portion of this multifaceted study, all participants were given a pretest to assess their levels of depression. No significant differences in depression were found between the experimental and control groups during the pretest. Participants in the experimental group were then asked to read an article suggesting that prejudice against their own racial group is severe and pervasive, while participants in the control group were asked to read an article suggesting that prejudice against a racial group other than their own is severe and pervasive. Upon measuring depression scores during the posttest period, the researchers discovered that those who had received the experimental stimulus (the article citing prejudice against their same racial group) reported greater depression than those in the control group. This is just one of many examples of social scientific experimental research.

In addition to the classic experimental design, there are two other ways of designing experiments that are considered to fall within the purview of “true” experiments (Babbie, 2010; Campbell & Stanley, 1963).Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth; Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally. They are the Solomon four-group design and the posttest-only control group design. In the former, four groups exist. Two groups are treated as they would be in a classic experiment. Another group receives the stimulus and is then given the posttest. The remaining group does not receive the stimulus but is given the posttest. Table 12.2 “Solomon Four-Group Design” illustrates the features of each of the four groups in the Solomon four-group design.

Finally, the posttest only control group is also considered a “true” experimental design though it lacks any pretest group. In this design, participants are assigned to either an experimental or a control group. Individuals are then measured on some dependent variable following the administration of an experimental stimulus to the experimental group. In theory, as long as the control and experimental groups have been determined randomly, no pretest is needed.

Time, other resources such as funding, and even one’s topic may limit a researcher’s ability to conduct a true experiment. For researchers in the medical and health sciences, conducting a true experiment could require denying needed treatment to patients, which is a clear ethical violation. Even those whose research may not involve the administration of needed medications or treatments may be limited in their ability to conduct a classic experiment. In social scientific experiments, for example, it might not be equitable or ethical to provide a large financial or other reward only to members of the experimental group. When random assignment of participants into experimental and control groups is not feasible, researchers may turn to a preexperimental designExperimental design used when random assignment of participants into experimental and control groups is not feasible. (Campbell & Stanley, 1963).Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally. However, this type of design comes with some unique disadvantages, which we’ll describe as we review the preexperimental designs available.

If we wished to measure the impact of some natural disaster, for example, Hurricane Katrina, we might conduct a preexperiment by identifying an experimental group from a community that experienced the hurricane and a control group from a similar community that had not been hit by the hurricane. This study design, called a static group comparisonAn experiment that includes a comparison control group that did not experience the stimulus; it involves experimental and control groups determined by a factor or factors other than random assignment., has the advantage of including a comparison control group that did not experience the stimulus (in this case, the hurricane) but the disadvantage of containing experimental and control groups that were determined by a factor or factors other than random assignment. As you might have guessed from our example, static group comparisons are useful in cases where a researcher cannot control or predict whether, when, or how the stimulus is administered, as in the case of natural disasters.

In cases where the administration of the stimulus is quite costly or otherwise not possible, a one-shot case studyAn experiment that contains no pretest and no control group. design might be used. In this instance, no pretest is administered, nor is a control group present. In our example of the study of the impact of Hurricane Katrina, a researcher using this design would test the impact of Katrina only among a community that was hit by the hurricane and not seek out a comparison group from a community that did not experience the hurricane. Researchers using this design must be extremely cautious about making claims regarding the effect of the stimulus, though the design could be useful for exploratory studies aimed at testing one’s measures or the feasibility of further study.

Finally, if a researcher is unlikely to be able to identify a sample large enough to split into multiple groups, or if he or she simply doesn’t have access to a control group, the researcher might use a one-group pre-/posttestAn experiment in which pre- and posttests are both taken but there is no control group. design. In this instance, pre- and posttests are both taken but, as stated, there is no control group to which to compare the experimental group. We might be able to study of the impact of Hurricane Katrina using this design if we’d been collecting data on the impacted communities prior to the hurricane. We could then collect similar data after the hurricane. Applying this design involves a bit of serendipity and chance. Without having collected data from impacted communities prior to the hurricane, we would be unable to employ a one-group pre-/posttest design to study Hurricane Katrina’s impact.

Table 12.3 “Preexperimental Designs” summarizes each of the preceding examples of preexperimental designs.

As implied by the preceding examples where we considered studying the impact of Hurricane Katrina, experiments do not necessarily need to take place in the controlled setting of a lab. In fact, many applied researchers rely on experiments to assess the impact and effectiveness of various programs and policies. You might recall our discussion of the police experiment described in Chapter 2 “Linking Methods With Theory”. It is an excellent example of an applied experiment. Researchers did not “subject” participants to conditions in a lab setting; instead, they applied their stimulus (in this case, arrest) to some subjects in the field and they also had a control group in the field that did not receive the stimulus (and therefore were not arrested).

Finally, a review of some of the strengths and weaknesses of experiments as a method of data collection is in order. A strength of this method, particularly in cases where experiments are conducted in lab settings, is that the researcher has substantial control over the conditions to which participants are subjected. Experiments are also generally easier to replicate than are other methods of data collection. Again, this is particularly true in cases where an experiment has been conducted in a lab setting.

As sociologists, who are especially attentive to how social context shapes social life, are likely to point out, a disadvantage of experiments is that they are rather artificial. How often do real-world social interactions occur in the same way that they do in a lab? Experiments that are conducted in applied settings may not be as subject to artificiality, though then their conditions are less easily controlled. Experiments also present a few unique concerns regarding validity. Problems of external validityThe extent to which the conditions of an experiment adequately represent those of the world outside the boundaries of the experiment. might arise when the conditions of an experiment don’t adequately represent those of the world outside the boundaries of the experiment. In the case of McCoy and Major’s (2003)McCoy, S. K., & Major, B. (2003). Group identification moderates emotional response to perceived prejudice. Personality and Social Psychology Bulletin, 29, 1005–1017. research on prejudice described earlier in this section, for example, the questions to ask with regard to external validity are these: Can we say with certainty that the stimulus applied to the experimental group resembles the stimuli that people are likely to encounter in their real lives outside of the lab? Will reading an article on prejudice against one’s race in a lab have the same impact that it would outside of the lab? This is not to suggest that experimental research is not or cannot be valid, but experimental researchers must always be aware that external validity problems can occur and be forthcoming in their reports of findings about this potential weakness. Concerns about internal validityThe extent to which we can be confident that an experiment’s stimulus actually produced the observed effect or whether something else caused the effect. also arise in experimental designs. These have to do with our level of confidence about whether the stimulus actually produced the observed effect or whether some other factor, such as other conditions of the experiment or changes in participants over time, may have produced the effect.

In sum, the potential strengths and weaknesses of experiments as a method of data collection in social scientific research include the following:

12.3 Ethnomethodology and Conversation Analysis

Though not unique methods of data collection per se, ethnomethodology and conversation analysis are unique enough, and prominent enough in sociology, that they warrant some dedicated attention in this text. EthnomethodologyThe study of how people construct and sustain their realities through conversation and gestures. refers to the study of everyday reality. Rather than assume that the purpose of social science is to understand some objective reality, ethnomethodologists investigate how people construct, prolong, and maintain their realities. The term ethnomethodology was coined by sociologist Harold Garfinkel (1967),Garfinkel, H. (1967). Studies in ethnomethodology. Englewood Cliffs, NJ: Prentice Hall. who, as described in his 2011 obituary, was a “sociologist who delved into the minutiae of everyday life” (Lynch, 2011).Lynch, M. (2011, July 13). Harold Garfinkel obituary. The Guardian. Retrieved from http://www.guardian.co.uk/education/2011/jul/13/harold-garfinkel-obituary Ethnomethodology’s emphasis on the everyday, and on ordinary people’s methods for producing order in their social worlds, is perhaps its most distinctive characteristic.

An example of ethnomethodological research is C. M. Scharff’s (2008)Scharff, C. M. (2008). Doing class: A discursive and ethnomethodological approach. Critical Discourse Studies, 5, 331–343. study of how young feminist women “do” social class. In her study, Scharff examines data from interviews with 40 German and British young women to understand how they “think, talk, and feel about feminism” (p. 334). By focusing in on language, talk, and interaction, Scharff argues that her account is ethnomethodological in nature. Kevin Whitehead (2009)Whitehead, K. (2009). “Categorizing the categorizer”: The management of racial common sense in interaction. Social Psychology Quarterly, 72, 325–342. also takes an ethnomethodological approach in his study of the social organization of race. In Whitehead’s words, he considers “one mechanism by which racial categories, racial ‘common sense,’ and thus the social organization of race itself, are reproduced in interaction” (p. 325).Whitehead, K. (2009). “Categorizing the categorizer”: The management of racial common sense in interaction. Social Psychology Quarterly, 72, 325–342. To study these processes, Whitehead analyzed the interactions and practices of participants in an employment “race training” workshop and found that individuals use race as a framework from which to understand their own and others’ actions, thereby reproducing race as a relevant social category.

Conversation analysisThe study of talk, including how talk progresses, how it is facilitated, and how it may be impeded. grew out of ethnomethodology (Schutt, 2006)Schutt, R. K. (2006). Investigating the social world: The process and practice of research (5th ed.). Thousand Oaks, CA: Sage. and thus shares its focus on the construction of reality as opposed to the discovery of reality. Conversation analysts focus specifically on talk in interaction: how talk progresses, how it is facilitated or impeded, how turns are taken in talk, and how these processes both shape and are shaped by social context. In conversation analysis, what people say is just as important as how they say it. Also important are the pauses people take in expressing themselves and how or whether they interrupt themselves or others while talking. Conversation analysts might study recordings of court proceedings or legislative debates to learn about the social construction of law and punishment. They might also study more simple interactions, such as a conversation between two people meeting for coffee.

Some research methods texts include coverage of ethnomethodology and conversation analysis in their presentations of qualitative data analysis (Schutt, 2006).Schutt, R. K. (2006). Investigating the social world: The process and practice of research (5th ed.). Thousand Oaks, CA: Sage. It makes sense to do so; both represent unique approaches to analyzing qualitative data. Yet they also rest upon particular ontological and epistemological assumptions that set them apart, in some ways at least, from more traditional mechanisms of analysis such as coding.