What you’ll learn to do: Evaluate and practice methods of analysis to assess the quality and reliability of a source
Sources come in all shapes and sizes. In order to use them effectively in your report writing, you will have to assess their quality and reliability.
In this section, we will look at a few of the more prominent and well-known techniques; these will be more than sufficient for most of your report writing needs. After you review and practice this material, you should feel confident in evaluating sources and will have the key tools to determine the best ones to use.
- Identify the seven pillars of information literacy
- Discuss the importance of evaluating sources and understanding biases
- Describe the components of the CRAAP analysis process
- Describe techniques to incorporate sources into your writing
People will often talk about the importance of information literacy, but just what does it mean to be information literate? The American Library Association defines information literacy as the ability to “recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information.”
The Society of College, National and University Libraries (SCONUL) 7 Pillars of Information Literacy provides a framework strategy for assembling material for your research project. If you follow this strategy, you can consider yourself information literate.
The pillars are as follows:
- Identify what you need to find out.
- Assess current knowledge, identify your personal knowledge gaps, and understand what types of information are available.
- Plan where you’ll locate data and how you’ll use it.
- Gather data needed, keeping track of where you found your information
- Evaluate both your research process and the information your find; compare and analyze data.
- Manage the information you’ve gathered professionally and ethically—cite all of your sources.
- Present the knowledge you’ve gained, disseminating information to others and apply your knowledge to your life.
Once you’ve evaluated your personal skill set, you can identify gaps in your current material or sources. This can be particularly helpful when determining whether any primary source research is required, and especially whether you would need to conduct any of your own fieldwork (i.e., surveys, interviews, observations, etc.).
This video provides an alternate presentation of the seven pillars, diving into each component a bit deeper.
Perhaps never before in mainstream discussion has the importance of evaluating sources been more important. The effect of “fake news,” and contested information is now a common feature of contemporary life. When evaluating sources, for which a variety of good techniques and rules of thumb exist, we argue that doing so is an essential part of critical thinking, which is the bedrock for good communication and report writing in any field.
The Human Fund
To illustrate the importance of evaluating sources, consider our case study with Martha of The Human Fund. Recall that she read a variety of secondary source materials after developing her research question. In addition to ensuring her sources were scoped properly—that they addressed elements of her research question—she would need to evaluate their authorship, determine how recent and reliable the information is, and understand any bias.
As our world becomes more and more connected by technology, our ability to evaluate and use information has become more difficult, but not impossible. It is essential to understand how these technologies work and how people use them.
Websites, broadly speaking, are perhaps the most difficult sources to evaluate; however, the following tips can act as basic guidelines:
- Consider the URL: generally speaking, .com, .org, .ac.uk, .edu and other more common domains are a bit more likely to have reliable and good content.
- What type of website is it? If it is a blog, social media site, or other tool for personal expression, proceed with caution. Much of the “fake news” problem is driven by sharing questionable material on social media.
- What is the main purpose or claim of the website? Be careful with websites interested in selling downloadable information sources, such as “How to Conduct Research” or “Make Money in Real Estate.” The content may be accurate and useful; however, the sheer abundance of poor sources means you should only use these types of data/sources with caution.
While we’ve boiled the evaluation of websites down to a few key tips, it is actually a complex topic that could fill books. For more information, check out Mike Caulfield’s Web Literacy for Student Fact-Checkers.
This book is (as stated in its own introduction) “an unabashedly practical guide for the student fact-checker. It supplements generic information literacy with the specific web-based techniques that can get you closer to the truth on the web more quickly.”
Understanding Bias in Your Sources
A word on bias: some consider bias to be a problem. However, we might argue here that bias is a normal part of life and human interaction. We are all biased by our upbringing, our experiences, and our perspectives. While any attempt to be objective in your analysis is a good thing, it can be just as useful to acknowledge your biases in your research and arm the reader or consumer of your material accordingly. In a way, this is a form of respect to your readership; you acknowledge their critical thinking role in consuming your material and also acknowledge that ruling out all bias—no matter how professional or scientific one’s research approach might be—is ultimately impossible.
The following video from Chris Flipp illustrates one way to acknowledge and represent bias. It uses a term called Bracketing, that comes from a qualitative research method/idea known as Phenomenology. Bracketing is the act of reviewing and gaining awareness around your preconceived notions of a given topic before pursuing further study. This awareness should help your evaluation of sources and keep you mentally engaged in the review of your own sentiment towards your data.
The following video from Practical Psychology illustrates various biases. Note how these biases could affect your thinking, and consider ways you might apply this awareness to evaluating sources:
There are several tools available when evaluating sources. Perhaps the most common and well known is the (well-titled) CRAAP Analysis. The University of Santa Cruz library has a full breakdown of the CRAAP method. Below is a summarized form; CRAAP stands for:
- Currency: How current is the source?
- Reliability: How important is the information, and has it been consistently presented?
- Authority: What is the source of the information?
- Accuracy: Judged against other sources (which themselves will need appropriate evaluation), how correct is the source?
- Purpose: What is the goal of the source—why was it created?
The following video from Wintec Library also discusses the analysis. Note that the video has no narration.
Lastly, Flinders University has an excellent commentary below on the use of Wikipedia. Depending on your organizational culture, Wikipedia might be sufficient as a source. This is hotly debated, but it all depends on the purpose of your writing. Not all of your business reports need to be lengthy and high caliber items supported by scholarly sources; you might find yourself writing a “quick” report within a day or two where your boss indicates that Wikipedia or some other introductory website (and encyclopedia maybe) is sufficient. It is important to point out that the report, while important, is not the end, it simply a means to making better decisions. Direction from your organization’s decision-makers is key. Flinder’s comments:
A note on Wikipedia. While you certainly would not cite a Wikipedia article as a credible source in your essay, Wikipedia can be a great place to start to get an understanding of a topic. It may also lead you to relevant, high quality resources. Try looking at the references for a Wikipedia page (at the bottom of each Wikipedia page) and assess the quality of the references you find. Many Wikipedia entries will cite scholarly resources (including books and journal articles) in their references, which if appropriate, can then be cited in your essay.
The Human Fund
If we return to Martha’s project, we can imagine her running each secondary source through a CRAAP analysis. Her sources are likely to be a mix of books, magazine articles, videos and other media. When considering what might make for a good source for Martha, we might use a reverse version the CRAAP analysis to illustrate her analysis:
How current is the source?
Martha is not likely to use anything older than 3 to 5 years. There is no hard-and-fast rule here, but civic politics, dynamic life, technology and other features of human experience are likely to render anything too much older than 5 years less helpful.
How important is the information, and has it been consistently presented?
Martha is likely to read or watch materials that are professionally put together and have a linked look and feel to the material.
What is the source of the information?
Martha is likely to look at sources that have a reputation of doing work in the homeless community or have done good work in other similar areas of social work.
Judged against other sources (which themselves will need appropriate evaluation), how correct is the source?
The bottom line here is whether the material has been reviewed by other experts. For scholarly work, we refer to this as “peer-reviewed.” Clearly not all of Martha’s sources need to be peer-reviewed; however, the more of her sources that are, the better her information will be. Flinders University recommends asking the following questions:
- Is it scholarly?
- Is the information supported by evidence?
- Is that evidence referenced by the source?
- Has the content been peer-reviewed or edited by a publisher?
- Can the information be verified by other literature on the same topic?
- Is the tone objective and impartial?
- Is it free from obvious errors such as spelling or grammar?
- Is it written by a scholar with expertise in the field?
What is the goal of the source—why was it created?
This is where intention and bias are more clear. For professional scholarly work, you will often see a note at the end of the document indicating any funding or entities that supported the work. This is there to inform the reader of external influences on the material. A professional author will work to limit his or her bias, or they will use an alternative technique, which is to discuss their bias in their work, and make their agenda clear to the reader.
Case Study: News Media Today
News media sources, such as a TV news report, must be carefully scrutinized. Here we refer to news media as major news networks, such as CNN or Fox News, local television news, and other televised news programs. These sources are unfortunately not above the tension and issues possible with less established sources. An April 2018 video criticizing Sinclair Broadcast Group highlights the dubious nature of our contemporary media landscape.
The following two videos discuss the fallout and tension:
Interestingly, the discussion has become deeply politicized. Whether the material presented is conservative or liberal is interesting, but our purpose here is to highlight how media is a contested space; it is just as difficult to get quality information here as anywhere else, and the information presented through media must be scrutinized like any other source.
As with evaluating websites, evaluating other media sources with the CRAAP method is appropriate. There are, however, a few added questions specific to media:
Who owns the outlet?
The above example is about Sinclair Broadcasting Group. Take a look at WebpageFX’s infographic of ownership of key outlets. As you view the infographic, consider applying the CRAAP Analysis to WebpageFX. Is it a good source? Why or why not?
Figure 1 shows the consolidation of media ownership in the United States over a 30 year span.With consolidation of outlets, the plurality of available media and variation in beliefs and political stances represented has diminished.
In plural democracies like the US and other Western nations, this is certainly a concern. To what extent do the media play a role in homogenizing American life? To what extent should we be concerned this affects our freedom and ability to live and exist with others with whom we might disagree, yet arguably should respect?
How long has the media outlet been around?
While not a perfect test of quality by any means, consider how something like 60 Minutes, the news magazine from CBS, has been on the air for over 40 years. Would this make it more or less reputable than, for example, a newer YouTube news channel, e.g., The Young Turks or The Rubin Report.
In the above discussion, we noted how quickly the use of media becomes politicized. Consider the use of mass media sources carefully given this potential problem. A good technique would be to simply address what you (as the researcher) think the bias might be. As we’ve discussed elsewhere, it’s not as simple as saying all bias is bad, but rather, bias is something we all encounter and have, and it is reasonable and useful to address bias head-on. The researcher should tell the reader what their biases are; the researcher should tell the reader what they think a given source’s biases are as well. The onus is on the reader always to think critically on the material presented.
Using Your Sources
There are three methods for referencing a source in your own text: quoting, paraphrasing, and summarizing.
Direct quotations are words and phrases that are taken directly from another source and then used word-for-word in your text. If you incorporate a direct quotation from another author’s text, you must put that quotation or phrase in quotation marks to indicate that it is not your language.
When writing direct quotations, you can use the source author’s name in the same sentence as the quotation to introduce the quoted text and to indicate the source in which you found the text. You should then include the page number or other relevant information in parentheses at the end of the phrase or use footnotes or end notes to cite the source. (The exact format will depend on the formatting style of your essay).
When paraphrasing, you may put any part of a source (such as a phrase, sentence, paragraph, or chapter) into your own words.
You may find that the original source uses language that is more clear, concise, or specific than your own language, in which case you should use a direct quotation, putting quotation marks around those unique words or phrases you don’t change. It is common to use a mixture of paraphrased text and quoted words or phrases, as long as the direct quotations are inside of quotation marks. You must still cite the source even if you rephrase their idea in your own words.
Summarizing involves distilling the main idea of a source into a much shorter overview. A summary outlines a source’s most important points and general position. When summarizing a source, it is still necessary to use a citation to give credit to the original author. You must reference the author or source in the appropriate citation at the end of the summary.
Integrating Material from Sources
Incorporating sources into your writing uses a general pattern.
- You make a claim or point, e.g., “The Human Fund helps >25% of the downtown Chicago homeless population.”
- You cite evidence, e.g. an MLA citation or APA citation, by embedding a hyperlink in a digital document, by paraphrasing, or by using a direct quote.
- You segue to another claim or new point.
The relationship between claim and evidence is key; for your writing to be effective, you must back up claims or knowledge with quality evidence (sources).
Integrating materials from sources into your own text can be tricky; if we consider the metaphor that writing a paper and including sources is a way of facilitating a conversation about a topic, it helps us to think about how this will work best . When you’re discussing a topic in person with one or more people, you will find yourself referring to outside sources: “When I was watching the news, I heard them say that . . . I read in the newspaper that . . . John told me that . . .” These kinds of phrases show instances of using a source in conversation and ways that we automatically shape our sentences to work references to the sources into the flow of conversation.
Think about this next time you try to work a source into a piece of writing: if you were speaking this aloud in conversation, how would you introduce the material to your listeners? What information would you give them in order to help them understand who the author was, and why their view is worth referencing? After giving the information, how would you then link it back to the point you were trying to make? Just as you would do this in a conversation if you found it necessary to reference a newspaper article or television show you saw, you also need to do this in your essays.
- American Library Association. Presidential Committee on Information Literacy. Final Report. (Chicago: American Library Association, 1989.) ↵
- Caulfield, Mike. "Why This Book?" Web Literacy for Student Fact-Checkers. Web. 30 June 2018. ↵
- Flinders University. "Evaluating your source." Web. http://flinders.libguides.com/evaluate. 18 June 2018. ↵
- Ibid. ↵