Zeya Guidelines

Project Overview:

In this task you will be asked to research and evaluate claims stated in various pieces of content (e.g., articles,
links, photos, or videos)

Why are we doing this?

We are doing this to better understand the information shared on Facebook, in terms of how individuals
assess its credibility after doing some research.

Big points to keep in mind:

This document contains ever-evolving and ever-improving guidance on how to think about the task's
questions and what the response options mean for evaluating article, photo or video content.

This isn't intended to be a strict set of guidelines, because this isn't intended to be a purely objective
task. There will be some variability between people in how they approach the task and how they
respond to the questions, and that is by design. However, try as much as possible to follow the
instructions laid out below and to answer each question to the best of your ability.

Starting Point:

This set of tasks will ask you to evaluate the central claim presented in a piece of content.

A central claim is a factual statement related to the content's main point or purpose that is falsifiable. By
falsifiable, we mean that you could assess the claim to be “true” or “false” (or somewhere in between)
based on evidence.

Example [IMAGE]

Photos show a giant floating ice photographed in Presumpscot River in Westbrook.

1
Zeya Guidelines

For example, for the content above you should focus on evaluating whether there is evidence that
indicates that these photos are actually photos of the giant ice disk described in the text of the status
update (e.g., rather than being photos of another place or thing). The claim is not only whether there
was “a giant ice disk is floating in the Presumpscot River Westbrook” as stated in the caption.

Example [URL]
Rolling Stone reported: “Anthony Bourdain Dead at 61 of Apparent Suicide.”
https://www.rollingstone.com/culture/culture-news/anthonybourdain-dead-at-61-of-apparent-suicide-
629356/

2
Zeya Guidelines

The main purpose of the article is to report that Anthony Bourdain has recently died and how he died.

The central claim in this case is that Anthony Bourdain died of suicide. In this case, the central claim has
two parts:

1) that Anthony Bourdain has died and

2) that his death was a suicide.

A given piece of content may contain many separate, distinct claims, but we want you focus on
identifying and evaluating the central claim (which is related to the main point of the content).

For example, although the above article about Anthony Bourdain's death also contains many statements
about his life and legacy (e.g., that he “worked in the dregs of the New York City restaurant scene for
years” and that he later became “the executive chef at the French bistro Brasserie Les Halles”), in this
task you should focus on evaluating the central claim only.

Disqualifying Issues

There are three reasons why content is disqualified from being rated:

1. It was not possible to access the content using the link, photo or video provided. This may be
because:
o The link is broken or the and the photo/video or caption does not render properly

3
Zeya Guidelines

o The content is missing, or you encounter a 'page does not exist' message
o Your web browser blocks the link or identifies it as unsafe
o The content is in a different language from what you are assigned to review
2. You are unable to read the full content because it is paywalled or requires a subscription. The
website asks you to pay for content, or it requires a subscription to access content and you do
not have one.
3. The content does not contain a falsifiable claim (i.e., the content does not contain a central
claim that can be assessed as “true” or “false” [or somewhere in between] based on evidence.)
Below are some examples of content types that often do not include a falsifiable claim.
o However, if you can identify a factual central claim in an item of these types, then you
should complete the rating task for that item rather than marking it as disqualifying
 Quizzes
 Even or sales flyers (e.g., an ad for a car dealership sales events posted on that
dealer's website, a sales flyer for a department store sale on that store's
website)
 Petitions
 Song lyrics
 Recipes
 Opinion-centered “listicles” (e.g., a page or list form article about “5 beauty
products you need to try now” or “25 of the funniest celebrity tweets from this
week”)
 Humorous memes or joke posts that do not make statements about specific
people, entities, or facts/statistics

Part 1: Context

POTENTIAL IMPACT OF THE CLAIM

PURPOSE: The purpose of the following question is to gauge how much it would matter if the central
claim turned out to be false.

Guidance for the following question:

• To answer the following question, you should consider all components of the piece of content
you are evaluating, including the photo itself, any text/caption in the photo, and the content
caption (displayed above the photo in the review tool, if present).
• You should be able to answer the following question relatively fast, looking only at the page title
and perhaps the first few lines of text or video/photo description.
• To answer this question, you may consider the potential impacts on the general public or
specific audiences who see the content, as well as on the individuals, groups, or organizations
described in the content.

4
Zeya Guidelines

• If you are rating a multi-part claim, you should provide the impact rating for part of the claim
that seems to matter the most (i.e., the highest impact claim)

Question 1.1 To what degree would it matter to someone's livelihood, decision-making, safety,
political and social worldview, or well-being if the information in this central claim was wrong or
misleading?
• Not a lot - It would not matter if the information were wrong or misleading.
• Somewhat - It may matter if the information were wrong or misleading. o e.g., sports, local
events, celebrity gossip or entertainment related content.
• A Great Deal - It would matter if the information were wrong or misleading. It could impact the
lives of those who see it or of individuals described in the claim.
o e.g., medical, legal, financial, political, commercial, or other topics where being
wrong might matter.
o (IMAGE) Content containing information, facts, or statistics that are important for
fostering an informed group of people, community, or nation. This could include
information related to government processes, natural disasters, international
relations, etc.

CONTENT TYPE

PURPOSE: The purpose of the following two questions is to provide context of the type of content you're
evaluating.

Guidance for the following two questions:

• To answer the following questions, you should read through the title and first few lines
of the article (or watch a bit of the video)
• We ask you to tell us whether the content you are evaluating is Parody or Satire:
o Parody refers to humorous imitations of serious literature, media, etc.
o Satire refers to humorous content that uses irony, sarcasm, or ridicule to
expose, denounce, or make fun of human stupidity, immorality, and/or
shortcomings.
• We also ask whether the content you are evaluating is Opinion:
o Opinion here means someone's personal opinion as written in the photo caption
or text OR both the Opinion section of professional news coverage (e.g., an “op-
ed” or editorial article) as well as someone's personal opinion as written in a
blog, social media post, personal webpage, or related non-journalistic coverage.
o However, some content may contain both opinion and factual statements (e.g.,
content may contain a factual central claim to support an opinion.)
o Therefore, you will still be asked to identify and evaluate the central claim for
content labeled as opinion.
5
Zeya Guidelines

o Additionally, not all personal webpages, blog posts or social media posts contain
opinion. You should only label content from these sources as opinion if it is
sharing an opinion

Question 1.2 Would a reasonable person conclude that the linked content is satire or parody?

• Yes, it is clear or obvious that the main purpose is satire or parody

• Yes, but it is not obvious that the main purpose is satire or parody
• No, it does not appear to be or is definitely not satire or parody
• Unable to determine.

Question 1.3 Would a reasonable person conclude that the linked content is opinion?
• Yes, it is opinion
• No, it does not appear to be opinion
• Unable to determine

Part 2: Claim Identification and Evidence Search

CLAIM IDENTIFICATION

PURPOSE: The purpose of the following questions is for you to: (1) identify the central claim made by
the content that you're evaluating, and (2) find evidence from another source regarding that central
claim, that would allow you to assess whether it is true, false, or somewhere in between

Guidance for the following question:

• The central claim should be one sentence or idea from the content you are evaluating, that is a
factual, falsifiable statement related to that content's main point or purpose.
o In photo content, the central claim is usually described in the content text (i.e., the text
caption of the photo and/or in text that is overlaid on the photo). You should consider
both types of text as well as the photo itself when identifying the central claim for a
piece of content.

6
Zeya Guidelines

7
Zeya Guidelines

8
Zeya Guidelines

9
Zeya Guidelines

10
Zeya Guidelines

11
Zeya Guidelines

o Usually the central claim is described or referenced in the piece of content's headline or
title.
o Sometimes the headline or title of the content contains a mixture of fact and opinion
(e.g., “Heartbreaking: baby bear who survived hunting wound cries while he's left to
die”). In these cases, the central claim is the factual portion of the statement (e.g.,
“baby bear who survived hunting wound cries while he's left to die.”).
o Sometimes the headline or title of the content is a question (e.g., “Is Austin, TX getting a
new name?”). In these cases, the central claim is the answer to that question provided
by content (e.g., “Austin, TX is not getting a new name”).
o Sometimes the headline or title of the content is phrased as an opinion statement or
doesn't contain a factual claim, but the content itself is rooted in a factual central claim
(e.g., an article titled “You should be worried about black mold” that reports that black
mold causes serious respiratory health problems). For these cases, you should submit
the central factual claim that the content is supported by or based on.
o Sometimes part of a claim may be omitted from a headline or title (e.g., “Why 'getting
lost in a book' is so good for you, according to science”), and needs to be found in the
body of the content.

12
Zeya Guidelines

o Sometimes a central claim is about something someone said (e.g., “After visiting
Colorado, Indiana lawmaker says medical marijuana is right for Indiana”). In this case,
the claim you should evaluate is whether that person made that statement.
o Sometimes a central claim is about something someone said AND whether what they
said is true or false (e.g., “Roubini Falsely Claims Crypto Currency is Centralized”). In this
case, the claim you should evaluate is whether the source's conclusion about what they
said is correct, as well as whether the person said that statement in the first place.

Question 2.1 Central Claim: What is the central claim you have identified from the original piece of
content?

EVIDENCE SEARCH

PURPOSE: The purpose of the following step is to find evidence from another source regarding the
central claim that you're evaluating, that would allow you to assess whether the central claim is true,
false, or somewhere in between.

Guidance for the following question:

• We ask you to find evidence for or against the central claim you've identified. By evidence, we
mean a statement, photo, video, audio, or statistic relevant to the central claim. This evidence
can either support the initial claim or go against it.
• To find evidence about the claim, please use the following approaches:
o [IMAGES] Try a reverse image search to locate sources that provide additional
information about the image and claim you are evaluating.
 To do a reverse image search you will need to save a copy of the image you are
evaluating. To do this, right click the image, select “save image as”, and save the
image. *Note: You must delete any images saved for reverse image search after
you are done searching. Please save all images to a single folder that you delete
regularly (at least daily).
 After you have saved the image, go to the reverse image search engine of your
choosing (e.g., https://images.google.com/) and upload the image to search.
 After searching, you should click into specific pages in the search results to see
what those sources say about the photo/its claim.
 Finally, select the source with the highest quality evidence to evaluate the claim
in the next steps. *Note: please make sure the source is discussing the same
photo as the one your evaluating, not another similar image.
 You should look for sources that provide additional information or reporting
about the photo and claim (for example, sources that describe what is shown in
the photo, when or where the photo was taken, whether the photo has been
altered/digitally manipulated, and/or whether the claim is true)

13
Zeya Guidelines

 *Note: You should not use a link to an identical copy of the photo content you
are evaluating that includes no additional information/reporting as an evidence
source.
o [URL] You could try a general web search on a search engine of your choice, using some
keywords from the central claim as your search terms.
 for example, in the case of the Anthony Bourdain example we described earlier:
 you might search: “Anthony Bourdain dead suicide” using your
preferred search engine.
 then click into specific pages in the search results to see what those
sources say about the claim
 finally, select the source with the highest quality evidence to evaluate
the claim in the next steps
o You could try a keyword search within the website of a particular source you trust as an
authority on information for topics related to the claim you're evaluating.
 for example, in the case of the claim “New study finds you can starve cancer
cells to death by cutting sugar from your diet”, you might choose to search for
information from Mayo Clinic.
 You could search “cut sugar starve cancer cells” in the search bar of the
Mayo Clinic's website
 You could also search “cut sugar starve cancer cells site: mayoclinic.org
“ at google.com

*Note: If you typically search using terms in a language that is different from the content you are
evaluating (e.g., you are rating Hindi content but you typically search using English search terms): If you
are struggling to find evidence, you should try searching using the same language used in the content.

Question 2.2 Search: What are the keywords you used to research this claim? If you searched multiple
times, enter just the keywords you used on your final/successful search. If you used a reverse image
search, please enter “reverse image search” in the text box.

Part 3: Evidence Description

EVIDENCE DESCRIPTION

PURPOSE:

[IMAGE] The purpose of the following questions is for you to describe the evidence that you found.

[URL] The purpose of the following question is for you to identify the central claim of the information
that you're evaluating.

14
Zeya Guidelines

Guidance for the following question:

• In the following questions, we ask you to describe the highest quality piece of evidence you
found about the central claim in your search. High quality evidence is evidence that is:
o From a source you trust (you should try to find evidence from a source you trust,
whenever possible). A trusted source might be:
 A source that you are familiar with and that you have found to provide reliable
information in the past
 A source that is a widely accepted authority on the topic (this may be based on
your prior knowledge or you may choose to do a quick search engine query for
the author or source to see if this is the case)
 A source that cites other widely accepted authorities or sources you trust to
support their claims
 A source that generally provides high quality, professional content (this may be
based on your prior knowledge or you may choose to look through additional
content or articles from the source to see if their other content is generally
credible and professionally written)
o Provides clear, credible facts/evidence for or against the claim
o speaks directly to the same event, time period, or photo as the original claim (Unless the
evidence you find is showing that the photo is of a different time, place or person than
described by the claim).
 You should check to be sure evidence you use is not outdated or about previous
events, especially for claims about current events.
• If you cannot find evidence about the claim from a source that you trust, you should try to find
the most relevant evidence about the claim you can find from any source, even one you don't
trust.
o In the task, you will be asked to note whether you do not trust the evidence provided by
the source because:
 You distrust the source (i.e., you are familiar with the source and know it
typically provides unreliable information)
 You do not know enough about the source to feel you can either trust or
distrust it
• If you are rating a multi-part claim and you feel that you need different sources of evidence to
support or disprove different parts of the claim, please paste each source into a separate entry
field.
o If you feel that one piece of evidence properly speaks to all parts of the central claim,
you only need to paste one evidence link.
• Finally, the following are not appropriate evidence sources:
o The same content you are currently evaluating—either from the same source (if the
source is identifiable) or an identical version of the content you are evaluating from

15
Zeya Guidelines

another source (e.g., you may not use a tweet or Facebook post of the same photo and
message you are evaluating as an evidence source).
o Other content from the same source as the content you are evaluating (if the source is
identifiable, e.g., from a logo on the photo or URL in the content text)
o The URL for your full search engine results (e.g.,
https://www.google.com/search?q=ice+disk+Westbrook+River)
o Social media posts, unless:
 You are rating a claim about whether a person said something on social media
and you are providing a link to the post referenced in the claim (however,
because people may delete social media posts, a claim shouldn't be labeled
false only because the tweet/post is no longer present)
 The post is from the official account for a person or organization who is an
authority on the claim's subject (e.g., a post from local emergency responders' office
would be an appropriate source for a claim about a local emergency in their district)

Question 3.1 Which of the following best describes the highest quality evidence you found about the
claim in your search?

• I found evidence from a source that I trust - I'm familiar with the source and know it usually
provides reliable information; OR I have researched the source and it seems to be an accepted
authority that generally provides high quality, professional content.
• I found evidence, but it's from a source I don't know enough about to trust or distrust - I'm
unfamiliar with the source and couldn't determine whether I should trust the information it
provides.
• I found evidence, but it's from a source I distrust - I'm familiar with the source and know it
provides unreliable or false information; OR I have researched the source and it seems to have a
negative reputation and/or generally provides low quality, false, or misleading content.
• I did not find any evidence about the claim - I have tried using reverse image search AND
keyword search (including a search using terms in the same language as the content I'm
evaluating), but I was unable to find any evidence about the claim.

Question 3.2A Evidence Link:

• Please paste the link for the highest quality evidence you found (Paste only the text of the
URL/link here. Do not include additional text from the webpage, article, etc.)

Question 3.2B Optional Additional Evidence Links:

• [OPTIONAL] If you are evaluating a multipart claim and found different evidence sources for
different parts of the claim, please paste the additional sources here
• [TEXT BOX]
• [TEXT BOX]

16
Zeya Guidelines

Question 3.3 Evidence Statement

• What evidence related to the claim does this link provide? For articles, please copy and paste 1
to 3 statements below. For images, videos or audio, please summarize the evidence in the first
text box.
• [TEXT BOX]
• [TEXT BOX]

Part 4: Evaluation

CENTRAL CLAIM EVAULATION

PURPOSE: The purpose of the following questions is for you to make an assessment about the central
claim you are evaluating based on the evidence you described above, and to provide some context for
that decision.

Guidance for the following question:

• Considering the research, you did about the central claim, we would like you to provide your
best judgement about whether the claim is True, False, or Misleading.
• The following definitions may help your decision making:
o True - based on your research, the central claim you are evaluating is factually accurate.
You found facts/evidence that support the claim.
o False - based on your research, the central claim is factually inaccurate. You found
facts/evidence that do not support the claim. [IMAGES] This includes evidence that
shows:
 The photo has been taken out of context (i.e. - you find evidence that shows
that the photo is of a different person/place/object/time than described in the
central claim)
 The photo has been digitally manipulated (i.e. - you find evidence that shows
the photo has been digitally manipulated/altered/morphed to show something
different than the original photo)
 The photo contains a fake quote (i.e. - the claim is about something someone
said, and you find evidence that shows the quote was never said by the person
described in the claim).
 The claim as stated in the content/its text is factually inaccurate o Misleading-
based on your research, the central claim you are evaluating is:
 Only partially accurate (e.g., it contains significant inaccuracies alongside some
accurate information, or it contains multiple photos and some are
authentic/accurately represented while others are misappropriated or altered)
 Only partially supported by evidence (e.g., only part of the claim is confirmed,
but there is no evidence about the other part(s) of the claim)
17
Zeya Guidelines

 About something someone said or claimed, and the quote is correct but is taken
out of context or has been significantly misrepresented
 Represented in significantly different ways in the text of the photo caption vs.
the text on the photo/image, and not all aspects of the claim are supported by
evidence based on your research (e.g., the aspect of the claim presented in the
caption is not supported by evidence, but the aspect of the claim presented in
the text on the photo/image is).
o Not Sure - based on the evidence you found in your research (or lack thereof), you do not feel
you can judge whether the central claim is true, false, or misleading.
• For photo content evaluated as False or Misleading, we will also ask you to indicate, if
applicable, if the content is False/Misleading because:
o The photo has been taken out of context (the photo shows a different
person/place/object/event/time than what is described by the claim)
o The photo has been digitally manipulated (the photo has been digitally
manipulated/altered/morphed to show something different than the original photo).
• After the final evaluation, there is also a space for you to provide optional additional information
about your evaluation. You can use this space to share:
o Issues you are having in general with the rating task
o Specific issues you encountered when rating this claim o additional context for why you
did or didn't trust a source (or why it was too difficult to decide if you should do either)
o Any additional information you'd like to share about your evaluation of this claim o
Anything else you think would be useful for us to know
Q4.1 Based on the research you did, what is your assessment of the central claim in the original
content?

• True - based on your research, the central claim you are evaluating is factually accurate. You
found facts/evidence that support the claim.
• Misleading - based on your research, the central claim: is only partially accurate (e.g., contains
some significant inaccuracies alongside some accurate information), is only partially supported
by evidence, takes out of context or significantly misrepresents something someone said or
claimed, or is represented in significantly different ways in the photo caption vs. the text of the
image OR the body of the original content you are evaluating (and only one of these is
supported by evidence).
• False - based on your research, the central claim is factually inaccurate. You found
facts/evidence that indicate the claim is without merit (e.g., you find evidence that the photo is
taken out of context, the photo has been digitally manipulated, the photo contains a fake quote,
or that claim as stated in the content/its text is factually inaccurate).
• Not Sure - based on the evidence (or lack thereof) you found in your research, you do not feel
you can judge whether the central claim is true, false, or misleading.

18
Zeya Guidelines

[IMAGE] Q4.2: [if Misleading or False is selected for Q4.1]: Based on the research you did, do either of
the following describe the content you are are evaluating? [This question is optional - select only if
applicable]

• Taken out of Context Photo - the photo shows a different person/place/object/event/time than
what is described by the claim
• Digitally Manipulated Photo - the photo has been digitally manipulated/altered/morphed to
show something different than the original photo

[URL] Q4.2: OPTIONAL: You may use the space below to briefly state any additional information or
details about your evaluation that you'd like to share:

• [TEXT BOX]

19