6/19/2018 Exp: Event Realness English 2.

0 - Quip

Exp: Event Realness English 2.0

VERSION 2.0,  6/20/18

Events Team
The mission of the Events Team is to help people do more things with their friends in the real world.  To that end, this team works on the tools and resources that allow
users to discover events that they might like to attend.  

Event Realness Experiment

One specific manner in which the Events Team enables this discovery process is by making a distinction between user-created events that are Real (happening in the
real world at a specific time) and those that are Non­Real (not happening in the real world).  Providing labels for these events helps train an algorithm to be more
accurate.  Furthermore, labeling these events does not delete them from Facebook.

Header
Upon entering the experiment, you will be provided with the following header.  Details for each area are defined below:

1. Event Name: The title of the event that is being evaluated.

2. Event Cover Photo:  Image that is uploaded to the event by user upon event creation.
3. Event Details: Details that are filled out by user during event creation.  This includes time, and owner info.
4. Event Description
5. Reject Button:  Rejects the job due to wrong language
6. Submit Button:  Completes the job

Experiment Flow
Below is an example of the possible questions that the rater can encounter when completing a job in this experiment. 

https://fb.quip.com/8SN7AZnb8z2I 1/8
6/19/2018 Exp: Event Realness English 2.0 - Quip

Experiment Instructions
1. Open job
2. Read the information presented in the header and evaluate based on the question below.  Do not open up the Facebook page of the event.  We only evaluate
based on the header information.
3. Answer the question below.

(Note: External links/sources are not allowed, we only evaluate the information provide of the event page.)

Q1: Is this event real?

As the only question within this experiment, we are determining if the event within the header is real or non­real.  When answering this question, please consider the
following:

● Cancelled events are still marked as real as long as they fulfill our Real event requirements.
● Evaluate the job by using only the event details in the header and/or the picture next to the header.  
● The language of the experiment should be present in either the event title or description.  If neither of these is present, then Reject the job.

ANSWER CHOICES

YES (Job Ends):  

● This is a Real event:  The reference event fulfills all of our requirements for time and activity.  If a user wanted to attend the event, they have sufficient
information to do so.

NO (Job Ends):  

● This Event is Non­Real:  The reference event does NOT fulfill our requirements for time and activity.  The event could be any of the following: fake, online,
does not provide sufficient information for a user to attend.

What is a Real Event?

A real event is a gathering of people at a pre-determined time for a specific activity.  We utilize these two elements in this experiment to define a Real event on Facebook.
 Essentially, the main question to ask when determining the “realness” of an event is the following:

Is this an activity in the real­world and if so, is there sufficient 
information for me to attend the event?
Requirements for each element are below.

Time
A pre-determined time for when people are to meet is required for an event to be real. When considering time of an event, please consider the following:

● Events that last up to 3 days from start to end are acceptable.  This accounts for the numerous events on Facebook that happen during the weekend (such as
festivals).
● Events that have a duration of 4 days or more are only considered events if there is enough activity within each day to justify having one event created for
the entire duration.  Investigate the event details as well as the activity in question.  Utilize your best judgement for cases that are not completely obvious.  
○ For example, a Yoga Class that lasts from Sunday through Saturday is unrealistic because typically such classes only last several hours rather than occur
across over multiple days.
○ However, a Yoga Retreat that lasts from Sunday through Saturday is acceptable because retreats typically possess durations of that length.
○ For further example of common acceptable events which are 4 days or longer, see the “Time Guide” below.
● In general, ensure that the time details listed for an event make sense for the actual activity
○ For example, a band performance should not be a month long in duration.
● Inspect the description for phrases such as “time to be determined at a later date.”  These indicate that the event creator still does not have a 
confirmed time for the event, therefore deeming it non-real (we don't want users to see events with inaccurate information).
○ It is acceptable, however, for the band lineup to be determined as long as the date and time are real.

https://fb.quip.com/8SN7AZnb8z2I 2/8
6/19/2018 Exp: Event Realness English 2.0 - Quip
● If an event such as a festival has only 2-3 hours of activity per day, considering it's a festival, we'll still accept that as Real.  
● If there is a long term artistic event, such as exhibition that lasts 3 months, there needs to be a job for each day it is open, not one for all the days it is open. If there
are multiple times at which to attend a show there needs to be a different job for each showing.
○ For example, the Mona Lisa and other DaVinci work is on display at the Moma for 4 months, however the job captures a single day it is real.
○ However, if Magic Mike is being performed at 6 pm and then again at 9 pm each performance needs its own job, you cannot have a single job going from 6
pm to midnight.

TIME GUIDE

Common event times found in the Event Realness Experiment

Typical Event
Event Types Event? Reason
Duration
Summer camps, trade shows, Typically these types of events have enough
1 4 days or more festivals, seminars, street markets, Yes activity that lasts the entirety of a day and lasts
retreats over several days.
2 Fitness Classes (Crossfit, Yoga, It would be unreasonable to find fitness calsses
1-3 Hours Yes
etc.) that last more than a day
3 3-6 Hours Concerts, Performances Yes

Activity 
The activity of an event represents what participants will do when they meet at the time of the event.  When considering the activity of an event, please consider the
following:

● The activity of an event does not need to be explicitly stated in the description.  This information can also be inferred through the event title or event
cover photo.
● The main intent of the event post does not need to be the activity itself.  If an event post is heavily promoting selling tickets to an event, we still accept this as
real because the event is still the main activity.
● We want to avoid events that more closely resemble an advertisement rather than a unique activity for the user.  This includes events created to promote a
sale, tour package, cause, etc.
● Use your best judgement to evaluate if an activity will be happening in real life or was created just for fun or as a joke/meme.
○ For example, for a Celion Dion Concert at a Best Buy in Idaho, it is unreasonable to take such an event seriously! 
● Recurring events are considered as a real event, as long as the event page it self only captures a single instance.
○ For example, a movie theater holds a Midnight B-Movie Showing every Wednesday. If the date and time are only for one specific Wednesday it is real. It's
usually created by a participated vendor/seller/special figure to let people know that they will be there/what's going on.
○ However, if the movie theater has the event last for a whole month, but practically the event is only happening every Wednesday (nothing is happening the
other days of the week) then it is non­real.

ACTIVITY GUIDE

Common activities found in the Event Realness Experiment

Event Category Common Examples Notes

1 Common Non-Real cases occur where an event is created to gain

Theater Performance Bands, Comedians, Dancers, Singers, 
support for an artist performing at a specific town/country
2 Nightclubs/Parties

3 Football, Baseball, Soccer, High

Athletic Contests
School, etc.
4 Commonly found in all locales.  In most cases, such events are
Festivals/Fair Da Lat Festival, Tet Festival, Book Fair, 
considered Real
These are fine as long as there is sufficient information to indicate
5 Protests/Marches that people will be gathering together in the real world in order to
protest or march
6 Ensure that the information provided is for one instance of a
Fitness Classes
reoccurring class and not a whole series
7 Same as above - ensure that the event is created for one instance
Workshops
rather than the whole series
8
Auctions Considered Real in most cases.

9 These are fine as Real events as long as it's a job that captures a
Art Exhibits single day of the exhibit, since such an event will commonly have
a duration that lasts over the course of multiple days
10 Garage Sales will be marked as real because they are distinct
Non-Commercial
Yard sale, warehouse sale, garage sale individual community gathering events when they are not
Garage Sale
considered a business promotional events.
11 The very first time people have the opportunity to visit this new
Grand Opening business. Grand Seasonal Opening and Grand Re-openings do
not qualify as real.
12

https://fb.quip.com/8SN7AZnb8z2I 3/8
6/19/2018 Exp: Event Realness English 2.0 - Quip
13
Fan Meeting  Considered Real Event

What is a Non-Real event?

Depending on the locale, you will encounter many events that may initially seem real, but then in reality do not satisfy our event realness requirements.  When
evaluating these posts, a helpful question to frame your thinking is as follows:

Is this an event I can attend and, if so, do I have sufficient information to 
do so?
That being said, the table below is a guide to the types of common non­real events that you may find in this experiment.  By no means is this list comprehensive;
therefore, refer back to the aforementioned question as needed.

Promotional
These types of will represent the majority of Non-Real cases that you'll encounter during this experiment.  These events typically will more closely resemble
advertisements than they do actual activities.   Common examples of such events are as follows:

● Company is using the event creation tool to advertise a discount/sale

● Travel agency creating an event for the purpose of selling a special tour package that is highly touristic and commercial. If it's an experience type of event, or an
educative/ecological walk, even tough they might call it tour, if it's something that would still hold interest for locals it's Real. Often non profit organizations,
government agencies or educators will host these, and that's a good indicator these should be marked Real.
● Seasonal/Grand Re-openings: 
○ A seasonal opening/grand re-opening does not qualify as real because the public have had the opportunity to visit the venues previously. 

https://fb.quip.com/8SN7AZnb8z2I 4/8
6/19/2018 Exp: Event Realness English 2.0 - Quip

Insufficient Time
These are events that have a start and end time that does not align with the actual activity for the event.  One common instance is a yoga class that has a duration that
lasts over several days.  Since this is not a realistic time duration for such an event, this would be considered Non-Real.  Currently, the occurrence of these events is rare.

Call to Action
These events are created for the sole purpose of receiving donations, votes, or any other type of contribution from a user without having any type of in person activity.
 However, if the event's activity is, for instance, a fundraiser that occurs at a real world location (such as a restaurant), this would be acceptable as Real.

Online
These events are Non-Real because they do not happen in the real world.  Therefore, if a user discovered this event on Facebook, he/she would be unable to attend the
event.  Common examples of online events:

● Live streams over the internet

● Radio/TV shows

Gather Support for Performer

Often, events will be created for the sole purpose of gathering support for the appearance of a specific artist/act/performer at a city, country, etc.  Since these events do
not confirm yet any concert happening at the real world, they are Non-Real.

Meme/Just for Fun/Spam

These are events that are created just for fun.  They may appear to be Real events (happen in the real world and have a date and time), but upon inspection of the
activity it is evident that is could not possibly be real.  Such events and their identification are up to your best judgement.  However, some common themes that occur
when labeling these events as Non-Real:

● Bands that are performing in unrealistic situations

https://fb.quip.com/8SN7AZnb8z2I 5/8
6/19/2018 Exp: Event Realness English 2.0 - Quip
● Events centered around popular internet memes

Major Holidays
Event creators will sometimes utilize the event creation tool to announce a large scale celebration or holiday.  Since these are typically announcements rather than
lead to a specific activity, we label these as Non-Real.

https://fb.quip.com/8SN7AZnb8z2I 6/8
6/19/2018 Exp: Event Realness English 2.0 - Quip

Time Details to Be Determined

These are events that involve real-world activities, but most likely were created with the purpose of publicizing the event without having concrete details.  Without
confirmed details, a user would not have the necessary knowledge to attend the event.  Such events are indicated by phrases like “time to be determine” within the
description.  Since the time of an event is required upon creation, it is likely that the event creator purposefully entered incorrect information during this process.

Schedule of Future Events

In some cases, event creators will create one singular event to advertise a schedule of upcoming events. These are Non-Real because an event should only represent one
activity at a time.

Appendix
Appendix 1.0 ­ Event Creation Tool

This is the Event Creation Tool that users encounter when creating a Facebook event.  Ultimately, the information that they provide through this UI is what raters will
see when completing the Event Realness experiment.

https://fb.quip.com/8SN7AZnb8z2I 7/8
6/19/2018 Exp: Event Realness English 2.0 - Quip

https://fb.quip.com/8SN7AZnb8z2I 8/8