LiDAR Instructions

Special Rules For This Task

 Parking Lots:
o If the Ego Vehicle is driving in, exiting, or entering a parking lot, then all cars,

pedestrians, and other objects need to be annotated. If there are any other

parking lots in the scene that aren’t connected to the parking lot the Ego Vehicle

is in, then annotate the parking lot with a ‘No Label Zone’

 Minimum LiDAR Points & occlusion:

o Any object that has 1 LiDAR or RADAR point at any time in the scene needs to be

annotated the entire time they are in camera view, even if they have 0 points. Use

interpolation, camera view, and your best judgment on correct cuboid position.

 Occlusion:
o If a cuboid becomes fully occluded during a frame, still annotate the object as

long as you can be reasonably sure you know the location and shape of the

object. Use your best judgment on correct cuboid position, sizing, and heading.

o If you cannot annotate the object due to high uncertainty (for example very long

occlusions), but can re-identify it later, start a new cuboid path with the next

frame where the object is visible. Be sure to merge the two cuboid paths

afterward, using the Merge This Cuboid Path with Another Path button.

 Cuboid Sizing:
o There should be no more than 20cm of space between the cuboid border and the

closest point on the object.

o Very small items that project off the object (such as side view mirrors and

antennas) should be left out if they will decrease the “tightness” of the bounding

box significantly.

o Exclude any echo, or noise caused by echo, from the cuboid.

 o Include any large objects on the back or on top of the vehicle. This includes

luggage, lumber, bicycles, etc.

 Towed objects:
o Annotate the towing vehicle and the towed object with separate cuboids.

Annotate each cuboid with their respective label. Examples:

 A semi-trailer: A “Vehicle - Truck” and a “Vehicle - Trailer”, with the

“Vehicle - Truck” having the attribute “Towing” and the “Vehicle - Trailer”

the attribute “Being Towed”.

 A tow truck towing a broken car: A “Vehicle - Tow Truck” and a “Vehicle -

Car”, with the “Vehicle - Tow Truck” having the attribute “Towing” and the

“Vehicle - Car” the attribute “Being Towed”.

 An RV towing a car: As “Vehicle - Truck” and a “Vehicle - Car”, with the

“Vehicle - Truck” having the attribute “Towing” and the “Vehicle - Car” the

attribute “Being Towed”.

 An bendy (articulated) bus: Two cuboids (or three, depending on the

number of segments), each labeled “Vehicle - Bendy Bus”, with the front

segment having the attribute “Towing” and the other cuboids the attribute

“Being Towed”.

 Pedestrian with Object: If more than one person are carrying a large object together, divide the
object (approximately) evenly between the persons. Example: If two people are carrying a couch,
create two cuboids, each of them with a pedestrian and half of the couch. Both cuboids get the
label “Pedestrian with object”.

Labels
Dynamic or stationary tracks
 Pedestrian - Adult: Human adult or older child (> 8 year old) on foot. The cuboid must contain all of
the pedestrians’ limbs and slight posture movements. Includes standing, sitting and lying humans.
 Pedestrian - Child: Human child (< 8 year old) on foot. The cuboid must contain all of the childs’
limbs and slight posture movements. Includes standing, sitting and lying humans.
 Pedestrian - Traffic Controller: Human or older child (> 8 year old) on foot, controlling traffic.
Includes traffic police, construction worker controlling traffic, crossing guard, or any other person
 controlling traffic. The cuboid must contain all of the pedestrians’ limbs and slight posture
movements. Cuboid must include signaling device, like batons, flags and signs. Cuboid must also
include any other object the Includes standing, sitting and lying humans.
 Pedestrian - Specialty: Police officer, construction worker, firemen, ambulance personnel, etc.
Does not include traffic police, crossing guards or any other human directing traffic. The cuboid
must contain all of the pedestrians’ limbs and slight posture movements. Includes standing, sitting
and lying humans.
 Pedestrian with Object - Adult: Human adult or older child (> 8 year old) on foot, holding a large
object. Examples include a baby, large box, or open umbrella; pushing a bicycle, motorcycle, cart,
stroller, or trash can; pulling a suitcase. This label also includes a knee scooter. The cuboid must
include both the pedestrian and the object. This does not include pedestrians carrying normal day-
to-day items (e.g. backpack, tote bag, purse). Does not include the leash of a pet.
 Pedestrian with Object - Child: Human child (=< 8 year old) on foot, holding a large object.
Examples include a baby, large box, or open umbrella; pushing a bicycle, motorcycle, cart, stroller,
or trash can; pulling a suitcase. The cuboid must include both the child and the object. This
does not include children carrying normal day-to-day items (e.g. backpack, tote bag, purse).
 Pedestrian with Object - Specialty: Police officer, construction worker, firemen, ambulance
personnel, etc. Human is holding a large object. Examples include a baby, large box, or open
umbrella; pushing a bicycle, motorcycle, cart, stroller, or trash can; pulling a suitcase. The cuboid
must include both the pedestrian and the object. Does not include traffic police, crossing guards or
any other human directing traffic. Includes standing, sitting and lying humans.
 Animal - Dogs/Cats: Small domestic animal such as dogs and cats, on foot. Includes animals on a
leash. If animal is leashed, cuboid must not include the leash. Does not include dead animals
(roadkill). Does not include animals being carried. Example: An adult walking a dog would yield two
cuboids, one “Animal - Dogs/Cats” and one “Pedestrian - Adult”. Both cuboids do not include the
leash.
 Animal - Other: Non-domestic animals on foot. Annotate all animals regardless of size. Includes
larger “domesticated” animals such as sheep, horses, cows and donkeys. Other examples include
birds on foot like ducks, deer, squirrels, skunks and raccoons. If the animal has a rider (like a horse
or donkey), the cuboid must include both animal and rider. Does not include flying birds.
Does not include dead animals (roadkill).
 Vehicle - Trailer: Any object designed to be towed by another vehicle. Object cannot move under
its own power. Includes boat on trailer, flatbed trailer, cement mixer, caravan, shipping container
on a trailer, trailers of 18-wheelers, bike trailer. Does not include boats, shipping containers etc. not
currently on a trailer. Does not include cars being towed. Trailers with other vehicles on top of them
should be annotated with only single cuboid for both the trailer and the vehicles atop of it, labeled
as ‘Vehicle - Trailer’.
 Vehicle - Wheelchair: Wheelchair. Includes both wheelchair with rider and without rider. If the
wheelchair has a rider, the cuboid must contain both the human and the wheelchair.
 Vehicle - Other Personal Mobility: Personal mobility vehicle like a skateboard, Segway, roller
blades. Includes both vehicles with rider and without rider. If the object has a rider, the cuboid must
contain both the human and the vehicle. Does not include wheelchairs, knee scooters, bicycles,
vespas, sitting scooters, all-terrain vehicles, trikes, and motorcycles.
 Vehicle - Bicycle: Bikes, including tandem and recumbent bicycles. Includes both bikes with rider
and without rider. If the bike has a rider, the cuboid must contain both the human and the bike.
 Vehicle - Motorcycle: Motorcycle. Includes both motorcycles with rider and without rider. If the
motorcycle has a rider, the cuboid must contain both the human and the motorcycle. Includes
vespas, sitting scooters. Does not includes all-terrain vehicles, trikes, and motorcycles with
sidecar.
 Vehicle - Other Motorized Riding: Three or four wheeled motorized vehicles with the rider outside
(on top). Includes all-terrain vehicles (ATVs), quads, trikes, and motorcycles with sidecar. Includes
both vehicles with rider and without rider. If the vehicle has a rider, the cuboid must contain both
 the human and the vehicle. Does not includes convertible cars, construction vehicles, tractors, or
any two-wheeled vehicles like motorcycles without sidecar, vespas, sitting scooters.
 Vehicle - Car: Normal sized vehicles, such as cars, pickup trucks, SUV’s, small passenger vans (<
9 people), small flatbed pickup trucks.
 Vehicle - Van: Large passenger vans, moving vans, delivery vans, news vans. Excludes small
passenger minivans.
 Vehicle - Truck: Large truck shaped vehicles such as freight trucks, oil tank trucks, large flatbed
trucks, large delivery trucks, large moving trucks, tractor units, garbage trucks, cement trucks, RVs.
Examples: A semi-trailer would be annotated as: A “Vehicle - Truck” and a “Vehicle - Trailer”, with
the “Vehicle - Truck” having the attribute “Towing” and the “Vehicle - Trailer” the attribute “Being
Towed”. Annotate an RV towing a car as “Vehicle - Truck” and a “Vehicle - Car”, with the “Vehicle -
Truck” having the attribute “Towing” and the “Vehicle - Car” the attribute “Being Towed”.
 Vehicle - Tow Truck: Large vehicle designed to tow other vehicles, like cars or even trucks.
Example: Annotate a tow truck towing a broken car as: A “Vehicle - Tow Truck” and a “Vehicle -
Car”, with the “Vehicle - Tow Truck” having the attribute “Towing” and the “Vehicle - Car” the
attribute “Being Towed”.
 Vehicle - Bus: Bus. Does not include bendy (articulated) busses or school busses.
 Vehicle - Bendy Bus: Bus that is composed of two or more connected segments. Each segment of
the bendy (articulated) bus must be annotated with its own cuboid.
 Vehicle - School Bus: Bus tasked with transporting children. Typically a yellow bus, with flashing
stop signs when dropping off/pickup up children.
 Vehicle - Construction: Vehicles primarily designed for construction. Typically very slow moving or
stationary. Includes cranes, caterpillars, bulldozers, excavators.
 Vehicle - Mobile Traffic Control: Mobile signs or boards (including electronic signs) that are meant
to direct traffic. Typically mounted on a trailer or on the back of a truck. Cuboid must include the
entire object. Does not include traffic control signs or boards that cannot move (i.e., have no
wheels).
 Vehicle - Other: Includes unusual vehicles such as forklifts, tanks, street sweepers, duck boats,
golf carts, tuk-tuks.
 Emergency - Police Vehicle: Police vehicles. Does not include police motorcycles.
 Emergency - Police Motorcycle: Police on motorcycle. The cuboid must include both the police
officer and the motorcycle.
 Emergency - Other: Non-police emergency vehicle. Examples include ambulances and fire trucks.
 Railway Vehicle: Any vehicle that travels on rails, like light rail, tram or train. For trains that consist
of several linked units, annotate each segment with a separate cuboid.
 Other Dynamic Object: Any object in motion left on the driveable surface that is too large to be
safely driven over. Includes balls rolling on the street, runaway strollers or shopping carts.

Stationary-only tracks
 Construction Object - Construction Cones and Poles: Any temporary cone or short temporary pole
(usually orange or striped and used in construction), placed to redirect traffic. Does not include
permanent structures. Only annotate these within 5 meters of the driveable region.
 Construction Object - Construction Zone Signs and Construction Sign Boards: Any construction
related signs or boards (including electronic signs) that are meant to direct traffic.
 Construction Object - Temporary Road Barrier: Temporary small walls used to change lane
assignment or mark road construction zones. Typically plastic and brightly colored about knee or
waist height. Only annotate these within 5 meters of the driveable region.
 Construction Object - Temporary Fence: Temporary walls used to block off construction zones for
buildings. Typically metal or wood and 2-3 meter high. Only annotate these within 5 meters of the
driveable region.
  Construction Object - Other: Any other large (larger than 0.5m high) construction object. Includes
stationary cement mixers, wreckage, generator, wheelbarrow. Does not include smaller items such
as sandbags, tools. Object must be inside the work zone area (marked by indicators like barrier or
cones). Only include these objects if they are in a work zone that occupies part of the normal
drivable region (road work, etc.)
 Static Road Object - Pushable or Pullable: Objects that can be moved by a pedestrian. Includes
garbage bins with wheels, shopping carts, wheelbarrows, moving carts etc. Object can be on the
road, on the sidewalk, or away from the driveable surface. Object must be larger than 50cm high.
Object must currently be stationary.
 Static Road Object - Other: Any object larger than 15cm obstructing the normal drivable region.
Includes dead animals, tree branches, trash bags, dumpsters, mattresses, ladder etc. Includes
shipping containers not on a trailer. Update: Include temporary objects on driveways.

No Label Zone: apply this label to parking lots.

 DO NOT label any other classes within the “No Label Zone”.
 When creating “No Label Zones”, create one large cuboid to cover the entire area.
 DO NOT make small strips to cover sections of a zone.
 Update: DO NOT annotate driveways into parking lots.
 Do not annotate parking lots that do not have any vehicles in them.
 Update: Do not annotate parking lots that are right next to the street (without driveway or curb
seperating them).
 No Label Zone should cover all stationary vehicles, alleys, or driveways within the parking lot.
 Make as many cuboids as necessary to cover the entire area without going over buildings,
sidewalks, or any other area that does not represent the zone being annotated.
 Cuboids are permitted to overlap slightly in order to fully contain the zone.
 DO label objects in parking lot driveways as they are exiting or entering parking lots.
o Update: Objects that enter a “No Label Zone” need to be fully annotated all the

way through.

o Start labeling objects when part of it exits a “No Label Zone”.

 DO label objects on sidewalks adjacent to but outside the “No Label Zone”.
 Bus stop roundabouts are not parking lots. Therefore, we should always annotate the vehicles in
that area.
 Gas Stations are not parking lots. We should always label each vehicle in the area as individual
vehicles.

Attributes
 For each vehicle, select the attribute "Is the vehicle towing or being towed?"

o Being Towed: The vehicle/trailer is being towed by another vehicle.

o Towing: The vehicle is towing. Trailers cannot be towing.

o Neither: The vehicle is not towing or being towed.

 For animals and vehicles that have the rider outside, select the attribute "Does the

vehicle or animal have a rider?"

o With Adult Rider: Vehicle currently has a rider. Rider is an adult or older child (>8

years old).

o With Child Rider: Vehicle currently has a rider. Rider is a child (=< 8 years old).

o Without Rider: Vehicle does not have a rider.

 For each object, select the attribute "What percent of this object is occluded?"

o Occlusion is visual (RGB camera) occlusion

o Select from the answers 0-20%, 20-40%, 40-60%, 60-80%, 80-100%.

o If the object is visible in more than one camera, use the least occluded answer

(lowest percentage).

o If the object is not visible in any camera, use the answer 80-100%.

o If the cuboid is not an object (No Label Zone), or if the object is too far to reliably

answer, select Can’t Tell.

o Annotate the occlusion for each frame individually.

 For each vehicle, select the attribute "Are the object’s turn signal indicator lights activated?"

o Signal indicator light (Left vs Right) is reasoned from the driver of the vehicle (not

ego or camera position).

o Use Both if the vehicle’s hazard lights are on.

o Use Can’t tell when the object is too far to reliably estimate the light status, or

does not have signal indicator lights.

o Annotate the light status for each frame individually.

 For each vehicle, select the attribute "Are any lights on?"

o Select On if any lights are on at any point in the sequence.

o Select Off when NO lights are on at point in the sequence

 o Use Can’t tell when the object is too far to reliably estimate the light status, or

does not have lights.

 For each vehicle, select the attribute "Are the object’s brake lights activated?"

o Annotate the light status (On, Off) for each frame individually.

o Use Can’t tell when the object is too far to reliably estimate the light status, or

does not have brake lights.

 For each vehicle, select the attribute "Are the object’s reverse lights activated?"

o Answer On if the reverse lights are on at ANY frame in the sequence.

o Use Can’t tell when the object is too far to reliably estimate the light status, or

does not have reverse lights.

 For each vehicle, select the attribute "Is the object parked?"

o A vehicle is parked if stopped or standing still, whether occupied or not, and does

not have the ability or intention to start moving again.

o Select No

o Select Yes if any of the following conditions apply for EVERY frame in the

sequence:

 Vehicle is parked on the side of a curb.

 Vehicle is parked in a parking lot.

 Vehicle is parked with or without a driver inside.

 Vehicle is stopped on the side of a freeway with or without hazard lights.

 Vehicle is parked in a driveway.

o Select No if any of the following conditions apply at ANY frame in the sequence:

 Movement of the vehicle is temporary obstructed (traffic light, traffic jam).

 The vehicle is waiting to enter or leave a gate booth lane.

 The vehicle is obeying the direction of a police officer, port security, or

other commission employee to control traffic.

  The vehicle is on the side of road to make way for construction or

emergency personnel (fire truck, ambulance, etc.)

 Vehicle is loading/unloading passengers or goods.

 Vehicle is stopped while making an (unprotected) turn in a traffic jam.

o Use Can’t tell when the object is too far to reliably estimate the parked status.