CSIDC01 Poznan

BlueEyes
Human Operator Monitoring System
Motivation What is BlueEyes ?
BlueEyes
Human-Operator Monitoring System
Team members Pawe Kowalik Piotr Kubiaczyk Krystian Nowak Stanisaw Osiski Tomasz Puak Jan Kniat, Ph. D.
Physiological foundations
System design System implementation
Future improvements
Project work summary System demonstration
Team mentor
Pozna University of Technology, Poland

http://www.put.poznan.pl
BlueEyes
Agenda
Motivation What is BlueEyes (and what is not) ? Physiological foundations System design System implementation Future improvements Project work summary System demonstration
Future improvements
BlueEyes
Motivation
Human error - a frequent reason of catastrophes and ecological disasters temporal indisposition weariness oversight Monitoring of conscious brain involvement Capable sensors available Lack of complex solution exploiting wireless link technology, database systems and AI algorithms
Future improvements
BlueEyes
What is BlueEyes ?
The complex solution for human-operator monitoring: conscious brain involvement monitoring triggering user-defined actions data recording
Future improvements
What is BlueEyes not?

Doesnt predict nor interfere with operators thoughts Cannot force directly the operator to work
BlueEyes
Who is BlueEyes for ?

Generic control rooms Power station Captain bridge Flight control centers Operating theatres anesthesiologists Common application A simpler system version for drivers
Future improvements
BlueEyes
BlueEyes - benefits
Prevention from dangerous incidents Minimization of ecological consequences financial loss a threat to a human life The reconstruction of the course of operators work
Future improvements
BlueEyes
Human brain
Preconscious brain Conscious brain
Future improvements
alogical automatic
manages the knowledge logical reasoning makes decisions
BlueEyes
Conscious brain - operation modes
Future improvements
Exploratory mode
Planning mode
BlueEyes
Eye movements
Retina - central vision (macula)
Central vision and the conscious brain

Saccade - a rapid eye jump to a new location assigned by the conscious attention process.
Future improvements
Saccades presence Exploratory mode
BlueEyes
The eye movement sensor

Eye position measuring - direct infrared oculography Oxy- and deoxyhemoglobin measurment Two axial accelerometer Ambient light sensor
Future improvements
Jazz Multisensor
BlueEyes
Wires, wires, wires...

A serious limitation of: mobility comfort safety
Future improvements
BlueEyes
Personal Area Network !

Solution: A wireless Personal Area Network
Future improvements
BlueEyes
Design objectives
A mobile data acquisition device
System design Objectives System overview DAU CSU System implementation Future improvements Project work summary System demonstration
A reliable real-time buffering, processing and recording A clear visualization interface Mechanisms for creating custom data analyzers To ensure a possibility of distributing the processing among a number of computers
BlueEyes
System overview
Data Acquisition Unit Central System Unit
Atmel 89C52 microcontroller Jazz Multisensor
Bluetooth
Bluetooth
Connection Manager Data Analysis
Visualization Module Data Logger
Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system.
BlueEyes
DAU - features
Data Acquisition Unit
Atmel 89C52 microcontroller Jazz Multisensor
Bluetooth
Lightweight Runs on batteries - low power consumption Easy to use - does not disturb the operator working ID cards for operator authorization Voice transmission using hardware PCM codec
BlueEyes
CSU - features
Connection management Data processing Visualization Data recording
Central System Unit
Bluetooth
Connection Manager Data Analysis
Visualization Module Data Logger
Access verification System maintenance
BlueEyes
CSU - saccade detector

Saccade detector and conscious brain involvement level calculation
Raw eye movement data
Velocity
System design Objectives System overview DAU CSU System implementation
Eye movement velocity 250 200 150 100 50 0 -50 -100 -150 -200 -250 0 8000
saccade
Differentiation
16000
Time [ms]
24000
32000
Visual attention level

60
Attention level (Lva)
Future improvements Project work summary System demonstration
50 40 30 20 10 0 0 8000 16000 Time [ms] 24000 32000
Conscious brain involvement level
BlueEyes
CSU - Visualization Module

Raw and processed data visualization using: VU-meters pie-charts time series
audio and video data playback
BlueEyes
Data security
Only registered mobile devices can connect to the system Bluetooth connection authentication Bluetooth connection encryption Access rights restrictions Personal and physiological data encryption
BlueEyes
Implementation - DAU
Made from scratch by the team
System design System DAU CSU Future improvements Project work summary
implementation
System demonstration
Bluetooth module inside Jazz sensor interface Microphone and earphone sockets 5-key keyboard Two lines, 8 characters LCD Powered from 6 AA batteries
BlueEyes
Implementation - DAU
implementation
PCM codec board Microcontroller board
Batteries and Bluetooth module voltage level monitor
BlueEyes
Implementation - CSU
Standard MDI user interface Custom-built Bluetooth communication routines Intel USB camera ID card programmer Developed using Visual C++ 6.0 Deployment environment: Windows 2000, MS SQL Server
implementation
BlueEyes
Future improvements
DAU small CMOS camera to monitor the operators point of gaze single PCB (SMD technlogy) low voltage ICs - LiIO batteries power CSU data mining algorithms advanced database encryption using e.g. AES algorithm
Future improvements Project work summary
BlueEyes
Project work summary

Bluetooth and PAN technology acquired New skills on designing software and hardware systems gained A mobile device implemented 31 KLOC of C++ code and 4 KLOC of 8051 assembler code written All parts of the Project Kit efficiently used New approach to system safety proposed and implemented
BlueEyes
BlueEyes
Future improvements
BlueEyes
Thank you for your attention
BlueEyes
Pozna University of Technology, Poland

http://www.put.poznan.pl
BlueEyes
DAU - components
microphone earphone
MC 145483 PCM codec Jazz Multisensor Beeper LCD display LED indicators
Bluetooth module
Atmel 89C52 microcontroller

UART
MAX232
BlueEyes
ID card interface
ID card
Simple keyboard
Data Acquisition Unit
UART
BlueEyes
CSU - Connection Manager

Operator Data Manager
Bluetooth
... ...
Bluetooth Connection Manager
Bluetooth
Initializes Bluetooth Data demultiplexer Data producer queues modules Inquires for new Separated Raw Data devices in range streams data Operator Buffer Manager Alerts Establishes Alert buffer connections ... Supports operator ... roaming Feeds Operator Raw Data Operator Buffer Manager with Manager Alerts Alert buffer physiological data
BlueEyes
CSU - Data Analysis

Custom data analyzers - decision tree based classifiers Example: Operator gesture recognition
(simplified) Vertical Acceleration variance value high Horizontal Acceleration variance high
N
Horizontal Acceleration variance high
N
YES
Y
NO
No answer
BlueEyes
CSU - Data Analysis

Custom data analyzers - decision tree based classifiers Automatic decision tree coefficients generation - C4.5 based algorithm
Data samples Assigned category
Yes No answer ... Yes No No answer
...
Decision tree coefficients
BlueEyes
CSU - intermodule communication

Uses buffered thread-safe queues and producer-consumer scheme
register at
Data consumer
creates Data producer
Data queue
Data Data consumer & Processing consumer producer
Data queue
Data consumer
Data Data consumer & Processing consumer producer Data queue
BlueEyes
CSU - intermodule communication

Connection Manager Operator manager (OM)
...
OM
OM
Alerts
Separated physiological data streams
Data Analysis
Visualization Module
Data Logger
Processed data
Recorded (off-line) data

CSIDC01 Poznan

Încărcat de

Informații document

Descriere originală:

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

CSIDC01 Poznan

Încărcat de

Drepturi de autor:

Formate disponibile

BlueEyes

Human Operator Monitoring System

Motivation What is BlueEyes ?

Pozna University of Technology, Poland

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

What is BlueEyes not?

Who is BlueEyes for ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

manages the knowledge logical reasoning makes decisions

Conscious brain - operation modes

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Central vision and the conscious brain

Saccades presence Exploratory mode

The eye movement sensor

Motivation What is BlueEyes ?

Wires, wires, wires...

Motivation What is BlueEyes ?

Personal Area Network !

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Atmel 89C52 microcontroller Jazz Multisensor

Connection Manager Data Analysis

Visualization Module Data Logger

Motivation What is BlueEyes ?

Atmel 89C52 microcontroller Jazz Multisensor

Motivation What is BlueEyes ?

Connection Manager Data Analysis

Visualization Module Data Logger

Access verification System maintenance

CSU - saccade detector

Motivation What is BlueEyes ?

Visual attention level

Attention level (Lva)

Future improvements Project work summary System demonstration

50 40 30 20 10 0 0 8000 16000 Time [ms] 24000 32000

Conscious brain involvement level

CSU - Visualization Module

Motivation What is BlueEyes ?

audio and video data playback

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

PCM codec board Microcontroller board

Batteries and Bluetooth module voltage level monitor

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Project work summary

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Motivation What is BlueEyes ?

Thank you for your attention

Pozna University of Technology, Poland

Atmel 89C52 microcontroller

Data Acquisition Unit

CSU - Connection Manager

Bluetooth Connection Manager