Deep Learning for Medical Data Analysis

Yeha Lee (yeha.lee@vuno.co)

 Contents

1. Deep Learning & Representation Learning

2. Background in Medical Domain

3. Deep Learning in Medicine

4. VUNO-net and VUNO-med

2
GPU Technology Conference 2015

3
A.I. is Eating the World

4
Success of Deep Learning

5
Brain-Inspired Learning

6
Representation Learning

7
 Hierarchical Representation

The structure of human brain for

visual information processing.
Objects

Data
Parts

Edges

8
 Deep Learning as a Record Breaker

Jump-up in Image Classification Jump-up in Speech Recognition 83.9%

0.9
85%
0.85
≥ 10% 75%
0.8
0.75 65%

0.7 Cloud computing big data deep learning

55%
0.65
0.6
2008 2014
SuperVision

SuperVision

ISI

ISI

ISI

ISI

LEAR-XRCE

LEAR-XRCE

LEAR-XRCE

LEAR-XRCE
University of Amsterdam 2004
OXFORD_VGG

OXFORD_VGG

OXFORD_VGG

1998
XRCE/INRIA

XRCE/INRIA
1990s

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

ILSVRC-2012 Task 1 (Classification) Result - 1.2 million training images, 1,000 classes Performance improvement in phoneme recognition (TIMIT Data)

9
Deep Learning as a Record Breaker

10
 Key Driving Factor

Big Data Computing Power Algorithm

11
Background in Medical Domain
 Current Clnical Workflow
6 year 1 year 4 year
Medical General Medical
School Admission Intern Resident
Practitioner Expert
11 Years

Lack of Experience Collective Intelligence?

13
 Human as a Doctor
•Quality of diagnosis is depend on…
•Physical and psychological conditions
•Lack of experience of patient cases
•Difference of theoretical background

Condition Experience Background

14
 Mistakes and Misdiagnosis
2015.6.17 뉴욕타임즈 기사
초기 환자를 폐렴으로 오진, 확진환자 162명의 절반이 해당 병원에서 발생

주요 암의 오진 원인 및 현황 (자료:소비자보호원)

15
Innovations in Medical History
1846
The Use of Anesthesia

1870s
Germ Theory of Disease

1970s
Randomized Clinical Trials

2015
Data-driven Healthcare

16
Mother of Medical Big Data?

What is the cause of death in field hospitals?

“Infection”
Dramatical decrease of death rate in 6 months

42% -> 2%

17
 Too Much Data, Really?

Digitalization is still going on

98%
Penetration rate of PACS: of Medical Data is Image!
Korea 70%, Germany 58%, Big medical data never retrieved
USA 52%, Japan 40%

18
How Machine Learning Work?

I made
Feature!

Domain Expert

19
Deep Learning in Medicine
 Why Deep Learning in Medical?
•Varieties of
•Modalities (x-ray, CT, PET, MRI, fMRI, Ultrasonography …)
•Target organs / tissues (Brain, Breast, Prostate …)

•For successful medical image analysis

•Extraction of most informative features is crucial

•Several Tasks
•Classification / Detection / Segmentation / Regstration …

21
 Diabetic Retinopathy Detection
• The most common diabetic eye disease, occurs when
blood vessels in the retina change
• the leading cause of blindness in the working-age
population

22
 Diabetic Retinopathy Detection
• Kaggle Competition (661 teams)
• Identify signs of diabetic retinopathy in eye images”
• Predicating the presence of diabetic retinopathy in each image on
a scale of 0 to 4

CNN based 

Classifier

1 : Mild DR 2 : Moderate DR

0 : No DR

3 : Severe DR Proliferative DR

Sample Data Score Board

23
 Breast Cancer-related Cell Detection
•Deep max-pooling CNN to detect mitosis in breast
histology images
•2012 ICPR Mitosis Detection Contest

D. C. Ciresan et. al., Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks, MICCAI, 2013

24
Breast Cancer-related Cell Detection

25
 Brain Image Segmentation
•Segment brain into CSF, WM and GM

W. Zhang et. al. Deep Convolutional Neural Networks for Multi-modality Isointense Infant Brain Image Segmentation, NeuroImage, 2015

26
 Brain Image Segmentation
•Using CNN with Multi-modal MR Images for Brain Image
Segmentation
•To segment brain into CSF, WM and GM, multi-modal inputs(T1, T2, FA)
are used
•CNN is trained with multi-modal brain MR patches for patch
classification
•From MR images of 8 infants 10,000 patches are generated for
training

27
VUNO-net and VUNO-med
 VUNO-net

Pylearn
Caffe

Torch VUNO-Net

DL4J
Cuda-
 RNNLIB Theano
ConvNet

29
 VUNO-net

Structure Output Learning

Convolution Softmax Multi-GPU Support

LSTM Regression Batch Normalization

MD-LSTM(2D) Connectionist Temporal Classification Parametric Rectifier

Pooling Initialization for Rectifier

Spatial Pyramid Pooling Stochastic Gradient Descent

Fully Connection Dropout

Open Source
Concatenation Data Augmentation
VUNO-net Only

30
 VUNO-net
• Multi-GPU Implementation using cuDNN

31
Performance

32
Our Vision

33
Content-based Medical Image Retrieval
시스템 프로세스

Similarity=91%
Similarity=91%
Similarity=71%

정량화/객관화
CT 환자 폐CT영상 유사도 계산 내용기반 검색
(Segmentation)

영상/컬러 값
histogram, moments, block-based

질감표시자
co-occurrence matrix, Fourier power spect
연구/개발 모듈

rum, Gabor features, Wavelet-based, Har

alick's statistical features, Tamura features
, Multiresolution auto-regression model

형태표시자
Edge histogram, Curvature scale space
, size, eccentricity, concavity

의료영상 딥러닝 기반 검색시스템

Classifier 개발
데이터 확보 폐질환 학습 모델 개발/검증

34
 Diffuse Interstitial Lung Disease
• DILD (미만성간질성폐질환)
- Diffuse: To become widely spread out
- Interstitium: The tissue and space around the air sacs of the lungs

• Disease Types

A region of lung tissue that

Consolidation has filled with liquid, a Honeycombing Clustered air spaces which
are usually subpleural and
condition marked by basal in distribution
induration of a normally
aerated lung

Emphysema Airflow limited state contained

Reticular Opacity An area of increased
within the disease state
pulmonary attenuation
known as COPD. An abnormal
caused by an
permanent enlargement of air
intraparenchymal process
spaces

Ground Glass Opacity Hazy area of increased

Normal
(GGO) attenuation in the lung with
preserved bronchial and
vascular markings

35
 Diffuse Interstitial Lung Disease

1 1
Emphysema GGO 2 1

2
2
3
2 3
1 3
3

4
4 4

4 5 5
Honeycombing Consolidation

36
ROI-based Classification

37
 ROI-based Classification

VUNO 2015 Chang et. al. 2013

Study design Training set Test set CNN SVM

GE GE(600) GE(600) 95.17±1.60 92.34±2.26

96.65% Siemens Siemens(600) Siemens(600) 96.00±0.70 91.53±2.07
91.1%
Integrated set GE + Siemens(1200) GE + Siemens(1200) 96.75±0.99 91.18±1.91

GE > Siemens GE(600) Siemens(600) 82.00±2.17 82.33±5.75

Siemens > GE Siemens(600) GE(600) 88.33±1.67 79.07±3.27

WR3 (World Record) - “A SVM classifier reduces interscanner variation in the HRCT classification of regional disease
WR3 VUNO
pattern in diffuse lung disease: Comparision to Bayesian classifier”, Chang, Yongjun et al., Med. Phys (2013)

38
Whole Lung Quantification

• Sliding window: Pixel level classification

• But, context information is more important

normal v.s. Emphysema

• Ongoing works

MD-LSTM Recurrent CNN

39
 Whole Lung Quantification
• How to apply the LSTM network to Image

• Context Informations

40
 Whole Lung Quantification
Example #1

Original Image (CT) VUNO Golden Standard

41
 Whole Lung Quantification
Example #2

Original Image (CT) VUNO Golden Standard

42
 Visualization
• Hand-crafted v.s. Learned Features

Texture-based Features CNN Features

43
 Visualization
• Layer-wise Features

Conv Layer 1 Conv Layer 2 Conv Layer 3 Conv Layer 4

44
 Visualization
• Mapping Into Patients’ Space

45
 Reference
1. W. Zhang et. al. Deep Convolutional Neural Networks for Multi-modality Isointense Infant Brain Image Segmentation, NeuroImage, 2015



2. D. C. Ciresan et. al., Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks, MICCAI, 2013



3. H. Suk et. al., Deep Learning-based Feature Representation for AD/MCI Classification, MICCAI, 2013



4. G. Wu et. al. Unsupervised Deep Feature Learning for Deformable Registration of MR Images, MICCAI, 2013



5. J. Wan et. al., Deep Learning for Content-based Image Retrieval : A Comprehensive Study, ACM Multimedia, 2014



6. L. R. Long et. al, Content-based Image Retrieval in Medicine, International Journal of Healthcare Information System and Informatics, 2009



7. T. M. Deserno et. al., Ontology of Gaps in Contents-based Image Retrieval, Journal of Digital Imaging, 2008

46