Project InnerEye

Medical Imaging AI to Empower Clinicians

Antonio Criminisi

Microsoft Healthcare NExT

Worrying headlines…

A New Algorithm Can Spot AI: It’s About to Cause a

Pneumonia Better Than a Radiologist Major Upheaval in Jobs

Healthcare explained: Could AI Skin Cancer Classification

replace radiology? With Deep Learning

Improving Palliative Care Stephen Hawking warns artificial

with Deep Learning intelligence could end mankind
…and discordant views between luminaries

Prof. Geoff Hinton remarking that Will radiologists be replaced by computers?

Debunking the hype of AI
“People should stop training radiologists now”
Prof Eliot Siegel, University of Maryland
[@machine learning conference]
[@Carestream blog RSNA16]
The InnerEye position InnerEye is a research project that develops
machine learning cloud services to

assist medical experts

in tasks of measurement, delineation and

quantitative temporal assessment

The InnerEye cloud services are intended to be integrated

as components into third-party medical software
Coming up today…

What is Artificial Intelligence ?

Machine learning ?
Computer Vision ?
Deep Learning ?

Are radiologists going to be

replaced by AI ?

How can AI benefit me ?

How can I get hold of such

technology ?
Overlapping fields

Artificial Intelligence Scary, but far in the future

(simulating human intelligence)

Robotics Machine learning

(hardware, mechanics) (tools/algorithms)

Deep learning
(one type of Less scary, practical and now!
ML algorithm)

Computer vision
(automatic
image analysis)
Project InnerEye’s focus - ML for radiological images

Machine learning
(tools/algorithms)

Less scary, practical and now!

Computer vision
(automatic
image analysis)
Machine learning is everywhere, and it is useful
Customers who bought this item also bought

Toy truck in yellow Pink elephant on wheels Buzzy Bee Giraffe on wheels
£7.99 £5.99 £6.99 £5.99
Microsoft Kinect
Microsoft Kinect – the visual recognition system

Machine learning
(pixel classification)

Input test depth image Body part segmentation

J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A.

Kipman, and A. Blake, Efficient Human Pose Estimation from Single Depth Images, in Trans. PAMI, IEEE, 2012
From Kinect depth images to radiological scans
Kinect depth images Magnetic resonance images

A. Criminisi, J. Shotton, Decision Forests for Computer Vision and Medical Image Analysis. Springer 2013
Project InnerEye Automatic, Semantic
a decade of research in Segmentation of
Radiological Images
The goal of automatic 3D segmentation
Two axial slices of the same CT scan Overlaid axial segmentations Natively 3D segmentation

Bladder

Prostate, seminal vesicles, rectum, bladder, left femur, right femur, skin
Why is voxel-wise semantic segmentation hard?

The challenge

• Same HU value for different anatomies

44 HU • Large deformations
• Implants
• Beam-hardening artefacts
• Different image resolution
• Image noise
44 HU • Presence/absence of contrast medium
• Different patient preparation
• …
 Our “ground-truth” labelled image dataset – hundreds of patients
axial + labels prostate seminal vesicles rectum bladder
patient 1
patient 2
patient 3

… … … … …
Modeling context via learned neighborhood patterns

Probe
voxels

Reference
voxel
Modeling context via learned neighborhood patterns

Features for
reference voxel
Training – learning to associate voxels with structures
Labelled training data A trained machine learning model
(a decision forest)
bladder

Training algorithm

sem. vesicles

rectum

prostate
Deep Decision Forests for semantic segmentation - training

Input CT image For each input voxel Output - probabilities @ layer 0

(and all its context features)

Z. Tu. Auto-context and Its Application to High-Level Vision tasks. CVPR 2008
 Deep Decision Forests for semantic segmentation - training
Input CT image
Appearance context

Output - probabilities @ layer 1

Input - probabilities @ layer 0

Semantic context
 Deep Decision Forests for semantic segmentation - training
Input CT image
Appearance context

Output - probabilities @ layer 2

Input - probabilities @ layer 1

Semantic context
Our trained Deep Decision Forest model
Forest layer 0

Probabilities @ layer 0

Forest layer 1

Probabilities @ layer 1

Input CT images

Forest layer 2

Probabilities @ layer 2

Forest layer 3

Segmentation
M. Fiterau, A. Criminisi, S. Rota Bulo’, P. Kontschierder, Deep Neural Decision Forests [Winner of the David Marr Prize]. ICCV 2015
Y. Ioannou, D. Robertson, R. Cipolla, A. Criminisi, Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups. CVPR 2017
Y. Ioannou, D. Robertson, D. Zikic, P. Kontschieder, J. Shotton, M. Brown, A. Criminisi, Decision Forests, Convolutional Networks and the Models in Between. ArXiv and Microsoft Tech Report 2015.
L. Le Folgoc, A. Nori, S. Ancha, A. Criminisi, Lifted Auto-Context Forests for Brain Tumour Segmentation. MICCAI 2016. BRATS workshop.
Decision Forests vs Neural Networks – what’s the difference?
7

Decision Tree Two-layer Perceptron 8

Any tree can be represented 1

2
0 3

4 Hidden activations
as a two-layer perceptron 6 1 1 2
1
(i.e. a neural network). 1 2 9
2 4
1
3
0
5 1 3 Output activations
1
3
0 1
9
3 Bias is depth of leaf
5
1
4
2
6 3
1
8 1 1
3 0 3
4

7 1
4
4

Algorithmically not much difference between forests and CNNs. However,

- Forests may require less training data - Forests may deal better with class imbalance
- Forests may be faster and do not require GPUs - In short, forests may be better suited for medical image analysis

I. K. Sethi. Entropy Nets: from Decision Trees to Neural Networks, Proc. IEEE (vol 78, Issue 10). Oct 1990
J. Welbl. Casting Random Forests as Artificial Neural Networks (and Profiting from It). Proc. GCPR. 2014.
Deep Decision Forests for semantic segmentation - runtime
Patient 1 Patient 2
@ runtime

Input CT image

Patient 3 Patient 4
Deep Decision Forests for semantic segmentation - runtime
Patient 1 Patient 2
@ runtime

Probabilities @ layer 0

Input CT image

Patient 3 Patient 4
Deep Decision Forests for semantic segmentation - runtime
Patient 1 Patient 2
@ runtime

Probabilities @ layer 1

Probabilities @ layer 0

Input CT image
Patient 3 Patient 4
Deep Decision Forests for semantic segmentation - runtime
Patient 1 Patient 2
@ runtime

Probabilities @ layer 2

Probabilities @ layer 1

Probabilities @ layer 0
Patient 3 Patient 4

Input CT image
Deep Decision Forests for semantic segmentation - runtime
Patient 1 Patient 2
@ runtime

Segmentation

Probabilities @ layer 2

Probabilities @ layer 1
Patient 3 Patient 4

Probabilities @ layer 0

Input CT image

The segmentation works despite anatomical variations. No need for atlases. No need for deformable registration. Probabilistic output (it encodes uncertainty).
Deep Decision Forests – comparison with inter-observer variability
Deep Decision Forests – comparison with ABAS algorithms
Project InnerEye Assistive AI for
Radiotherapy Planning
InnerEye – assistive AI for radiotherapy planning
The clinical workflow for image guided radiotherapy Fraction 1
Delivery of
therapy

Fraction 2
Delivery of
therapy

Fraction 3
Delivery of
therapy
The InnerEye app – an investigational device

Live Demo

NOT FOR CLINICAL USE

NO FDA clearance NOT FOR CLINICAL USE

Project InnerEye Assistive AI for
Quantitative Radiology
Quantitative tracking of disease progression
MR T1 contrast Pseudoprogression? Pseudoresponse? Hyperprogression?

Tumor responding. How much? How quickly? … … …

… … … tumor recurring.
Quantitative tracking of disease progression
MR T1 contrast with overlaid tumor segmentation
 Quantitative tracking of disease progression
Volume of active tumor
Volume cc

time
1 2 3 4 5 6 7 8 9
 Quantitative tracking of disease progression
MR Flair with overlaid oedema segmentation

Oedema
Active tumor
Volume cc

No visible tumor, but

non negligible oedema

time
1 2 3 4 5 6 7 8 9 10 11 12 13
Tissue-specific segmentation of glioblastoma in multi-modal MR

Six input MR modalities: T1, T1gad, T2, FLAIR + 2 diffusion images. Segmentation of: oedema, active tumor and necrosis
What is Artificial Intelligence ?
Machine learning ?
Computer Vision ?
Deep Learning ?

Are radiologists going to be

replaced by AI ?

How can AI benefit me ?

How can I get hold of such

technology ?
What is Artificial Intelligence ?
Machine learning ? Machine learning provides advanced, practical
Computer Vision ? and useful tools for medical image analysis
Deep Learning ?

Are radiologists going to be We are implementing assistive machine learning,

replaced by AI ? to amplify the skills of medical experts

How can AI benefit me ? Potentially more efficient, quantitative image

analysis workflow in the clinic

How can I get hold of such If you are a medical software provider, contact us
technology ?
InnerEyeInfo@microsoft.com
Private Preview - integrating InnerEye services within your products

Your medical software

Radiomics
Quantitative monitoring
Radiotherapy planning
…

InnerEye Service

InnerEyeInfo@microsoft.com

The InnerEye cloud services are intended to be integrated as components into third-party medical software.
The InnerEye team

InnerEyeInfo@microsoft.com
With huge thanks to our two main clinical partners

Dr Raj Jena Dr Mark Phillips

Dr Andrew Hoole Dr Meghan Macomber
Dr Marina Romanchikova Dr David Haynor
Dr David Noble William Logan
Dr Stephen Price Patty Sponseller
Dr Ferdia Gallagher Matt Nyflot
Prof Fiona Gilbert Mazen Zawaideh
Jay Liao
Prof Paul Kinahan
 Project InnerEye
Medical Imaging AI to Empower Clinicians

InnerEyeInfo@microsoft.com