+DUGZDUH$FFHOHUDWRUIRU6LPLODULW\%DVHG
'DWD'HGXSH
'RQJ\DQJ/L4LQJER:DQJ&\ULO*X\RW$VKZLQ1DUDVLPKD'HMDQ9XFLQLF=YRQLPLU%DQGLF4LQJ<DQJ

'HSWRI(OHFWULFDO&RPSXWHUDQG%LRPHGLFDO(QJLQHHULQJ8QLYRI5KRGH,VODQG.LQJVWRQ5,

6WRUDJH$UFKLWHFWXUH5HVHDUFK*URXS+*67D:HVWHUQ'LJLWDO&RPSDQ\6DQ-RVH&$


$EVWUDFW'DWD GHGXSOLFDWLRQ KDV SURYHQ LPSRUWDQW LQ
EDFNXS VWRUDJH V\VWHPV DV ODUJH DPRXQW RI LGHQWLFDO RU VLPLODU
GDWDFKXQNVH[LVW5HFHQWVWXGLHVKDYHVKRZQWKHJUHDWSRWHQWLDO
RI GDWD GHGXSOLFDWLRQ LQ SULPDU\ VWRUDJH DQG VWRUDJH FDFKHV
'HGXSOLFDWLRQV LQ WKHVH HQYLURQPHQWV UHTXLUH KLJK VSHHG
SURFHVVLQJQRWWRGUDJGRZQSURGXFWLRQSHUIRUPDQFH7KLVSDSHU
SUHVHQWV D KDUGZDUH DFFHOHUDWRU IRU VLPLODULW\ EDVHG GDWD
GHGXSOLFDWLRQ ,W LPSOHPHQWV WKUHH FRPSXWHLQWHQVLYH NHUQHO
PRGXOHV WR LPSURYH WKURXJKSXW DQG ODWHQF\ LQ GHGXSH V\VWHPV
VNHWFK FRPSXWDWLRQ IRU GDWD EORFNV LQGH[ VHDUFKLQJ IRU
UHIHUHQFH EORFN DQG GHOWD HQFRGLQJ RYHU VLPLODU EORFNV
$GRSWLQJSLSHOLQHGFRPSXWDWLRQDQGSDUDOOHOGDWDORRNXSDFURVV
PXOWLSOH KDUGZDUH PRGXOHV RXU +: GHVLJQ LV FDSDEOH RI
SURFHVVLQJ KLJK WKURXJKSXW GDWD WUDIILF E\ ZRUNLQJ RQ PXOWLSOH
GDWD XQLWV FRQFXUUHQWO\ WKXV HQDEOLQJ ZLUH VSHHG GHGXSH IRU
GDWD VWUHDP ZKHUH VLPLODU EORFNV SUHVHQW 8VLQJ D 3& KRVW
V\VWHPFRQQHFWHGWRWKH)3*$EDVHGDFFHOHUDWRUWKURXJKD3&,H
*HQ  LQWHUIDFH RXU H[SHULPHQWV VKRZ WKDW WKH VLPLODULW\
EDVHG GDWD GHGXSHSHUIRUPVEHWWHULQ GDWDUHGXFWLRQUDWLR
WKDQFRQYHQWLRQDOGHGXSHWHFKQLTXHVWKDWORRNDWLGHQWLFDOEORFNV
RQO\ %\ FRPSDULQJ WKH KDUGZDUH LPSOHPHQWDWLRQ ZLWK LWV
VRIWZDUH FRXQWHUSDUW WKH H[SHULPHQWDO UHVXOWV VKRZ WKDW RXU
SUHOLPLQDU\ )3*$ LPSOHPHQWDWLRQ ZLWK PD[LPXP FORFN VSHHG
RI0+]DFKLHYHVDWOHDVWWLPHVLPSURYHPHQWLQODWHQF\RYHU
WKHVRIWZDUHLPSOHPHQWDWLRQUXQQLQJRQVWDWHRIDUWVHUYHUV
.H\ZRUGV'DWD GHGXSOLFDWLRQ )LQJHUSULQWLQJ PXOWLSOH
ORRNXSGLFWLRQDU\&DFKH3&,H
, ,1752'8&7,21
'DWDGHGXSOLFDWLRQKDVEHFRPHLQFUHDVLQJO\LPSRUWDQWGXH
WR H[SORVLYH GDWD JURZWK LQ WKH ,QWHUQHW ZRUOG ,W KDV EHHQ
KLJKO\ VXFFHVVIXO LQ HQWHUSULVH EDFNXS HQYLURQPHQWV >@
7\SLFDOO\ FRPSDQLHV FRQGXFW GDLO\ LQFUHPHQWDO EDFNXSV DQG
ZHHNO\IXOOEDFNXSVWRDUFKLYHWKHLUGDWD7KHJUHDWDPRXQWRI
GXSOLFDWHG GDWD OHW WKH XVH RI GHGXSOLFDWLRQ WHFKQRORJ\
ZLGHVSUHDGLQHQWHUSULVHEDFNXSV\VWHPV>@
7KH VXFFHVV RI GDWD GHGXSOLFDWLRQ LQ EDFNXS V\VWHPV
LQVSLUHG D QXPEHU RI HIIRUWV WR LQWURGXFH GHGXSOLFDWLRQ LQWR
SULPDU\ VWRUDJH >@>@ 8QOLNH LQ WKH EDFNXS V\VWHP WKH
SULPDU\ VWRUDJH LQ D SURGXFWLRQ HQYLURQPHQW SUHVHQWV WKH
GHGXSOLFDWLRQ IXQFWLRQDOLW\ ZLWK PXOWLSOH XQLTXH FKDOOHQJHV
)LUVWO\ D SULPDU\ VWRUDJH GRHV QRW KDYH DV PXFK GXSOLFDWHG
GDWD DV LQ WKH EDFNXS V\VWHPV 'DWD VHQW WR SULPDU\ VWRUDJH
FRPHV IURP XVHU OHYHO DSSOLFDWLRQV VXFK DV GDWDEDVH DQG ILOH
V\VWHPV 7KH PDLQ RSHUDWLRQV IRU WKHVH NLQGV RI DSSOLFDWLRQV
978-1-4673-7891-8/15/$31.00 2015 IEEE 224

DUH PRGLI\ DGG DQG GHOHWH  7KHVH RSHUDWLRQV PRVWO\
JHQHUDWH  VLPLODU GDWD EORFNV LQVWHDG RI LGHQWLFDO EORFNV 7KLV
SURSHUW\ OHDGV XV WR ORRN DW GHGXSOLFDWLRQ WHFKQLTXHV DW VXE
EORFNOHYHO7KHVHFRQGFKDOOHQJHLVWKHGLIIHUHQWUHTXLUHPHQW
IRU WKH SHUIRUPDQFH PHWULFV ZKHUH GHGXSOLFDWLRQ LQ EDFNXS
VWRUDJH LV PRUH WKURXJKSXW GHPDQGLQJ WKDQ WKH ODWHQF\
VHQVLWLYHQDWXUHLQKHUHQWLQWKHSULPDU\VWRUDJHRIDSURGXFWLRQ
V\VWHP 7KH WKLUG FKDOOHQJH LV WKH OLPLWDWLRQ RI UHVRXUFHV
'HGXSH V\VWHP LQ WKH 3ULPDU\ VWRUDJH RIWHQ VKDUHV KDUGZDUH
DQGVRIWZDUHUHVRXUFHVRIWKHSURGXFWLRQV\VWHPZKLOHEDFNXS
GHGXSOLFDWLRQ V\VWHP KDV LWV RZQ UHVRXUFHV 7DNLQJ VHUYHU
UHVRXUFHVVXFKDVWKH&38DQGWKH5$0UHVRXUFHVWRSHUIRUP
GHGXSOLFDWLRQ PD\ VORZ GRZQ WKH DSSOLFDWLRQ SHUIRUPDQFH
UXQQLQJRQWKHVHUYHUZKLFKLVXQGHVLUDEOH
)LOHV RU GDWD EORFNV DUH IUHTXHQWO\ PRGLILHG DQG
UHDVVHPEOHG LQ WKHLU OLIH F\FOH :LWK WKH GLIIHUHQFHV GHWHFWHG
EHWZHHQ QHDUGXSOLFDWH GDWD EORFNV GHOWD HQFRGLQJ FDQ EH
HPSOR\HG WR GHGXSHGDWDDWRUEHORZ WKH ILOH OHYHO DV ZHOODV
EORFN OHYHO 7KH EDVLF RSHUDWLRQ RI GHOWD HQFRGLQJ LV WR NHHS
RQH FRS\ RI VDPH FRQWHQWV LQ RQH RI WKH WZR VLPLODU GDWD
EORFNV ZKLOH RQO\ VDYLQJ D UHIHUHQFH WR WKH GDWD LQ WKH RWKHU
EORFN3KLOLS6KLODQHHWDOEXLOWDGHGXSHVWRUDJHXVLQJWKHGHOWD
HQFRGLQJ VFKHPH >@ DQG VKRZHG WKDW WKH DGGLWLRQDO
GHGXSOLFDWLRQ EHQHILW GXH WR WKH GHOWD HQFRGLQJ LV  WLPHV
LPSURYHPHQW RQ WRS RI WKH FRQYHQWLRQDO GHGXSH WHFKQLTXHV
+RZHYHU WKH WKURXJKSXW RI WKH FXUUHQWO\ VRIWZDUH EDVHG
GHGXSHSURFHVVUDQJHVIURP0%VWR0%VZKLFKDUHQRW
LGHDOIRUSULPDU\VWRUDJHRUFDFKHV\VWHPVZKLFKGHPDQGFORVH
WR JLJDE\WH SHU VHFRQG WKURXJKSXW DQG VXEPLOOLVHFRQG LQ
ODWHQF\>@
,Q RUGHU WR PDNH VLPLODULW\ EDVHG GHGXSH WHFKQRORJLHV
DSSOLFDEOHWRSULPDU\VWRUDJHVRUFDFKHVKDUGZDUHDFFHOHUDWLRQ
VKRXOG EH H[SORUHG 1RW RQO\ FDQ D KDUGZDUH LPSOHPHQWDWLRQ
RIIHU KLJK VSHHG GHGXSH EXW DOVR RIIORDG GHGXSH IXQFWLRQV
IURPVHUYHUVVRWKDWDSSOLFDWLRQSHUIRUPDQFHLVQRWQHJDWLYHO\
LPSDFWHG ,Q WKLV SDSHU ZH SUHVHQW D KDUGZDUH DFFHOHUDWRU
GHVLJQ IRU VLPLODULW\ EDVHG GHGXSH IRU SULPDU\ VWRUDJHV DQG
VWRUDJH FDFKHV 8WLOL]LQJ SLSHOLQH SURFHVVLQJ DQG SDUDOOHO
VWUXFWXUHV RXU GHVLJQ SURYLGHV KLJK WKURXJKSXW DQG IDVW
UHVSRQVHWLPH7KHSURSRVHGDUFKLWHFWXUHLVLPSOHPHQWHGRQD
;LOLQ[ 9LUWH[ )3*$ GHYHORSPHQW ERDUG 7KUHH PDMRU
KDUGZDUH PRGXOHV IRU WKH GHGXSH V\VWHP ZHUH IXOO\ WHVWHG WR
EHIXQFWLRQDO([WHQVLYHH[SHULPHQWVKDYHEHHQFDUULHGRXWWR
HYDOXDWH WKH SHUIRUPDQFH RI HDFK PRGXOH LQ WHUPV RI ODWHQF\
DQG WKH GHGXSH UDWLR RI WKH RYHUDOO V\VWHP DV FRPSDUHG WR
VRIWZDUHLPSOHPHQWDWLRQV2XUH[SHULPHQWDOUHVXOWVVKRZWKDW
WKHKDUGZDUHLPSOHPHQWDWLRQSURYLGHVDWOHDVWWLPHVVSHHGXS
LQ WHUPV RI ODWHQF\ RYHU LWV VRIWZDUH FRXQWHUSDUW ZKLOH WKH
GHGXSHUDWLRLVFRPSDUDEOH:HDOVRVKRZWKDWVLPLODULW\EDVHG
GHGXSHRIIHUVPRUHGDWDUHGXFWLRQWKDQWKHW\SLFDOGHGXSH
WHFKQLTXHV
7KLVSDSHUPDNHVWKHIROORZLQJFRQWULEXWLRQV
)LJXUH*HQHUDOZRUNIORZIRUKDUGZDUHDFFHOHUDWRU
 'HVLJQ DQG LPSOHPHQWDWLRQ RI KDUGZDUH VROXWLRQV IRU
WKUHH PDMRU PRGXOHV RI VLPLODULW\ EDVHG GDWD GHGXSOLFDWLRQ
ILQJHUSULQW FRPSXWDWLRQ WR GHULYH WKH VNHWFK RI D GDWD EORFN
LQGH[LQJVWUXFWXUHDQGVHDUFKORJLFIRUILQGLQJUHIHUHQFHEORFNV
WKDW DUH XVHG DV EDVHV IRU GHOWD HQFRGLQJ DQG KDUGZDUH GHOWD
HQFRGLQJORJLF
,QVLPLODULW\EDVHGGHGXSOLFDWLRQDQHZEORFNVHDUFKHVIRU
D QHDUGXSOLFDWH EORFN LQ D VHW RI UHIHUHQFH EORFNV E\
FRPSDULQJWKHLUVNHWFKHV,IDPDWFKHGVNHWFKLVIRXQGLQDOLVW
RIUHIHUHQFHEORFNVDGHOWDHQFRGLQJLVSHUIRUPHGDJDLQVWWKH
IRXQG UHIHUHQFH EORFN DQG RQO\ WKH GHOWD LV VWRUHG ZLWK D
SRLQWHU WR WKH UHIHUHQFH EORFN 7KHUHIRUH VLPLODULW\ EDVHG
 ,QWHJUDWLRQ RI WKH KDUGZDUH PRGXOHV LQWR VRIWZDUH GHGXSHUHTXLUHVWKUHHNH\IXQFWLRQVFDOFXODWHWKHVNHWFKRID
GHGXSH SODWIRUP >@ 7KH LQWHJUDWHG V\VWHP LV VKRZQ WR EORFN  VHOHFW DQG VWRUH UHIHUHQFH EORFNV DJDLQVW ZKLFK WKH
IXQFWLRQFRUUHFWO\DQGHIILFLHQWO\ GHOWD HQFRGLQJ ZLOO EH SHUIRUPHG DIWHU D PDWFKHG VNHWFK LV
3HUIRUPDQFHHYDOXDWLRQVKDYHEHHQFDUULHGRXWXVLQJUHDO
ZRUOGGDWDVHWV:HFRQGXFWHGH[WHQVLYHH[SHULPHQWVWRVKRZ
WKHDFKLHYDEOHVSHHGXSDQGGDWDUHGXFWLRQUDWLRVDVFRPSDUHG
WRWKHH[LVWLQJVROXWLRQV
7KHUHVWRIWKLVSDSHULVRUJDQL]HGDVIROORZV,Q6HFWLRQ
WKHUHODWHG EDFNJURXQG ZRUNLVSUHVHQWHG 6HFWLRQ SUHVHQWV
RXU GHVLJQ RI WKH  KDUGZDUH PRGXOHV 7KH )3*$
LPSOHPHQWDWLRQWKHWHVWVHWXSDQGWKHH[SHULPHQWDOUHVXOWVDUH
GHWDLOHGDQGGLVFXVVHGLQ6HFWLRQ:HFRQFOXGHRXUSDSHULQ
6HFWLRQ
,, %$&.*5281'
$ 6WDQGDUGGHGXSH
$ W\SLFDO SURFHVV RI GDWD GHGXSOLFDWLRQ LQYROYHV WKH
IROORZLQJ SURFHVV )LUVWO\ LW VSOLWV ILOHV LQWR PXOWLSOH FKXQNV
DQG JHQHUDWHV D ILQJHUSULQW IRU HDFK FKXQN 7KH ILQJHUSULQW
XVXDOO\LVDVWURQJKDVKGLJHVWRIWKHFKXQN,IWZRILQJHUSULQWV
PDWFK LW PHDQV WKHLU FRQWHQWV DUH GXSOLFDWH :KHQ D QHZ
LQFRPLQJ FKXQNV ILQJHUSULQW PDWFKHV DQ H[LVWLQJ RQH LQ WKH
GHGXSOLFDWLRQ V\VWHP RQO\ WKH FKXQNV PHWDGDWD VXFK DV ILOH
QDPH RU /%$ DQG D UHIHUHQFH WR WKH H[LVWLQJ FRQWHQW ZLOO EH
VWRUHG>@
% 6LPLODULW\EDVHGGHGXSH
,WLVRIWHQWKHFDVHWKDWGDWDFKXQNVDUHIUHTXHQWO\PRGLILHG
E\ FXW LQVHUW GHOHWH RU XSGDWH D SDUW RI WKH FRQWHQW 7KRXJK
VOLJKWO\ FKDQJHG FKXQN ZLOO JHQHUDWH GLIIHUHQW VWURQJ KDVKHV
DQGFRXOGQRWEHLQGH[HGE\VWDQGDUGGHGXSHWKHVNHWFKRIWKH
FKXQNPD\VWD\WKHVDPHLIDZHDNHUKDVKIXQFWLRQLVXVHG>@
6XFK ZHDNHU KDVK VNHWFKHV W\SLFDOO\ FRQVLVW RI VHYHUDO 5DELQ
ILQJHUSULQWV DQG KDYH WKH SURSHUW\ WKDW LI WZR FKXQNV VKDUH D
VNHWFKWKHQWKH\KDYHDORWRIVDPHFRQWHQWLHWKH\DUHOLNHO\
QHDUGXSOLFDWHV1RWHWKDWZHZLOOXVHWKHWHUPVFKXQNDQG
EORFNLQWHUFKDQJHDEO\LQWKLVSDSHUWRUHIHUWRWKHEDVLFXQLW
RIGDWDGHGXSOLFDWLRQ
225
IRXQGGHOWDHQFRGLQJ
& 'HOWDHQFRGLQJ
)RUWZRQHDUGXSOLFDWHILOHVIROGDQGIQHZGHOWDHQFRGLQJLVWR
FRPSXWH D PLQLPDO VL]H RI IGHOWD E\ ZKLFK IQHZ FRXOG EH
UHFRQVWUXFWHG ZLWK IROG >@ 'HOWD HQFRGLQJ FRQVWUXFWV D
GLFWLRQDU\RIREVHUYHGVHTXHQFHVDQGORRNVIRUUHSHWLWLRQVDVLW
JRHV ,W ZULWHV WKH QXPEHU RI WKH GLFWLRQDU\ HQWU\ ZKHQ D
UHSHWLWLRQ HQFRXQWHUHG DQG VWRUH D XQLTXH WRNHQ LI QR PDWFK
KDSSHQHG7KHRXWSXWWKXVFRQVLVWVRIDSSURSULDWHO\ODEHOHGIQHZ
DQGUHIHUHQFHVWRWKHUHSHWLWLRQVLQWKHIROG
7KRXJK H[WHQVLYH ZRUN KDV EHHQ GRQH RQ KDUGZDUH
FRPSUHVVLRQQRQHRIWKHPZHUHGHVLJQHGVSHFLILFDOO\IRUGHOWD
HQFRGLQJ WKDW FDQ EH XVHG IRU GHGXSH V\VWHPV > @ $OVR
FXUUHQWKDUGZDUHEDVHGGHOWDFRPSUHVVLRQKDVWRFRPSUHVVGDWD
FKXQN IQHZ E\WH E\ E\WH ,W WDNHV . ORRSV WR FRPSXWH IGHOWD
ZKLFK PD\ SRVH DV D SHUIRUPDQFH ERWWOHQHFN IRU KLJK
WKURXJKSXW VWRUDJH V\VWHPV 7\SLFDOO\ ,2 EXVHV DUH XVXDOO\
PRUHWKDQRQHE\WHLQZLGWK$QHQFRGLQJXQLWZKRVHODWHQF\
LQFUHDVHV OLQHDUO\ ZLWK WKH LQSXW OHQJWK LV QRW DFFHSWDEOH IRU
PRGHUQ GDWD VWRUDJH DSSOLFDWLRQV ,QVSLUHG E\ :. DOJRULWKPV
IRU&RPSUHVVHG&DFKLQJLQ9LUWXDO0HPRU\6\VWHPV>@ZH
FKRRVHPXOWLSOHE\WHVDVWKHWRNHQVL]HIRUHQFRGLQJDQGXVHD
SDUDOOHOKDUGZDUHVWUXFWXUHWRFDUU\RXWWKHGHOWDHQFRGLQJRYHU
PXOWLSOHE\WHVFRQFXUUHQWO\
,,, '(6,*1$1'237,0,=$7,21
7KHKDUGZDUHDFFHOHUDWLQJVLPLODULW\EDVHGGDWDGHGXSHE\
IROORZLQJ WKUHH PRGXOHV VNHWFK FRPSXWLQJ UHIHUHQFH EORFN
LQGH[LQJ DQGGHOWDHQFRGLQJ$V VKRZQ LQ )LJ  GDWDEORFNV
JR WKURXJK VNHWFK FRPSXWDWLRQ DQG UHIHUHQFH EORFN LQGH[LQJ
SURFHVVWKUHHNLQGVRIEORFNDUHFODVVLILHG
 5HIHUHQFHEORFNLWVGDWDFRQWHQWLVSRSXODULHPDQ\
GDWDEORFNVVKDUHVRPHFRQWHQWZLWKLW
 $VVRFLDWHEORFNVLWVGDWDFRQWHQWLVLGHQWLILHGDVVLPLODU
WRRQHRIWKHH[LVWLQJUHIHUHQFHEORFNV
 )LJXUH$QH[DPSOHVKRZLQJWZRVKLQJOHV
 ,QGHSHQGHQW EORFN LWV GDWD FRQWHQW LV FROG ZKLFK
PHDQV LW GRHV QRW VKDUH VLJQLILFDQW DPRXQW RI GDWD
FRQWHQWZLWKDQ\RIWKHUHIHUHQFHEORFNV
7KHVH WKUHH NLQGV RI EORFN DUH WUHDWHG LQ GLIIHUHQW ZD\V
LQGHSHQGHQWEORFNVDUHGLUHFWO\VWRUHGLQWRPDLQVWRUDJHWKXV
PDNLQJ OLWWOH RU QR FRQWULEXWLRQ WR GDWD UHGXFWLRQ UHIHUHQFH )LJXUH%ORFNGLDJUDPRIKDUGZDUHGHVLJQIRUVNHWFK
EORFNVDUHFDFKHGVLQFHWKH\DUHIUHTXHQWO\DFFHVVHGDVVRFLDWH FRPSXWDWLRQZLWKILQJHUSULQWSLSHOLQHDQGVNHWFKVHOHFWLRQ
EORFNVDQGWKHUHIHUHQFHEORFNVWKDWDUHVLPLODUWRWKHPDUHIHG FRQGXFLYH WR REWDLQ PRUH HYHQO\ GLVWULEXWHG DQG ORZHU
WRGHOWDHQFRGLQJZKHUHWKHUHVXOWLQJGHOWDLVVWRUHGLQWRPDLQ PD[LPXPSLSHOLQHVWDJHODWHQF\6XFKVFKHPHLVUHIHUUHGWRDV
VWRUDJH 7KH IROORZLQJ VHFWLRQV H[SODLQ WKH GHVLJQ RI HDFK VSOLWIUHVKLQRXUGHVLJQ
PRGXOHWKDWLVLQFKDUJHRILGHQWLI\LQJWKHGLIIHUHQWEORFNVDQG
$VVKRZQLQ)LJEDVHGRQWKHSURSHUW\RIIUHVKVKLIW
HQVXLQJHQFRGLQJLIDSSOLFDEOH
DQGVSOLWIUHVKZHPD\GHVLJQDSLSHOLQHVWUXFWXUHWRSURYLGH
$ 6NHWFK&RPSXWDWLRQ D OLQH VSHHG 5DELQ ILQJHUSULQW FRPSXWDWLRQ $ UDQGRP
5DELQ ILQJHUSULQW LV D SRO\QRPLDO PRGXOR RSHUDWLRQ WKDW VDPSOLQJ WHFKQLTXH VXFK DV RQH XWLOL]LQJ 0LQZLVH WKHRU\ LV
FDQEHWUDQVIRUPHGLQWRPXOWLSOHELW;25FDOFXODWLRQVWKDWDUH WKHQ XVHG WR VHOHFW DPRQJ DOO 5DELQ ILQJHUSULQWV WR IRUP D
VXLWDEOH IRU KDUGZDUH LPSOHPHQWDWLRQ >@ ,Q RXU GHGXSH VNHWFK IRU WKH FXUUHQW GDWD EORFN >@ DV VKRZQ LQ )LJ 
V\VWHP WKH VNHWFK RI D GDWD EORFN LV GHULYHG E\ FRPSXWLQJ 'HWDLOHGGHVLJQFDQEHIRXQGIURPRXUSUHYLRXVZRUN>@
5DELQ ILQJHUSULQW RQ HYHU\ IL[VL]HG VOLGLQJ ZLQGRZ HJ 
E\WHVORQJUHIHUUHGWRDVVKLQJOHRYHUWKHGDWDVWUHDP)LJ % 5HIHUHQFHEORFNLQGH[
VKRZV WZR VXFK FRQVHFXWLYH VKLQJOHV$[ DQG %[:H ILUVW $IWHU WKH VNHWFK IRU HDFK EORFN LV FDOFXODWHG ZH XVH WKH
FRPSXWH)3$WKHILQJHUSULQWRI$[WKHILUVWVKLQJOHXVLQJ VNHWFKHV WR UHSUHVHQW HDFK GDWD EORFN DQG NHHS WUDFN RI ,2
SDUDOOHO ;25 FDOFXODWLRQV IRU SRO\QRPLDO PRGXODWLRQ DFFHVV ZLWK UHJDUGLQJ WR WKHVH VNHWFKHV %DVHG RQ WKH FRQWHQW
IROORZLQJ(TXDWLRQ ORFDOLW\ LH DFFHVV IUHTXHQF\ DQG UHFHQF\ RI GDWD FRQWHQWV
>@ZHVHOHFWDQGFDFKHWKHPRVWSRSXODUEORFNVDVUHIHUHQFH
)3  $ = $ PRG 3 
EORFNV )RU H[DPSOH WKHUH DUH XS WR WZR WKRXVDQGV LQ RXU
2QFH WKH ILQJHUSULQW IRU $[ LV FDOFXODWHG 5DELQ H[SHULPHQWDO VHWXS7KHVHUHIHUHQFH EORFNVDQG WKHLU VNHWFKHV
ILQJHUSULQW RI %[ )3% FDQ EH REWDLQHG E\ DSSO\LQJ WKH DUH VWRUHG LQ D UHIHUHQFH OLVW (YHU\ QHZO\ JHQHUDWHG EORFN
IRUPXODLQ(TXDWLRQ VNHWFK LV XVHG DV D NH\ WR TXHU\ WKH UHIHUHQFH OLVW WR ILQG D
PDWFK :KHQ KLW WKH LQFRPLQJ EORFN LV DQ DVVRFLDWHG EORFN
)3  %  = % PRG 3 = : [  8 [j

 PRG 3 +  [j

$ PRG 3  PRG 3  DQGZLOOEHGHOWDHQFRGHGDJDLQVWWKHPDWFKLQJUHIHUHQFHEORFN
7KHUHPDLQLQJGDWDDQGSRLQWHUVIRUHQFRGHGGHOWDVLQVWHDGRI
:KHUH [j = [  PRG 3  WKH ILQJHUSULQW RI $[ LV FDOFXODWHG WKH RULJLQDO .% EORFN DUH VWRUHG LQ WKH SULPDU\ VWRUDJH RU
XVLQJDOORIWKHELWVLQVKLQJOH$[DQGZHFDOOWKHIXQFWLRQDV FDFKH
IUHVK 7KH ILQJHUSULQW RI %[ RQ WKH RWKHU KDQG LV
FDOFXODWHGXVLQJWKHILQJHUSULQWRI$DORQJZLWKWKHILUVWE\WH ,Q RXU GHVLJQ ZH DVVXPH WKDW D VNHWFK FRQWDLQV 
RI $ DQG WKH ODVWE\WHRI %:H FDOO LW D VKLIW IXQFWLRQ
1RWH WKDW IUHVK ORJLF WDNHV PRUH WLPH WKDQ WKH VKLIW ORJLF 7R
RSWLPL]HWKHSLSHOLQHLPSOHPHQWDWLRQZHFDQIXUWKHUVSOLWWKH
IUHVK ORJLF LQWR PXOWLSOH VXEPRGXOHV WR UHGXFH WKH
FRPSXWDWLRQ FRPSOH[LW\ RI IUHVK IXQFWLRQV WKHUHIRUH
)LJXUH5DELQILQJHUSULQWSLSHOLQH
226
ILQJHUSULQWVHDFKRIZKLFKLVRQHE\WHORQJ,IWZRGDWDEORFNV
KDYHQPDWFKHGILQJHUSULQWVEHWZHHQWKHLUUHVSHFWLYHVNHWFKHV
ZKHUHQFDQEHIURPWRZHGHFODUHWKH\DUHQHDUGXSOLFDWH
EORFNVQLVUHIHUUHGWRDVVLPLODULW\WKUHVKROG2QFHVXFKQHDU
GXSOLFDWH EORFN LV IRXQG LQ WKH UHIHUHQFH LQGH[ WKH
FRUUHVSRQGLQJ UHIHUHQFH EORFN ZLOO EH UHDG RXW DQG GHOWD
HQFRGLQJDJDLQVWLWLVSHUIRUPHG
7R ILQG D PDWFK DPRQJ PDQ\ UHIHUHQFH EORFNV LQ D VKRUW
WLPHZHEXLOGDKDUGZDUHVHDUFKVWUXFWXUHDVVKRZQLQ)LJ
7KHUHIHUHQFHOLVWWDEOHRQWKHXSSHUULJKWFRUQHURIWKHILJXUH
VWRUHVWKH PHWDGDWDRI WKH. UHIHUHQFHEORFNV(DFKHQWU\ LQ
WKLVWDEOHFRQWDLQVWKH%VNHWFKRIRQHRI.UHIHUHQFHEORFNV
DSRLQWHUWRWKHUHIHUHQFHEORFNGDWDDQGRWKHUQHFHVVDU\VWDWXV
LQIRUPDWLRQ WKHUHIRUH WKH UHIHUHQFH OLVW WDEOH PDLQWDLQV 
VXFKHQWULHVLQPHPRU\7RWKHOHIWRIWKHUHIHUHQFHOLVWWDEOHLQ
WKHILJXUHLVWKHKDVKLQGH[WDEOH7KLVKDVKLQGH[WDEOHVWRUHV
SRLQWHUVWRWKHUHIHUHQFHOLVWWDEOH(QWU\KLQWKLVWDEOHSRLQWVDW
 )LJXUH%ORFNGLDJUDPRIUHIHUHQFHEORFNLQGH[LQJ
WKH UHIHUHQFH OLVW WDEOH HQWU\ WKDW FRQWDLQV WKH VNHWFK ZKRVH
FKRVHQ Q ILQJHUSULQWV KDYH KDVK YDOXH K 6LQFH WKHUH DUH &Q
DYDLODEOHFRPELQDWLRQRIQILQJHUSULQWVZHKDYH&QJURXSVLQ
WKH KDVK LQGH[ WDEOH +DVK NH\ LV JHQHUDWHG IURP Q E\WH
SHUPXWDWLRQRI%VLJQDWXUHZHXVH&5&LPSOHPHQWDWLRQZLWK
DELWSRO\QRPLDOWRJHQHUDWHFRUUHVSRQGLQJKDVKNH\
,Q RUGHU WR DYRLG XQQHFHVVDU\ VHDUFKHV DQG FRPSDULVRQV
ZH DGRSW &Q EORRP ILOWHUV WR SURPSWO\ WHUPLQDWH WKH VHDUFK
SURFHVVRQFHZHILQGWKHUHLVQRPDWFKLQWKHUHIHUHQFHOLVW$V
VKRZQLQWKHILJXUHWKH%ORRPILOWHUVFDQTXLFNO\SURQRXQFHD
QHJDWLYHUHVXOWRQHDFKRIWKH&QILQJHUSULQWSHUPXWDWLRQVE\
FKHFNLQJ WKH KDVK YDOXH RI DQ LQFRPLQJ EORFN >@ WKH VDPH
KDVK YDOXH XVHG WR EXLOG WKH KDVK LQGH[ WDEOH 7KH UHIHUHQFH
VHDUFKVWUXFWXUHVKRZQLQ)LJDUHXVHGZLWKWZRSXUSRVHVLQ
SURFHVVUHSHDWVXQWLODPDWFKLVIRXQGRUDOOSRVLWLYHRXWSXWVRI
WKH%ORRPILOWHUVKDYHEHHQH[KDXVWHG
,GHQWLI\DQHZUHIHUHQFHEORFNDQGDGGLWWRWKHOLVW:KHQ
WKHUHIHUHQFHEORFNOLVWZDVLQLWLDOO\SRSXODWHGRUZKHQDQHZ
UHIHUHQFHEORFNLVDGGHGWRWKHDOUHDG\SRSXODWHGOLVWWRUHSODFH
DQ/58/38>@UHIHUHQFHEORFNZHQHHGWRJRWKURXJKWKLV
VHDUFK VWUXFWXUH )LUVW WKH VNHWFK RI WKH QHZ UHIHUHQFH EORFN
QHHGVWREHDGGHGWRWKHUHIHUHQFHOLVWWDEOH6XSSRVHWKHVNHWFK
LVWREHVWRUHGLQHQWU\1RIWKHUHIHUHQFHOLVWWDEOH1FRXOGEH
D QHZ HQWU\ RU DQ H[LVWLQJ HQWU\ FRQWDLQLQJ WKH /58/38
UHIHUHQFHEORFNWKDWVKRXOGEHUHSODFHG:HWKHQFRPSXWH&Q
KDVKYDOXHVFRUUHVSRQGLQJWRWKH&QQVLJQDWXUHSHUPXWDWLRQV
:HVWRUH1LQWRDOO&QHQWULHVPDWFKLQJWKHKDVKYDOXHVLQWKH
+DVKLQGH[WDEOH)LQDOO\WKHFRUUHVSRQGLQJELWVLQWKH%ORRP
PLQGWRVHDUFKIRUDUHIHUHQFHEORFNVLPLODUWRDQLQFRPLQJ ILOWHUVZLOOEHVHWUHIOHFWLQJWKHQHZPHPEHURILQWKHUHIHUHQFH
EORFN DQG  DGGLQJ D QHZ UHIHUHQFH EORFN LQWR WKH UHIHUHQFH
EORFNOLVW7KHGHWDLOHGRSHUDWLRQVWKURXJKWKLVVHDUFKVWUXFWXUH
DUHVKRZQLQ)LJ
6HDUFK IRU D VLPLODU UHIHUHQFH EORFN VR DV WR LGHQWLI\ DQ
DVVRFLDWHG EORFN 8SRQ DUULYDO RI D QHZ EORFN LWV VNHWFK LV
FDOFXODWHG XVLQJ WKH ILQJHUSULQW FLUFXLW GHVFULEHG LQ WKH
SUHYLRXV VHFWLRQ 7KH UHVXOWDQW % VNHWFK LV HQWHUHG WR WKH
VHDUFK ORJLF DV VKRZQ DW WKH ULJKW VLGH RI )LJ  Q E\WH
VLJQDWXUHVDUHSLFNHGXSIURPVLJQDWXUHVLQWKHVNHWFKWRIRUP
&QSHUPXWDWLRQVLQWRWDO(DFKRIWKHVHQE\WHVHOHFWLRQVJRHV
WKURXJK WKH KDVK IXQFWLRQ WR GHULYH D ELW KDVK YDOXH 7KLV
EORFNOLVW>@
7DNLQJ DGYDQWDJH RI )3*$V SDUDOOHO FRPSXWDWLRQ RXU
KDUGZDUH GHVLJQ IRU UHIHUHQFH LQGH[ PRGXOH DOORZV SDUDOOHO
VHDUFK DPRQJ DOO WKH &Q SDUDOOHO EORRP ILOWHUV DW VDPH WLPH
2QFH D PDWFK FDVH LV IRXQG UHIHUHQFH LQGH[ FDQ VSRW WKH
ORFDWLRQLQWKHUHIHUHQFHOLVW
& 'HOWDHQFRGLQJ
7KH GDWD WUDQVIHU LQWHUIDFH RQ RXU KDUGZDUH SODWIRUP
EHWZHHQWKHDFFHOHUDWRUDQGWKHKRVWLV%LQZLGWK,QRUGHUWR
KDVK YDOXH LV IHG LQWR WKH %ORRP ILOWHU WR TXLFNO\ GHWHUPLQH 5DZGDWD'Z GRXEOHZRUG
ZKHWKHULWKDVDSRVVLEOHPDWFKLQWKHUHIHUHQFHEORFNOLVW:H
WKHQSLFNXSWKHILUVWSRVLWLYHUHVXOWDPRQJDOOWKH%ORRPILOWHUV 2IIVHW         
%ONUHI 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z
WRSLFNXSWKHLQGH[SRLQWLQJDWWKHFRUUHVSRQGLQJHQWU\LQWKH %ONQHZ 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z
UHIHUHQFH OLVW WDEOH 7KH FRUUHVSRQGLQJ % VNHWFK LQ WKH
UHIHUHQFH OLVW WDEOH LV UHDG RXW WR DFWXDOO\ FRPSDUH ZLWK WKH 5HVXOW
VNHWFK RI WKH LQFRPLQJ DVVRFLDWHG EORFN  ,I WKH QXPEHU RI 2IIVHW         
PDWFKHGVLJQDWXUHVH[FHHGVQWKHQWKHVHDUFKLVDKLW:HZLOO IODJ         
IROORZ WKH SRLQWHU VWRUHG LQ WKH HQWU\ WR UHDG WKH UHIHUHQFH LQGH[         
EORFN GDWD WR SHUIRUP GHOWD HQFRGLQJ 2WKHUZLVH LI WKH WZR OHQJWK         
VNHWFKHV GR QRW KDYH Q RU PRUH PDWFKLQJ VLJQDWXUHV ZH ZLOO PLVV    'Z  'Z   
ORRN DW WKH QH[W SRVLWLYH RXWSXW RI WKH %ORRP ILOWHUV DQG WKH )LJXUH$QH[DPSOHVKRZLQJKDUGZDUHGHOWDHQFRGLQJ

227
DFKLHYH D ZLUH VSHHG HQFRGLQJ IRU VLPLODULW\ EDVHG GDWD
GHGXSOLFDWLRQ ZH ORRNV IRU HYHU\ % UHSHWLWLRQV IURP QHDU
GXSOLFDWHEORFNV)LJLOOXVWUDWHVWKHGHOWDHQFRGLQJDOJRULWKP
ZLWKWZRGDWDEORFNVDVLQSXW%ONUHILVDUHIHUHQFHEORFNWREH
GHOWD HQFRGHG DJDLQVW ,W LV ORDGHG LQWR WKH GLFWLRQDU\ ILUVW
%ONQHZ LV WKH DVVRFLDWHG EORFN WR EH FRPSUHVVHG ,W LV IHG LQWR
WKHGHOWDFRPSUHVVRUIROORZLQJWKHUHIHUHQFHEORFN:KLOHWKH
WZR EORFNV IORZ WKURXJK WKH FRPSUHVVRU UHSHWLWLRQV EHWZHHQ
WKH WZR EORFNV ZHUH VHDUFKHG 6KRZQ LQ )LJ  DV WKH WZR
EORFNVZHUHVFDQQHGFRPSUHVVHGGDWDLVRXWSXWIURPWKHGHOWD
HQFRGHU 7KH HQFRGHG GDWD FRQVLVWV RI DQ RIIVHW D IODJ DQ
LQGH[ DQG WKH OHQJWK )RU H[DPSOH LQ GDWD EORFN %ONQHZ GDWD
ZRUGV 'Z DQG 'Z PDWFKHG GDWD VWUHDP VWDUWLQJ IURP
SRVLWLRQ  ZLWK OHQJWK  LQ UHIHUHQFH EORFN %ONUHI $QG KHQFH
WKH HQFRGHG RXWSXW LV     DV VKRZQ LQ WKH ILJXUH
+RZHYHUGDWDZRUG'ZLQ%ONQHZKDVQRPDWFKLQ%ONUHIWKH
HQFRGHG RXW LV WKHQ   'Z ,Q JHQHUDO HQFRGHU RXWSXW
FRQWDLQV IRXU ILHOGV RI LQIRUPDWLRQ ZKLOH VFDQQLQJ WKH GDWD
EORFN7KHIODJILHOGLQGLFDWHVZKHWKHUWKHFXUUHQWVWULQJKDVD
PDWFKLQWKH%ONUHI,WLVVHWWRLIDPDWFKVWULQJLVIRXQGLQWKH
%ONUHI IRUWKHFXUUHQWVWULQJEHLQJVFDQQHGDQGFOHDUWRLIQR
PDWFKLVIRXQG7KHLQGH[ILHOGJLYHVWKHVWDUWLQJSRVLWLRQRI
WKH PDWFKHG VWULQJ LQ %ONUHI 7KH OHQJWK ILHOG UHFRUGV WKH WRWDO
OHQJWK RI WKH PDWFKHG VWULQJ 7KH PLVV ILHOG UHFRUGV WKH GDWD
ZKLFKGRHVQRWDSSHDULQ%ONUHIZKHQWKHIODJILHOGLV
'HOWD HQFRGLQJ VWDUWV DIWHU %ONUHI  VWRUHV HYHU\ % VKLQJOH
LQWR WKHUHIHUHQFHGLFWLRQDU\ VR WKDW QRQHRI %ONUHI GDWD VWULQJV
ZLOO EH PLVVHG GXULQJ WKH VHDUFK 7KH SURFHVV LV UHIOHFWHG LQ
$OJRULWKP
$OJRULWKP6LQJOHGLFWLRQDU\HQFRGLQJ
LIUHIHUHQFHEORFNWKHQ
IRUL WREORFNVL]HGR
'LFWLRQDU\>L@ %ONUHI>,LL@
)LJXUH'HOWDFRPSUHVVLRQHQJLQHIRUHYHU\E\WHGDWDTXDQWXP
$VVKRZQLQ$OJRULWKPDVLQJOHODUJHGLFWLRQDU\LVVSOLW
LQWR  VPDOOHU GLFWLRQDULHV VR WKDW ERWK SRSXODWLQJ %ONUHI
GLFWLRQDULHV DQG GHOWD HQFRGLQJ RI %ONQHZ FDQ EH SURFHVVHG LQ
SDUDOOHO DPRQJ WKH GLFWLRQDULHV KDUGZDUH )RU HDFK VPDOOHU
GLFWLRQDU\ XSGDWLQJ DQG VHDUFKLQJ ZHUH GRQH WKURXJK WZR
SDUDOOHO DQG SLSHOLQHG VWUXFWXUHV DV VKRZQ LQ )LJ  2QH
SLSHOLQH SRSXODWHV WKH GLFWLRQDU\ XVLQJ WKH UHIHUHQFH EORFN
ZKLOH WKH RWKHU VFDQV WKH LQFRPLQJ GDWD EORFN WR EH
FRPSUHVVHG 7KH PDWFKHG UHIHUHQFH EORFN %ONUHI WDNHV WKH
UHIHUHQFHSLSHOLQHWRORDGWKHUHIHUHQFHGDWDLQWRWKHGLFWLRQDU\
(DFK % GDWD LQ WKH %ONUHI  VKLIWHG E\ RQH E\WH DW D WLPH LV
KDVKHG LQWR  ELWV KDVK YDOXH WKDW LV XVHG WR VHDUFK IRU D
PDWFKLQJ VWULQJ LQ WKH GLFWLRQDU\ 7KH GLFWLRQDU\ KDV 
HQWULHVDQGVWRUHVHYHU\%GDWDRI%ONUHIVHTXHQWLDOO\7RDYRLG
OLQHDU VHDUFK ZH XVH DQRWKHU5$0 WREXLOGWKH KDVK WDEOH ,W
KDVHQWULHVUHSUHVHQWLQJELWVKDVKYDOXHVRIDQ%LQSXW
+DVKWDEOH>KDVKBIXQF%ONUHI>LLL@@ LHQGIRU
HOVH
$ KDVK HQWU\ FRUUHVSRQGV WR  ELWV UHIHUHQFH DGGUHVV YDOXH
IRUL WREORFNVL]HGR IURPWRGLFWLRQDU\LQGH[
+DVKBLQGH[ +DVKWDEOH>KDVKBIXQF%ONQHZ>LhLh@
6WULQJPDWFKZLWK'LFWLRQDU\>+DVKBLQGH[@
(QFRGLQJ
HQGIRU
HQGLI
$VVKRZQLQ$OJRULWKPHQFRGLQJRID%ONQHZZLWKVL]H6
WDNHV6FORFNF\FOHVFORVHWRDZLUHVSHHGIRU%EXVZLGWK
+RZHYHUORDGLQJDQGVFDQQLQJ%ONUHIUXQVWLPHVVORZHUWKDQ
EXV GDWD IORZ 7KXV ZH SURYLGH $OJRULWKP  ZLWK PXOWLSOH
GLFWLRQDULHVWRSHUIRUPSDUDOOHOVWRUHDQGVHDUFK
$OJRULWKP0XOWLSOHGLFWLRQDULHVHQFRGLQJ
LIUHIHUHQFHEORFNWKHQIRUP WRGR
IRUL WREORFNVL]HGR
'LFWLRQDU\>P@>L@ %ONUHI>LhPLhP@
$IWHU WKH UHIHUHQFH EORFN ILQLVKHV XSGDWLQJ WKH KDVK WDEOH
DQG WKH GLFWLRQDU\ WKH%ONQHZ JRHV WKURXJK WKHGHOWD HQFRGLQJ
SLSHOLQH VR WKDW LW GRHV D TXLFN VHDUFK IRU UHSHDWHG VWULQJV
WKURXJKWKHIDVWVHDUFKVWUXFWXUH$ELWZLVHFRPSDULVRQLVDOVR
SHUIRUPHG WR PDNH VXUH WKH WZR VWULQJV PDWFK H[DFWO\ ELW E\
ELW2QFHDPDWFKLVIRXQGDVHTXHQWLDOVHDUFKLVSHUIRUPHGWR
PD[LPL]H WKH PDWFK OHQJWK 7KH VHDUFK UHVXOWV DUH WKHQ
HQFRGHGEDVHGRQWKHHQFRGLQJVFKHPHGHVFULEHGDERYH
6WULQJ PDWFKLQJ LV GRQH IRU HYHU\ % VKLQJOH LH
VXEVHTXHQWVKLQJOHVLQDGDWDEORFNDUHVKLIWHGE\MXVWRQHE\WH
DW D WLPH 6LQFH RXU EXV ZLGWK LV % RQH VHW RI GHOWD
FRPSUHVVLRQ HQFRGLQJ HQJLQH FDQQRW NHHS SDFH ZLWK WKH GDWD
+DVKWDEOH>P@>KDVKBIXQF%ONUHI>LhPLhP@@ L
HQGIRU
HQGIRU
HOVHIRUP WRGR
IRUL WREORFNVL]HGR
+DVKBLQGH[>P@ +DVKWDEOH>KDVKIXQF%ONQHZ>LhLh@
6WULQJPDWFKZLWK'LFWLRQDU\>P@>+DVKBLQGH[>P@@
(QFRGLQJ
HQGIRU
HQGIRU
)LJXUH3DUDOOHOGHOWDFRPSUHVVLRQHQFRGLQJVWUXFWXUH
HQGLI
IRUHYHU\RQHE\WHVKLIWVKLQJOHV

228
 Sketch computing
20
Software

Total execusion time (second)

Hardware
15

10

)LJXUH([SHULPHQWSODWIRUPIRUKDUGZDUHDFFHOHUDWLQJ
0
Linux kernel Japanese Census Google books
(2 .1GB) (489MB) (1GB)
WUDQVIHU VSHHG RI WKH EXV 7R KDYH D FRPSUHVVLRQ HQFRGLQJ )LJXUH6NHWFKFRPSXWLQJWLPHRQWKUHHGDWDVHWV
KDUGZDUHWKDWLVFDSDEOHRIZLUHVSHHGRSHUDWLRQZHFRQVWUXFW
 PRGXOHV WKDW ZRUN LQ SDUDOOHO DV VKRZQ LQ )LJ  (DFK HVVHQWLDO WR VWRUDJH ,2 SHUIRUPDQFH :H ILUVW HYDOXDWH
FKDQQHO VWRUHV DQG HQFRGHV RQH VKLQJOH 7KH  SDUDOOHO ODWHQFLHVRIWKHLQGLYLGXDOIXQFWLRQVLQRXUGHVLJQ
FKDQQHOV ZRUN FRQFXUUHQWO\ RQ  VKLQJOHV WKDW DUH RQH E\WH
VKLIWIURPHDFKRWKHU
,9 ,03/(0(17$7,21$1'(9$/8$7,21
$ ([SHULPHQWDOVHWXS
7KH WKUHH PDMRU KDUGZDUH PRGXOHV IRU VLPLODULW\ EDVHG
GHGXSH DV GLVFXVVHG DERYH DUH EXLOW RQ ;LOLQ[ 0/
GHYHORSPHQW ERDUG ZLWK D 9LUWH[ /;7 )3*$ 7KH
PD[LPXPFORFNVSHHGLV0+]IRUVRPHRIWKHORJLFGHVLJQ
$V D KDUGZDUH FRSURFHVVRU LW FRQQHFWV WR WKH KRVW WKURXJK D
3&,H*HQEULGJH$GDWDGHGXSOLFDWLRQVRIWZDUHVLPXODWRU
 6NHWFKFRPSXWLQJODWHQF\
7KHILUVWIXQFWLRQIRUVLPLODULW\EDVHGGHGXSHLVILQJHUSULQW
FRPSXWDWLRQ WR GHULYH VNHWFKHV IRU GDWD EORFNV 2XU ILUVW
H[SHULPHQW LV WR PHDVXUH WKH WLPHV WDNHQ IRU FRPSXWLQJ WKH
5DELQ ILQJHUSULQWV DQG GHULYLQJ D VNHWFK IRU HDFK GDWD EORFN
)LJVKRZVWKHPHDVXUHGODWHQF\IRUFRPSXWLQJVNHWFKHVRI
.% EORFNV XVLQJ VRIWZDUH LPSOHPHQWDWLRQ DQG RXU )3*$
LPSOHPHQWDWLRQ UHVSHFWLYHO\ 7KUHH VHWV RI EDUV DUH VKRZQ LQ
WKLV ILJXUH FRUUHVSRQGLQJ WR WKH WKUHH GDWD VHWV /LQX[ NHUQHO
-DSDQHVH &HQVXV GDWD DQG *RRJOH ERRN UHVSHFWLYHO\ ,W LV
VKRZQLQWKHILJXUHWKDWRXUKDUGZDUHLPSOHPHQWDWLRQUXQQLQJ
>@ LV UXQQLQJ RQ WKH KRVW 3& ZLWK ,QWHO5 FRUH70  'XR DW 0+] FORFN UHGXFHV WKH WRWDO FRPSXWDWLRQ WLPH
&38(ZLWK*+]DQG*%'5$0)LJVKRZVWKH VXEVWDQWLDOO\$WOHDVWDQRUGHURIPDJQLWXGHLPSURYHPHQWKDV
EORFNGLDJUDPRIKRZWKHKDUGZDUHPRGXOHVDUHFRQQHFWHGWR EHHQ REVHUYHG IURP RXU H[SHULPHQWV DOWKRXJK WKH VRIWZDUH
WKHKRVWV\VWHP)RUWKHSXUSRVHRISHUIRUPDQFHHYDOXDWLRQDQG
FRPSDULVRQ ZH LQVWDOOHG WKH VWDQGDUG GHGXSH VRIWZDUH
GRZQORDGHG IURP >@ %\ VWDQGDUG GHGXSH ZH PHDQ WKH
GHGXSH IXQFWLRQ WKDWSHUIRUP GDWDUHGXFWLRQRQO\ RQ LGHQWLFDO
GDWDFKXQNV $Q RSHQ VRXUFHVRIWZDUHSDFNDJH >@ WKDWGRHV
VLPLODULW\EDVHGGHGXSHZDVDOVRLQVWDOOHGLQRUGHUWRHYDOXDWH
WKHHIILFLHQF\DQGHIIHFWLYHQHVVRIRXUQHZO\GHVLJQKDUGZDUH
PRGXOHV 7KHUHIRUH LQ WKH IROORZLQJ GLVFXVVLRQV ZH ZLOO
FRPSDUH WKUHH GHGXSH V\VWHPV WKH VWDQGDUG GHGXSH WKH
VRIWZDUHVLPLODULW\EDVHGGHGXSHDQGRXUKDUGZDUHPRGXOHV
2QHLPSRUWDQWFKDOOHQJHIRUHYDOXDWLQJGHGXSHHIILFLHQF\LV
WKHGDWDVHWVXVHGIRUWKHHYDOXDWLRQ3XEOLVKHG,2WUDFHVDQG
VWDQGDUGEHQFKPDUNVGRQRWSURYLGHUHDOGDWDFRQWHQWVWKDWDUH
HVVHQWLDO IRU HYDOXDWLQJ GDWD UHGXQGDQFLHV :LWK WKH ODFN RI
VWDQGDUG EHQFKPDUN GDWD VHWV ZH FROOHFWHG WKUHH UHDO ZRUOG
GDWD VHWV  *% /LQX[ VRXUFH FRGH ZLWK GLIIHUHQW YHUVLRQV
0%-DSDQHVH&HQVXVGDWDDQG*%*RRJOHERRNV7KHVH
WKUHHGDWDVHWVKDYHGLIIHUHQWDPRXQWVRIGDWDGXSOLFDWLRQVDQG
YHUVLRQUDQRQWKH*+],QWHO5FRUHVHUYHUZLWKQRRWKHU
DSSOLFDWLRQV UXQQLQJ )RU H[DPSOH IRU /LQX[ NHUQHO GDWD VHW
XVLQJVRIWZDUHILQJHUSULQWFRPSXWDWLRQWRGHULYHVNHWFKHVRIDOO
GDWD EORFNV WDNHV RYHU  VHFRQGV ZKLOH RXU KDUGZDUH
LPSOHPHQWDWLRQ WDNHV RQO\  VHFRQGV  7KH DYHUDJH GHOD\
IRU FRPSXWLQJ D VNHWFK RI D .% EORFN LV DERXW  XV XVLQJ
WKH KDUGZDUH PRGXOH %XW LW WDNHV RYHU  XV WR GR WKH VDPH
XVLQJWKHVRIWZDUHPRGXOH)RUKLJKSHUIRUPDQFHVWRUDJHVXFK
DV 66' WKLV GLIIHUHQFH FDQ KDYH D VLJQLILFDQW LPSDFW RQ WKH
SURGXFWLRQ SHUIRUPDQFH RI GLVN ,2V 1RW RQO\ GRHV WKH
KDUGZDUH LPSOHPHQWDWLRQ VSHHGXS ILQJHUSULQW FRPSXWDWLRQ
JUHDWO\ EXW DOVR RIIORDGV WKH FRPSXWDWLRQ WR WKH DFFHOHUDWRU
DOORZLQJ WKH VHUYHU &38 WR FRQFHQWUDWH RQ DSSOLFDWLRQ
Reference block index
40
Software
35 Hardware
30
Latency (ns)

UHSUHVHQWGLIIHUHQWW\SHVRIUHDOZRUOGGDWD2XUSXUSRVHKHUHLV 25

WR YDOLGDWH WKDW RXU QHZ KDUGZDUH GHVLJQ FDQ SHUIRUP WKH 20
GHGXSHIXQFWLRQRIH[LVWLQJVRIWZDUHSDFNDJHVDQGWRVKRZWKH 15
IHDVLELOLW\RIXVLQJWKHKDUGZDUHDFFHOHUDWRUWRFDUU\RXWRQOLQH
GHGXSHIRUSULPDU\VWRUDJHDQGVWRUDJHFDFKHV 10

% 3HUIRUPDQFHFRPSDULVRQV 0
4 5 6 7 8
6LQFHGDWDGHGXSHIRUSULPDU\VWRUDJHDQGFDFKHVLVRQWKH Similarity threshold
FULWLFDOSDWKIRUSURGXFWLRQ,2VPLQLPL]LQJGHGXSHODWHQF\LV )LJXUH$YHUDJHODWHQFLHVRIUHIHUHQFHEORFNVHDUFKIRU
GLIIHUHQWVLPLODULW\WKUHVKROGV

229
SHUIRUPDQFH Delta compression

50 Software
 5HIHUHQFHEORFNLQGH[ODWHQF\ Hardware

7KH VHFRQG IXQFWLRQ LV UHIHUHQFH EORFN VHDUFK WR ILQG WKH 40
EHVWUHIHUHQFHEORFNIRUGHOWDHQFRGLQJ:HPHDVXUHGWKHWLPH

Latency (us)
LWWDNHVWRVHDUFKIRUDPDWFKHGUHIHUHQFHEORFNXSRQHDFKQHZ 30

FRPLQJEORFN)LJVKRZVWKHPHDVXUHGUHVXOWVIRUGLIIHUHQW
VLPLODULW\WKUHVKROGYDOXHVZKLOHUHIHUHQFHEORFNLQGH[PRGXOH 20

LVUXQQLQJDW0+]FORFNVSHHG7KHVLPLODULW\WKUHVKROGLV
XVHG WR GHWHUPLQH KRZ PDQ\ ILQJHUSULQWV LQ D VNHWFK VKRXOG 10

PDWFK EHIRUH D GHOWD HQFRGLQJ LV SHUIRUPHG 7KH ORZHU WKH

0
WKUHVKROGYDOXHLVWKHKLJKHUWKHFKDQFHWRILQGVLPLODUEORFNV 4 5 6 7 8
+RZHYHULWDOVRLQFUHDVHVWKHFKDQFHRIIDOVHSRVLWLYHLHWZR Similarity threshold

EORFNV DUH FRQVLGHUHG VLPLODU EDVHG RQ D IHZ ILQJHUSULQWV )LJXUH'HOWDFRPSUHVVLRQWLPHFRPSDULVRQEHWZHHQ

PDWFK EXW WKH\ DUH QRW GHOWD FRPSUHVVLEOH +LJKHU WKUHVKROG KDUGZDUHDQGVRIWZDUHZLWKGLIIHUHQWWKUHVKROGV
YDOXHRQWKHRWKHUKDQGJLYHVDEHWWHUFKDQFHWKDWWZREORFNV UHIHUHQFH EORFN XVLQJ RXU KDUGZDUH FRPSUHVVRU :H DOVR
DUH GHOWD FRPSUHVVLEOH EHFDXVH WKH\ KDYH PRUH PDWFKHG PHDVXUHG WKH VDPH GHOWD HQFRGLQJ WLPH XVLQJ WKH VRIWZDUH
ILQJHUSULQWV LQ WKHLU UHVSHFWLYH VNHWFKHV %XW VRPH GHOWDFRPSUHVVRU0LQL/=2>@7KHSHUIRUPDQFHFRPSDULVRQ
FRPSUHVVLEOH EORFNV PD\ EH PLVVHG LI WKH WKUHVKROG YDOXH LV LVVKRZQLQ)LJ7KHGHOWDHQFRGLQJWLPHYDULHVGHSHQGLQJ
WRRKLJK RQWKHFKRVHQVLPLODULW\WKUHVKROGYDOXHV,WLVFOHDUIURPWKLV
)URP)LJZHFDQVHHWKDWRXUKDUGZDUHLPSOHPHQWDWLRQ ILJXUHWKDWWKHKDUGZDUHFRPSUHVVRUVKRZVWRWLPHVIDVWHU
RI UHIHUHQFH EORFN VHDUFK WDNHV OHVV WLPH WKDQ LWV VRIWZDUH SHUIRUPDQFH WKDQ LWV VRIWZDUH FRXQWHUSDUW 7KLV VXEVWDQWLDO
FRXQWHUSDUW +RZHYHU WKH ODWHQF\ UHGXFWLRQ LV QRW DV VSHHGXS RQ GHOWD HQFRGLQJ LV YHU\ LPSRUWDQW IRU GHGXSH RQ
VXEVWDQWLDO DV WKH ILQJHUSULQW FRPSXWDWLRQ SDUW )URP RXU SULPDU\ VWRUDJH DQG VWRUDJH FDFKHV ,W LV QRWLFHG WKDW WKH
H[SHULPHQWVZHREVHUYHGWZRUHDVRQVIRUWKLV)LUVWRIDOORXU HQFRGLQJWLPHRIWKHKDUGZDUHPRGXOHLVDERXWXVZKLOHWKH
KDUGZDUH GHVLJQ RQ WKLV SDUW LV VWLOO SUHOLPLQDU\ DQG WKHUH DUH VRIWZDUH PRGXOH WDNHV DURXQG  XV WKDW FRXOG WDNH D
URRPV IRU RSWLPL]DWLRQ JLYHQ PRUH WLPH 7KH VRIWZDUH VLJQLILFDQWSDUWRIDQ,2UHVSRQVHWLPH
LPSOHPHQWDWLRQRQWKHRWKHUKDQGVLVSUHWW\PDWXUHZLWKPDQ\
EXLOW LQ RSWLPL]DWLRQV 6HFRQGO\ WKH ODWHQF\ WLPH LV RQ WKH & 'DWDGHGXSH5DWLR
RUGHURI WKHVRI QDQRVHFRQGV7KHUHLV QRW PXFK URRP IRU ,Q RUGHU WR YDOLGDWH GHGXSH FDSDELOLW\ RI RXU KDUGZDUH
LPSURYHPHQW ZLWK KDUGZDUH LPSOHPHQWDWLRQ VLQFH WKH FORFN GHVLJQ ZH FDUULHG RXW H[SHULPHQWV WR PHDVXUH WKH GDWD
VSHHG RI WKH )3*$ RQ ZKLFK ZH LPSOHPHQWHG WKH FLUFXLW LV UHGXFWLRQ UDWLR RI WKH KDUGZDUH GHGXSH V\VWHP :H FRPSDUH
0+]ZKLOHWKH&38XVHGWRUXQWKHVRIWZDUHPRGXOHLVD WKLVUDWLRZLWKPDWXUHVRIWZDUHGHGXSHV\VWHPV7KHSXUSRVHLV
PXOWLFRUH ZLWK *+]FORFN VSHHG )RUWXQDWHO\ WKLVSDUWRI WR PDNH VXUH WKDW WKH KLJK VSHHG KDUGZDUH FDQ DFKLHYH WKH
WKH SURFHVVLQJ VKRXOG QRW GUDJ GRZQ WKH ,2 SHUIRUPDQFH H[SHFWHGGDWDUHGXFWLRQV)LJDVKRZVWKHGDWDUHGXFWLRQ
VLJQLILFDQWO\VLQFHLWLVLQQDQRVHFRQGUDQJHDQHJOLJLEOHWLPH UDWLRVRIVLPLODULW\EDVHGGHGXSHIRUERWKVRIWZDUHSDFNDJHDQG
IRUGLVN,2RSHUDWLRQV WKHKDUGZDUHLPSOHPHQWDWLRQIRU/LQX[NHUQHOVGDWDVHW,WFDQ
EHVHHQLQWKLVILJXUHWKDWWKHGDWDUHGXFWLRQUDWLRVRIWKHWZR
 'HOWDHQFRGLQJODWHQF\ V\VWHPV DUH FRPSDUDEOH IRU DOO VLPLODULW\ WKUHVKROG YDOXHV
FRQVLGHUHG :H QRWLFHG WKDW IRU ORZHU VLPLODULW\ WKUHVKROG
 7KHWKLUGIXQFWLRQLVGHOWDHQFRGLQJ:HPHDVXUHGWKHGHOWD
YDOXHV VXFK DV  VRIWZDUH SDFNDJH GRHV D OLWWOH ELW EHWWHU MRE
HQFRGLQJ WLPH RI HDFK .% DVVRFLDWHG EORFN DJDLQVW D .%

Linux kernel Japansese census Google book

3 3 3
Dedupe ratio = before size / after size

Dedupe ratio = before size / after size

Dedupe ratio = before size / after size

Software Software Software

Hardware Hardware Hardware
2.5 2.5 2.5

2 2 2

1.5 1.5 1.5

1 1 1
4 5 6 7 8 4 5 6 7 8 4 5 6 7 8
Similarity threshold Similarity threshold Similarity threshold

D EF
)LJXUH6LPLODULW\EDVHGGDWDUHGXFWLRQFRPSDULVRQEHWZHHQKDUGZDUHDQGVRIWZDUHIRUWKUHHGDWDVHWVZLWKGLIIHUHQWVLPLODULW\WKUHVKROGV

230
WKDQ WKH KDUGZDUH LPSOHPHQWDWLRQ 2XU DQDO\VLV RI WKH
KDUGZDUH GHVLJQ DQG WKH VRIWZDUH SDFNDJH VXJJHVWV WKH
IROORZLQJ UHDVRQV IRU WKLV )LUVW RI DOO ZLWK WKH VRIWZDUH
FRPSUHVVRU GDWD HQFRGLQJ FDQ EH GRQH ERWK LQQHU EORFN DQG
LQWHU EORFNV EHWZHHQ WKH UHIHUHQFH EORFN DQG WKH DVVRFLDWHG
EORFN ,Q WKH KDUGZDUH LPSOHPHQWDWLRQ RQ WKH RWKHU KDQGV
RQO\ LQWHU EORFN FRPSUHVVLRQ LV SHUIRUPHG )XUWKHU
LPSURYHPHQWRQWKHKDUGZDUHGHVLJQLVSRVVLEOH6HFRQGO\IRU
VPDOOHU YDOXH WKUHVKROG VRIWZDUH FDQ ZRUN PXFK KDUGHU ZLWK
PRUH LWHUDWLRQV WR ILQG VWULQJ PDWFKHV ZLWKLQ DQG EHWZHHQ
EORFNV 7KH KDUGZDUH LPSOHPHQWDWLRQ ZLOO SHUIRUP MXVW RQH
SDVVDQGLWPD\PLVVVRPHVXEVWULQJPDWFKHV
2QHDGYDQWDJHRIKDUGZDUHLPSOHPHQWDWLRQLVRIIORDGLQJRIWKH
V\VWHPUHVRXUFHVRIWKHKRVWVHUYHUVRWKDWDSSOLFDWLRQVFDQUXQ
IDVWHU2XUSURSRVHGDUFKLWHFWXUHPDNHVWKHKLJKSHUIRUPDQFH
DFKLHYDEOH
,Q GDWD EORFN VNHWFK FRPSXWDWLRQ PRGXOH ZH SLFNHG D
UDQGRP LUUHGXFLEOH SRO\QRPLDO +RZHYHU IRU WKH NLQG RI
DSSOLFDWLRQV ZH WDUJHW UHGXFLEOH SRO\QRPLDOV FDQ EH DGRSWHG
WRR ,Q WKDW FDVH WKH GDWD ILQJHUSULQWV PD\ UXQ LQWR PRUH
FRQIOLFWV DQG WKH SURSHUW\ PD\ EH EHQHILFLDO IRU ODWHU GDWD
SURFHVVLQJ 7KH KDUGZDUH PRGXOH GHVLJQ GXH WR WKLV FKDQJH
ZLOOEHVLPSOHDQGIDVW
)RU UHIHUHQFH LQGH[LQJ GHVLJQ VSOLWWLQJ % EORFN VNHWFK
)LJEDQG)LJFVKRZWKHGDWDUHGXFWLRQUDWLRVRI
WKHRWKHUWZRGDWDVHWV-DSDQHVHFHQVXVGDWDDQG*RRJOHERRN
UHVSHFWLYHO\ 6LPLODU REVHUYDWLRQV DUH REWDLQHG WR WKDW RI )LJ
LQWR &QSDUDOOHOEORRP ILOWHUV FDQDYRLG VHDUFKLQJ &Q FRSLHV
RI UHIHUHQFH EORFN VNHWFK OLVW DW VDPH WLPH &RPSDUHG WR
PXOWLSOH FRSLHV RI WZR WKRXVDQG HQWULHV UHIHUHQFH OLVW WKH
 D 7KH GDWD UHGXFWLRQ UDWLRV RI WKH VRIWZDUH SDFNDJH DQG
KDUGZDUH LPSOHPHQWDWLRQ DUH IDLUO\ FORVH EXW WKHUH DUH VPDOO
GLIIHUHQFHVEHWZHHQWKHWZRIRUVPDOOHUWKUHVKROGYDOXHV
:HKDYHDOVRFDUULHGRXWH[SHULPHQWVWRFRPSDUHWKHGDWD
UHGXFWLRQ UDWLRV RI VWDQGDUG GHGXSH DQG VLPLODULW\ EDVHG
GHGXSH7KHPHDVXUHGUHVXOWVDUHVKRZQLQ)LJ)URPWKLV
ILJXUH RQH FDQ VHH WKDW WKH VLPLODULW\ EDVHG GHGXSH VKRZV
EHWWHU GDWD UHGXFWLRQ WKDQ VWDQGDUG GHGXSH EHFDXVH RI WKH
H[LVWHQFH RI VLPLODU GDWD EORFNV )URP RXU H[SHULPHQWV ZH
REVHUYHG DERXW  EHWWHU GDWD UHGXFWLRQ RI VLPLODULW\ EDVHG
GHGXSH WKDQ VWDQGDUG GHGXSH :H EHOLHYH VXFK LPSURYHPHQW
VKRXOG EH PXFK ELJJHU LQ UHDO ZRUOG HQYLURQPHQWV DQG LQ
SURGXFWLRQ V\VWHPV 0RUH VLPLODULW\ H[LVWV LQ UHDO ZRUOG GDWD
VXFK DV GDWDEDVHV ELJ GDWD ODUJH ILOHV VHQVRU GDWD DQG GDWD
EHLQJ SURFHVVHG E\ VHUYHUV $V D UHVXOW VLPLODULW\ EDVHG GDWD
GHGXSHVKRXOGSHUIRUPPXFKEHWWHULQWHUPVRIGDWDUHGXFWLRQ
' 'LVFXVVLRQV
%RWK VRIWZDUH LPSOHPHQWDWLRQ DQG KDUGZDUH
LPSOHPHQWDWLRQKDYHWKHLUUHVSHFWLYHDGYDQWDJHVGHSHQGLQJRQ
WKH V\VWHP HQYLURQPHQWV 6RIWZDUH LV IOH[LEOH WR LPSOHPHQW
EXW GRHV QHHG KLJKO\ VNLOOIXO KDQGWXQLQJ WR UHDFK LWV
SRWHQWLDOV 6XFK GHVLJQV DOVR UHTXLUH FKDQJHV LQ WKH 26 RU
UHTXHVW XVH RI VSHFLDO OLEUDULHV ,W WDNHV WKH V\VWHP UHVRXUFHV
VXFKDV&38DQGPHPRU\IURPWKHDSSOLFDWLRQV2QWKHRWKHU
KDQGWKHKDUGZDUHDSSURDFKGRHVUHTXLUHGHVLJQFXVWRPL]DWLRQ
HLWKHURQDVWRUDJHGHYLFHD+%$RUDQGHGLFDWHGDFFHOHUDWRU
Standard dedupe vs hardware
3
Dedupe ratio = before size / after size
2.5
2 )XWXUH RSWLPL]DWLRQ FDQ VWLOO EH DFKLHYHG E\ LPSURYLQJ
1.5
1
Linux kernel Japanese Census Google books
(2 .1GB)
)LJXUH&RPSDULVRQEHWZHHQVWDQGDUGGHGXSHDQGKDUGZDUHGHGXSH
IRUWKUHHGDWDVHWV
Standard dedupe
Hardware similarity=8
Hardware similarity=7
Hardware similarity=6
Hardware similarity=5
Hardware similarity=4
(489MB) (1GB)
231
KDUGZDUH UHVRXUFH FRQVXPSWLRQ RI EORRP ILOWHUV LV PXFK
VPDOOHU,QDGGLWLRQEHFDXVHWKHSUHFRQGLWLRQRIWKHUHIHUHQFH
OLVW LW GRHV QRW H[LVW WZR % VNHWFKHV WKDW VKDUHG WKH VDPH Q
E\WHVGDWD>@WKHFRQIOLFWRIKDVKLQGH[WDEOHLVORZVRWKDW
QRPXOWLSOHKDVKHQWULHVZLOOPDSLQWRRQHORFDWLRQVLQFHWKHUH
LV QR VDPH Q E\WH VNHWFK SHUPXWDWLRQ H[LVWV LQ WKH UHIHUHQFH
OLVW7KXVRXUGHVLJQFDQILQGSRWHQWLDOPDWFKEORFNLQDVKRUW
WLPH
+LJKHUGHGXSHUDWLRLVSRVVLEOHLIRXUGHOWDHQFRGHULVWXQHG
WRORRNGXSOLFDWLRQVLQVLGHDQDVVRFLDWHGEORFNLWVHOILQDGGLWLRQ
WR GXSOLFDWHV EHWZHHQ WKH DVVRFLDWHG EORFN DQG LWV UHIHUHQFH
EORFN 2XU FXUUHQW LPSOHPHQWDWLRQ RI WKH GHOWD FRPSUHVVLRQ
HQFRGLQJ RQO\ ILQG UHSHDWHG GDWD VWULQJV IURP WKH UHIHUHQFH
EORFN EXW GRHV QRW VHDUFK IURP LQVLGH WKH DVVRFLDWHG EORFN
LWVHOI :H EHOLHYH GHGXSH FDQ EH LPSURYHG E\ VHDUFKLQJ
UHSHDWHG VWULQJV IURP ERWK WKH UHIHUHQFH EORFN DQG WKH
DVVRFLDWHGEORFN
9 &21&/86,21$1')8785(:25.6
:H KDYH SURSRVHG D KDUGZDUH DFFHOHUDWRU WR VSHHG XS
VLPLODULW\ EDVHG GHGXSOLFDWLRQ IRU SULPDU\ VWRUDJHV RU VWRUDJH
FDFKHV  7KH QHZ KDUGZDUH GHVLJQV KDYH DFKLHYHG GHGXSH
UDWLRVWKDWDUHFRPSDUDEOHWRPDWXUHVRIWZDUHLPSOHPHQWDWLRQV
ZKLOH SURYLGLQJ  WR  WLPHV KLJKHU VSHHG WKDQ WKH VRIWZDUH
LPSOHPHQWDWLRQV $ ZRUNLQJ SURWRW\SH KDV EHHQ EXLOW XVLQJ
9LUWH[ )3*$ XQGHU ;LOLQ[ HQYLURQPHQW ([WHQVLYH
H[SHULPHQWVKDYHEHHQFDUULHGRXWE\FRQQHFWLQJWKHSURWRW\SH
WRDKRVWVHUYHUWKURXJKD3&,HLQWHUIDFHWRWHVWWKHYDOLGLW\DQG
SHUIRUPDQFHRIWKHKDUGZDUHLPSOHPHQWDWLRQ2XUH[SHULPHQWV
KDYH GHPRQVWUDWHG WKH SHUIRUPDQFH LQ WHUPV RI ODWHQF\ DQG
GDWDUHGXFWLRQUDWLRRIWKHKDUGZDUHGHVLJQ7KHDUFKLWHFWXUHLV
UHDGLO\ DSSOLFDEOH LQ SULPDU\ GDWD VWRUDJHV DQG FDFKLQJ LQ
K\EULGGDWDVWRUDJHV\VWHPV
KDUGZDUH GHOWD FRPSUHVVLRQ UDWLR 7KHUH LV D URRP IRU
LPSURYHPHQW IRU UHIHUHQFH EORFN LQGH[LQJ WR HOLPLQDWH RU
PLQLPL]H WKH XVDJH RI )3*$ 5$0 UHVRXUFHV IRU WKH KDVK
WDEOH:HDUHFXUUHQWO\ZRUNLQJRQLQWHJUDWLQJDOOWKHKDUGZDUH
PRGXOHV WRJHWKHU WR IRUP D FRPSOHWH GHGXSH V\VWHP IRU
SULPDU\VWRUDJHVDQGVWRUDJHFDFKHV

  ,VVXH1RYHPEHU
>@ 7RUVWHQ 6XHO 1DVLU 0HPRQ $OJRULWKPV IRU 'HOWD &RPSUHVVLRQ DQG
$&.12:/('*0(17 5HPRWH )LOH 6\QFKURQL]DWLRQ LQ :RUN VXSSRUWHG E\ 16) &$5((5
$ZDUG16)&&5DQGE\,QWHO&RUSRUDWLRQ
7KH DXWKRUV DUH JUDWHIXO WR 0U -LQJ <DQJ IRU KLV KHOS LQ >@ (-HEDPDODU /HDYOLQH '$VLU $QWRQ\ *QDQD 6LQJK +DUGZDUH
LQVWDOOLQJ WKH VRIWZDUH LQ RXU H[SHULPHQWV DQG SURYLGLQJ ,PSOHPHQWDWLRQ RI /=0$ 'DWD &RPSUHVVLRQ $OJRULWKP
VXJJHVWLRQV DQG FRPPHQWV 7KLV UHVHDUFK LV VXSSRUWHG LQ SDUW ,QWHUQDWLRQDO-RXUQDORI$SSOLHG,QIRUPDWLRQ6\VWHPV,-$,6,661
E\WKH+*6785,UHVHDUFKFRQWUDFW16)JUDQWV&&) S9ROXPH1R0DUFK
DQG&&)$Q\RSLQLRQVILQGLQJVDQGFRQFOXVLRQVRU >@ : &XL 1HZ /=: 'DWD &RPSUHVVLRQ $OJRULWKP DQG ,WV )3*$
UHFRPPHQGDWLRQV H[SUHVVHG LQ WKLV PDWHULDO DUH WKRVH RI WKH ,PSOHPHQWDWLRQWK3LFWXUH&RGLQJ6\PSRVLXP
DXWKRUV DQG GR QRW QHFHVVDULO\ UHIOHFW WKH YLHZV RI WKH
1DWLRQDO6FLHQFH)RXQGDWLRQ
5()(5(1&(6
>@ .LUDQ 6ULQLYDVDQ 7LP %LVVRQ *DUWK *RRGVRQ DQG .DODGKDU
9RUXJDQWL 1HW$SS ,QF L'HGXS /DWHQF\DZDUH ,QOLQH 'DWD
'HGXSOLFDWLRQ IRU 3ULPDU\ 6WRUDJH 86(1,; FRQIHUHQFH RQ +RW
7RSLFVLQ6WRUDJHDQG)LOH6\VWHPV
>@ 3KLOLS6KLODQH*UDQW:DOODFH0DUN+XDQJDQG:LQGVRU+VX'HOWD
&RPSUHVVHG DQG 'HGXSOLFDWHG 6WRUDJH 8VLQJ 6WUHDP,QIRUPHG
/RFDOLW\ LQ +RW6WRUDJH
 3URFHHGLQJV RI WKH WK 86(1,; 7KURXJKSXW 'DWD 6WRUDJHV WKH UG ,((( ,QWHUQDWLRQDO 6\PSRVLXP
FRQIHUHQFHRQ+RW7RSLFVLQ6WRUDJHDQG)LOH6\VWHPV
>@ -LQJ[LQ )HQJ -LUL 6FKLQGOHU $ 'HGXSOLFDWLRQ 6WXG\ IRU +RVWVLGH
&DFKHV ZLWK '\QDPLF :RUNORDGV LQ 9LUWXDOL]HG 'DWD &HQWHU
>@ 3DXO5:LOVRQ6FRWW).DSODQDQG<DQQLV6PDUDJGDNLV7KH&DVH
IRU&RPSUHVVHG&DFKLQJLQ9LUWXDO0HPRU\6\VWHPV3URFHHGLQJVRI
WKH86(1,;$QQXDO7HFKQLFDO&RQIHUHQFH
>@ 0 5DELQ )LQJHUSULQWLQJ E\ UDQGRP SRO\QRPLDOV $LNHQ
&RPSXWDWLRQ/DERUDWRU\8QLY
>@ $=%URGHU0&KDULNDU$0)ULH]HDQG00LW]HQPDFKHU0LQ
:LVH,QGHSHQGHQW3HUPXWDWLRQV&RPSXWHUDQG6\VWHP6FLHQFHVSS

>@ 'RQJ\DQJ /L4LQJER:DQJ&\ULO *X\RW $VKZLQ1DUDVLPKD'HMDQ
9XFLQLFDQG=YRQLPLU%DQGLF4LQJ<DQJ $3DUDOOHODQG3LSHOLQHG
$UFKLWHFWXUH IRU $FFHOHUDWLQJ )LQJHUSULQW &RPSXWDWLRQ LQ +LJK
RQ)LHOG3URJUDPPDEOH &XVWRP &RPSXWLQJ 0DFKLQHV )&&0 
3DSHU6HVVLRQ7
>@ 4<DQJDQG-5HQ,&$6+,QWHOOLJHQWO\FRXSOHGDUUD\RIVVGDQG
(QYLURQPHQWV 0DVV 6WRUDJH 6\VWHPV DQG 7HFKQRORJLHV 0667 S KGG +LJK 3HUIRUPDQFH &RPSXWHU $UFKLWHFWXUH +3&$  ,(((
a0D\ WK,QWHUQDWLRQDO6\PSRVLXPRQ,(((SS
>@ -LDQ /LX <XQSHQJ &KDL ;LDR 4LQ <XDQ ;LDR 3/&FDFKH >@ /XNDV .HNHO\ 0DUWLQ =DGQN -LU 0DWRXVHN -DQ .RUHQHN
(QGXUDEOH66'FDFKHIRUGHGXSOLFDWLRQEDVHGSULPDU\VWRUDJH0DVV )DVW ORRNXS IRU G\QDPLF SDFNHW ILOWHULQJ LQ )3*$ 'HVLJQ DQG
6WRUDJH6\VWHPVDQG7HFKQRORJLHV0667Sa 'LDJQRVWLFV RI (OHFWURQLF &LUFXLWV  6\VWHPV WK ,QWHUQDWLRQDO
6\PSRVLXPRQ$SULOSS
>@ 0LQ )X 'DQ )HQJ <X +XD ;XELQ +H =XRQLQJ &KHQ :HQ ;LD
)DQJWLQJ +XDQJ DQG 4LQJ /LX  $FFHOHUDWLQJ 5HVWRUH DQG *DUEDJH >@ 0LQ)X'HVWRU*LWKXEKWWSVJLWKXEFRPIRP\GHVWRU
&ROOHFWLRQ LQ 'HGXSOLFDWLRQEDVHG %DFNXS 6\VWHPV YLD ([SORLWLQJ >@ 0DUNXV )UDQ] ;DYHU -RKDQQHV 2EHUKXPHU PLQL VXEVHW RI WKH /=2
+LVWRULFDO,QIRUPDWLRQLQ86(1,;$7&
 UHDOWLPHGDWDFRPSUHVVLRQOLEUDU\LQ*LWKXE
>@ -DPHV-+XQW.LHP3KRQJ9R:DOWHU) 7LFK\'HOWDDOJRULWKPV KWWSVJLWKXEFRPGIHOLQWREOHQGHUJLWWUHHPDVWHUH[WHUQO]RPLQLO]R
DQ (PSULFDO $QDO\VLV $&0 7UDQVDFWLRQV RQ 6RIWZDUH (QJLQHHULQJ 
DQG0HWKRGRORJ\SS
>@ 3KLOLS6KLODQH0DUN+XDQJ*UDQW:DOODFHDQG:LQGVRU+VX:$1
2SWLPL]HG 5HSOLFDWLRQ RI %DFNXS 'DWDVHWV 8VLQJ 6WUHDP,QIRUPHG
'HOWD&RPSUHVVLRQ$&07UDQVDFWLRQVRQ6WRUDJH726S9ROXPH


232