Documente Academic
Documente Profesional
Documente Cultură
'DWD'HGXSH
'RQJ\DQJ/L4LQJER:DQJ&\ULO*X\RW$VKZLQ1DUDVLPKD'HMDQ9XFLQLF=YRQLPLU%DQGLF4LQJ<DQJ
'HSWRI(OHFWULFDO&RPSXWHUDQG%LRPHGLFDO(QJLQHHULQJ8QLYRI5KRGH,VODQG.LQJVWRQ5,
6WRUDJH$UFKLWHFWXUH5HVHDUFK*URXS+*67D:HVWHUQ'LJLWDO&RPSDQ\6DQ-RVH&$
$EVWUDFW'DWD GHGXSOLFDWLRQ KDV SURYHQ LPSRUWDQW LQ DUH PRGLI\ DGG DQG GHOHWH 7KHVH RSHUDWLRQV PRVWO\
EDFNXS VWRUDJH V\VWHPV DV ODUJH DPRXQW RI LGHQWLFDO RU VLPLODU JHQHUDWH VLPLODU GDWD EORFNV LQVWHDG RI LGHQWLFDO EORFNV 7KLV
GDWDFKXQNVH[LVW5HFHQWVWXGLHVKDYHVKRZQWKHJUHDWSRWHQWLDO SURSHUW\ OHDGV XV WR ORRN DW GHGXSOLFDWLRQ WHFKQLTXHV DW VXE
RI GDWD GHGXSOLFDWLRQ LQ SULPDU\ VWRUDJH DQG VWRUDJH FDFKHV EORFNOHYHO7KHVHFRQGFKDOOHQJHLVWKHGLIIHUHQWUHTXLUHPHQW
'HGXSOLFDWLRQV LQ WKHVH HQYLURQPHQWV UHTXLUH KLJK VSHHG IRU WKH SHUIRUPDQFH PHWULFV ZKHUH GHGXSOLFDWLRQ LQ EDFNXS
SURFHVVLQJQRWWRGUDJGRZQSURGXFWLRQSHUIRUPDQFH7KLVSDSHU VWRUDJH LV PRUH WKURXJKSXW GHPDQGLQJ WKDQ WKH ODWHQF\
SUHVHQWV D KDUGZDUH DFFHOHUDWRU IRU VLPLODULW\ EDVHG GDWD VHQVLWLYHQDWXUHLQKHUHQWLQWKHSULPDU\VWRUDJHRIDSURGXFWLRQ
GHGXSOLFDWLRQ ,W LPSOHPHQWV WKUHH FRPSXWHLQWHQVLYH NHUQHO V\VWHP 7KH WKLUG FKDOOHQJH LV WKH OLPLWDWLRQ RI UHVRXUFHV
PRGXOHV WR LPSURYH WKURXJKSXW DQG ODWHQF\ LQ GHGXSH V\VWHPV
'HGXSH V\VWHP LQ WKH 3ULPDU\ VWRUDJH RIWHQ VKDUHV KDUGZDUH
VNHWFK FRPSXWDWLRQ IRU GDWD EORFNV LQGH[ VHDUFKLQJ IRU
DQGVRIWZDUHUHVRXUFHVRIWKHSURGXFWLRQV\VWHPZKLOHEDFNXS
UHIHUHQFH EORFN DQG GHOWD HQFRGLQJ RYHU VLPLODU EORFNV
$GRSWLQJSLSHOLQHGFRPSXWDWLRQDQGSDUDOOHOGDWDORRNXSDFURVV GHGXSOLFDWLRQ V\VWHP KDV LWV RZQ UHVRXUFHV 7DNLQJ VHUYHU
PXOWLSOH KDUGZDUH PRGXOHV RXU +: GHVLJQ LV FDSDEOH RI UHVRXUFHVVXFKDVWKH&38DQGWKH5$0UHVRXUFHVWRSHUIRUP
SURFHVVLQJ KLJK WKURXJKSXW GDWD WUDIILF E\ ZRUNLQJ RQ PXOWLSOH GHGXSOLFDWLRQ PD\ VORZ GRZQ WKH DSSOLFDWLRQ SHUIRUPDQFH
GDWD XQLWV FRQFXUUHQWO\ WKXV HQDEOLQJ ZLUH VSHHG GHGXSH IRU UXQQLQJRQWKHVHUYHUZKLFKLVXQGHVLUDEOH
GDWD VWUHDP ZKHUH VLPLODU EORFNV SUHVHQW 8VLQJ D 3& KRVW )LOHV RU GDWD EORFNV DUH IUHTXHQWO\ PRGLILHG DQG
V\VWHPFRQQHFWHGWRWKH)3*$EDVHGDFFHOHUDWRUWKURXJKD3&,H UHDVVHPEOHG LQ WKHLU OLIH F\FOH :LWK WKH GLIIHUHQFHV GHWHFWHG
*HQ LQWHUIDFH RXU H[SHULPHQWV VKRZ WKDW WKH VLPLODULW\
EHWZHHQ QHDUGXSOLFDWH GDWD EORFNV GHOWD HQFRGLQJ FDQ EH
EDVHG GDWD GHGXSHSHUIRUPVEHWWHULQ GDWDUHGXFWLRQUDWLR
WKDQFRQYHQWLRQDOGHGXSHWHFKQLTXHVWKDWORRNDWLGHQWLFDOEORFNV
HPSOR\HG WR GHGXSHGDWDDWRUEHORZ WKH ILOH OHYHO DV ZHOODV
RQO\ %\ FRPSDULQJ WKH KDUGZDUH LPSOHPHQWDWLRQ ZLWK LWV EORFN OHYHO 7KH EDVLF RSHUDWLRQ RI GHOWD HQFRGLQJ LV WR NHHS
VRIWZDUH FRXQWHUSDUW WKH H[SHULPHQWDO UHVXOWV VKRZ WKDW RXU RQH FRS\ RI VDPH FRQWHQWV LQ RQH RI WKH WZR VLPLODU GDWD
SUHOLPLQDU\ )3*$ LPSOHPHQWDWLRQ ZLWK PD[LPXP FORFN VSHHG EORFNV ZKLOH RQO\ VDYLQJ D UHIHUHQFH WR WKH GDWD LQ WKH RWKHU
RI0+]DFKLHYHVDWOHDVWWLPHVLPSURYHPHQWLQODWHQF\RYHU EORFN3KLOLS6KLODQHHWDOEXLOWDGHGXSHVWRUDJHXVLQJWKHGHOWD
WKHVRIWZDUHLPSOHPHQWDWLRQUXQQLQJRQVWDWHRIDUWVHUYHUV HQFRGLQJ VFKHPH >@ DQG VKRZHG WKDW WKH DGGLWLRQDO
GHGXSOLFDWLRQ EHQHILW GXH WR WKH GHOWD HQFRGLQJ LV WLPHV
.H\ZRUGV'DWD GHGXSOLFDWLRQ )LQJHUSULQWLQJ PXOWLSOH LPSURYHPHQW RQ WRS RI WKH FRQYHQWLRQDO GHGXSH WHFKQLTXHV
ORRNXSGLFWLRQDU\&DFKH3&,H +RZHYHU WKH WKURXJKSXW RI WKH FXUUHQWO\ VRIWZDUH EDVHG
GHGXSHSURFHVVUDQJHVIURP0%VWR0%VZKLFKDUHQRW
, ,1752'8&7,21 LGHDOIRUSULPDU\VWRUDJHRUFDFKHV\VWHPVZKLFKGHPDQGFORVH
WR JLJDE\WH SHU VHFRQG WKURXJKSXW DQG VXEPLOOLVHFRQG LQ
'DWDGHGXSOLFDWLRQKDVEHFRPHLQFUHDVLQJO\LPSRUWDQWGXH ODWHQF\>@
WR H[SORVLYH GDWD JURZWK LQ WKH ,QWHUQHW ZRUOG ,W KDV EHHQ
KLJKO\ VXFFHVVIXO LQ HQWHUSULVH EDFNXS HQYLURQPHQWV >@ ,Q RUGHU WR PDNH VLPLODULW\ EDVHG GHGXSH WHFKQRORJLHV
7\SLFDOO\ FRPSDQLHV FRQGXFW GDLO\ LQFUHPHQWDO EDFNXSV DQG DSSOLFDEOHWRSULPDU\VWRUDJHVRUFDFKHVKDUGZDUHDFFHOHUDWLRQ
ZHHNO\IXOOEDFNXSVWRDUFKLYHWKHLUGDWD7KHJUHDWDPRXQWRI VKRXOG EH H[SORUHG 1RW RQO\ FDQ D KDUGZDUH LPSOHPHQWDWLRQ
GXSOLFDWHG GDWD OHW WKH XVH RI GHGXSOLFDWLRQ WHFKQRORJ\ RIIHU KLJK VSHHG GHGXSH EXW DOVR RIIORDG GHGXSH IXQFWLRQV
ZLGHVSUHDGLQHQWHUSULVHEDFNXSV\VWHPV>@ IURPVHUYHUVVRWKDWDSSOLFDWLRQSHUIRUPDQFHLVQRWQHJDWLYHO\
LPSDFWHG ,Q WKLV SDSHU ZH SUHVHQW D KDUGZDUH DFFHOHUDWRU
7KH VXFFHVV RI GDWD GHGXSOLFDWLRQ LQ EDFNXS V\VWHPV GHVLJQ IRU VLPLODULW\ EDVHG GHGXSH IRU SULPDU\ VWRUDJHV DQG
LQVSLUHG D QXPEHU RI HIIRUWV WR LQWURGXFH GHGXSOLFDWLRQ LQWR VWRUDJH FDFKHV 8WLOL]LQJ SLSHOLQH SURFHVVLQJ DQG SDUDOOHO
SULPDU\ VWRUDJH >@>@ 8QOLNH LQ WKH EDFNXS V\VWHP WKH VWUXFWXUHV RXU GHVLJQ SURYLGHV KLJK WKURXJKSXW DQG IDVW
SULPDU\ VWRUDJH LQ D SURGXFWLRQ HQYLURQPHQW SUHVHQWV WKH UHVSRQVHWLPH7KHSURSRVHGDUFKLWHFWXUHLVLPSOHPHQWHGRQD
GHGXSOLFDWLRQ IXQFWLRQDOLW\ ZLWK PXOWLSOH XQLTXH FKDOOHQJHV ;LOLQ[ 9LUWH[ )3*$ GHYHORSPHQW ERDUG 7KUHH PDMRU
)LUVWO\ D SULPDU\ VWRUDJH GRHV QRW KDYH DV PXFK GXSOLFDWHG KDUGZDUH PRGXOHV IRU WKH GHGXSH V\VWHP ZHUH IXOO\ WHVWHG WR
GDWD DV LQ WKH EDFNXS V\VWHPV 'DWD VHQW WR SULPDU\ VWRUDJH EHIXQFWLRQDO([WHQVLYHH[SHULPHQWVKDYHEHHQFDUULHGRXWWR
FRPHV IURP XVHU OHYHO DSSOLFDWLRQV VXFK DV GDWDEDVH DQG ILOH HYDOXDWH WKH SHUIRUPDQFH RI HDFK PRGXOH LQ WHUPV RI ODWHQF\
V\VWHPV 7KH PDLQ RSHUDWLRQV IRU WKHVH NLQGV RI DSSOLFDWLRQV DQG WKH GHGXSH UDWLR RI WKH RYHUDOO V\VWHP DV FRPSDUHG WR
225
)LJXUH$QH[DPSOHVKRZLQJWZRVKLQJOHV
,QGHSHQGHQW EORFN LWV GDWD FRQWHQW LV FROG ZKLFK
PHDQV LW GRHV QRW VKDUH VLJQLILFDQW DPRXQW RI GDWD
FRQWHQWZLWKDQ\RIWKHUHIHUHQFHEORFNV
7KHVH WKUHH NLQGV RI EORFN DUH WUHDWHG LQ GLIIHUHQW ZD\V
LQGHSHQGHQWEORFNVDUHGLUHFWO\VWRUHGLQWRPDLQVWRUDJHWKXV
PDNLQJ OLWWOH RU QR FRQWULEXWLRQ WR GDWD UHGXFWLRQ UHIHUHQFH )LJXUH%ORFNGLDJUDPRIKDUGZDUHGHVLJQIRUVNHWFK
EORFNVDUHFDFKHGVLQFHWKH\DUHIUHTXHQWO\DFFHVVHGDVVRFLDWH FRPSXWDWLRQZLWKILQJHUSULQWSLSHOLQHDQGVNHWFKVHOHFWLRQ
EORFNVDQGWKHUHIHUHQFHEORFNVWKDWDUHVLPLODUWRWKHPDUHIHG FRQGXFLYH WR REWDLQ PRUH HYHQO\ GLVWULEXWHG DQG ORZHU
WRGHOWDHQFRGLQJZKHUHWKHUHVXOWLQJGHOWDLVVWRUHGLQWRPDLQ PD[LPXPSLSHOLQHVWDJHODWHQF\6XFKVFKHPHLVUHIHUUHGWRDV
VWRUDJH 7KH IROORZLQJ VHFWLRQV H[SODLQ WKH GHVLJQ RI HDFK VSOLWIUHVKLQRXUGHVLJQ
PRGXOHWKDWLVLQFKDUJHRILGHQWLI\LQJWKHGLIIHUHQWEORFNVDQG
$VVKRZQLQ)LJEDVHGRQWKHSURSHUW\RIIUHVKVKLIW
HQVXLQJHQFRGLQJLIDSSOLFDEOH
DQGVSOLWIUHVKZHPD\GHVLJQDSLSHOLQHVWUXFWXUHWRSURYLGH
$ 6NHWFK&RPSXWDWLRQ D OLQH VSHHG 5DELQ ILQJHUSULQW FRPSXWDWLRQ $ UDQGRP
5DELQ ILQJHUSULQW LV D SRO\QRPLDO PRGXOR RSHUDWLRQ WKDW VDPSOLQJ WHFKQLTXH VXFK DV RQH XWLOL]LQJ 0LQZLVH WKHRU\ LV
FDQEHWUDQVIRUPHGLQWRPXOWLSOHELW;25FDOFXODWLRQVWKDWDUH WKHQ XVHG WR VHOHFW DPRQJ DOO 5DELQ ILQJHUSULQWV WR IRUP D
VXLWDEOH IRU KDUGZDUH LPSOHPHQWDWLRQ >@ ,Q RXU GHGXSH VNHWFK IRU WKH FXUUHQW GDWD EORFN >@ DV VKRZQ LQ )LJ
V\VWHP WKH VNHWFK RI D GDWD EORFN LV GHULYHG E\ FRPSXWLQJ 'HWDLOHGGHVLJQFDQEHIRXQGIURPRXUSUHYLRXVZRUN>@
5DELQ ILQJHUSULQW RQ HYHU\ IL[VL]HG VOLGLQJ ZLQGRZ HJ
E\WHVORQJUHIHUUHGWRDVVKLQJOHRYHUWKHGDWDVWUHDP)LJ % 5HIHUHQFHEORFNLQGH[
VKRZV WZR VXFK FRQVHFXWLYH VKLQJOHV$[ DQG %[:H ILUVW $IWHU WKH VNHWFK IRU HDFK EORFN LV FDOFXODWHG ZH XVH WKH
FRPSXWH)3$WKHILQJHUSULQWRI$[WKHILUVWVKLQJOHXVLQJ VNHWFKHV WR UHSUHVHQW HDFK GDWD EORFN DQG NHHS WUDFN RI ,2
SDUDOOHO ;25 FDOFXODWLRQV IRU SRO\QRPLDO PRGXODWLRQ DFFHVV ZLWK UHJDUGLQJ WR WKHVH VNHWFKHV %DVHG RQ WKH FRQWHQW
IROORZLQJ(TXDWLRQ ORFDOLW\ LH DFFHVV IUHTXHQF\ DQG UHFHQF\ RI GDWD FRQWHQWV
>@ZHVHOHFWDQGFDFKHWKHPRVWSRSXODUEORFNVDVUHIHUHQFH
)3 $ = $ PRG 3
EORFNV )RU H[DPSOH WKHUH DUH XS WR WZR WKRXVDQGV LQ RXU
2QFH WKH ILQJHUSULQW IRU $[ LV FDOFXODWHG 5DELQ H[SHULPHQWDO VHWXS7KHVHUHIHUHQFH EORFNVDQG WKHLU VNHWFKHV
ILQJHUSULQW RI %[ )3% FDQ EH REWDLQHG E\ DSSO\LQJ WKH DUH VWRUHG LQ D UHIHUHQFH OLVW (YHU\ QHZO\ JHQHUDWHG EORFN
IRUPXODLQ(TXDWLRQ VNHWFK LV XVHG DV D NH\ WR TXHU\ WKH UHIHUHQFH OLVW WR ILQG D
PDWFK :KHQ KLW WKH LQFRPLQJ EORFN LV DQ DVVRFLDWHG EORFN
)3 % = % PRG 3 = : [ 8 [j
PRG 3 + [j
$ PRG 3 PRG 3 DQGZLOOEHGHOWDHQFRGHGDJDLQVWWKHPDWFKLQJUHIHUHQFHEORFN
7KHUHPDLQLQJGDWDDQGSRLQWHUVIRUHQFRGHGGHOWDVLQVWHDGRI
:KHUH [j = [ PRG 3 WKH ILQJHUSULQW RI $[ LV FDOFXODWHG WKH RULJLQDO .% EORFN DUH VWRUHG LQ WKH SULPDU\ VWRUDJH RU
XVLQJDOORIWKHELWVLQVKLQJOH$[DQGZHFDOOWKHIXQFWLRQDV FDFKH
IUHVK 7KH ILQJHUSULQW RI %[ RQ WKH RWKHU KDQG LV
FDOFXODWHGXVLQJWKHILQJHUSULQWRI$DORQJZLWKWKHILUVWE\WH ,Q RXU GHVLJQ ZH DVVXPH WKDW D VNHWFK FRQWDLQV
RI $ DQG WKH ODVWE\WHRI %:H FDOO LW D VKLIW IXQFWLRQ ILQJHUSULQWVHDFKRIZKLFKLVRQHE\WHORQJ,IWZRGDWDEORFNV
1RWH WKDW IUHVK ORJLF WDNHV PRUH WLPH WKDQ WKH VKLIW ORJLF 7R KDYHQPDWFKHGILQJHUSULQWVEHWZHHQWKHLUUHVSHFWLYHVNHWFKHV
RSWLPL]HWKHSLSHOLQHLPSOHPHQWDWLRQZHFDQIXUWKHUVSOLWWKH ZKHUHQFDQEHIURPWRZHGHFODUHWKH\DUHQHDUGXSOLFDWH
IUHVK ORJLF LQWR PXOWLSOH VXEPRGXOHV WR UHGXFH WKH EORFNVQLVUHIHUUHGWRDVVLPLODULW\WKUHVKROG2QFHVXFKQHDU
FRPSXWDWLRQ FRPSOH[LW\ RI IUHVK IXQFWLRQV WKHUHIRUH GXSOLFDWH EORFN LV IRXQG LQ WKH UHIHUHQFH LQGH[ WKH
FRUUHVSRQGLQJ UHIHUHQFH EORFN ZLOO EH UHDG RXW DQG GHOWD
HQFRGLQJDJDLQVWLWLVSHUIRUPHG
7R ILQG D PDWFK DPRQJ PDQ\ UHIHUHQFH EORFNV LQ D VKRUW
WLPHZHEXLOGDKDUGZDUHVHDUFKVWUXFWXUHDVVKRZQLQ)LJ
7KHUHIHUHQFHOLVWWDEOHRQWKHXSSHUULJKWFRUQHURIWKHILJXUH
VWRUHVWKH PHWDGDWDRI WKH. UHIHUHQFHEORFNV(DFKHQWU\ LQ
WKLVWDEOHFRQWDLQVWKH%VNHWFKRIRQHRI.UHIHUHQFHEORFNV
DSRLQWHUWRWKHUHIHUHQFHEORFNGDWDDQGRWKHUQHFHVVDU\VWDWXV
LQIRUPDWLRQ WKHUHIRUH WKH UHIHUHQFH OLVW WDEOH PDLQWDLQV
VXFKHQWULHVLQPHPRU\7RWKHOHIWRIWKHUHIHUHQFHOLVWWDEOHLQ
WKHILJXUHLVWKHKDVKLQGH[WDEOH7KLVKDVKLQGH[WDEOHVWRUHV
)LJXUH5DELQILQJHUSULQWSLSHOLQH
SRLQWHUVWRWKHUHIHUHQFHOLVWWDEOH(QWU\KLQWKLVWDEOHSRLQWVDW
226
)LJXUH%ORFNGLDJUDPRIUHIHUHQFHEORFNLQGH[LQJ
WKH UHIHUHQFH OLVW WDEOH HQWU\ WKDW FRQWDLQV WKH VNHWFK ZKRVH SURFHVVUHSHDWVXQWLODPDWFKLVIRXQGRUDOOSRVLWLYHRXWSXWVRI
FKRVHQ Q ILQJHUSULQWV KDYH KDVK YDOXH K 6LQFH WKHUH DUH &Q WKH%ORRPILOWHUVKDYHEHHQH[KDXVWHG
DYDLODEOHFRPELQDWLRQRIQILQJHUSULQWVZHKDYH&QJURXSVLQ
WKH KDVK LQGH[ WDEOH +DVK NH\ LV JHQHUDWHG IURP Q E\WH ,GHQWLI\DQHZUHIHUHQFHEORFNDQGDGGLWWRWKHOLVW:KHQ
SHUPXWDWLRQRI%VLJQDWXUHZHXVH&5&LPSOHPHQWDWLRQZLWK WKHUHIHUHQFHEORFNOLVWZDVLQLWLDOO\SRSXODWHGRUZKHQDQHZ
DELWSRO\QRPLDOWRJHQHUDWHFRUUHVSRQGLQJKDVKNH\ UHIHUHQFHEORFNLVDGGHGWRWKHDOUHDG\SRSXODWHGOLVWWRUHSODFH
DQ/58/38>@UHIHUHQFHEORFNZHQHHGWRJRWKURXJKWKLV
,Q RUGHU WR DYRLG XQQHFHVVDU\ VHDUFKHV DQG FRPSDULVRQV VHDUFK VWUXFWXUH )LUVW WKH VNHWFK RI WKH QHZ UHIHUHQFH EORFN
ZH DGRSW &Q EORRP ILOWHUV WR SURPSWO\ WHUPLQDWH WKH VHDUFK QHHGVWREHDGGHGWRWKHUHIHUHQFHOLVWWDEOH6XSSRVHWKHVNHWFK
SURFHVVRQFHZHILQGWKHUHLVQRPDWFKLQWKHUHIHUHQFHOLVW$V LVWREHVWRUHGLQHQWU\1RIWKHUHIHUHQFHOLVWWDEOH1FRXOGEH
VKRZQLQWKHILJXUHWKH%ORRPILOWHUVFDQTXLFNO\SURQRXQFHD D QHZ HQWU\ RU DQ H[LVWLQJ HQWU\ FRQWDLQLQJ WKH /58/38
QHJDWLYHUHVXOWRQHDFKRIWKH&QILQJHUSULQWSHUPXWDWLRQVE\ UHIHUHQFHEORFNWKDWVKRXOGEHUHSODFHG:HWKHQFRPSXWH&Q
FKHFNLQJ WKH KDVK YDOXH RI DQ LQFRPLQJ EORFN >@ WKH VDPH KDVKYDOXHVFRUUHVSRQGLQJWRWKH&QQVLJQDWXUHSHUPXWDWLRQV
KDVK YDOXH XVHG WR EXLOG WKH KDVK LQGH[ WDEOH 7KH UHIHUHQFH :HVWRUH1LQWRDOO&QHQWULHVPDWFKLQJWKHKDVKYDOXHVLQWKH
VHDUFKVWUXFWXUHVKRZQLQ)LJDUHXVHGZLWKWZRSXUSRVHVLQ +DVKLQGH[WDEOH)LQDOO\WKHFRUUHVSRQGLQJELWVLQWKH%ORRP
PLQGWRVHDUFKIRUDUHIHUHQFHEORFNVLPLODUWRDQLQFRPLQJ ILOWHUVZLOOEHVHWUHIOHFWLQJWKHQHZPHPEHURILQWKHUHIHUHQFH
EORFN DQG DGGLQJ D QHZ UHIHUHQFH EORFN LQWR WKH UHIHUHQFH EORFNOLVW>@
EORFNOLVW7KHGHWDLOHGRSHUDWLRQVWKURXJKWKLVVHDUFKVWUXFWXUH
DUHVKRZQLQ)LJ 7DNLQJ DGYDQWDJH RI )3*$V SDUDOOHO FRPSXWDWLRQ RXU
KDUGZDUH GHVLJQ IRU UHIHUHQFH LQGH[ PRGXOH DOORZV SDUDOOHO
6HDUFK IRU D VLPLODU UHIHUHQFH EORFN VR DV WR LGHQWLI\ DQ VHDUFK DPRQJ DOO WKH &Q SDUDOOHO EORRP ILOWHUV DW VDPH WLPH
DVVRFLDWHG EORFN 8SRQ DUULYDO RI D QHZ EORFN LWV VNHWFK LV 2QFH D PDWFK FDVH LV IRXQG UHIHUHQFH LQGH[ FDQ VSRW WKH
FDOFXODWHG XVLQJ WKH ILQJHUSULQW FLUFXLW GHVFULEHG LQ WKH ORFDWLRQLQWKHUHIHUHQFHOLVW
SUHYLRXV VHFWLRQ 7KH UHVXOWDQW % VNHWFK LV HQWHUHG WR WKH
VHDUFK ORJLF DV VKRZQ DW WKH ULJKW VLGH RI )LJ Q E\WH & 'HOWDHQFRGLQJ
VLJQDWXUHVDUHSLFNHGXSIURPVLJQDWXUHVLQWKHVNHWFKWRIRUP 7KH GDWD WUDQVIHU LQWHUIDFH RQ RXU KDUGZDUH SODWIRUP
&QSHUPXWDWLRQVLQWRWDO(DFKRIWKHVHQE\WHVHOHFWLRQVJRHV EHWZHHQWKHDFFHOHUDWRUDQGWKHKRVWLV%LQZLGWK,QRUGHUWR
WKURXJK WKH KDVK IXQFWLRQ WR GHULYH D ELW KDVK YDOXH 7KLV
KDVK YDOXH LV IHG LQWR WKH %ORRP ILOWHU WR TXLFNO\ GHWHUPLQH 5DZGDWD'Z GRXEOHZRUG
ZKHWKHULWKDVDSRVVLEOHPDWFKLQWKHUHIHUHQFHEORFNOLVW:H
WKHQSLFNXSWKHILUVWSRVLWLYHUHVXOWDPRQJDOOWKH%ORRPILOWHUV 2IIVHW
%ONUHI 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z
WRSLFNXSWKHLQGH[SRLQWLQJDWWKHFRUUHVSRQGLQJHQWU\LQWKH %ONQHZ 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z 'Z
UHIHUHQFH OLVW WDEOH 7KH FRUUHVSRQGLQJ % VNHWFK LQ WKH
UHIHUHQFH OLVW WDEOH LV UHDG RXW WR DFWXDOO\ FRPSDUH ZLWK WKH 5HVXOW
VNHWFK RI WKH LQFRPLQJ DVVRFLDWHG EORFN ,I WKH QXPEHU RI 2IIVHW
PDWFKHGVLJQDWXUHVH[FHHGVQWKHQWKHVHDUFKLVDKLW:HZLOO IODJ
IROORZ WKH SRLQWHU VWRUHG LQ WKH HQWU\ WR UHDG WKH UHIHUHQFH LQGH[
EORFN GDWD WR SHUIRUP GHOWD HQFRGLQJ 2WKHUZLVH LI WKH WZR OHQJWK
VNHWFKHV GR QRW KDYH Q RU PRUH PDWFKLQJ VLJQDWXUHV ZH ZLOO PLVV 'Z 'Z
ORRN DW WKH QH[W SRVLWLYH RXWSXW RI WKH %ORRP ILOWHUV DQG WKH )LJXUH$QH[DPSOHVKRZLQJKDUGZDUHGHOWDHQFRGLQJ
227
DFKLHYH D ZLUH VSHHG HQFRGLQJ IRU VLPLODULW\ EDVHG GDWD
GHGXSOLFDWLRQ ZH ORRNV IRU HYHU\ % UHSHWLWLRQV IURP QHDU
GXSOLFDWHEORFNV)LJLOOXVWUDWHVWKHGHOWDHQFRGLQJDOJRULWKP
ZLWKWZRGDWDEORFNVDVLQSXW%ONUHILVDUHIHUHQFHEORFNWREH
GHOWD HQFRGHG DJDLQVW ,W LV ORDGHG LQWR WKH GLFWLRQDU\ ILUVW
%ONQHZ LV WKH DVVRFLDWHG EORFN WR EH FRPSUHVVHG ,W LV IHG LQWR
WKHGHOWDFRPSUHVVRUIROORZLQJWKHUHIHUHQFHEORFN:KLOHWKH
WZR EORFNV IORZ WKURXJK WKH FRPSUHVVRU UHSHWLWLRQV EHWZHHQ
WKH WZR EORFNV ZHUH VHDUFKHG 6KRZQ LQ )LJ DV WKH WZR
EORFNVZHUHVFDQQHGFRPSUHVVHGGDWDLVRXWSXWIURPWKHGHOWD
HQFRGHU 7KH HQFRGHG GDWD FRQVLVWV RI DQ RIIVHW D IODJ DQ
LQGH[ DQG WKH OHQJWK )RU H[DPSOH LQ GDWD EORFN %ONQHZ GDWD
ZRUGV 'Z DQG 'Z PDWFKHG GDWD VWUHDP VWDUWLQJ IURP
SRVLWLRQ ZLWK OHQJWK LQ UHIHUHQFH EORFN %ONUHI $QG KHQFH
WKH HQFRGHG RXWSXW LV DV VKRZQ LQ WKH ILJXUH
+RZHYHUGDWDZRUG'ZLQ%ONQHZKDVQRPDWFKLQ%ONUHIWKH
HQFRGHG RXW LV WKHQ 'Z ,Q JHQHUDO HQFRGHU RXWSXW )LJXUH'HOWDFRPSUHVVLRQHQJLQHIRUHYHU\E\WHGDWDTXDQWXP
FRQWDLQV IRXU ILHOGV RI LQIRUPDWLRQ ZKLOH VFDQQLQJ WKH GDWD
EORFN7KHIODJILHOGLQGLFDWHVZKHWKHUWKHFXUUHQWVWULQJKDVD $VVKRZQLQ$OJRULWKPDVLQJOHODUJHGLFWLRQDU\LVVSOLW
PDWFKLQWKH%ONUHI,WLVVHWWRLIDPDWFKVWULQJLVIRXQGLQWKH LQWR VPDOOHU GLFWLRQDULHV VR WKDW ERWK SRSXODWLQJ %ONUHI
%ONUHI IRUWKHFXUUHQWVWULQJEHLQJVFDQQHGDQGFOHDUWRLIQR GLFWLRQDULHV DQG GHOWD HQFRGLQJ RI %ONQHZ FDQ EH SURFHVVHG LQ
PDWFKLVIRXQG7KHLQGH[ILHOGJLYHVWKHVWDUWLQJSRVLWLRQRI SDUDOOHO DPRQJ WKH GLFWLRQDULHV KDUGZDUH )RU HDFK VPDOOHU
WKH PDWFKHG VWULQJ LQ %ONUHI 7KH OHQJWK ILHOG UHFRUGV WKH WRWDO GLFWLRQDU\ XSGDWLQJ DQG VHDUFKLQJ ZHUH GRQH WKURXJK WZR
OHQJWK RI WKH PDWFKHG VWULQJ 7KH PLVV ILHOG UHFRUGV WKH GDWD SDUDOOHO DQG SLSHOLQHG VWUXFWXUHV DV VKRZQ LQ )LJ 2QH
ZKLFKGRHVQRWDSSHDULQ%ONUHIZKHQWKHIODJILHOGLV SLSHOLQH SRSXODWHV WKH GLFWLRQDU\ XVLQJ WKH UHIHUHQFH EORFN
ZKLOH WKH RWKHU VFDQV WKH LQFRPLQJ GDWD EORFN WR EH
'HOWD HQFRGLQJ VWDUWV DIWHU %ONUHI VWRUHV HYHU\ % VKLQJOH FRPSUHVVHG 7KH PDWFKHG UHIHUHQFH EORFN %ONUHI WDNHV WKH
LQWR WKHUHIHUHQFHGLFWLRQDU\ VR WKDW QRQHRI %ONUHI GDWD VWULQJV UHIHUHQFHSLSHOLQHWRORDGWKHUHIHUHQFHGDWDLQWRWKHGLFWLRQDU\
ZLOO EH PLVVHG GXULQJ WKH VHDUFK 7KH SURFHVV LV UHIOHFWHG LQ (DFK % GDWD LQ WKH %ONUHI VKLIWHG E\ RQH E\WH DW D WLPH LV
$OJRULWKP KDVKHG LQWR ELWV KDVK YDOXH WKDW LV XVHG WR VHDUFK IRU D
$OJRULWKP6LQJOHGLFWLRQDU\HQFRGLQJ PDWFKLQJ VWULQJ LQ WKH GLFWLRQDU\ 7KH GLFWLRQDU\ KDV
LIUHIHUHQFHEORFNWKHQ HQWULHVDQGVWRUHVHYHU\%GDWDRI%ONUHIVHTXHQWLDOO\7RDYRLG
IRUL WREORFNVL]HGR OLQHDU VHDUFK ZH XVH DQRWKHU5$0 WREXLOGWKH KDVK WDEOH ,W
'LFWLRQDU\>L@ %ONUHI>,LL@ KDVHQWULHVUHSUHVHQWLQJELWVKDVKYDOXHVRIDQ%LQSXW
+DVKWDEOH>KDVKBIXQF%ONUHI>LLL@@ LHQGIRU
HOVH
$ KDVK HQWU\ FRUUHVSRQGV WR ELWV UHIHUHQFH DGGUHVV YDOXH
IRUL WREORFNVL]HGR IURPWRGLFWLRQDU\LQGH[
+DVKBLQGH[ +DVKWDEOH>KDVKBIXQF%ONQHZ>LhLh@ $IWHU WKH UHIHUHQFH EORFN ILQLVKHV XSGDWLQJ WKH KDVK WDEOH
6WULQJPDWFKZLWK'LFWLRQDU\>+DVKBLQGH[@
(QFRGLQJ DQG WKH GLFWLRQDU\ WKH%ONQHZ JRHV WKURXJK WKHGHOWD HQFRGLQJ
HQGIRU SLSHOLQH VR WKDW LW GRHV D TXLFN VHDUFK IRU UHSHDWHG VWULQJV
HQGLI WKURXJKWKHIDVWVHDUFKVWUXFWXUH$ELWZLVHFRPSDULVRQLVDOVR
SHUIRUPHG WR PDNH VXUH WKH WZR VWULQJV PDWFK H[DFWO\ ELW E\
$VVKRZQLQ$OJRULWKPHQFRGLQJRID%ONQHZZLWKVL]H6 ELW2QFHDPDWFKLVIRXQGDVHTXHQWLDOVHDUFKLVSHUIRUPHGWR
WDNHV6FORFNF\FOHVFORVHWRDZLUHVSHHGIRU%EXVZLGWK PD[LPL]H WKH PDWFK OHQJWK 7KH VHDUFK UHVXOWV DUH WKHQ
+RZHYHUORDGLQJDQGVFDQQLQJ%ONUHIUXQVWLPHVVORZHUWKDQ HQFRGHGEDVHGRQWKHHQFRGLQJVFKHPHGHVFULEHGDERYH
EXV GDWD IORZ 7KXV ZH SURYLGH $OJRULWKP ZLWK PXOWLSOH
GLFWLRQDULHVWRSHUIRUPSDUDOOHOVWRUHDQGVHDUFK 6WULQJ PDWFKLQJ LV GRQH IRU HYHU\ % VKLQJOH LH
VXEVHTXHQWVKLQJOHVLQDGDWDEORFNDUHVKLIWHGE\MXVWRQHE\WH
$OJRULWKP0XOWLSOHGLFWLRQDULHVHQFRGLQJ
LIUHIHUHQFHEORFNWKHQIRUP WRGR
DW D WLPH 6LQFH RXU EXV ZLGWK LV % RQH VHW RI GHOWD
IRUL WREORFNVL]HGR FRPSUHVVLRQ HQFRGLQJ HQJLQH FDQQRW NHHS SDFH ZLWK WKH GDWD
'LFWLRQDU\>P@>L@ %ONUHI>LhPLhP@
+DVKWDEOH>P@>KDVKBIXQF%ONUHI>LhPLhP@@ L
HQGIRU
HQGIRU
HOVHIRUP WRGR
IRUL WREORFNVL]HGR
+DVKBLQGH[>P@ +DVKWDEOH>KDVKIXQF%ONQHZ>LhLh@
6WULQJPDWFKZLWK'LFWLRQDU\>P@>+DVKBLQGH[>P@@
(QFRGLQJ
HQGIRU
HQGIRU
)LJXUH3DUDOOHOGHOWDFRPSUHVVLRQHQFRGLQJVWUXFWXUH
HQGLI
IRUHYHU\RQHE\WHVKLIWVKLQJOHV
228
Sketch computing
20
Software
10
)LJXUH([SHULPHQWSODWIRUPIRUKDUGZDUHDFFHOHUDWLQJ
0
Linux kernel Japanese Census Google books
(2 .1GB) (489MB) (1GB)
WUDQVIHU VSHHG RI WKH EXV 7R KDYH D FRPSUHVVLRQ HQFRGLQJ )LJXUH6NHWFKFRPSXWLQJWLPHRQWKUHHGDWDVHWV
KDUGZDUHWKDWLVFDSDEOHRIZLUHVSHHGRSHUDWLRQZHFRQVWUXFW
PRGXOHV WKDW ZRUN LQ SDUDOOHO DV VKRZQ LQ )LJ (DFK HVVHQWLDO WR VWRUDJH ,2 SHUIRUPDQFH :H ILUVW HYDOXDWH
FKDQQHO VWRUHV DQG HQFRGHV RQH VKLQJOH 7KH SDUDOOHO ODWHQFLHVRIWKHLQGLYLGXDOIXQFWLRQVLQRXUGHVLJQ
FKDQQHOV ZRUN FRQFXUUHQWO\ RQ VKLQJOHV WKDW DUH RQH E\WH
VKLIWIURPHDFKRWKHU 6NHWFKFRPSXWLQJODWHQF\
7KHILUVWIXQFWLRQIRUVLPLODULW\EDVHGGHGXSHLVILQJHUSULQW
,9 ,03/(0(17$7,21$1'(9$/8$7,21 FRPSXWDWLRQ WR GHULYH VNHWFKHV IRU GDWD EORFNV 2XU ILUVW
H[SHULPHQW LV WR PHDVXUH WKH WLPHV WDNHQ IRU FRPSXWLQJ WKH
$ ([SHULPHQWDOVHWXS 5DELQ ILQJHUSULQWV DQG GHULYLQJ D VNHWFK IRU HDFK GDWD EORFN
7KH WKUHH PDMRU KDUGZDUH PRGXOHV IRU VLPLODULW\ EDVHG )LJVKRZVWKHPHDVXUHGODWHQF\IRUFRPSXWLQJVNHWFKHVRI
GHGXSH DV GLVFXVVHG DERYH DUH EXLOW RQ ;LOLQ[ 0/ .% EORFNV XVLQJ VRIWZDUH LPSOHPHQWDWLRQ DQG RXU )3*$
GHYHORSPHQW ERDUG ZLWK D 9LUWH[ /;7 )3*$ 7KH LPSOHPHQWDWLRQ UHVSHFWLYHO\ 7KUHH VHWV RI EDUV DUH VKRZQ LQ
PD[LPXPFORFNVSHHGLV0+]IRUVRPHRIWKHORJLFGHVLJQ WKLV ILJXUH FRUUHVSRQGLQJ WR WKH WKUHH GDWD VHWV /LQX[ NHUQHO
$V D KDUGZDUH FRSURFHVVRU LW FRQQHFWV WR WKH KRVW WKURXJK D -DSDQHVH &HQVXV GDWD DQG *RRJOH ERRN UHVSHFWLYHO\ ,W LV
3&,H*HQEULGJH$GDWDGHGXSOLFDWLRQVRIWZDUHVLPXODWRU VKRZQLQWKHILJXUHWKDWRXUKDUGZDUHLPSOHPHQWDWLRQUXQQLQJ
>@ LV UXQQLQJ RQ WKH KRVW 3& ZLWK ,QWHO5 FRUH70 'XR DW 0+] FORFN UHGXFHV WKH WRWDO FRPSXWDWLRQ WLPH
&38(ZLWK*+]DQG*%'5$0)LJVKRZVWKH VXEVWDQWLDOO\$WOHDVWDQRUGHURIPDJQLWXGHLPSURYHPHQWKDV
EORFNGLDJUDPRIKRZWKHKDUGZDUHPRGXOHVDUHFRQQHFWHGWR EHHQ REVHUYHG IURP RXU H[SHULPHQWV DOWKRXJK WKH VRIWZDUH
WKHKRVWV\VWHP)RUWKHSXUSRVHRISHUIRUPDQFHHYDOXDWLRQDQG YHUVLRQUDQRQWKH*+],QWHO5FRUHVHUYHUZLWKQRRWKHU
FRPSDULVRQ ZH LQVWDOOHG WKH VWDQGDUG GHGXSH VRIWZDUH DSSOLFDWLRQV UXQQLQJ )RU H[DPSOH IRU /LQX[ NHUQHO GDWD VHW
GRZQORDGHG IURP >@ %\ VWDQGDUG GHGXSH ZH PHDQ WKH XVLQJVRIWZDUHILQJHUSULQWFRPSXWDWLRQWRGHULYHVNHWFKHVRIDOO
GHGXSH IXQFWLRQ WKDWSHUIRUP GDWDUHGXFWLRQRQO\ RQ LGHQWLFDO GDWD EORFNV WDNHV RYHU VHFRQGV ZKLOH RXU KDUGZDUH
GDWDFKXQNV $Q RSHQ VRXUFHVRIWZDUHSDFNDJH >@ WKDWGRHV LPSOHPHQWDWLRQ WDNHV RQO\ VHFRQGV 7KH DYHUDJH GHOD\
VLPLODULW\EDVHGGHGXSHZDVDOVRLQVWDOOHGLQRUGHUWRHYDOXDWH IRU FRPSXWLQJ D VNHWFK RI D .% EORFN LV DERXW XV XVLQJ
WKHHIILFLHQF\DQGHIIHFWLYHQHVVRIRXUQHZO\GHVLJQKDUGZDUH WKH KDUGZDUH PRGXOH %XW LW WDNHV RYHU XV WR GR WKH VDPH
PRGXOHV 7KHUHIRUH LQ WKH IROORZLQJ GLVFXVVLRQV ZH ZLOO XVLQJWKHVRIWZDUHPRGXOH)RUKLJKSHUIRUPDQFHVWRUDJHVXFK
FRPSDUH WKUHH GHGXSH V\VWHPV WKH VWDQGDUG GHGXSH WKH DV 66' WKLV GLIIHUHQFH FDQ KDYH D VLJQLILFDQW LPSDFW RQ WKH
VRIWZDUHVLPLODULW\EDVHGGHGXSHDQGRXUKDUGZDUHPRGXOHV SURGXFWLRQ SHUIRUPDQFH RI GLVN ,2V 1RW RQO\ GRHV WKH
KDUGZDUH LPSOHPHQWDWLRQ VSHHGXS ILQJHUSULQW FRPSXWDWLRQ
2QHLPSRUWDQWFKDOOHQJHIRUHYDOXDWLQJGHGXSHHIILFLHQF\LV JUHDWO\ EXW DOVR RIIORDGV WKH FRPSXWDWLRQ WR WKH DFFHOHUDWRU
WKHGDWDVHWVXVHGIRUWKHHYDOXDWLRQ3XEOLVKHG,2WUDFHVDQG DOORZLQJ WKH VHUYHU &38 WR FRQFHQWUDWH RQ DSSOLFDWLRQ
VWDQGDUGEHQFKPDUNVGRQRWSURYLGHUHDOGDWDFRQWHQWVWKDWDUH
HVVHQWLDO IRU HYDOXDWLQJ GDWD UHGXQGDQFLHV :LWK WKH ODFN RI Reference block index
VWDQGDUG EHQFKPDUN GDWD VHWV ZH FROOHFWHG WKUHH UHDO ZRUOG 40
Software
GDWD VHWV *% /LQX[ VRXUFH FRGH ZLWK GLIIHUHQW YHUVLRQV 35 Hardware
0%-DSDQHVH&HQVXVGDWDDQG*%*RRJOHERRNV7KHVH 30
WKUHHGDWDVHWVKDYHGLIIHUHQWDPRXQWVRIGDWDGXSOLFDWLRQVDQG
Latency (ns)
UHSUHVHQWGLIIHUHQWW\SHVRIUHDOZRUOGGDWD2XUSXUSRVHKHUHLV 25
WR YDOLGDWH WKDW RXU QHZ KDUGZDUH GHVLJQ FDQ SHUIRUP WKH 20
GHGXSHIXQFWLRQRIH[LVWLQJVRIWZDUHSDFNDJHVDQGWRVKRZWKH 15
IHDVLELOLW\RIXVLQJWKHKDUGZDUHDFFHOHUDWRUWRFDUU\RXWRQOLQH
GHGXSHIRUSULPDU\VWRUDJHDQGVWRUDJHFDFKHV 10
% 3HUIRUPDQFHFRPSDULVRQV 0
4 5 6 7 8
6LQFHGDWDGHGXSHIRUSULPDU\VWRUDJHDQGFDFKHVLVRQWKH Similarity threshold
FULWLFDOSDWKIRUSURGXFWLRQ,2VPLQLPL]LQJGHGXSHODWHQF\LV )LJXUH$YHUDJHODWHQFLHVRIUHIHUHQFHEORFNVHDUFKIRU
GLIIHUHQWVLPLODULW\WKUHVKROGV
229
SHUIRUPDQFH Delta compression
50 Software
5HIHUHQFHEORFNLQGH[ODWHQF\ Hardware
7KH VHFRQG IXQFWLRQ LV UHIHUHQFH EORFN VHDUFK WR ILQG WKH 40
EHVWUHIHUHQFHEORFNIRUGHOWDHQFRGLQJ:HPHDVXUHGWKHWLPH
Latency (us)
LWWDNHVWRVHDUFKIRUDPDWFKHGUHIHUHQFHEORFNXSRQHDFKQHZ 30
FRPLQJEORFN)LJVKRZVWKHPHDVXUHGUHVXOWVIRUGLIIHUHQW
VLPLODULW\WKUHVKROGYDOXHVZKLOHUHIHUHQFHEORFNLQGH[PRGXOH 20
LVUXQQLQJDW0+]FORFNVSHHG7KHVLPLODULW\WKUHVKROGLV
XVHG WR GHWHUPLQH KRZ PDQ\ ILQJHUSULQWV LQ D VNHWFK VKRXOG 10
2 2 2
1 1 1
4 5 6 7 8 4 5 6 7 8 4 5 6 7 8
Similarity threshold Similarity threshold Similarity threshold
D EF
)LJXUH6LPLODULW\EDVHGGDWDUHGXFWLRQFRPSDULVRQEHWZHHQKDUGZDUHDQGVRIWZDUHIRUWKUHHGDWDVHWVZLWKGLIIHUHQWVLPLODULW\WKUHVKROGV
230
WKDQ WKH KDUGZDUH LPSOHPHQWDWLRQ 2XU DQDO\VLV RI WKH 2QHDGYDQWDJHRIKDUGZDUHLPSOHPHQWDWLRQLVRIIORDGLQJRIWKH
KDUGZDUH GHVLJQ DQG WKH VRIWZDUH SDFNDJH VXJJHVWV WKH V\VWHPUHVRXUFHVRIWKHKRVWVHUYHUVRWKDWDSSOLFDWLRQVFDQUXQ
IROORZLQJ UHDVRQV IRU WKLV )LUVW RI DOO ZLWK WKH VRIWZDUH IDVWHU2XUSURSRVHGDUFKLWHFWXUHPDNHVWKHKLJKSHUIRUPDQFH
FRPSUHVVRU GDWD HQFRGLQJ FDQ EH GRQH ERWK LQQHU EORFN DQG DFKLHYDEOH
LQWHU EORFNV EHWZHHQ WKH UHIHUHQFH EORFN DQG WKH DVVRFLDWHG
,Q GDWD EORFN VNHWFK FRPSXWDWLRQ PRGXOH ZH SLFNHG D
EORFN ,Q WKH KDUGZDUH LPSOHPHQWDWLRQ RQ WKH RWKHU KDQGV
UDQGRP LUUHGXFLEOH SRO\QRPLDO +RZHYHU IRU WKH NLQG RI
RQO\ LQWHU EORFN FRPSUHVVLRQ LV SHUIRUPHG )XUWKHU
DSSOLFDWLRQV ZH WDUJHW UHGXFLEOH SRO\QRPLDOV FDQ EH DGRSWHG
LPSURYHPHQWRQWKHKDUGZDUHGHVLJQLVSRVVLEOH6HFRQGO\IRU
WRR ,Q WKDW FDVH WKH GDWD ILQJHUSULQWV PD\ UXQ LQWR PRUH
VPDOOHU YDOXH WKUHVKROG VRIWZDUH FDQ ZRUN PXFK KDUGHU ZLWK
FRQIOLFWV DQG WKH SURSHUW\ PD\ EH EHQHILFLDO IRU ODWHU GDWD
PRUH LWHUDWLRQV WR ILQG VWULQJ PDWFKHV ZLWKLQ DQG EHWZHHQ
SURFHVVLQJ 7KH KDUGZDUH PRGXOH GHVLJQ GXH WR WKLV FKDQJH
EORFNV 7KH KDUGZDUH LPSOHPHQWDWLRQ ZLOO SHUIRUP MXVW RQH
ZLOOEHVLPSOHDQGIDVW
SDVVDQGLWPD\PLVVVRPHVXEVWULQJPDWFKHV
)RU UHIHUHQFH LQGH[LQJ GHVLJQ VSOLWWLQJ % EORFN VNHWFK
)LJEDQG)LJFVKRZWKHGDWDUHGXFWLRQUDWLRVRI
LQWR &QSDUDOOHOEORRP ILOWHUV FDQDYRLG VHDUFKLQJ &Q FRSLHV
WKHRWKHUWZRGDWDVHWV-DSDQHVHFHQVXVGDWDDQG*RRJOHERRN
UHVSHFWLYHO\ 6LPLODU REVHUYDWLRQV DUH REWDLQHG WR WKDW RI )LJ RI UHIHUHQFH EORFN VNHWFK OLVW DW VDPH WLPH &RPSDUHG WR
PXOWLSOH FRSLHV RI WZR WKRXVDQG HQWULHV UHIHUHQFH OLVW WKH
D 7KH GDWD UHGXFWLRQ UDWLRV RI WKH VRIWZDUH SDFNDJH DQG
KDUGZDUH LPSOHPHQWDWLRQ DUH IDLUO\ FORVH EXW WKHUH DUH VPDOO KDUGZDUH UHVRXUFH FRQVXPSWLRQ RI EORRP ILOWHUV LV PXFK
GLIIHUHQFHVEHWZHHQWKHWZRIRUVPDOOHUWKUHVKROGYDOXHV VPDOOHU,QDGGLWLRQEHFDXVHWKHSUHFRQGLWLRQRIWKHUHIHUHQFH
OLVW LW GRHV QRW H[LVW WZR % VNHWFKHV WKDW VKDUHG WKH VDPH Q
:HKDYHDOVRFDUULHGRXWH[SHULPHQWVWRFRPSDUHWKHGDWD E\WHVGDWD>@WKHFRQIOLFWRIKDVKLQGH[WDEOHLVORZVRWKDW
UHGXFWLRQ UDWLRV RI VWDQGDUG GHGXSH DQG VLPLODULW\ EDVHG QRPXOWLSOHKDVKHQWULHVZLOOPDSLQWRRQHORFDWLRQVLQFHWKHUH
GHGXSH7KHPHDVXUHGUHVXOWVDUHVKRZQLQ)LJ)URPWKLV LV QR VDPH Q E\WH VNHWFK SHUPXWDWLRQ H[LVWV LQ WKH UHIHUHQFH
ILJXUH RQH FDQ VHH WKDW WKH VLPLODULW\ EDVHG GHGXSH VKRZV OLVW7KXVRXUGHVLJQFDQILQGSRWHQWLDOPDWFKEORFNLQDVKRUW
EHWWHU GDWD UHGXFWLRQ WKDQ VWDQGDUG GHGXSH EHFDXVH RI WKH WLPH
H[LVWHQFH RI VLPLODU GDWD EORFNV )URP RXU H[SHULPHQWV ZH
REVHUYHG DERXW EHWWHU GDWD UHGXFWLRQ RI VLPLODULW\ EDVHG +LJKHUGHGXSHUDWLRLVSRVVLEOHLIRXUGHOWDHQFRGHULVWXQHG
GHGXSH WKDQ VWDQGDUG GHGXSH :H EHOLHYH VXFK LPSURYHPHQW WRORRNGXSOLFDWLRQVLQVLGHDQDVVRFLDWHGEORFNLWVHOILQDGGLWLRQ
VKRXOG EH PXFK ELJJHU LQ UHDO ZRUOG HQYLURQPHQWV DQG LQ WR GXSOLFDWHV EHWZHHQ WKH DVVRFLDWHG EORFN DQG LWV UHIHUHQFH
SURGXFWLRQ V\VWHPV 0RUH VLPLODULW\ H[LVWV LQ UHDO ZRUOG GDWD EORFN 2XU FXUUHQW LPSOHPHQWDWLRQ RI WKH GHOWD FRPSUHVVLRQ
VXFK DV GDWDEDVHV ELJ GDWD ODUJH ILOHV VHQVRU GDWD DQG GDWD HQFRGLQJ RQO\ ILQG UHSHDWHG GDWD VWULQJV IURP WKH UHIHUHQFH
EHLQJ SURFHVVHG E\ VHUYHUV $V D UHVXOW VLPLODULW\ EDVHG GDWD EORFN EXW GRHV QRW VHDUFK IURP LQVLGH WKH DVVRFLDWHG EORFN
GHGXSHVKRXOGSHUIRUPPXFKEHWWHULQWHUPVRIGDWDUHGXFWLRQ LWVHOI :H EHOLHYH GHGXSH FDQ EH LPSURYHG E\ VHDUFKLQJ
UHSHDWHG VWULQJV IURP ERWK WKH UHIHUHQFH EORFN DQG WKH
DVVRFLDWHGEORFN
' 'LVFXVVLRQV
%RWK VRIWZDUH LPSOHPHQWDWLRQ DQG KDUGZDUH 9 &21&/86,21$1')8785(:25.6
LPSOHPHQWDWLRQKDYHWKHLUUHVSHFWLYHDGYDQWDJHVGHSHQGLQJRQ
WKH V\VWHP HQYLURQPHQWV 6RIWZDUH LV IOH[LEOH WR LPSOHPHQW :H KDYH SURSRVHG D KDUGZDUH DFFHOHUDWRU WR VSHHG XS
EXW GRHV QHHG KLJKO\ VNLOOIXO KDQGWXQLQJ WR UHDFK LWV VLPLODULW\ EDVHG GHGXSOLFDWLRQ IRU SULPDU\ VWRUDJHV RU VWRUDJH
SRWHQWLDOV 6XFK GHVLJQV DOVR UHTXLUH FKDQJHV LQ WKH 26 RU FDFKHV 7KH QHZ KDUGZDUH GHVLJQV KDYH DFKLHYHG GHGXSH
UHTXHVW XVH RI VSHFLDO OLEUDULHV ,W WDNHV WKH V\VWHP UHVRXUFHV UDWLRVWKDWDUHFRPSDUDEOHWRPDWXUHVRIWZDUHLPSOHPHQWDWLRQV
VXFKDV&38DQGPHPRU\IURPWKHDSSOLFDWLRQV2QWKHRWKHU ZKLOH SURYLGLQJ WR WLPHV KLJKHU VSHHG WKDQ WKH VRIWZDUH
KDQGWKHKDUGZDUHDSSURDFKGRHVUHTXLUHGHVLJQFXVWRPL]DWLRQ LPSOHPHQWDWLRQV $ ZRUNLQJ SURWRW\SH KDV EHHQ EXLOW XVLQJ
HLWKHURQDVWRUDJHGHYLFHD+%$RUDQGHGLFDWHGDFFHOHUDWRU 9LUWH[ )3*$ XQGHU ;LOLQ[ HQYLURQPHQW ([WHQVLYH
H[SHULPHQWVKDYHEHHQFDUULHGRXWE\FRQQHFWLQJWKHSURWRW\SH
Standard dedupe vs hardware WRDKRVWVHUYHUWKURXJKD3&,HLQWHUIDFHWRWHVWWKHYDOLGLW\DQG
3
SHUIRUPDQFHRIWKHKDUGZDUHLPSOHPHQWDWLRQ2XUH[SHULPHQWV
Dedupe ratio = before size / after size
Standard dedupe
Hardware similarity=8 KDYH GHPRQVWUDWHG WKH SHUIRUPDQFH LQ WHUPV RI ODWHQF\ DQG
2.5
Hardware similarity=7 GDWDUHGXFWLRQUDWLRRIWKHKDUGZDUHGHVLJQ7KHDUFKLWHFWXUHLV
Hardware similarity=6 UHDGLO\ DSSOLFDEOH LQ SULPDU\ GDWD VWRUDJHV DQG FDFKLQJ LQ
Hardware similarity=5 K\EULGGDWDVWRUDJHV\VWHPV
Hardware similarity=4
2 )XWXUH RSWLPL]DWLRQ FDQ VWLOO EH DFKLHYHG E\ LPSURYLQJ
KDUGZDUH GHOWD FRPSUHVVLRQ UDWLR 7KHUH LV D URRP IRU
LPSURYHPHQW IRU UHIHUHQFH EORFN LQGH[LQJ WR HOLPLQDWH RU
1.5 PLQLPL]H WKH XVDJH RI )3*$ 5$0 UHVRXUFHV IRU WKH KDVK
WDEOH:HDUHFXUUHQWO\ZRUNLQJRQLQWHJUDWLQJDOOWKHKDUGZDUH
PRGXOHV WRJHWKHU WR IRUP D FRPSOHWH GHGXSH V\VWHP IRU
1
Linux kernel Japanese Census Google books SULPDU\VWRUDJHVDQGVWRUDJHFDFKHV
(2 .1GB) (489MB) (1GB)
)LJXUH&RPSDULVRQEHWZHHQVWDQGDUGGHGXSHDQGKDUGZDUHGHGXSH
IRUWKUHHGDWDVHWV
231
,VVXH1RYHPEHU
>@ 7RUVWHQ 6XHO 1DVLU 0HPRQ $OJRULWKPV IRU 'HOWD &RPSUHVVLRQ DQG
$&.12:/('*0(17 5HPRWH )LOH 6\QFKURQL]DWLRQ LQ :RUN VXSSRUWHG E\ 16) &$5((5
$ZDUG16)&&5DQGE\,QWHO&RUSRUDWLRQ
7KH DXWKRUV DUH JUDWHIXO WR 0U -LQJ <DQJ IRU KLV KHOS LQ >@ (-HEDPDODU /HDYOLQH '$VLU $QWRQ\ *QDQD 6LQJK +DUGZDUH
LQVWDOOLQJ WKH VRIWZDUH LQ RXU H[SHULPHQWV DQG SURYLGLQJ ,PSOHPHQWDWLRQ RI /=0$ 'DWD &RPSUHVVLRQ $OJRULWKP
VXJJHVWLRQV DQG FRPPHQWV 7KLV UHVHDUFK LV VXSSRUWHG LQ SDUW ,QWHUQDWLRQDO-RXUQDORI$SSOLHG,QIRUPDWLRQ6\VWHPV,-$,6,661
E\WKH+*6785,UHVHDUFKFRQWUDFW16)JUDQWV&&) S9ROXPH1R0DUFK
DQG&&)$Q\RSLQLRQVILQGLQJVDQGFRQFOXVLRQVRU >@ : &XL 1HZ /=: 'DWD &RPSUHVVLRQ $OJRULWKP DQG ,WV )3*$
UHFRPPHQGDWLRQV H[SUHVVHG LQ WKLV PDWHULDO DUH WKRVH RI WKH ,PSOHPHQWDWLRQWK3LFWXUH&RGLQJ6\PSRVLXP
DXWKRUV DQG GR QRW QHFHVVDULO\ UHIOHFW WKH YLHZV RI WKH >@ 3DXO5:LOVRQ6FRWW).DSODQDQG<DQQLV6PDUDJGDNLV7KH&DVH
1DWLRQDO6FLHQFH)RXQGDWLRQ IRU&RPSUHVVHG&DFKLQJLQ9LUWXDO0HPRU\6\VWHPV3URFHHGLQJVRI
WKH86(1,;$QQXDO7HFKQLFDO&RQIHUHQFH
>@ 0 5DELQ )LQJHUSULQWLQJ E\ UDQGRP SRO\QRPLDOV $LNHQ
5()(5(1&(6 &RPSXWDWLRQ/DERUDWRU\8QLY
>@ .LUDQ 6ULQLYDVDQ 7LP %LVVRQ *DUWK *RRGVRQ DQG .DODGKDU >@ $=%URGHU0&KDULNDU$0)ULH]HDQG00LW]HQPDFKHU0LQ
9RUXJDQWL 1HW$SS ,QF L'HGXS /DWHQF\DZDUH ,QOLQH 'DWD :LVH,QGHSHQGHQW3HUPXWDWLRQV&RPSXWHUDQG6\VWHP6FLHQFHVSS
'HGXSOLFDWLRQ IRU 3ULPDU\ 6WRUDJH 86(1,; FRQIHUHQFH RQ +RW
7RSLFVLQ6WRUDJHDQG)LOH6\VWHPV >@ 'RQJ\DQJ /L4LQJER:DQJ&\ULO *X\RW $VKZLQ1DUDVLPKD'HMDQ
>@ 3KLOLS6KLODQH*UDQW:DOODFH0DUN+XDQJDQG:LQGVRU+VX'HOWD 9XFLQLFDQG=YRQLPLU%DQGLF4LQJ<DQJ $3DUDOOHODQG3LSHOLQHG
&RPSUHVVHG DQG 'HGXSOLFDWHG 6WRUDJH 8VLQJ 6WUHDP,QIRUPHG $UFKLWHFWXUH IRU $FFHOHUDWLQJ )LQJHUSULQW &RPSXWDWLRQ LQ +LJK
/RFDOLW\ LQ +RW6WRUDJH
3URFHHGLQJV RI WKH WK 86(1,; 7KURXJKSXW 'DWD 6WRUDJHV WKH UG ,((( ,QWHUQDWLRQDO 6\PSRVLXP
FRQIHUHQFHRQ+RW7RSLFVLQ6WRUDJHDQG)LOH6\VWHPV RQ)LHOG3URJUDPPDEOH &XVWRP &RPSXWLQJ 0DFKLQHV )&&0
3DSHU6HVVLRQ7
>@ -LQJ[LQ )HQJ -LUL 6FKLQGOHU $ 'HGXSOLFDWLRQ 6WXG\ IRU +RVWVLGH
&DFKHV ZLWK '\QDPLF :RUNORDGV LQ 9LUWXDOL]HG 'DWD &HQWHU >@ 4<DQJDQG-5HQ,&$6+,QWHOOLJHQWO\FRXSOHGDUUD\RIVVGDQG
(QYLURQPHQWV 0DVV 6WRUDJH 6\VWHPV DQG 7HFKQRORJLHV 0667 S KGG +LJK 3HUIRUPDQFH &RPSXWHU $UFKLWHFWXUH +3&$ ,(((
a0D\ WK,QWHUQDWLRQDO6\PSRVLXPRQ,(((SS
>@ -LDQ /LX <XQSHQJ &KDL ;LDR 4LQ <XDQ ;LDR 3/&FDFKH >@ /XNDV .HNHO\ 0DUWLQ =DGQN -LU 0DWRXVHN -DQ .RUHQHN
(QGXUDEOH66'FDFKHIRUGHGXSOLFDWLRQEDVHGSULPDU\VWRUDJH0DVV )DVW ORRNXS IRU G\QDPLF SDFNHW ILOWHULQJ LQ )3*$ 'HVLJQ DQG
6WRUDJH6\VWHPVDQG7HFKQRORJLHV0667Sa 'LDJQRVWLFV RI (OHFWURQLF &LUFXLWV 6\VWHPV WK ,QWHUQDWLRQDO
6\PSRVLXPRQ$SULOSS
>@ 0LQ )X 'DQ )HQJ <X +XD ;XELQ +H =XRQLQJ &KHQ :HQ ;LD
)DQJWLQJ +XDQJ DQG 4LQJ /LX $FFHOHUDWLQJ 5HVWRUH DQG *DUEDJH >@ 0LQ)X'HVWRU*LWKXEKWWSVJLWKXEFRPIRP\GHVWRU
&ROOHFWLRQ LQ 'HGXSOLFDWLRQEDVHG %DFNXS 6\VWHPV YLD ([SORLWLQJ >@ 0DUNXV )UDQ] ;DYHU -RKDQQHV 2EHUKXPHU PLQL VXEVHW RI WKH /=2
+LVWRULFDO,QIRUPDWLRQLQ86(1,;$7&
UHDOWLPHGDWDFRPSUHVVLRQOLEUDU\LQ*LWKXE
>@ -DPHV-+XQW.LHP3KRQJ9R:DOWHU) 7LFK\'HOWDDOJRULWKPV KWWSVJLWKXEFRPGIHOLQWREOHQGHUJLWWUHHPDVWHUH[WHUQO]RPLQLO]R
DQ (PSULFDO $QDO\VLV $&0 7UDQVDFWLRQV RQ 6RIWZDUH (QJLQHHULQJ
DQG0HWKRGRORJ\SS
>@ 3KLOLS6KLODQH0DUN+XDQJ*UDQW:DOODFHDQG:LQGVRU+VX:$1
2SWLPL]HG 5HSOLFDWLRQ RI %DFNXS 'DWDVHWV 8VLQJ 6WUHDP,QIRUPHG
'HOWD&RPSUHVVLRQ$&07UDQVDFWLRQVRQ6WRUDJH726S9ROXPH
232