CHINMAYAINSTITUTEOFTECHNOLOGY KANNUR

SEMINARREPORT On SKINPUT Presentedby DEEPAKVADAKKEVEETTIL MCAFOURTHSEMESTER

SCHOOLOFCOMPUTERSCIENCE AND INFORMATIONTECHNOLOGY

CHINMAYAINSTITUTEOFTECHNOLOGY KANNUR SCHOOLOFCOMPUTERSCIENCE AND INFORMATIONTECHNOLOGY

DECLARATION
IDEEPAKVADAKKEVEETTIL,FOURTHSemesterMCA,studentofChinmayaInstituteof Technology,doherebydeclarethattheSeminarReportentitledSKINPUTistheoriginalwork carriedoutbymeunderthesupervisionofMr.SHAJEER.MPtowardspartialfulfillmentofthe requirementofMCADegree.

SignatureoftheStudent

Kannur: Date:

CHINMAYAINSTITUTEOFTECHNOLOGY KANNUR SCHOOLOFCOMPUTERSCIENCE AND INFORMATIONTECHNOLOGY CERTIFICATE

ThisistocertifythattheSeminarReporttitledSKINPUTwaspreparedandpresentedbyDEEPAK VADAKKEVEETTILoftheSchoolofComputerScienceandInformationTechnology,Chinmaya InstituteofTechnologyinpartialfulfillmentoftherequirementasasubjectundertheUniversityof KannurduringtheFOURTHsemester.

FacultyinCharge

Kannur Date:

ACKNOWLEDGEMENT
First of all, I express my sincere thanks to Principal Dr.K.K.Falgunanfor giving permissiontoconductthisseminarandgivinglotofencouragement. IamthankfultoallfacultymembersinMCAdepartmentfortheircooperationtowardsthis seminarpresentations. Atlast,butnottheleastIwouldliketoexpressmythankstoomnipotentgodgivethe strengthtoconducttheseminar,andalsosincerethankstomyfriends,whohaveparticipatedin theseminarandencouragedmemuch.

ABSTRACT
We present Skinput, a technology that appropriates the human body for acoustic transmission,allowingtheskintobeusedasaninputsurface.Inparticular,weresolvethelocation offingertapsonthearmandhandbyanalyzingmechanicalvibrationsthatpropagatethroughthe body.Wecollectthesesignalsusinganovelarrayofsensorswornasanarmband.Thisapproach providesanalwaysavailable,naturallyportable,andonbodyfingerinputsystem.Weassessthe capabilities,accuracyandlimitationsofourtechniquethroughatwopart,twentyparticipantuser study.Tofurtherillustratetheutilityofourapproach,weconcludewithseveralproofofconcept applicationswedeveloped. AuthorKeywords Bioacoustics, finger input, buttons, gestures, onbody interaction, projected displays, audio interfaces.

CONTENTS

1.

INTRODUCTION

1 2 4 4 5 7 7 9 11 13 15 18 20 21

2. RELATED WORK 3. SKINPUT 4. BIO-ACOUSTIC 5. SENSING 6. ARMBAND PROTOTYPE 7. PROCESSING 8. EXPERIMENT 9. DESIGN AND SETUP 10. RESULTS 11. SUPPLEMENTALEXPERIMENTS 12. EXAMPLEINTERFACESANDINTERACTIONS 13. CONLUSION 14. REFERENCE

INTRODUCTION Deviceswithsignificantcomputationalpowerandcapabilitiescannowbeeasilycarriedon ourbodies.However,theirsmallsizetypicallyleadstolimitedinteractionspaceandconsequently diminishestheirusabilityandfunctionality.Sincewecannotsimplymakebuttonsandscreenslarger withoutlosingtheprimarybenefitofsmallsize,weconsideralternativeapproachesthatenhance interactionswithsmallmobilesystems. One option is to opportunistically appropriate surface area from the environment for interactivepurposes.Forexample,describesatechniquethatallowsasmallmobiledevicetoturn tablesonwhichitrestsintoagesturalfingerinputcanvas.However,tablesarenotalwayspresent, and in a mobile context, users are unlikely to want to carry appropriated surfaces with them. However,thereisonesurfacethathasbeenpreviousoverlookedasaninputcanvas,andonethat happenstoalwaystravelwithus:ourskin.Appropriatingthehumanbodyasaninputdeviceis appealingnotonlybecausewehaveroughlytwosquaremetersofexternalsurfacearea,butalso becausemuchofitiseasilyaccessiblebyourhands.Furthermore,proprioceptionoursenseof howourbodyisconfiguredinthreedimensionalspaceallowsustoaccuratelyinteractwithour bodiesinaneyesfreemanner.Forexample,wecanreadilyflickeachofourfingers,touchthetipof ournose,andclapourhandstogetherwithoutvisualassistance.Fewexternalinputdevicescan claimthisaccurate,eyesfreeinputcharacteristicandprovidesuchalargeinteractionarea. In this paper, we present our work on Skinput a method that allows the body to be appropriatedforfingerinputusinganovel,noninvasive,wearablebioacousticsensor. Thecontributionsofthispaperare: 1) We describe the design of a novel, wearable sensor for bioacoustic signal acquisition (Figure1). 2) Wedescribeananalysisapproachthatenablesoursystemtoresolvethelocationoffinger tapsonthebody. 3) Weassesstherobustnessandlimitationsofthissystemthroughauserstudy. 4) We explore the broader space of bioacoustic input through prototype applications and additionalexperimentation.

Awearable,bioacousticsensingarraybuiltintoanarmband.Sensingelementsdetectvibrations transmittedthroughthebody.Thetwosensorpackagesshownaboveeachcontainfive,specially weighted,cantileveredpiezofilms,responsivetoaparticularfrequencyrange. RELATEDWORK AlwaysAvailableInput TheprimarygoalofSkinputistoprovideanalwaysavailablemobileinputsystemthatis, aninputsystemthatdoesnotrequireausertocarryorpickupadevice.Anumberofalternative approacheshavebeenproposedthatoperateinthisspace.Techniquesbasedoncomputervisionare popular.These,however,arecomputationallyexpensiveanderrorproneinmobilescenarios.Speech inputisalogicalchoiceforalwaysavailableinput,butislimitedinitsprecisioninunpredictable acousticenvironments,andsuffersfromprivacyandscalabilityissuesinsharedenvironments. Otherapproaches havetakenthe form ofwearablecomputing.This typicallyinvolves a physicalinputdevicebuiltinaformconsideredtobepartofonesclothing.Forexample,glove based input systems allow users to retain most of their natural hand movements, but are cumbersome,uncomfortable,anddisruptivetotactilesensation.PostandOrthpresentasmart fabricsystemthatembedssensorsandconductorsintofabric,buttakingthisapproachtoalways availableinputnecessitatesembeddingtechnologyinallclothing,whichwouldbeprohibitively complexandexpensive.TheSixthSenseprojectproposesamobile,alwaysavailableinput/output capabilitybycombiningprojectedinformationwithacolormarkerbasedvisiontrackingsystem. Thisapproachisfeasible,butsuffersfromseriousocclusionandaccuracylimitations.Forexample, determining whether, e.g., a finger has tapped a button, or is merely hovering above it, is extraordinarilydifficult.Inthepresentwork,webrieflyexplorethecombinationofonbodysensing withonbodyprojection. 2

BioSensing Skinputleveragesthenaturalacousticconductionpropertiesofthehumanbodytoprovide aninputsystem,andisthusrelatedtopreviousworkintheuseofbiologicalsignalsforcomputer input.Signalstraditionallyusedfordiagnosticmedicine,suchasheartrateandskinresistance,have beenappropriatedforassessingausersemotionalstate.Thesefeaturesaregenerallysubconsciously drivenandcannotbecontrolledwithsufficientprecisionfordirectinput.Similarly,brainsensing technologies such as electroencephalography (EEG) and functional near infrared spectroscopy (fNIR)havebeenusedbyHCIresearcherstoassesscognitiveandemotionalstate;thisworkalso primarilylookedatinvoluntarysignals.Incontrast,brainsignalshavebeenharnessedasadirect input for use by paralyzed patients, but direct brain computer interfaces (BCIs) still lack the bandwidth required for everyday computing tasks, and require levels of focus, training, and concentrationthatareincompatiblewithtypicalcomputerinteraction.Therehasbeenlesswork relatingtotheintersectionoffingerinputandbiologicalsignals.Researchershaveharnessedthe electrical signals generated by muscle activation during normal hand movement through electromyography. At present, however, this approach typically requires expensive amplification systemsandtheapplicationofconductivegelforeffectivesignalacquisition,whichwouldlimitthe acceptabilityofthisapproachformostusers. TheinputtechnologymostrelatedtoourownisthatofAmentoetal,whoplacedcontact microphonesonauserswristtoassessfingermovement.However,thisworkwasneverformally evaluated,asisconstrainedtofingermotionsinonehand.TheHambonesystememploysasimilar setup, and through an HMM, yields classification accuracies around 90% for four gestures. Performanceoffalsepositiverejectionremainsuntestedinbothsystemsatpresent.Moreover,both techniquesrequiredtheplacementofsensorsneartheareaofinteraction,increasingthedegreeof invasivenessandvisibility. Finally, bone conduction microphones and headphones now common consumer technologiesrepresentanadditionalbiosensingtechnologythatisrelevanttothepresentwork. Theseleveragethefactthatsoundfrequenciesrelevanttohumanspeechpropagatewellthrough bone. Bone conduction microphones are typically worn near the ear, where they can sense vibrationspropagatingfromthemouthandlarynxduringspeech.Boneconductionheadphones sendsoundthroughthebonesoftheskullandjawdirectlytotheinnerear,bypassingtransmission ofsoundthroughtheairandouterear,leavinganunobstructedpathforenvironmentalsounds.

AcousticInput Ourapproachisalsoinspiredbysystemsthatleverageacoustictransmissionthroughinput surfaces.Paradisoetal.[21]measuredthearrivaltimeofasoundatmultiplesensorstolocatehand tapsonaglasswindow.Ishiietal.[12]useasimilarapproachtolocalizeaballhittingatable,for computeraugmentationofarealworldgame.Bothofthesesystemsuseacoustictimeofflightfor localization,whichweexplored,butfoundtobeinsufficientlyrobustonthehumanbody,leadingto thefingerprintingapproachdescribedinthispaper. SKINPUT Toexpandtherangeofsensingmodalitiesforalwaysavailableinputsystems,weintroduce Skinput,anovelinputtechniquethatallowstheskintobeusedasafingerinputsurface.Inour prototypesystem,wechoosetofocusonthearm.Thisisanattractiveareatoappropriateasit provides considerable surface area for interaction, including a contiguous and flat area for projection. Furthermore, the forearm and hands contain a complex assemblage of bones that increasesacousticdistinctivenessofdifferentlocations.Tocapturethisacousticinformation,we developedawearablearmbandthatisnoninvasiveandeasilyremovable(Figures1and5). Inthissection,wediscussthemechanicalphenomenathatenablesSkinput,withaspecific focusonthemechanicalpropertiesofthearm.ThenwewilldescribetheSkinputsensorandthe processing techniques we use to segment, analyze, and classify bioacoustic signals. Figure 2. Transversewavepropagation:Fingerimpactsdisplacetheskin,creatingtransversewaves(ripples). Thesensorisactivatedasthewavepassesunderneathit. BIOACOUSTICS Whenafingertapstheskin,severaldistinctformsofacousticenergyareproduced.Some energyisradiatedintotheairassoundwaves;thisenergyisnotcapturedbytheSkinputsystem. Amongtheacousticenergytransmittedthroughthearm,themostreadilyvisiblearetransverse waves,createdbythedisplacementoftheskinfromafingerimpact(Figure2).Whenshotwitha highspeedcamera,theseappearasripples,whichpropagateoutwardfromthepointofcontact.The amplitudeoftheseripplesiscorrelatedtoboththetappingforceandtothevolumeandcompliance ofsofttissuesundertheimpactarea.Ingeneral,tappingonsoftregionsofthearmcreateshigher amplitudetransversewavesthantappingonboneyareas,whichhavenegligiblecompliance. In addition to the energy that propagates on the surface of the arm, some energy is 4

transmittedinward,towardtheskeleton(Figure3).Theselongitudinal(compressive)wavestravel throughthesofttissuesofthearm,excitingthebone,whichismuchlessdeformablethenthesoft tissuebutcanrespondtomechanicalexcitationbyrotatingandtranslatingasarigidbody.This excitation vibrates soft tissues surrounding the entire length of the bone, resulting in new longitudinalwavesthatpropagateoutwardtotheskin. Wehighlightthesetwoseparateformsofconductiontransversewavesmovingdirectlyalong the arm surface, and longitudinal waves moving into and out of the bone through soft tissues becausethesemechanismscarryenergyatdifferentfrequenciesandoverdifferentdistances. Roughlyspeaking,higherfrequenciespropagatemorereadilythroughbonethanthroughsofttissue, andboneconductioncarriesenergyoverlargerdistancesthansofttissueconduction.Whilewedo notexplicitlymodelthespecificmechanismsofconduction,ordependonthesemechanismsforour analysis,wedobelievethesuccessofourtechniquedependsonthecomplexacousticpatternsthat resultfrommixturesofthesemodalities. Similarly,wealsobelievethatjoints playanimportantroleinmakingtappedlocations acoustically distinct. Bones are held together by ligaments, and joints often include additional biologicalstructuressuchasfluidcavities.Thismakesjointsbehaveasacousticfilters.Insome cases,thesemaysimplydampenacoustics;inothercases,thesewillselectivelyattenuatespecific frequencies,creatinglocationspecificacousticsignatures.

SENSING Tocapturetherichvarietyofacousticinformationdescribedintheprevioussection,we evaluated many sensing technologies, including bone conduction microphones, conventional 5

microphonescoupledwithstethoscopes,piezocontactmicrophones,andaccelerometers.However, these transducers were engineered for very different applications than measuring acoustics transmittedthroughthehumanbody.Assuch,wefoundthemtobelackinginseveralsignificant ways.Foremost,mostmechanicalsensorsareengineeredtoproviderelativelyflatresponsecurves overtherangeoffrequenciesthatisrelevanttooursignal.Thisisadesirablepropertyformost applicationswhereafaithfulrepresentationofaninputsignaluncoloredbythepropertiesofthe transducerisdesired.However,becauseonlyaspecificsetoffrequenciesisconductedthroughthe arminresponsetotapinput,aflatresponsecurveleadstothecaptureofirrelevantfrequenciesand thustoahighsignaltonoiseratio. WhileboneconductionmicrophonesmightseemasuitablechoiceforSkinput,thesedevices aretypicallyengineeredforcapturinghumanvoice,andfilteroutenergybelowtherangeofhuman speech(whoselowestfrequencyisaround85Hz).Thusmostsensorsinthiscategorywerenot especiallysensitivetolowerfrequencysignals(e.g.,25Hz),whichwefoundinourempiricalpilot studiestobevitalincharacterizingfingertaps.

Figure4.Responsecurve(relativesensitivty)ofthesensingelementthatresonatesat78Hz. Figure4showstheresponsecurveforoneofoursensors,tunedtoaresonantfrequencyof78Hz. Thecurveshowsa~14dBdropoff 20Hzawayfromtheresonantfrequency.Additionally,the cantileveredsensorswerenaturallyinsensitivetoforcesparalleltotheskin(e.g.,shearingmotions causedbystretching).Thus,theskinstretchinducedbymanyroutinemovements(e.g.,reachingfor a doorknob) tends to be attenuated. However, the sensors are highly responsive to motion perpendiculartotheskinplaneperfectforcapturingtransversesurfacewaves(Figure2)and longitudinal waves emanating from interior structures. Finally, our sensor design is relatively inexpensiveandcanbemanufacturedinaverysmallformfactors,renderingitsuitableforinclusion infuturemobiledevices.

ARMBANDPROTOTYPE Ourfinalprototype,showninFigures1and5,featurestwoarraysoffivesensingelements, incorporatedintoanarmbandformfactor.Thedecisiontohavetwosensorpackageswasmotivated byourfocusonthearmforinput.Inparticular,whenplacedontheupperarm(abovetheelbow), wehopedtocollectacousticinformationfromthefleshybicepareainadditiontothefirmerareaon theundersideofthearm,withbetteracousticcouplingtotheHumerus,themainbonethatruns fromshouldertoelbow.Whenthesensorwasplacedbelowtheelbow,ontheforearm,onepackage waslocatedneartheRadius,thebonethatrunsfromthelateralsideoftheelbowtothethumbside ofthewrist,andtheotherneartheUlna,whichrunsparalleltothisonthemedialsideofthearm closesttothebody.Eachlocationthusprovidedslightlydifferentacousticcoverageandinformation, helpfulindisambiguatinginputlocation. Basedonpilotdatacollection,weselectedadifferentsetofresonantfrequenciesforeach sensor package (Table 1). We tuned the upper sensor package to be more sensitive to lower frequencysignals,astheseweremoreprevalentinfleshierareas.Conversely,wetunedthelower sensorarraytobesensitivetohigherfrequencies,inordertobettercapturesignalstransmitted though(denser)bones.

PROCESSING In our prototype system, we employ a Mackie Onyx 1200F audio interface to digitally capture data from the ten sensors. This was connected via Firewire to a conventional desktop computer, where a thin client written in C interfaced with the device using the Audio Stream Input/Output(ASIO)protocol. 7

Eachchannelwassampledat5.5kHz,asamplingratethatwouldbeconsideredtoolowfor speech or environmental audio, but was able to represent the relevant spectrum of frequencies transmittedthroughthearm.Thisreducedsampleratemakesourtechniquereadilyportableto embeddedprocessors.Forexample,theATmega168processoremployedbytheArduinoplatform cansampleanalogreadingsat77kHzwithnolossofprecision,andcouldthereforeprovidethefull samplingpowerrequiredforSkinput. Datawas then sentfromourthinclient overa local sockettoourprimaryapplication, writteninJava.Thisprogramperformedthreekeyfunctions.First,itprovidedalivevisualizationof the data from our ten sensors, which was useful in identifying acoustic features. Second, it segmentedinputsfromthedatastreamintoindependentinstances(taps).Third,itclassifiedthese inputinstances. Theaudiostreamwassegmentedintoindividualtapsusinganabsoluteexponentialaverage ofalltenchannels.Whenanintensitythresholdwasexceeded,theprogramrecordedthetimestamp asapotentialstartofatap.Iftheintensitydidnotfallbelowasecond,independentclosing thresholdbetween100msand700msaftertheonsetcrossing,theeventwasdiscarded.Ifstartand endcrossingsweredetectedthatsatisfiedthesecriteria,theacousticdatainthatperiod(plusa60ms bufferoneitherend)wasconsideredaninputevent(Figure6,verticalgreenregions).Although simple, this heuristic proved to be highly robust, mainly due to the extreme noise suppression providedbyoursensingapproach.Afteraninputhasbeensegmented,thewaveformsareanalyzed. Thehighlydiscretenatureoftaps(i.e.pointimpacts)meantacousticsignalswerenotparticularly expressiveovertime.Signalssimplydiminishedinintensityovertime.Thus,featuresarecomputed overtheentireinputwindowanddonotcaptureanytemporaldynamics.

Weemployabruteforcemachinelearningapproach,computing186featuresintotal,many ofwhicharederivedcombinatorially.Forgrossinformation,weincludetheaverageamplitude, standarddeviationandtotal(absolute)energyofthewaveformsineachchannel(30features).From these,wecalculateallaverageamplituderatiosbetweenchannelpairs(45features).Wealsoinclude anaverageoftheseratios.Wecalculatea256pointFFTforalltenchannels,althoughonlythe lowertenvaluesareused,yielding100features.ThesearenormalizedbythehighestamplitudeFFT valuefoundonanychannel.Wealsoincludethecenterofmassofthepowerspectrumwithinthe same0Hzto193Hzrangeforeachchannel,aroughestimationofthefundamentalfrequencyofthe signaldisplacingeachsensor.Subsequentfeatureselectionestablishedtheallpairsamplituderatios andcertainbandsoftheFFTtobethemostpredictivefeatures. These 186 features are passed to a Support Vector Machine (SVM) classifier. A full description ofSVMs is beyondthe scopeof this paper.Oursoftwareuses the implementation provided in the Weka machine learning toolkit. It should be noted, however, that other, more sophisticatedclassificationtechniquesandfeaturescouldbeemployed.Thus,theresultspresented inthispapershouldbeconsideredabaseline. BeforetheSVMcanclassifyinputinstances,itmustfirstbetrainedtotheuserandthe sensorposition.Thisstagerequiresthecollectionofseveralexamplesforeachinputlocationof interest.WhenusingSkinputtorecognizeliveinput,thesame186acousticfeaturesarecomputed ontheflyforeachsegmentedinput.ThesearefedintothetrainedSVMforclassification.Weuse aneventmodelinoursoftwareonceaninputisclassified,aneventassociatedwiththatlocationis instantiated.Anyinteractivefeaturesboundtothateventarefired.Ascanbeseeninourvideo,we readilyachieveinteractivespeeds. EXPERIMENT Participants Toevaluatetheperformanceofoursystem,werecruited13participants(7female)fromthe GreaterSeattlearea.Theseparticipantsrepresentedadiversecrosssectionofpotentialagesand bodytypes.Agesrangedfrom20to56(mean38.3),andcomputedbodymassindexes(BMIs) rangedfrom20.5(normal)to31.9(obese). ExperimentalConditions Weselectedthreeinputgroupingsfromthemultitudeofpossiblelocationcombinationsto 9

test.Webelievethatthesegroupings,illustratedinFigure7,areofparticularinterestwithrespectto interfacedesign,andatthesametime,pushthelimitsofoursensingcapability.Fromthesethree groupings,wederivedfivedifferentexperimentalconditions,describedbelow. Fingers(FiveLocations) Onesetofgestureswetestedhadparticipantstappingonthetipsofeachoftheirfivefingers (Figure 6, Fingers). The fingers offer interesting affordances that make them compelling to appropriateforinput.Foremost,theyprovideclearlydiscreteinteractionpoints,whichareeven alreadywellnamed(e.g.,ringfinger).Inadditiontofivefingertips,thereare14knuckles(five major,nineminor),which,takentogether,couldoffer19readilyidentifiableinputlocationsonthe fingersalone.Second,wehaveexceptionalfingertofingerdexterity,asdemonstratedwhenwe countbytappingonourfingers.Finally,thefingersarelinearlyordered,whichispotentiallyuseful forinterfaceslikenumberentry,magnitudecontrol(e.g.,volume),andmenuselection. Atthesametime,fingersareamongthemostuniformappendagesonthebody,withallbut the thumb sharing a similar skeletal and muscular structure. This drastically reduces acoustic variationandmakesdifferentiatingamongthemdifficult.Additionally,acousticinformationmust crossasmanyasfive(fingerandwrist)jointstoreachtheforearm,whichfurtherdampenssignals. Forthisexperimentalcondition,wethusdecidedtoplacethesensorarraysontheforearm,just below the elbow. Despite these difficulties, pilot experiments showed measureable acoustic differencesamongfingers,whichwetheorizeisprimarilyrelatedtofingerlengthandthickness, interactions with the complex structure of the wrist bones, and variations in the acoustic transmissionpropertiesofthemusclesextendingfromthefingerstotheforearm. WholeArm(FiveLocations)

Figure7:Thethreeinputlocationsetsevaluatedinthestudy. Anothergesturesetinvestigatedtheuseoffiveinputlocationsontheforearmandhand:arm,wrist, 10

palm,thumbandmiddle finger(Figure7,Whole Arm).Weselectedthese locations fortwo importantreasons.First,theyaredistinctandnamedpartsofthebody(e.g.,wrist).Thisallowed participants to accurately tap these locations without training or markings. Additionally, these locationsprovedtobeacousticallydistinctduringpiloting,withthelargespatialspreadofinput points offering further variation. We used these locations in three different conditions. One conditionplacedthesensorabovetheelbow,whileanotherplaceditbelow.Thiswasincorporated into the experiment to measure the accuracy loss across this significant articulation point (the elbow).Additionally,participantsrepeatedthelowerplacementconditioninaneyesfreecontext: participants were told to close their eyes and face forward, both for training and testing. This conditionwasincludedtogaugehowwelluserscouldtargetonbodyinputlocationsinaneyesfree context(e.g.,driving). Forearm(TenLocations) Inanefforttoassesstheupperboundofourapproachssensingresolution,ourfifthand finalexperimentalconditionusedtenlocationsonjusttheforearm(Figure6,Forearm).Notonly wasthisaveryhighdensityofinputlocations(unlikethewholearmcondition),butitalsoreliedon aninputsurface(theforearm)withahighdegreeofphysicaluniformity(unlike,e.g.,thehand).We expected that these factors would make acoustic sensing difficult. Moreover, this location was compellingduetoitslargeandflatsurfacearea,aswellasitsimmediateaccessibility,bothvisually and for finger input. Simultaneously, this makes for an ideal projection surface for dynamic interfaces. Tomaximizethesurfaceareaforinput,weplacedthesensorabovetheelbow,leavingthe entireforearmfree.Ratherthannamingtheinputlocations,aswasdoneinthepreviouslydescribed conditions,weemployedsmall,coloredstickerstomarkinputtargets.Thiswasbothtoreduce confusion (since locations on the forearm do not have common names) and to increase input consistency. As mentioned previously, we believe the forearm is ideal for projected interface elements;thestickersservedaslowtechplaceholdersforprojectedbuttons. DESIGNANDSETUP Weemployedawithinsubjectsdesign,witheachparticipantperformingtasksineachofthe fiveconditionsinrandomizedorder:fivefingerswithsensorsbelowelbow;fivepointsonthewhole armwiththesensorsabovetheelbow;thesamepointswithsensorsbelowtheelbow,bothsighted andblind;andtenmarkedpointsontheforearmwiththesensorsabovetheelbow. 11

Participantswereseatedinaconventionalofficechair,infrontofadesktopcomputerthat presentedstimuli.Forconditionswithsensorsbelowtheelbow,weplacedthearmband~3cmaway fromtheelbow,withonesensorpackageneartheradiusandtheotherneartheulna.Forconditions withthesensorsabovetheelbow,weplacedthearmband~7cmabovetheelbow,suchthatone sensorpackagerestedonthebiceps.Righthandedparticipantshadthearmbandplacedontheleft arm,whichallowedthemtousetheirdominanthandforfingerinput.Fortheonelefthanded participant,weflippedthesetup,whichhadnoapparenteffectontheoperationofthesystem. Tightness of the armband was adjusted to be firm, but comfortable. While performing tasks, participants could place their elbow on the desk, tucked against their body, or on the chairs adjustablearmrest;mostchosethelatter. Procedure Foreachcondition,theexperimenterwalkedthroughtheinputlocationstobetestedand demonstrated finger taps on each. Participants practiced duplicating these motions for approximatelyoneminutewitheachgestureset.Thisallowedparticipantstofamiliarizethemselves withournamingconventionsandtopracticetappingtheirarmandhandswithafingeronthe oppositehand.Italsoallowedustoconveytheappropriatetapforcetoparticipants,whooften initiallytappedunnecessarilyhard. Totrainthesystem,participantswereinstructedtocomfortablytapeachlocationtentimes, withafingeroftheirchoosing.Thisconstitutedonetraininground.Intotal,threeroundsoftraining datawerecollectedperinputlocationset(30examplesperlocation,150datapointstotal).An exceptiontothisprocedurewasinthecaseofthetenforearmlocations,whereonlytworoundswere collectedtosavetime(20examplesperlocation,200datapointstotal).Totaltrainingtimeforeach experimentalconditionwasapproximatelythreeminutes. WeusedthetrainingdatatobuildanSVMclassifier.Duringthesubsequenttestingphase, wepresentedparticipantswithsimpletextstimuli(e.g.tapyourwrist),whichinstructedthem wheretotap.Theorderofstimuliwasrandomized,witheachlocationappearingtentimesintotal. Thesystemperformedrealtimesegmentationandclassification,andprovidedimmediatefeedback to the participant. We provided feedback so that participants could see where the system was makingerrors.Ifaninputwasnotsegmented,participantscouldseethisandwouldsimplytap again.Overall,segmentationerrorrateswerenegligibleinallconditions,andnotincludedinfurther analysis. 12

Accuracyofthethreewholearmcentricconditions.Errorbarsrepresentstandarddeviation.

RESULTS Inthis section,wereportontheclassificationaccuracies forthetestphases inthefive different conditions. Overall, classification rates were high, with an average accuracy across conditionsof87.6%.Additionally,wepresentpreliminaryresultsexploringthecorrelationbetween classificationaccuracyandfactorssuchasBMI,age,andsex. FiveFingers Despite multiple joint crossings and ~40cm ofseparation between the input targets and sensors, classification accuracy remained high for the fivefinger condition, averaging 87.7% (SD=10.0%,chance=20%)acrossparticipants.Segmentation,asinotherconditions,wasessentially perfect. Inspectionoftheconfusionmatricesshowednosystematicerrorsintheclassification,with errorstendingtobeevenlydistributedovertheotherdigits.Whenclassificationwasincorrect,the systembelievedtheinputtobeanadjacentfinger60.5%ofthetime;onlymarginallyaboveprior probability(40%).Thissuggeststhereareonlylimitedacousticcontinuitiesbetweenthefingers. Theonlypotentialexceptiontothiswasinthecaseofthepinky,wheretheringfingerconstituted 63.3%percentofthemisclassifications. WholeArm Participants performed three conditions with the wholearm location configuration. The belowelbow placement performed the best, posting a 95.5% (SD=5.1%, chance=20%) average

13

accuracy.Thisisnotsurprising,asthisconditionplacedthesensorsclosertotheinputtargetsthan theotherconditions.Movingthesensorabovetheelbowreducedaccuracyto88.3%(SD=7.8%, chance=20%),adropof7.2%.Thisisalmostcertainlyrelatedtotheacousticlossattheelbowjoint andtheadditional10cmofdistancebetweenthesensorandinputtargets.Figure8showsthese results. The eyesfree input condition yielded lower accuracies than other conditions, averaging 85.0% (SD=9.4%, chance=20%). This represents a 10.5% drop from its vision assisted, but otherwiseidenticalcounterpartcondition.Itwasapparentfromwatchingparticipantscompletethis conditionthattargetingprecisionwasreduced.Insightedconditions,participantsappearedtobe abletotaplocations with perhaps a 2cmradius oferror.Although notformallycaptured,this marginoferrorappearedtodoubleortriplewhentheeyeswereclosed.Webelievethatadditional trainingdata,whichbettercoverstheincreasedinputvariability,wouldremovemuchofthisdeficit. Wewouldalsocautiondesignersdevelopingeyesfree,onbodyinterfacestocarefullyconsiderthe locationsparticipantscantapaccurately.

Forearm Classificationaccuracyforthetenlocationforearmconditionstoodat81.5%(SD=10.5%, chance=10%),asurprisinglystrongresultforaninputsetwedevisedtopushoursystemssensing limit.Followingtheexperiment,weconsidereddifferentwaystoimproveaccuracybycollapsingthe ten locations intolargerinputgroupings.Thegoal ofthis exercisewas toexplorethetradeoff betweenclassificationaccuracyandnumberofinputlocationsontheforearm,whichrepresentsa particularlyvaluableinputsurfaceforapplicationdesigners.Wegroupedtargetsintosetsbasedon 14

whatwebelievedtobelogicalspatialgroupings.Inadditiontoexploringclassificationaccuracies for layouts that we considered to be intuitive, we also performed an exhaustive search (programmatically)overallpossiblegroupings.Formostlocationcounts,thissearchconfirmedthat our intuitive groupings were optimal; however, this search revealed one plausible, although irregular,layoutwithhighaccuracyatsixinputlocations. Unlikeinthefivefingerscondition,thereappearedtobesharedacoustictraitsthatledtoa higher likelihood of confusion with adjacent targets than distant ones. This effect was more prominentlaterallythanlongitudinally.Figure9illustratesthiswithlateralgroupingsconsistently outperforming similarly arranged, longitudinal groupings. This is unsurprising given the morphologyofthearm,withahighdegreeofbilateralsymmetryalongthelongaxis. BMIEffects Earlyon,wesuspectedthatouracousticapproachwassusceptibletovariationsinbody composition.Thisincluded,mostnotably,theprevalenceoffattytissuesandthedensity/massof bones.These,respectively,tendtodampenorfacilitatethetransmissionofacousticenergyinthe body. To assess how these variations affected our sensing accuracy, we calculated each participantsbodymassindex(BMI)fromselfreportedweightandheight.Dataandobservations fromtheexperimentsuggestthathighBMIiscorrelatedwithdecreasedaccuracies.Theparticipants withthethreehighestBMIsproducedthethreelowestaverageaccuracies.Figure10illustratesthis significantdisparityhereparticipantsareseparatedintotwogroups,thosewithBMIgreaterand lessthantheUSnationalmedian,ageandsexadjusted(F1,12=8.65,p=.013). Otherfactorssuchasageandsex,whichmaybecorrelatedtoBMIinspecificpopulations, mightalsoexhibitacorrelationwithclassificationaccuracy.Forexample,inourparticipantpool, malesyieldedhigherclassificationaccuraciesthanfemales,butweexpectthatthisisanartifactof BMIcorrelationinoursample,andprobablynotaneffectofsexdirectly. SUPPLEMENTALEXPERIMENTS Weconducted a series ofsmaller, targetedexperiments toexplore the feasibility ofour approachforotherapplications.Inthefirstadditionalexperiment,whichtestedperformanceofthe systemwhileuserswalkedandjogged,werecruitedonemale(age23)andonefemale(age26)fora singlepurposeexperiment.Fortherestoftheexperiments,werecruitedsevennewparticipants(3 15

female,meanage26.9)fromwithinourinstitution.Inallcases,thesensorarmbandwasplacedjust belowtheelbow.Similartothepreviousexperiment,eachadditionalexperimentconsistedofa trainingphase,whereparticipantsprovidedbetween10and20examplesforeachinputtype,anda testingphase,inwhichparticipantswerepromptedtoprovideaparticularinput.Asbefore,input orderwasrandomized;segmentationandclassificationwereperformedinrealtime. WalkingandJogging As discussed previously, acousticallydriven input techniques are often sensitive to environmentalnoise.Inregardtobioacousticsensing,withsensorscoupledtothebody,noise createdduringothermotionsisparticularlytroublesome,andwalkingandjoggingrepresentperhaps the most common types of wholebody motion. This experiment explored the accuracy of our systeminthesescenarios. Eachparticipanttrainedandtestedthesystemwhilewalkingandjoggingonatreadmill. Threeinputlocationswereusedtoevaluateaccuracy:arm,wrist,andpalm.Additionally,therateof falsepositivesandtruepositiveswascaptured.Thetestingphasetookroughlythreeminutesto complete.Themalewalkedat2.3mphandjoggedat4.3mph;thefemaleat1.9and3.1mph, respectively.Inbothwalkingtrials,thesystemneverproducedafalsepositiveinput.Meanwhile, true positive accuracy was 100%. Classification accuracy for the inputs (e.g., a wrist tap was recognizedasawristtap)was100%forthemaleand86.7%forthefemale(chance=33%). Inthejoggingtrials,thesystemhadfourfalsepositiveinputevents(twoperparticipant) over six minutes of continuous jogging. Truepositive accuracy, as with walking, was 100%. Consideringthatjoggingisperhapsthehardestinputfilteringandsegmentationtest,weviewthis resultasextremelypositive.Classificationaccuracy,however,decreasedto83.3%and60.0%forthe maleandfemaleparticipantsrespectively. Althoughthenoisegeneratedfromthejoggingalmostcertainlydegradedthesignal,we believethechiefcause forthis decrease wasthe quality ofthe training data. Participants only providedtenexamplesforeachofthreetestedinputlocations.Furthermore,thetrainingexamples werecollectedwhileparticipantswerejogging.Thus,theresultingtrainingdatawasnotonlyhighly variable,butalsosparseneitherofwhichisconducivetoaccuratemachinelearningclassification. Webelievethatmorerigorouscollectionoftrainingdatacouldyieldevenstrongerresults. SingleHandedGestures Intheexperiments discussedthus far,weconsideredonlybimanualgestures,wherethe 16

sensorfreearm,andinparticularthefingers,areusedtoprovideinput.However,therearearange ofgesturesthatcanbeperformedwithjustthefingersofonehand.Thiswasthefocusof[2], althoughthisworkdidnotevaluateclassificationaccuracy. We conducted three independent tests to explore onehanded gestures. The first had participantstaptheirindex,middle,ringandpinkyfingersagainsttheirthumb(akintoapinching gesture) ten times each. Our system was able to identify the four input types with an overall accuracyof89.6%(SD=5.1%,chance=25%).Werananidenticalexperimentusingflicksinsteadof taps(i.e.,usingthethumbasacatch,thenrapidlyflickingthefingersforward).Thisyieldedan impressive96.8%(SD=3.1%,chance=25%)accuracyinthetestingphase.

Thismotivatedustorunathirdandindependentexperimentthatcombinedtapsandflicks intoasinglegestureset.Participantsretrainedthesystem,andcompletedanindependenttesting round.Evenwitheightinputclassesinveryclosespatialproximity,thesystemwasabletoachievea remarkable 87.3% (SD=4.8%, chance=12.5%) accuracy. This result is comparable to the aforementioned ten location forearm experiment (which achieved 81.5% accuracy), lending credence to the possibility of having ten or more functions on the hand alone. Furthermore, proprioceptionofourfingersonasinglehandisquiteaccurate,suggestingamechanismforhigh accuracy,eyesfreeinput. SurfaceandObjectRecognition Duringpiloting,itbecameapparentthatoursystemhadsomeabilitytoidentifythetypeof materialonwhichtheuserwasoperating.Usingasimilarsetuptothemainexperiment,weasked participants to tap their index finger against 1) a finger on their other hand, 2) a paper pad approximately 80 pages thick, and 3) an LCD screen. Results show that we can identify the contactedobjectwithabout87.1%(SD=8.3%,chance=33%)accuracy.Thiscapabilitywasnever consideredwhendesigningthesystem,sosuperioracousticfeaturesmayexist.Evenasaccuracy 17

standsnow,thereareseveralinterestingapplicationsthatcouldtakeadvantageofthisfunctionality, including workstations or devices composed of different interactive surfaces, or recognition of differentobjectsgraspedintheenvironment. IdentificationofFingerTapType Userscantapsurfaceswiththeirfingersinseveraldistinctways.Forexample,onecanuse thetipoftheirfinger(potentiallyeventheirfingernail)orthepad(flat,bottom)oftheirfinger.The formertendstobequiteboney,whilethelattermorefleshy.Itisalsopossibletousetheknuckles (bothmajorandminormetacarpophalangealjoints). Toevaluateourapproachsabilitytodistinguishtheseinputtypes,wehadparticipantstapon atablesituatedinfrontoftheminthreeways:fingertip,fingerpad,andmajorknuckle.Aclassifier trainedonthisdatayieldedanaverageaccuracyof89.5%(SD=4.7%,chance=33%)duringthe testingperiod. Thisabilityhasseveralpotentialuses.Perhapsthemostnotableistheabilityforinteractive touchsurfacestodistinguishdifferenttypesoffingercontacts.Oneexampleinteractioncouldbe thatdoubleknockingonanitemopensit,whileapadtapactivatesanoptionsmenu. SegmentingFingerInput A pragmatic concern regarding the appropriation of fingertips for input was that other routinetaskswouldgeneratefalsepositives.Forexample,typingonakeyboardstrikesthefinger tipsinaverysimilarmannertothefingertipinputweproposedpreviously.Thus,wesetoutto explorewhetherfingertofingerinputsoundedsufficientlydistinctsuchthatotheractionscouldbe disregarded. Asaninitialassessment,weaskedparticipantstotaptheirindexfinger20timeswitha fingerontheirotherhand,and20timesonthesurfaceofatableinfrontofthem.Thisdatawas usedtotrainourclassifier.Thistrainingphasewasfollowedbyatestingphase,whichyieldeda participantwideaverageaccuracyof94.3%(SD=4.5%,chance=50%). EXAMPLEINTERFACESANDINTERACTIONS We conceived and built several prototype interfaces that demonstrate our ability to appropriate the human body, in this case the arm, and use it as an interactive surface. These interfacescanbeseeninFigure11,aswellasintheaccompanyingvideo. 18

Whilethebioacousticinputmodalityisnotstrictlytetheredtoaparticularoutputmodality, webelievethesensorformfactorsweexploredcouldbereadilycoupledwithvisualoutputprovided byanintegratedpicoprojector.Therearetwonicepropertiesofwearingsuchaprojectiondeviceon the arm that permit us to sidestep many calibration issues. First, the arm is a relatively rigid structuretheprojector,whenattachedappropriately,willnaturallytrackwiththearm.Second, sincewehavefinegrainedcontrolofthearm,makingminuteadjustmentstoaligntheprojected imagewiththearmistrivial. Toillustratetheutilityofcouplingprojectionandfingerinputonthebody(asresearchers haveproposedtodowithprojectionandcomputervisionbasedtechniques[19]),wedeveloped threeproofofconceptprojectedinterfacesbuiltontopofoursystemsliveinputclassification.In thefirstinterface,weprojectaseriesofbuttonsontotheforearm,onwhichausercanfingertapto navigateahierarchicalmenu(Figure11,left).Inthesecondinterface,weprojectascrollingmenu (center),whichausercannavigatebytappingatthetoporbottomtoscrollupanddownoneitem respectively.Tappingontheselecteditemactivatesit.Inathirdinterface,weprojectanumeric keypadonauserspalmandallowthemtotaponthepalmto,e.g.,dialaphonenumber(right).To emphasizetheoutputflexibilityofapproach,wealsocoupledourbioacousticinputtoaudiooutput. Inthiscase,theusertapsonpresetlocationsontheirforearmandhandtonavigateandinteractwith anaudiointerface.

19

CONCLUSION Inthispaper,wehavepresentedourapproachtoappropriatingthehumanbodyasaninput surface. We have described a novel, wearable bioacoustic sensing array that we built into an armbandinordertodetectandlocalizefingertapsontheforearmandhand.Resultsfromour experimentshaveshownthatoursystemperformsverywellforaseriesofgestures,evenwhenthe bodyisinmotion.Additionally,wehavepresentedinitialresultsdemonstratingotherpotentialuses ofourapproach,whichwehopetofurtherexploreinfuturework.Theseincludesinglehanded gestures,tapswithdifferentpartsofthefinger,anddifferentiatingbetweenmaterialsandobjects. Weconcludewithdescriptionsofseveralprototypeapplicationsthatdemonstratetherichdesign spacewebelieveSkinputenables.

20

REFERENCES 1. Ahmad,F.,andMusilek,P.AKeystrokeandPointerControlInputInterfaceforWearable Computers.InProc.IEEEPERCOM06,211. 2. Amento,B.,Hill,W.,andTerveen,L.TheSoundofOneHand:AWristmountedBio acousticFingertipGestureInterface.InCHI02Ext.Abstracts,724725. 3. Argyros, A.A., and Lourakis, M.I.A. Visionbased Interpretation of Hand Gestures for RemoteControlofaComputerMouse.InProc.ECCV2006WorkshoponComputerVision inHCI,LNCS3979,4051. 4. Burges,C.J.ATutorialonSupportVectorMachinesforPatternRecognition.DataMining andKnowledgeDiscovery,2.2,June1998,121167. 5. Clinical Guidelines on the Identification, Evaluation, and Treatment of Overweight and ObesityinAdults.NationalHeart,LungandBloodInstitute.Jun.17,1998. 6. Deyle, T., Palinko, S., Poole, E.S., and Starner, T. Hambone: A BioAcoustic Gesture Interface.InProc.ISWC'07.18. 7. Erol,A.,Bebis,G.,Nicolescu,M.,Boyle,R.D.,andTwombly,X.Visionbasedhandpose estimation:Areview.ComputerVisionandImageUnderstanding.108,Oct.,2007. 8. Fabiani, G.E. McFarland, D.J. Wolpaw, J.R. and Pfurtscheller, G. Conversion of EEG activityintocursormovementbyabraincomputerinterface(BCI).IEEETrans.onNeural SystemsandRehabilitationEngineering,12.3,3318.Sept.2004.

21