Running Head: MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM

Mental Models, Affordances and the Livescribe Pen and Notebook System Armen J. Chakmakjian Bentley University

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM Abstract

This paper is an expert review of what mental models and affordances the Livescribe Pen and Notebook system provides the user. The Livescribe pen can record pen strokes as well as audio and synchronizes their replay. The combination of pen and notebook not only provides affordances and live within the mental model of writing with a pen and paper, but also augments that model by allowing the replay of the sound when taking notes. The specialized notebook page is a surface that has both an area to write as an affordance consistent with the mental model of writing, but also presents several affordances to control the pens operations. Some of these controls are familiar to the users mental model of recording audio, while others are unique to the operation of the pen and must be learned. This paper starts by describing memory and categorization, mental models and affordances, and then delves into the mental model of writing with an instrument on a surface. A description the Livescribe pen and notebook system technology follows, then an analysis of that functionality in terms of mental models and affordances and the usability of the system. The paper closes with some recommendations for the Livescribe system to be easier to learn.

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM Mental Models, Affordances and the Livescribe Pen and Notebook System Recognition of things we encounter as humans requires that we retain some amount of those observations for later use. (Anderson, 1983) The categorical storage of this information, alternatively called scripts, schemas or frames, are sequences of actions can be retrieved to fill in missing information as new situations are encountered. (Long, 1989) Spreading activation is a process that plays a pivotal role in the retrieval of those schemas. (Anderson, 1983). Those retrieved schemas are can be referred to as either affordances by cognitive psychologists or mental models by systems designers. As James Gibson points out, affordance is a word he made up to describe a concept wasnt in the dictionary at that time since was no word that described the concept that an environment affords things to an animal that could be either beneficial or unfavorable in result. (Gibson, 1979, p.127) At about the same time, Donald Norman wrote about mental models, in which a target system and a user interact to inform and evolve the users cognitive processes to get a workable result in that interaction (Norman, 1983).

This paper starts by describing memory and categorization, mental models and affordances, and then delves into the mental model of writing with an instrument on a surface. A description the Livescribe pen and notebook system technology follows, then an analysis of that functionality in terms of mental models and affordances and the usability of the system. The paper closes with some recommendations for the Livescribe system to be easier to learn. Survey of Research Memory, Categorization, Schemas, and Semantic Networks Humans have a conceptual system that stores experiences in categorical fashion. (Barsalou, 2008) However, in order to categorize that stimulus, it first must be perceived and stored. Anderson proposed that retention of these experiences comes through the compilation of cognitive units that are recordings of an external event or an internal computation. These units are transient and will eventually be encoded into a long-term memory structure called a trace. The ability to recall a trace is subject to the frequency of repeated associations with that original trace. (Anderson, 1983) In order to retrieve and use that perceptual data, the brain applies a heuristic in which new information that is perceived as relevant is associated with existing traces in process is called Spreading Activation. (Crestani, 1997) Spreading activation was proposed by Quillian when doing research into simulating memory search and computer comprehension. In his research, memory is described as a semantic network containing factual assertions about the environment. (Quillian, 1969) This theory studied search in that network of information, observing when two or more nodes (concepts) were traversed in that network and an intersection resulted. (Collins & Loftus, 1975) These paths are being traversed in parallel, and the excitation of related paths or related nodes is known as semantic priming; it allows for the quicker processing of that

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM association. Semantic priming effects arise from the similarity between the new input (the target) and the network nodes (the prime) that are activated. (Masson, 1995) Humans respond to the world based on a continual categorization and classification of memories. The persistent economy of an organism to gather as much information while conserving its finite resources leads to the concept of categories; categories reduce the infinite

properties and differences among stimuli, the traces, to something that is cognitively manageable. (Rosch, 1999) From a cognitive point of view, humans interact in a symbolic environment and not the physical environment presented to them. (Carley & Palmquist, 1992) Familiarity being the goal, humans relate to new objects by associating them with other physical objects for which they have an internal representation, or a metaphor for classification. (Jonassen & Henning, 1996) Schemas are the categorical storage of this information as sequences of actions that can be retrieved or modified later. (Long, 1989) The concept of schemas is best understood in a discussion of affordances and mental models. Affordances In his seminal work, The Ecological Approach To Visual Perception, Gibson posits that animals, including humans, move through a medium (air most of the time, or water if they are swimming) that contains substances. Substances are perceived by means of their surfaces. (Gibson, 1979, p.32) Visual perception of a surface relies on processing of light emanating or being reflected from that surface. The distinguishing properties of color, texture and the like define the surface. (Gibson, 1979, p24) Jones points out that Gibsons idea that an objects affordances are perceived separately from an objects properties was not what Gibson originally thought and his theory evolved in the years of writing and research in this area between 1966 and 1979. (Jones, 2003) Eleanor Gibson attempted to clarify how affordances are learned. In order to perceive what the affordance is for a particular object, the actor must also understand the environment in which it is resident, their own activity in that environment, and the ensuing results as they interact in the environment. This is essentially how babies learn their surroundings with their minimal exploratory abilities such as seeing and sucking. (Gibson, E., 2000) Ellis and Tucker point out that once the affordance is perceived, it also carries with it the range of possible actions, such as a chair being a thing upon which you sit, or a floor being something on which you stand, or the difference between the uses of a fork and spoon. However, their research points them to the conclusion that the list of possible actions actually depends on the users nervous system disposing of the possible uses of the object. (Ellis & Tucker, 2000) In this way they may be going where Gibson would not, to the idea that perceptual processes known as mental models championed by Donald Norman.

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM Mental Models A mental model is a construct that combines the previously described schema with a

process for manipulating the information in the schema. (Merrill, 2000) Donald Norman, taking a systems level approach, describes mental models as the result of a people interacting with a target system. He notes with that cursory definition several factors need to be considered. These factors to be considered are the target system itself, the conceptual model of the target system (in other words the designers viewpoint of how it will be used), the users mental model (which evolves with interaction with the target) and the scientists conceptual model of what they are observing. (Norman, 1983) This would imply that all those mental models interact and affect each other. From this explanation it could be also extrapolated here that natural things, such as a stone, do not start with a conceptual model (design intent) associated with them until interaction, discounting for the moment any metaphysical or religious purpose that might be ascribed to them. Norman also points out that mental models evolve inherently through contact between user and object and are highly influenced by the nature of that interaction as well as the users own previous experiences. (Norman, 1986) Keiras and Bovair point out in their research, when presenting a new device to a user, even a metaphorically familiar one with new features, those extra features must be presented in such a way that they themselves support inferences about the specific resulting actions. (Keiras and Bovair, 1984) This is important when redefining or enhancing the function of the familiar pen and paper system. The Mental Model Writing with Pen and Paper Writing, as the creative process, is a set of activities in which the writer takes on a goal of organizing a set of thinking processes within the network of their goals and composes them on a surface or task environment. (Flower & Hayes, 1981) This very well describes the mental model of the tasks associated with achieving the goal of written composition; it also touches on the affordances in the writers environment. Gibson neatly describes the specific act of writing with a pen in his theory of affordances: A hand-held tool of enormous importance is one that, when applied to a surface, leaves traces and thus affords trace-making. The tool may be a stylus, brush, crayon, pen, or pencil, but if it marks a surface it can be use to depict and to write, to represent scenes and to specify words. (Gibson, 1979, p.134) Description of the Livescribe Pen and Notebook System The Livescribe Pen and notebook system was introduced in 2008 (Livescribe, Inc.). The writing capture system was based on the handwriting capture technology invented by the members of the Anoto AB team, through many antecedent technologies and patents. (Pettersson & Edso, 2003; Petterrson & Bjrklund, 2006) The Anoto technology allowed pen strokes to be captured on dot paper. The deeper purpose of this handwriting capture technology was ascribed to inventor

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM Christer Fhrus who envisioned in 1996 a high tech pen that could get the paperwork done more efficiently. (Anoto AB, 2012) Livescribe combined audio recording with Anotos

technologies to allow for the ability to simultaneously record handwriting and synchronize it with the concurrent sound track in the vicinity of the user. (Margraff & Chisolm, 2008) The most obvious application of synchronized audio and handwriting is a student taking notes during a lecture with the associated synchronized audio and replaying it later. The Livescribe System in Terms of Affordances and Mental Models In terms of affordances, the pen offers the existing Gibsons trace-making concept previously quoted in this paper. Consistent with Gibsons theory, as Norman points out, some of the affordances are not readily, visually apparent, but the feedback in other non-visual ways of the system may indicate to the user what the function is. (Norman, 1999) Figure 1 is a picture of the pens interfaces and controls. The not-visually-apparent affordances are the microphone and speakers, as well as the camera in the tip. These affordances become apparent in conjunction with the use of the notebook. The power button affords the system to be turned on, but it also has a secondary function. When pressed and held, it will begin to do an audio recording even with the paper is not present. In this case the user must learn that function; it seems slightly disconnected from the mental model of what a power button does.
LCD display embedded camera Power button

Ballpoint ink rell

speaker

microphone

Figure 1: Livescribe Pen Interface and Controls The display is a visual affordance that gives the user familiar information such as the time, as well as the amount of battery power remaining. The mental model of a battery symbol is a familiar metaphor for a user. It is clearly a metaphor since a real battery doesnt actually drain power like a water level decreasing in a vessel. The notebook page, in Figure 2, is an affordance consistent with Gibsons trace-making model as previously described. What the user does not readily see is the dot field on the paper that is used by the camera in the tip of the pen to capture the location of pen strokes on the page. The user would get a sense that the writing was being recorded when they attach the pen to a computer and upload the information. The desktop software renders a copy of the page with their exact strokes. Thus the modern mental model of a page being scanned is inferred. The audio affordance of the pen and notebook system is invoked by a set of controls on the paper surface. Metaphorically, these controls look like controls on any modern recording device and operate within that mental model with one exception. Although there are controls for record, pause and stop functions consistent to the users mental model, the play control is not

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM available near them as would be expected. That interaction or affordance is invoked by tapping

on any pen stroke. If a recording is available, the user will hear the audio emanating from the pen speaker. The user will also note that there are other controls on the bottom of the page to control the speed of playback and playback location. Finally, augmenting the written record while the audio is playing is another affordance that creates its own mental model. Once the user discovers this feature, they will have learned that the synchronization of audio and pen stroke can be enhanced.

Figure 2: Livescribe Notebook page showing interface and controls Recommendations The Livescribe Pen and Notebook System has many useful features that are learnable and somewhat consistent with the metaphors of writing and recording audio. Invoking other nonstandard functions is less obvious. Playback is controlled on the page by either tapping on the pen stroke or on a percentage gauge near the bottom of the page. This gauge at the bottom disengages the temporal correspondence between pen stroke and audio because the writing surface offers no indication or feedback in that regard. The suggestion would be to have the feature reversed so when the gauge is tapped during audio playback previously invoked by tapping on a pen stroke, the pen display shows the percent location of the audio playing for that page. That might be more consistent with the mental model of the action the user might be trying to accomplish. The function whereby audio can be recorded even when a notebook is not present is interesting but difficult for someone to learn. This is accomplished by holding down the power button which results in the pen is automatically put into audio recording mode. Later using

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM playback, if a user writes in a notebook the written and auditory records would become synchronized. This is a useful feature, but one that would be difficult to teach someone without documentation. The affordance here, synchronization after a non-written recording, has no parallel or metaphor that would be invoked in order to learn it except after it is attempted. The closest metaphor would be in the movie industry where a sound track (such as the background music) is superimposed on video during editing. It might be better to have the pen query the user

through the display and accompanying audio if the pen detects through its camera that the pen cap was still attached for several seconds after power on asking the user to tap the power button to invoke that function after power up. If the pen cap were removed, the prompt would be stopped. However, significant research would be required to come up with more inferable solution. Conclusion This paper studied the memory, categorization, schemas, spreading activation, mental models and affordances. The paper briefly described the Livescribe pen and notebook system technology, and then described that functionality in terms of mental models and affordances. The paper closes with observations about features that are difficult to learn because they have no corresponding metaphor except to the extent that they are learned when the situation occurs to use them. The recommendation is to remove the playback gauge at the bottom of the page as it is not particular usable or understandable in the model that the pen presents. The feature in which the power button held down starting non-attached notebook recording should also be rethought.

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM References Anderson, J. R. (1983). A spreading activation theory of memory. Journal of verbal learning and verbal behavior, 22(3), 261-295. Anoto AB. History. (2012). Retrieved from <http://www.anoto.com/history-2.aspx>. Barsalou, L. W. (2008). Cognitive and Neural Contributions to Understanding the Conceptual System. Current Directions In Psychological Science (Wiley-Blackwell), 17(2), 9195. doi:10.1111/j.1467-8721.2008.00555.x Carley, K., & Palmquist, M. (1992). Extracting, representing, and analyzing mental models. Social Forces, 70(3), 601-636. Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological review, 82(6), 407. Crestani, F. (1997). Application of spreading activation techniques in information retrieval. The Artificial Intelligence Review,11(6), 453-482. doi: http://dx.doi.org/10.1023/A:1006569829653 Ellis, R., & Tucker, M. (2000). Micro affordance: The potentiation of components of action by seen objects. British Journal of Psychology, 91(4), 451-471. Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing.College composition and communication, 32(4), 365-387. Gibson, E. J. (2000). Where is the information for affordances?. Ecological Psychology, 12(1), 53-56. Gibson, J. J. The Ecological Approach to Visual Perception. Houghton Mifflin, Boston, 1979. Jonassen, D. H., & Henning, P. (1996). Mental models: Knowledge in the head and knowledge in the world. In Proceedings of the 1996 international conference on Learning sciences (pp. 433-438). International Society of the Learning Sciences. Jones, K. S. (2003). What is an affordance?. Ecological psychology, 15(2), 107-114. Kieras, D. E., & Bovair, S. (1984). The role of a mental model in learning to operate a device. Cognitive science, 8(3), 255-273.

Livescribe, Inc. Livescribe FAQ. (2010). Retrieved from <http://www.livescribe.com/enus/presskit/assets/Livescribe_FAQ_071910.pdf>. Long, D. (1989). Second Language Listening Comprehension: A Schema-Theoretic Perspective. Modern Language Journal, 73(1), 32. Marggraff, James & Chisolm, A. (2008). Interactive apparatus with recording and playback capability usable with encoded writing medium. US: Patent 7453447B2. Masson, M. E. (1995). A distributed memory model of semantic priming.Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(1), 3.

MENTAL MODELS, AFFORDANCES & LIVESCRIBE SYSTEM Merrill, M. D. (2000). Knowledge objects and mental models. In Advanced Learning

10

Technologies, 2000. IWALT 2000. Proceedings. International Workshop on (pp. 244-246). IEEE. Norman, D. (1983). Some observations on mentai models. Mental models, 7. Norman, D. A. (1986). Cognitive engineering. User centered system design, 31-61. Norman, D. A. (1999). Affordance, conventions, and design. interactions, 6(3), 38-43. Pettersson, M. & Edso, T. (2003). Determination of a position code. US: Patent 6548768. Pettersson, M. & Bjrklund, A. (2006). Method and device for decoding a positioncoding pattern. US: Patent 7999798. Quillian, M. R. (1969). The teachable language comprehender: a simulation program and theory of language. Communications of the ACM, 12(8), 459-476. Rosch, E. (1999). Principles of categorization. Concepts: core readings, 189-206.