Multitouch April 9

Extension
module
for GIMP
2010
An Extension module that would allow gimp to receive multiple inputs and
hence allow manipulations via Multitouch gestures. GIMP
Abstract
My Proposal is to make Gnome Image Manipulation Program to be able to accept multiple
inputs and hence be able to take actions based on the gestures by multiple inputs or single
input. This will also allow bi-manual inputs.

Multitouch can be great motivating factor behind this proposal, ever since I used image editing
applications on my I-phone which enables Multitouch supports then felt how effortlessly
editing is that never found on my system using keyboard and mouse.Also it was not done
before with any other Image editing application. A powerful image editing application with
multiple touch capable user interface will ease the process of editing to very high extent.

 Drawing with hands on touch surface will be more intuitive when compared to
traditional mouse-keyboard drawing. GIMP can be used as drawing book for children in
schools as drawing on GIMP will be more natural. Drawing on GIMP will also facilitate them
more color combination, shapes of brushes and option to go back if something go wrong.
 More than one user will be able to work on the same table, also will be able to share their work
by simple gestures.
 Growing multitouch gadgets require application that support more than one inputs. GIMP will
be first to implement of its type.

The reason behind to choosing Gimp not to start developing something from scratch is due it
it’s easy to start UI. By experience I’ve felt that gimp is perfect to support Multitouch and image
editing will be fun using cool gestures over it.

Hence idea is simple to develop extensions with new gesture–sets that’d easy to use image
editing on Multitouch hardware’s.

Introduction to development
My proposed project will be developed in a C++ makefile environment that can compile cross-
platform. This project will be developed through a series of iterations, with a concrete
deliverable at every stage.

Implementing multiple inputs in gimp demands its customized user interface, a gesture
recognition toolkit that will be recognizing gestures and support for present and future
multitouch hardware.
Customized User Interface:

The whole user interface would comprise of two categories of windows, first category would be
Type 1 windows. These windows will be having only single input support i.e. they will not be
having more than one inputs and no gesture recognition will be done when focus would be on
these windows. Tool selection window, layers selection window and all other windows where
more than one input is not possible will be these windows. So on these windows touch will be
simulated as mouse inputs which can be read by gtk+ to reduce complexity.

Type 2 windows will be having full-fledged gesture support. These windows are the canvas
windows which will be containing images and drawing windows.

Type 1 and Type 2 windows

Also Type 2 windows will be supporting multiuser interface. Multiuser interface facilitates
collaborative work simultaneously allows user to share their work intuitively on the same touch
surface.
 Multiuser collaboration on Gimp, sharing of work using gestures

Gimp present user interface supports this concept as it is not having one window composed of
everything. Whenever a new image if opened, it opens it in different window. This naturally
aids the concept of multiuser interface as it requires personal canvas window for each user. So
making this user interface possible is not big daunting task.

Gesture recognition toolkit:

I would be creating a language / framework independent Gesture Recognition toolkit that takes
OSC messages formatted with TUIO specification as input and outputs recognized gestures via
OSC protocol. I will use the gesture recognition toolkit AMELiA to describe models specifically
for the domain of multitouch gestures. So input OSC messages are used to receive touch
information and interpret it into events such as a click, drag, pinch, rotation, zoom etc.

Then implementation of TUIO C++ with gtk+ will be done, so that gesture recognition toolkit
can communicate with gtk+.
http://www.tuio.org/api/cpp/index.html
This will facilitate gtk+ receive gestures with location in x-y coordinates. Whenever a gesture will
be recognized, corresponding event will be generated from gtk+.

The procedure would be to initiate gtk+ events with gestures defined for specific tasks that would map
to three primary tasks – Drawing, workspace Navigation and multiple user interaction.

The Drawing Mode would allow drawing via Multitouch. To keep the interface intuitive a new toolbar
menu set will be created. This has a certain advantage over the traditional mouse system since it’d
minimize the need to access the toolbar again and again for commonly used tasks.

TOUCH SET 1 - Workspace Navigation- This includes Gestures to ZOOM-IN , ZOOM-OUT, PAN , AND
ORBIT and taking measurements- without having the need to select any of the sub-options.

 Rotation of canvas window

 Panning – Single Touch drag

Few transformation Tool implementations via multitouch gestures

TOUCH SET 2 - Drawing- This shall enable drawing using gestures via GIMP- The tentative plan for the
gestures has been described in diagrams below
 Drawing gestures

Using selection tool

Support for present and future multitouch hardware
Initially development will be started with support to tuio based touch surfaces, like all open source
multitouch hardware. Later support for other hardware will be extended like mac touch, wii mote touch
devices, , and hp slate touch input. This can be done by enabling gesture recognition toolkit to receive
input from these devices.

TimeLine
1st Month
 Further discuss the idea with mentors.
 Prepare Final tentative list of features/gestures implementations.
 A simple learning and recognition system for non-parameterized, fixed dimensionality
gestures.
 Integrate TUIO C++ client to GIMP. Make GIMP receive TUIO messages.
 Basic changes to GIMP GUI.

2nd Month
 First part of implementation: GIMP communication with single touch using TUIO
 Finalize base UI button-set.
 Adding varying dimensional gestures to the model. Gestures that have taps, multiple
touch-up and touch-down events will be learnt and recognized
 Map proposed gestures to common tasks. Apply to Orbit, Zoom, Pan etc.

3rd Month
 Testing- test out on FTIR table.
 Improve upon the usability factor.
 Work on documentation.
 Test on Multitouch setup, Release packaged module to community.
 Fine tune the working related to Multitouching gestures. Reduce Confusion.
 Debug, Reduce code complexity.
 Testing
 Improve documentation, Release
Personal Profile
Name: Alok Mishra
Email: hayalok@gmail.com
Time zone: +530 GMT
Blog URL: ialok.wordpress.com
Location: New Delhi, India

I am a 21 year old student from New Delhi, India pursuing Bachelor of Engineering degree in
Electronics and Communication Engineering at Govt. Engineering College Bikaner, Rajasthan
Technical University. I have been involved with Open Source communities online for over three
years. Last year’s Google Summer of Code got me introduced to NUI Group, a part since.
Thereafter I made a FTIR table in college and began experimenting with real Multitouching and
realized how it is going to revolutionize the way we interact with our computers.

At the age of 17 I won an interschool Flash Website Design competition that got further got me
interested into UIs, and programming. During my curriculum I’ve taken courses on object-
oriented programming in C++, Data Structures and Algorithms, Perl, UML, Computer Graphics,
and Architecture. Apart I’ve been experimenting with Touchlib’s AS3 Library apps by extending
the premade apps.

I can program in AS3, C, C++, Ruby and PHP. Since I am already well-versed in the community
interactions/dynamics of OSS projects, I will have absolutely no trouble in mingling with the
GIMP community and working with the infrastructure (I am familiar with CVS, SVN etc.), already
in place. I have contributed to a cross platform multitouch library python pymt. As I have
already started to became familiar with the API of GTK+ and GIMP development environment
well before two months from start of program, coding could be started right away, which shall
give me extra time in hand to try provide support feature for all present hardware, as proposed
above.

Recently participation in some intercollegiate technical festivals, fetched my team First Prize at
IIT Kanpur and BITS Pilani’s technical festival, where I exhibited our table with NUI’s apps and
demonstrated how to make a Flash App from scratch.
References
When It Gets More Difficult, Use Both Hands – Exploring Bimanual Curve Manipulation
Russell Owen, Gordon Kurtenbach, George Fitzmaurice, Thomas Baudel, Bill Buxton
http://www.billbuxton.com/Bimanual2005.pdf

Interaction Techniques for 3D Modeling on Large Displays -Tovi Grossman, Ravin

Balakrishnan, Gordon Kurtenbach, George Fitzmaurice, Azam Khan, Bill Buxton