Light Robotics - Structure-mediated Nanobiophotonics

Light Robotics Structure-mediated Nanobiophotonics

Jesper Glückstad

Darwin Palima

Series Editor

David L. Andrews

Table of Contents

Cover

Title page

Copyright

List of Contributors

Biographies

Preface

Introduction

Part 1: Elements of light robotics

Chapter 1: Human gesture recognition for optical manipulation and its future nanobiophotonics applications

Abstract

1. Optical tweezers basics

2. Measurement of position and force

3. System design and instrumentation of optical manipulation systems

4. Human interfaces

5. Control with peripheral devices

6. 3D control

7. Haptics

8. Internet control—controlling systems remotely

9. Future directions

Chapter 2: Laser-based assembler and microfluidic applications

Abstract

1. Introduction in microfluidics

2. Generation of microstructures with two-photon polymerization

3. Assembling techniques

4. Applications for assembled microstructures

5. Conclusion and outlook

Chapter 3: Optomechanical microtools and shape-induced forces

Abstract

1. Introduction and background

2. Theory

3. Experimental realizations

4. Applications

5. Conclusions and future prospects

Chapter 4: Optically driven rotating micromachines

Abstract

1. Introduction

2. Optical angular momentum

3. Principles of design

4. Computational modeling

5. Fabrication

6. Applications

7. Conclusions

Chapter 5: Electromagnetic wave enabled micro/nanorobotic devices and their applications

Abstract

1. Introduction

2. Optically enabled robotic micro/nanodevices

3. Plasmonically enabled robotic micro/nanodevices

4. Electrically and optoelectronically enabled robotic micro/nanodevices

Part 2: Expanding possibilities

Chapter 6: Complex polymer microtools for on-demand contact-free applications

Abstract

1. Introduction

2. Applications

3. Conclusion and outlook

Chapter 7: Optically driven micro- and nanorobots

Abstract

1. Overview of optically driven micro- and nanorobots

2. Multiple-trap manipulation system for light robotics

3. Fabrication technology for light robotics

4. Sensing technology for light robotics

5. Biomedical applications of light robotics

Chapter 8: Enhanced second-harmonic generation in lithium niobate nanowires used for localized light delivery

Abstract

1. Introduction

2. State of the art

3. Theoretical background

4. Sample fabrication

5. Experimental setup

6. Experimental results and discussion

7. Conclusion and outlook

Acknowledgments

Chapter 9: Next generation light robotics

Abstract

1. Introduction

2. Scaling of physical effects

3. Light-driven microrobots

4. Microrobots with various functional loads

5. Conclusion and outlook

Part 3: Biophotonics applications

Chapter 10: Optical techniques and microtools for subcellular delivery and sampling

Abstract

1. Introduction and overview

2. Direct optically triggered membrane effects

3. Secondary target techniques

4. Conclusions

Chapter 11: Optically controlled fusion of selected cells and vesicles using plasmonic nanoheaters

Abstract

1. Introduction

2. Optical control of plasmonic nanoheaters

3. Fusion of pure membranes

4. Fusion of pure membrane systems to living cells

5. Fusion of two selected living cells

6. Conclusion and outlook

Acknowledgments

Chapter 12: The application of optically trapped force probes in single molecule biology experiments

Abstract

1. Introduction

2. Force probe design

3. Experimental setup

4. Force probe calibration

5. Single molecule force experiment

6. Future directions

Chapter 13: Application of optical tweezers for biochemical and thermal cell stimulation

Abstract

1. Introduction: cell migration and local stimulation of single cells

2. Methods

3. Results

4. Summary, conclusions, and outlook

Acknowledgments

Chapter 14: Controlling autonomous nanobiorobots by optical micromanipulation

Abstract

1. Introduction

2. Nanomotors

3. Bacterial microsystems

4. Bacterial adhesion and patterning

5. Optical manipulation of bacteria

6. Functional optically assembled systems based on biomolecular motors

7. Conclusions

Index

Copyright

Elsevier

Radarweg 29, PO Box 211, 1000 AE Amsterdam, Netherlands

The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

Copyright © 2017 Elsevier Ltd. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-7020-7096-9

For information on all Elsevier publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Matthew Deans

Acquisition Editor: Simon Holt

Editorial Project Manager: Sabrina Webber

Production Project Manager: Nicky Carter

Designer: Greg Harris

Typeset by Thomson Digital

List of Contributors

Badri L. Aekbote,     Biological Research Centre, Szeged, Hungary

Fumihito Arai,     Nagoya University, Nagoya, Japan

Andrew Rafael Bañas,     Technical University of Denmark, Lyngby, Denmark

Azra Bahadori,     Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark

Álvaro Barroso Peña,     Institute of Applied Physics, University of Muenster, Muenster, Germany

Poul M. Bendix,     Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark

Konrad Berghoff,     University of Bayreuth, Bayreuth, Germany

Ann A.M. Bui,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

András Buzás,     Biological Research Centre, Szeged, Hungary

David Carberry,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Duncan Casey,     Centre for Functional Nanomaterials, University of Bristol, United Kingdom

Cornelia Denz,     Institute of Applied Physics, University of Muenster, Muenster, Germany

Cemal Esen,     Applied Laser Technologies, Ruhr-Universität Bochum, Universitätsstraße, Bochum, Germany

Donglei (Emma) Fan

Materials Science and Engineering Program

The University of Texas at Austin, Austin, TX, United States

Lisa Gebhardt,     University of Bayreuth, Bayreuth, Germany

Jesper Glückstad,     Technical University of Denmark, Lyngby, Denmark

Rachel Grange,     Optical Nanomaterial Group, Institute for Quantum Electronics, Zurich, Switzerland

Wolfgang Gross,     University of Bayreuth, Bayreuth, Germany

Jianhe Guo,     Materials Science and Engineering Program, The University of Texas at Austin, Austin, TX, United States

Simon Hanna,     University of Bristol, Bristol, United Kingdom

Takeshi Hayakawa,     Nagoya University, Nagoya, Japan

Antoine Houillot,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Jannis Köhler,     Applied Laser Technologies, Ruhr-Universität Bochum, Universitätsstraße, Bochum, Germany

Neil M. Kad,     University of Kent, Canterbury, Kent, United Kingdom

Anatolii V. Kashchuk,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Lóránd Kelemen,     Biological Research Centre, Szeged, Hungary

Steve Keller,     University of Bayreuth, Bayreuth, Germany

Holger Kress,     University of Bayreuth, Bayreuth, Germany

Sarah I. Ksouri,     Applied Laser Technologies, Ruhr-Universität Bochum, Universitätsstraße, Bochum, Germany

Hisataka Maruyama,     Nagoya University, Nagoya, Japan

Mark Neil,     Imperial College London, London, United Kingdom

Timo A. Nieminen,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Lene B. Oddershede,     Niels Bohr Institute, University of Copenhagen, Copenhagen, Denmark

Pál Ormos,     Biological Research Centre, Szeged, Hungary

Andreas Ostendorf,     Applied Laser Technologies, Ruhr-Universität Bochum, Universitätsstraße, Bochum, Germany

Darwin Palima,     Technical University of Denmark, Lyngby, Denmark

David Phillips,     University of Glasgow, United Kingdom

Mark R. Pollard,     DFM A/S, K. Lyngby, Denmark

Daryl Preece,     University of California San Diego, La Jolla, CA, United States

Halina Rubinsztein-Dunlop,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Anton Sergeyev,     Optical Nanomaterial Group, Institute for Quantum Electronics, Zurich, Switzerland

Linda Shi,     University of California San Diego, La Jolla, CA, United States

Stephen Simpson,     Institute of Scientific Instruments of the CAS, Brno, Czech Republic

Alexander B. Stilgoe,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Mark Jayson Villangca,     Technical University of Denmark, Lyngby, Denmark

Gaszton Vizsnyiczai,     Biological Research Centre, Szeged, Hungary

Douglas Wylie,     Imperial College London, London, United Kingdom

Shu Zhang,     The University of Queensland, St Lucia, Brisbane, QLD, Australia

Gordon Zyla,     Applied Laser Technologies, Ruhr-Universität Bochum, Universitätsstraße, Bochum, Germany

Biographies

Jesper Glückstad is Professor and Group Leader at the Department of Photonics Engineering at the Technical University of Denmark. He established the Programmable Phase Optics Laboratory in Denmark in the late 1990s, and served as Guest Professor in Biophotonics at Lund Institute of Technology, Sweden, 2006–11. In 2004 he received the prestigious Doctor of Science (DSc) degree from the Technical University of Denmark. Prior to his achievements in Denmark, he was a visiting scientist at Hamamatsu Photonics Central Research Laboratories and in the Physics Department at Osaka University in Japan. He is the year 2000 recipient of the Danish Optical Society Award and was elected as Scientist of the Year in 2005 by Dir. Ib Henriksen’s Foundation in Denmark. Prof. Glückstad is a 2010 elected Fellow of the OSA and a Fellow of the SPIE as the first from Denmark. Between 2012 and 2014, he served on the prestigious SPIE Fellows committee. Since he obtained his PhD at the Niels Bohr Institute in 1994, he has published more than 300 journal articles and international conference papers among them several in Nature journals. He holds more than 30 international patent inventions to his name and is founder of the 2011 spin-out OptoRobotix ApS and its recent associated tech-transfer unit GPC Photonics.

Darwin Palima is Associate Professor in the Department of Photonics Engineering at the Technical University of Denmark where he teaches a course he created on Biophotonics and Optical engineering. He has pioneered new scientific directions at the programmable phase optics laboratory and, as lab responsible, closely mentors the group’s PhD students and postdocs. He collaborates extensively with Jesper Glückstad from joint supervision of PhD students to joint authorships of scientific papers, conference presentations, patent applications, and popular articles. Having authored a physics textbook before moving to Denmark, he played a key role in their jointly authored monograph on Generalized Phase Contrast. The present volume is another product of their collaboration. Darwin concurrently taught at the Philippine Science High School when he worked through his PhD at the University of the Philippines proposing improvements to computer-generated holography. His interest in structured light started in the mid-1990s when he studied optical vortices for his BSc thesis.

Preface

Scientific disciplines constantly evolve and create new offspring—subdisciplines—that combine the favorable characteristics from its forerunners. The merger of biology and photonics has within the last decade produced one such offspring, Biophotonics, which harnesses light to study biological materials. More recently we have seen the exciting merger of biophotonics with contemporary nanophotonics into so-called nanobiophotonics culminating with the 2014 Chemistry Nobel Prize for superresolution microscopy—now simply coined nanoscopy. The usage of the term Biophotonics in scientific papers curiously began to shoot up after Prasad’s 2003 book Introduction to Biophotonics, the first monograph on the field. The aforementioned correlation can mean two things: (1) The book was written at an opportune time when biophotonics was actually taking off; and (2) the book helped clarify the scope of the emerging field for disparate researchers who then contributed to its growth upon realizing how their activities are united in a new context.

This book was written along similar lines on the borderline between a plurality of emerging scientific subdisciplines. After years of working on light-driven trapping and manipulation, we can see that a confluence of developments is now ripe for the emergence of a new area that can contribute to nanobiophotonics—Light Robotics—which combines advances in microfabrication and optical micromanipulation together with intelligent control ideas from robotics. This volume collects expert contributions from various areas that are coherently coming together through light robotics. We cover the fundamental aspects of optical trapping systems, microfabrication, and microassembly, and present theoretical principles and experimental illustrations for optimizing the optical force and torque. We also present an array of various new functionalities that are enabled by these new designed structures—light-driven microrobots. Finally, we cover various nanobiophotonics applications demonstrating the use of biophysical tools utilizing light robotics concepts.

We have endeavored to make the book accessible to a broad audience from advanced undergraduates and graduate students to practitioners and researchers not only in nanobiophotonics and micro- and nanotechnology, but also to other areas in optics and photonics, mechanical engineering, control and instrumentation engineering, and related fields. We hope that this book is able to do justice in presenting a clearer picture of this emerging field, which is essential to igniting the needed synergy between various stakeholders in the development of this field. We are fortunate to be taking these first steps together with all the contributors from across four continents to whom we owe a debt of gratitude for the time, effort, and expertise devoted into formulating their respective chapters. Thank you very much!

Jesper Glückstad

Darwin Palima

Technical University of Denmark, Kgs. Lyngby, Denmark

May 2017

Introduction

Darwin Palima

Jesper Glückstad

Technical University of Denmark, Kgs. Lyngby, Denmark

The 1920 science fiction play, R.U.R, introduced the word robot to describe mass-produced synthetic humans [1]. The fascination with intelligent humanoid robots continues today and remains a favorite theme in science fiction. Beyond fiction, the modern definition of a robot, according to ISO 8373:2012 [2], is an actuated mechanism programmable in two or more axes with a degree of autonomy, moving within its environment, to perform intended tasks and robotics is the science and practice of designing, manufacturing, and applying robots. A robotic device is similar to a robot but may lack in the number of programmable axes or autonomy, that is, teleoperated devices controlled by human operators as we see in robotic surgery [3]. This book shows that various light-based technologies are now enabling functionalities in what can be aptly recognized as an emerging field of light robotics. In the new context of light robotics, it will suffice to use robots and robotic devices interchangeably.

Light robotics refers to the use of light to realize functionalities associated with robotics and robotic devices or, alternately, importing ideas from robotics to develop new light-based functionalities. This book focuses on functionalities that are relevant to biophotonics at the micro- and nanoscale. However, just as what we call optical tweezers does not look remotely close to mechanical tweezers, one can expect that many of the microrobots that we will encounter in this book do not resemble conventional robots or robotic devices. Nonetheless, a closer look will reveal that they have a role to play in realizing light-based robotics for biophotonics.

A robot needs a control system to monitor and regulate its functions and provide an interface to users and other equipment, if necessary. However, working in the micro- and nanoscale biophotonics regime imposes restrictive conditions on how much hardware and software we can fit into the tiny microstructures—the microrobots. We can work around this basic space constraint by off-loading the intensive computing and control aspects to regular computers. This off-loading is analogous to how compact, highly portable devices gain access to sophisticated computational power via cloud computing services. We can further optimize the limited space by also removing the onboard power.

So, how can we actuate microrobots having no onboard processor and power source? Light is a useful agent for remote actuation. Using light as information carrier would have worked to transmit coded instructions to an onboard processor on a powered robot [4]. Without onboard power, we could use light as energy carrier to supply power remotely via onboard photovoltaics [5]. Without onboard circuitry, we can still use light to carry energy to directly drive light-sensitive structures using materials that can convert optical to mechanical energy via photomechanics [6,7]. Taking off from optical trapping and optical micromanipulation, this book focuses on approaches that exploit light’s momentum to drive microscopic objects designed as microrobots [8]. Most modern optical micromanipulation derive inspiration from Arthur Ashkin’s pioneering work in 1970, which presented the possibility of using optical radiation pressure to trap and manipulate microparticles [9]. One popular implementation, the optical tweezers, uses a sharply focused beam to work as actuator to trap and manipulate microparticles in 3D [10,11]. However, there are many possible implementation geometries, each having their respective strengths and weaknesses [12]. Designing bespoke light robotics systems must consider the application-specific requirements and constraints.

Fig. 1 presents a schematic showing the different elements in a light robotics system. One of the key elements is controlled light delivery to actuate and adjust the position of a microrobot. The microrobot is shown carrying a functional load, analogous to an end effector—the device installed at the end of robotic arms that interacts with the environment to perform its task. The microrobot and its functional load may themselves be fabricated or assembled using controlled light delivery. With clever design, the microrobot and its functional load can enable structure-mediated access either to nanoscopic length scales beyond the reach of diffraction-limited light beams or to minimize direct exposure when probing light-sensitive processes. Controlled light delivery may also be used to directly interact with the system as well as to activate and regulate the functional load, which can be designed either to deliver stimuli or collect information, that is, for sensing, detection, imaging, or spectroscopy in conjunction with an auxiliary system. In general, there can be a swarm of microrobots simultaneously working at different tasks. These tasks can be geared toward biophotonics, a field that aims to harness light to image, detect, and manipulate biological matter for fundamental studies and various applications [13]. This book explores the various aspects of the light robotics system and presents illustrative examples harnessing their utility for biophotonics.

Figure 1   Schematic of a Light Robotics System (inset artist’s rendition courtesy of A. Bañas).

Optical tweezers are precision biophysical tools that can measure the mechanical properties of biomolecules. This is typically done by tethering them onto microspheres held by optical tweezers exerting calibrated optical forces. This illustrates a light robotics approach in biophotonics. This use of optically trapped microspheres is a well-developed subject in optical trapping and is treated at length in several books on optical tweezers [14,15]. In this book, our focus is on leveraging advances in microfabrication to design microrobots that elevate the level of control and realize new functionalities. We will still discuss trapped microspheres but the focus is on alternate functionalities that they can offer beyond calibrated optical forces.

Telerobotics has proven indispensable for distances that are orders of magnitude larger than what we can conveniently access, as the success of the mars exploration rovers shows [4]. However, working at orders of magnitude smaller length scales arguably presents a similar challenging inaccessibility and remoteness. Thus, nanomanipulation techniques are also referred to as telenanorobotics approaches, for instance, when using atomic force microscopy (AFM) as the nanorobot [16–18]. Besides autonomous nanoscopic machines, an alternate definition of nanorobots and nanorobotics includes complete systems, regardless of actual size, that can manipulate nanoscopic entities with nanoscale precision. We adopt this duality in this book and use microrobots and microrobotics when referring both to the tiny light-driven microstructures as well as the complete systems used to optically control and manipulate them.

The book in brief

Back in 2005, Elliot Botvinick and Michael Berns demonstrated RoboLase, a teleoperated system at the University of California, where internet-based users as far away as Australia could control the RoboLase microscope and its laser beams for ablation and optical micromanipulation [19]. In Chapter 1, project scientists at Michael Berns’ lab discuss the basic concepts and design considerations in optical trapping and manipulation systems, including the RoboLase. The chapter provides special focus on the different human interfaces that are being explored to help operators achieve intuitive control in light robotics.

Versatile technologies for microfabrication and microassembly are essential for rapidly prototyping new design ideas in light robotics. In this regard, Chapter 2 discusses the fundamentals of two-photon fabrication, a 3D laser direct writing technique, as well as various light-based assembling techniques that exploit the pick-and-place robotic functionality of optical tweezers for bottom-up fabrication using basic components. This chapter outlines different methods for fixed assembly and illustrates examples of detachable assembly for reconfigurability in lab-on-chip applications. The authors use holographic optical tweezers to simultaneously generate multiple individually controllable optical tweezers for experimental demonstrations.

Chapter 3 details the theory on the behavior of complex shapes in optical fields and discusses their rigorous force calibration. It introduces the concept of shape-induced forces, which treats the shape of the microobject as a design parameter that can be optimized for applications, for example, to maintain the same force even when the object moves relative to the illuminating beam. The chapter describes the fabrication, 3D tracking and optical control of these microtools together with a range of demonstration experiments including 3D surface imaging for applications in nanobiophotonics.

Designing nonspherical structures enables control not only over the optical force but the optical torque as well. Chapter 4 reviews the basic theory/principles for rotating microstructures with light and presents heuristic principles for designing light-driven micromachines. Asymmetry is a key element: A spherically symmetric object cannot be rotated by optical illumination. The chapter illustrates these concepts for a prototypical optically driven micromachine, which it follows through fabrication and experimental demonstration. To appreciate these developments in a larger context, Chapter 5 reviews the recent progress in the development of electromagnetic wave enabled micro/nanomechanical motors. To provide expanded benchmarking, the chapter also considers recent developments in plasmonics-enabled nanorotors as well as optoelectronic and electric robotics micro- and nanodevices.

The combination of advances in microfabrication with optical manipulation creates various novel functionalities for light robotics and we see some of them in several of the chapters that follow. Chapter 6 presents two-photon fabricated structures that are functionalized with specific physical, chemical, and biochemical methods to create complex microtools. Light-driven structures are used in synthetic model systems for understanding the dynamics of self-propelled microswimmers and hydrodynamic synchronization. Moreover, fabrication incorporates various functional elements into the light-actuated microrobot, such as waveguides, SERS probes and even contact-based grippers for handling cells with minimal illumination.

Chapter 7 demonstrates various light driven micro- and nanorobots for cellular transportation, force application, cell surgery, and extra- and intracellular measurements. Structures doped with temperature-sensitive dyes can act as local temperature sensors. Nanowires exhibiting high photothermal effect work as efficient and highly localized light-activated heaters that are steered by microrobots, for example, to porate cell membranes. The chapter discusses the fabrication of microrobots carrying nanowires as their functional load, which opens a gateway for bringing advances in nanowires into light robotics. To present a glimpse of opportunities available in nanowire photonics [20], Chapter 8 discusses the optimization of lithium niobate nanowires for second harmonic generation and waveguiding and experimentally demonstrates that the guided second harmonic light can be used for localized fluorescence excitation. Finally, Chapter 9 looks at the various opportunities available from metallic nanolayers that light robotics can potentially tap from. For illustration, the chapter considers a hollow microrobot with a metallic disc nanolayer deposited inside. Experimental demonstrations illustrate that this effectively works as a light-activated syringe that a microrobot can use to load and unload cargo. This can be especially useful for cargo that may be difficult to trap, for example, due to size or poor refractive index contrast.

The final section of the book illustrates the strong potential of light robotics in biophotonics through various example applications demonstrating the use of light robotics concepts for developing biophysical tools to the investigation of various aspects of biophotonics.

Chapter 10 sets the stage and examines an important area where light robotics can play a strong role: investigating biological processes with resolution in the single-cell and subcellular levels to reveal details that are lost in standard protocols employing a large ensemble of cells. The chapter surveys various tools and techniques that interact with the cell membrane, disrupting it either via direct laser illumination or structure-mediated approach. Different opportunities are illustrated, such as optical poration and lysis alongside surface chemistry and coatings capable of causing structural effects within a target cell’s plasma membrane. Chapter 11 explores in detail the use of gold nanoparticles as functional elements that can be positioned and heated with light to perturb cellular membranes. Plasmonic effects turn the gold nanoparticles into efficient nanoheaters that can weld cells in an altogether different manner: instead of bonding cells together, it triggers the fusion of the cell membranes to combine intracellular materials into a single viable hybrid cell. This hybridization offers new opportunities in basic and applied cellular research. The technique also works for introducing foreign materials into cells by fusing them with synthesized vesicles. Going further down the size scale, Chapter 12 uses light robotics in a single-molecule application. This chapter takes readers through the various development aspects from fabrication to the calibration and final application of optically trapped probes in single-molecule biology force measurements using DNA filaments and a DNA repair protein.

Chapter 13 explores biophysical tools created using two types of functional loads: heaters and chemical sources. Here, light robotics is used to control, both temporally and spatially, the temperature and chemical gradients established in a cellular environment. The chapter also develops various optical and numerical analysis tools to quantitatively characterize the cellular response to the thermal and chemical gradients, respectively called thermotaxis and chemotaxis. The promising results in this chapter indicate that light robotics can play a role in understanding the basic processes governing cellular migration, which can have rich implications from controlling biofilm formation to regulating immune system response. However, this understanding can also have a symbiotic benefit to the light robotics field, particularly in developing autonomous microrobots. Chapter 14 discusses work along this direction. As we saw in Fig. 1, controlled light delivery can be used to fabricate/assemble microrobots that, in turn, may also be steered and functionalized by light. However, light scattering makes controlled light delivery increasingly difficult as we go deeper into biological matter. Thus, working inside thick biological tissues will require either a light delivery system beyond the current state of the art or autonomous micrororobots that are capable of autonomous navigation. Given the space constraints discussed earlier, we can turn to biomimetics for solutions that mimic processes governing cellular migration through various field gradients. Chapter 14 uses controlled light delivery to assemble a hybrid microrobot that consists of a designed structure (e.g., a drug carrier) and several bacteria preferentially arranged to propel the microrobot using their flagellar motion. Moving forward requires developing microrobots that can efficiently respond and move according to task-specific gradients.

Outlook: challenges and opportunities

The various elements of light robotics outlined in Fig. 1 present various challenges and opportunities. Advancing light robotics can benefit from improvements in optical manipulation–for example, designed structures that are highly efficient in translating incident light into usable optical force/torque and new strategies for controlled delivery that can work reliably through highly scattering biological media. However, optical manipulation is just one aspect of light robotics. Further progress will require support from proactive research and development of compatible light sources, sensors, and other allied technologies to equip the microrobots with needed functionalities. Optical microfabrication and microassembly can be enriched by ideas from other microfabrication and assembly techniques to develop the next generation of microrobots. However, real progress requires not just isolated advances in these subdisciplines but also dedicated efforts at system integration. Moreover, applications in biophotonics will require a strong collaboration between light robotics tool developers and biological end-users. Newly developed materials, tools, and techniques need to be subjected to rigorous biocompatibility and cell viability studies. Collaboration would also be essential in guiding the creation of light robotics tools that genuinely address authentic users’ needs. Bringing these tools out of the optics and photonics labs and into biophotonics end-users, will require instruments that can be operated intuitively by the end-users and so attention to ergonomics and intuitive human interfaces is needed as well.

This book aims to promote this collaboration. We have collected contributions from various traditional research fields and presented how they can coherently come together through light robotics. We hope this book can contribute to igniting a similar synergy between the different stakeholders, which will be needed to accelerate developments in this new field. There are also opportunities for making headway through light robotics in other application areas beyond biophotonics.

References

[1] Capek K, R.U.R. (Rossum’s Universal Robots), Prague, CZ, 1920.

[2] ISO 8373:2012—Robots and robotic devices—Vocabulary. Available from: http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=55890

[3] Lanfranco AR, Castellanos AE, Desai JP, Meyers WC. Robotic surgery: a current perspective. Ann Surg. 2004;239:14–21.

[4] Mars exploration rover mission: home. Available from: http://mars.nasa.gov/mer/home/

[5] Jones BE, McKenzie JS. A review of optical actuators and the impact of micromachining. Sensors Actuators A Phys. 1993;37:202–207.

[6] Yu Y, Nakano M, Ikeda T. Photomechanics: directed bending of a polymer film by light. Nature. 2003;425:145.

[7] Han DD, Zhang YL, Ma JN, Liu YQ, Han B, Sun HB. Light-mediated manufacture and manipulation of actuators. Adv Mater. 2016;28:8328–8343.

[8] Palima D, Glückstad J. Gearing up for optical microrobotics: micromanipulation and actuation of synthetic microstructures by optical forces. Laser Photon Rev. 2013;7:478–494.

[9] Ashkin A. Acceleration and trapping of particles by radiation pressure. Phys Rev Lett. 1970;24:156–159.

[10] Ashkin A, Dziedzic JM, Bjorkholm JE, Chu S. Observation of a single-beam gradient force optical trap for dielectric particles. Opt Lett. 1986;11:288.

[11] Ashkin A, Dziedzic JM. Optical trapping and manipulation of viruses and bacteria. Science. 1987;235:1517–1520.

[12] Glückstad J, Palima D. Generalized phase contrast: applications in optics and photonics. In: Springer Series in optical sciences, vol 146, Netherlands:Springer; 2009.

[13] Shen X, Van Wijk R, eds. Biophotonics: optical science and engineering for the 21st century. New York: Springer; 2005.

[14] Padgett M, Molloy J, McGloin D, eds. Optical Tweezers: Methods and Applications, vol. 20105644. Boca Raton: Chapman and Hall/CRC; 2010.

[15] Jones PH, Marago OM, Volpe G. Optical tweezers: principles and applications. Cambridge: Cambridge University Press; 2015.

[16] Sitti M, Hashimoto H. Tele-nanorobotics using an atomic force microscope as a nanorobot and sensor. Adv Robot. 1998;13:417–436.

[17] Sitti M, Hashimoto H. Tele-nanorobotics using atomic force microscope. In: Proceedings of 1998 IEEE/RSJ international conference on intelligent robots and systems. Innovations in theory, practice and applications (Cat. No. 98CH36190). IEEE 1998; 3. pp. 1739–46.

[18] Xie H, Onal C, Régnier S, Sitti M. Atomic force microscopy based nanorobotics. In: Springer tracts in advanced robotics, vol. 71. Berlin, Heidelberg: Springer; 2012.

[19] Botvinick EL, Berns MW. Internet-based robotic laser scissors and tweezers microscopy. Microsc Res Tech. 2005;68:65–74.

[20] Yan R, Gargas D, Yang P. Nanowire photonics. Nat. Photonics. 2009;3:569–576.

Part 1

Elements of light robotics

Chapter 1: Human gesture recognition for optical manipulation and its future nanobiophotonics applications

Chapter 2: Laser-based assembler and microfluidic applications

Chapter 3: Optomechanical microtools and shape-induced forces

Chapter 4: Optically driven rotating micromachines

Chapter 5: Electromagnetic wave enabled micro/nanorobotic devices and their applications

Chapter 1

Human gesture recognition for optical manipulation and its future nanobiophotonics applications

Daryl Preece

Linda Shi    University of California San Diego, La Jolla, CA, United States

Abstract

The optical trapping community has been some of the earliest adopters of new computer interface technologies. In this guide to interfaces for light-driven nanorobotics, we delve into the science of making such interfaces work. We start by reviewing the basic principles of optical trapping before explaining the basics of creating robust and practical user interfaces for light-driven systems. Finally, we review the various different approaches for creating outstanding and interface technology for nanorobotics.

Keywords

haptics

kinect

multitouch

optical tweezers

telepresence

VR

trapping

gesture recognition

automation

micromanipulation

natural user interface

Chapter Outline

1 Optical Tweezers Basics

1.1 Optical Tweezers

1.2 Optical Gradient Force

1.3 Practical Setup

1.4 Forces

2 Measurement of Position and Force

2.1 Drag Force Method

2.2 Equipartition

2.3 Langevin Method

2.4 Light Deflection Method

3 System Design and Instrumentation of Optical Manipulation Systems

3.1 System Design

3.2 System Implementation

4 Human Interfaces

4.1 Software Control of Optical Manipulation Systems

5 Control with Peripheral Devices

6 3D Control

6.1 Gathering Spatial Information

6.2 Supplying 3D Information

7 Haptics

8 Internet Control—Controlling Systems Remotely

9 Future Directions

References 

1. Optical tweezers basics

1.1. Optical tweezers

With the invention of the laser, it became practical to focus high intensities of light down to a spot size of a few hundred nanometers. At this scale, optical forces become significant enough to move small particles. Arthur Ashkin did the first work in this area while working at Bell Labs in 1970. Ashkin used a 514 nm (CW) argon ion laser to push transparent latex spheres suspended in water and a counter-propagating geometry to produce a stable optical trap [1]. In 1971, Ashkin published a further paper showing that particles could also be levitated by what he called an optical fountain [2]. However, Ashkin noticed that particles were also pulled into the center of the beam. The force behind this phenomenon, the optical gradient force, seemed to pull small dielectric particles from 10 μm down to 25 nm into the brightest part of the beam. In 1986, Ashkin exploited this force to create the single-beam gradient trap [3]. The trap gave researchers control over particles that were previously too small to manipulate by other means allowing them to exert piconewton forces over nanometer length scales.

1.2. Optical gradient force

To understand the single-beam gradient trap in an intuitive way, one must imagine a particle in close proximity to a highly focused beam of light (Fig. 1.1). We consider that the refractive index of the particle is higher than the refractive index of the surrounding medium. When light hits the particle, some proportion of the light will pass through the spherical surface of the particle causing it to act as a lens. If the particle is displaced sideways from the center of the beam, transmitted light will be directed in the same sense as the displacement. The resultant force on the particle due to this deflection is backward toward the center of the beam (Fig. 1.1B and C). In other words in the direction of the positive gradient of intensity and it is thus known as the gradient force. The rest of the light, which is reflected or has momentum contributions, which are not toward the field gradient, will contribute to a scattering force. The scattering force is proportional to the optical intensity and points in the direction of the incident light [4]. Since the gradient force is greater than scattering force, the net result is a force pulling the particle along the gradient of intensity and toward the highest intensity.

Figure 1.1   Diagram Showing a Particle at Different Places in Focused Gaussian Laser Beam

Light exits the particle with a different momentum than it enters with causing the particle to move. (A), (B), (C), and (D) show a particle at different points in the beam with ray tracing of the refracted light. As rays are redirected by the particle, the momentum components of the rays change, the resultant force on the particle is shown by the arrows at the center of each particle..

If the trapping beam is highly focused, then the particle will also experience a force along the axis of the beam propagation. Since the convergence or divergence of the exiting rays change with the particle’s distance from the focal point so do the axial momentum components (Fig. 1.1A and D). This means that for a particle below the focus of the beam, the forces resolve to push the particle back toward the focus of the beam and vice versa for particles above the beam focus. The particle can thus be trapped in three dimensions.

1.3. Practical setup

It is fairly easy to construct a simple single beam gradient force trap. In order to create a highly focused beam, a microscope objective is usually used. This can be used both for trapping and viewing of the sample. The objective lens should be of high numerical aperture (NA) to ensure that the gradient force outweighs both gravity and the scattering force. It should be noted that many tweezers systems tend to be inverted in order that the scattering force acts against gravity allowing lower laser powers to be used. For increased NA, water or oil immersion objectives are often used [5]. The practical NA of the beam is usually fully exploited by overfilling the back aperture of the objective lens, though this must be weighed against power considerations [6] (Fig. 1.2).

Figure 1.2   A Basic Optical Tweezers System With a 1064 nm Laser Source, Beam Steering Mirror, 4f Imaging System, Dichroic Mirror, CCD Camera, High NA Objective Lens, Illumination Light, and a Condenser Lens

The sample may be held on a translation stage, which may be moved to enable viewing of different parts of the sample and can be used to translate the optical trap to areas of interest. Standard Köhler illumination may be used to illuminate the sample. Though other modalities of optical trap exist, such as fiber traps [7], the single-beam gradient trap represents the most common type of optical trap. Often the single beam trap is extended to counter propagating optical traps, which can provide a greater optical force at a lower NA [8,9].

1.4. Forces

A number of different ways have been developed to parameterize optical forces. Ashkin developed the simple metric of the force efficiency Q to do this [4]. The time-averaged force acting upon a particle can be expressed as:

where n is the refractive index of the surrounding medium, P is the beam power at the focus, c is the speed of light in free space, and Q is the dimensionless force efficiency. The force efficiency can also be viewed as the fractional momentum change per photon.

For smaller particles, Rayleigh or dipole scattering models may provide a better way of predicting optical forces. Though designed with a point dipole in mind, these models can produce good results for particles less than a wavelength of light in size [10]. For bigger particles, Mie scattering theory can be used to predict optical forces [11]. For particles of sufficient size, ray tracing may be applicable [12–14]. When dealing with more complex problems, such as nonspherical or complex particles, a T-matrix method may be used [15,16]. For complicated shapes, the Maxwell stress tensor method may also be useful [17].

2. Measurement of position and force

Precise control of light-driven systems cannot be accomplished without an accurate understanding of forces present in the system. This is particularly challenging when dealing with objects that may move only a matter of nanometers. In optical tweezers, displacements may be sensed to nanometer accuracy and forces have been calibrated down to 25 fN [18], making optical traps extremely sensitive sensors.

Empirically, it has been shown [19] that an optically trapped particle behaves like a damped mass in a parabolic energy well where displacements from the trap center are small. This behavior is analogous to a mass attached to a Hookean spring. However, Simmons and coworkers found (for 1 μ m beads) that for displacements that are greater than half a bead radii, the restoring force rapidly becomes nonlinear reaching a peak at roughly the bead radius. This is consistent with calculations done by Ashkin [4] but this distance may expand as the bead diameter shrinks into the Rayleigh regime (a<<λ).

The force constant κ is proportional to the trap stiffness which depends on a number of factors, such as the trapping geometry; particle size and refractive index; refractive index of the media; and the overall laser power. If the force constant can be calibrated, it is possible to measure the absolute optical force exerted on an object.

A variety of techniques have been developed for ascertaining the forces exerted by optical traps. We will now discuss several commonly used methodologies, which have been widely employed. In choosing an appropriate method, it is important to consider external noise sources, such as vibrations, measurement time scales, and equipment constraints, such as stage frequency response or data acquisition bandwidth. Generally, all methods can be used in most trapping systems but some may require specific equipment in order to get precise measurements. The reader is encouraged to explore the references for more information.

2.1. Drag force method

In most cases, the optical force is not the only force acting on the particle. Thermal forces bombard the particle causing it to move by Brownian motion. This motion is impeded by the viscous drag of the particle. For a particle in an over damped media where there is no turbulence (i.e., low Reynolds number, Re < 1), the drag force can be represented by the Stokes drag force that linearly scales with velocity.

Where a is the radius of the particle, η is the dynamic viscosity of the media, and v is the velocity of the particle through the surrounding medium.

In order to calibrate the force with respect to movement of the particle or object, the trap stiffness of optical traps must be determined. One method is to move the entire microscope stage with the particle on it as was done by Simmons and coworkers [19]. Such a movement creates a Stokes drag force which acts in opposition to the gradient force.

Thus the trap stiffness may be obtained by measurement of the stage velocity and particle displacement as long as the particle radius and fluid viscosity are known. Such an approach requires less actuator bandwidth than many other methods since measurements can be taken relatively slowly.

2.2. Equipartition

where kB is the Boltzmann constant, T where σ²is the variance in the position of the particle. Setting the two equations as equal and rearranging we get:

This allows the trap stiffness to be determined without any need for knowledge of the viscosity. This equation can then be applied in each dimension in which particle position can be measured. It should be noted that in order for this technique to be valid, a critical threshold of measurements is needed such that the variance and mean squared displacement of the particle converge to a constant ratio. Since this technique relies on accurate measurement of the variance, care must also be taken to avoid low-frequency noise due to laser instability, mechanical drift or thermal fluctuations which would give the appearance of a lower apparent trap stiffness [20].

2.3. Langevin method

A micron-sized particle held in an optical trap in an over damped media, such as water is often subject to thermal excitation through continuous bombardment by fast-moving molecules. If the particle is weakly held in the optical trap the particle’s movement traces out a constrained Brownian walk in a potential well. The Brownian motion is easily noticeable and can be detected by cameras and photodiodes. The thermal force takes the form:

Here ξ(t) is a random Gaussian process, T is the absolute temperature, and kB , and the inertial force to give the following Langevin equation:

can be dropped since it is negligible in a low Reynolds number media. Equations of this form have standard solutions. It can be shown that the time-dependent autocorrelation function of such an equation is given by:

This indicates the correlation between the position of a particle at an initial time and subsequent time t. By taking the Fourier transform of the autocorrelation function, one can derive an expression for a power spectrum of the particle movement. The power spectrum takes the form of a Lorentzian curve.

Fitting such a function allows the determination of trap stiffness. The corner frequency fc of such a curve is given by:

It is also possible to determine the trap stiffness using the equipartition theorem by integrating the area under the Lorentzian fit [21] (Fig. 1.3).

Figure 1.3   Power Spectral Density Plot of a 5 μm Silica Bead in Water Held by an Optical Trap

The smooth line is a fitted Lorentzian curve that can be used to accurately find the value of the trap stiffness

2.4. Light deflection method

Though most methods for calibration of the trap stiffness are indirect, it is possible to measure the force directly. As has been mentioned earlier, the force exerted on the particle is proportional to the deflection of the incident beam. Thus if the light deflected by the particle can be measured by a photodiode or position sensitive detector the momentum transfer to the trapped object can be directly calculated, as noted in [22–24]. This is advantageous since prior knowledge of the particle or the medium is not required. However, a well-engineered light collection system is required since such a technique requires full knowledge of all the light scattered by the particle.

3. System design and instrumentation of optical manipulation systems

Automation is ubiquitous in modern complex systems, such as vehicles, robots, and biomedical systems, and optical manipulation systems are no exception. Researchers and engineers often use control theory during the design of a system. In this section, we will try to cover some basic concepts in the design and implementation of optical tweezers systems. These concepts are often overlooked in literature however are crucial to the realization of high quality optical trapping systems particularly those where robotic operations are required.

3.1. System design

The design of modern complex systems is a process of selecting suitable devices and interfacing these devices in order to achieve desired performance. Most systems are built on vibration-isolated tables. With computers and wires either suspended above or below the table. The current trend is to use a dedicated, microprocessor-based control system for each device [25]. There are many commercial devices available that can be used directly in an optical manipulation system. Selection of the appropriate hardware components requires a basic knowledge of the parameters used to evaluate the equipment and how they affect the performance of a system [25,26].

For most optical manipulation systems, the main components are:

1. Microscope: Modern microscopes have become an indispensable part of modern research. Products from the major microscope manufactures, such as Nikon, Leica, Olympus, and Zeiss provide durable mechanical components, interchangeable modules, and computer controlled features. It is important to carefully compare the magnification, resolution, working distance, illumination, and imaging sources when choosing the microscope brand. Ultimately the limit on the resolution of any optical system is set by the diffraction of light; an optical system that performs to this level is termed diffraction limited. In this case, the spatial resolution is given by the Rayleigh criterion:

Where λ is the wavelength of light being imaged by the microscope, d is the smallest resolvable distance between two points in the sample, and NA is the numerical aperture of the microscope objective. Similarly when a Gaussian laser beam with wavelength λ is used to trap a particle, 2d is the diameter of a focused laser spot at the focus.

In the case of commercial microscopes, a beam for optical trapping can often be inserted through a viewing or illumination port. This may, however, present difficulties in alignment or imaging. An alternative is to insert a thick dichroic mirror directly below the objective holder. This can then be coupled to an external optical train. In the past, this was done with custom made parts but now several microscope companies provide integration for such beams.

2. Computer: A powerful computer is the central part of many systems. Used to receive, save, and process signals from the detectors, cameras, and sensors that generate commands to both activate the actuators, and to communicate with controllers. Processing speed and storage capacity are key factors when choosing a computer. Another important factor to consider is the interfacing of hardware devices. For instance, many older hardware companies use PCI and PCIe bus lines to communicate with the computer but nowadays computer manufacturers only provide one PCIe slot and no PCI slots. Though laptops are replacing desktops in many work environments It is still comparatively difficult to interface many control systems to laptops. However, recently there has been a growth in USB control. So future systems may be increasingly portable.

3. Camera: Quality imaging is a critical component in the optical manipulation systems as it is used to detect particle motion and, by extension, force [27,28]. There are many digital cameras exclusively designed for microscopes, such as Hamamatsu, Photometrics, Nikon, Andor, Sony, Cohu. There are two factors to consider when selecting a digital microscope camera: sensitivity and resolution. The sensor is classified as either a CCD or a CMOS, both of which convert photons into electrical signals. The larger the sensor or the bigger the pixels are, the more sensitive it will be. The resolution is the total number of pixels that actually form the image. Make sure the camera resolution exceeds the resolution of the microscope’s optics by at least a factor of 2 (Nyquist limit). For force calibration or particle tracking applications, high-speed cameras may be necessary especially when multiple objects or robotic actuators must be tracked at once (see sensors). Several companies make high-speed cameras, such as AVT, Mikrotron, and Photron. Since the exposure time decreases as the frame rate increases, sufficiently strong illumination is also a prerequisite to prevent darkening of the image.

4. Actuators: The actuator is the device that mechanically drives a dynamic system. Actuating devices include stepper motors, DC motors, AC motors, solenoids, voice coil actuators, and relays [25]. Proper selection of actuators and their drive systems during the design process is very important. The parameters to select the best actuators are accuracy, sensitivity, noise, stability, bandwidth, speed, oscillation. Commonly used actuators in an optical manipulation system include microscope stages [25,29,30], sensor-based piezo actuators [25,31–34], rotary stages [30], piezo-mirrors [34], fast steering mirrors [30], SLMs [35–37], AOMs [19], and galvanometer scanning mirrors [38]. The most relevant specifications used in selecting actuators and their controllers are travel range, resolution, repeatability, stability, speed, computer interface, and compatibility between commercial software.

5. Sensors: A sensor is a device used to measure a given physical quantity [29]. Choosing the appropriate sensor depends on its sensitivity, resolution, bandwidth, measuring range, and dynamic responses. In optical manipulation systems sensors are commonly used to measure position and force. The force can be measured by a calibrated fast responding sensor. It is then possible to modulate the force by feedback with actuators [35].

The most commonly used sensors for force measurement are quadrant photodiode detectors (QPDs) and high-speed cameras:

a. QPDs: These detectors consist of a photodiode divided into four identical sectors. A photodiode makes use of the photovoltaic effect, in which a photon falling onto the sensing surface induces the release of an electron from a valence band. Photodiodes divided into four sectors can reveal the relative motion of a light beam in vertical and horizontal directions. By measuring the voltage differences between quadrants, the user can tell how the beam is moving as shown in Franck 2010 [33].

b. Cameras: High-speed cameras are reliable alternatives for the direct characterization of the optical trapping force and particle motion, replacing indirect motion measurements often performed by quadrant detectors [27]. Such applications require particle positions to be logged in the kiloHertz range and above.