Biography
Hrvoje Benko is a Director of Research Science at Meta Reality Labs Research and an Affiliate Professor at the University of Washington Information School. At Meta, he leads research on novel interaction techniques, devices, and interfaces for augmented reality (AR) and virtual reality (VR). He is responsible for setting the vision and research direction of the HCI research organization at Meta Reality Labs Research as well as driving the research and development on novel wristband input and output solutions for AR and VR experiences. Prior to joining Meta, he was a Principal Researcher at Microsoft Research, where he conducted highly influential research on topics ranging from multi-touch interaction and room-scale virtual environments to haptic devices and perceptual illusions. Benko received a PhD in Computer Science from Columbia University.
Benko and his colleagues won the UIST 2022 Lasting Impact Award for OmniTouch, the first wearable projector & depth-camera system that enabled touch interaction everywhere, and developed the open-source RoomAlive Toolkit, which enables the creation of room-sized dynamic projection-mapping experiences that don’t require headsets or glasses. His team’s work on a research prototype multi-touch mouse was reported on in a paper that received the UIST 2009 best paper award and was later commercialized as the Microsoft Touch Mouse. His team’s novel approach to visualizing contact points on a multi-touch display to allow users to better understand their interactions was incorporated in the Windows OS. Benko has received many best paper awards and honorable mentions at CHI, UIST, ISS, and CSCW, has co-authored more than 100 scientific articles, and is a co-inventor on more than 70 issued patents.
Benko has been an active participant in the SIGCHI community, serving UIST as General Chair (2014), Program Co-Chair (2012), and member of the steering committee; CHI as Associate Chair for many years; and TOCHI as Associate Editor since 2016 and Information Director (2016–2018).
Education
- Ph D, Computer Science, Columbia University, 2007
- MSC, Computer Science, Columbia University, 2003
- BSC, Computer Engineering, Lehigh University, 2001
Awards
- Lasting Impact Award - ACM UIST, 2022
Consulting and Other Projects
- Microsoft (RoomAlive Toolkit), 2016
- Microsoft (Mouse 2.0 project shipped as Microsoft Touch Mouse)
- Microsoft (Ripples project shipped in Microsoft Surface SDK and ultimately incorporating in Windows OS)
Publications and Contributions
-
Conference PaperGaze as an Indicator of Input Recognition Errors (2022)Proceedings of the ACM on Human-Computer Interaction, 6(ETRA)
-
Conference PaperOptimizing the Timing of Intelligent Suggestion in Virtual Reality (2022)Proceedings of ACM User Interface Systems and Technology
-
Conference PaperRIDS: Implicit Detection of a Selection Gesture Using Hand Motion Dynamics During Freehand Pointing in Virtual Reality (2022)Proceedings of ACM User Interface Software and Technology
-
Conference PaperArmstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality (2021)Proceedings of ACM CHI
-
Conference PaperElectroRing: Subtle Pinch and Touch Detection with a Ring (2021)Proceedings of ACM CHI
-
Conference PaperFalse Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error Preference (2021)Proceedings of ACM UIST
-
Conference PaperRotoWrist: Continuous Infrared Wrist Angle Tracking using a Wristband (2021)Proceedings of ACM VRST
-
Conference PaperTowards gaze-based prediction of the intent to interact in virtual reality (2021)Proceedings of ACM Symposium on Eye Tracking Research & Applications
-
Conference PaperUnderstanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality (2021)Proceedings of ACM CHI
-
Conference PaperAcustico: Surface Tap Detection and Localization using Wrist-based Acoustic TDOA Sensing (2020)Proceedings of ACM UIST
-
Conference PaperHead-Coupled Kinematic Template Matching: A Prediction Model for Ray Pointing in VR (2020)Proceedings of ACM
-
Conference PaperInvestigating Remote Tactile Feedback for Mid-Air Text-Entry in Virtual Reality (2020)Proceedings of IEEE ISMAR
-
Conference PaperActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces (2019)Proceedings of ACM UIST 2019
-
Conference PaperBack-of-Device Force Feedback Improves Touch Screen Interaction for Mobile Devices (2019)IEEE Transactions on Haptics
-
Conference PaperLearning Cooperative Personalized Policies from Gaze Data (2019)Proceedings of ACM UIST
-
Conference PaperPerformance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality (2019)Proceedings of IEEE ISMAR
-
Conference PaperPseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display Ratio (2019)Proceedings of ACM CHI
-
Conference PaperTasbi: Multisensory Squeeze and Vibrotactile Wrist Haptics for Augmented and Virtual Reality (2019)Proceedings of World Haptics Conference
-
Conference PaperEnabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane Simulation (2018)Proceedings of ACM CHI
-
Conference PaperCLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality (2018)Proceedings of ACM CHI
-
Conference PaperHaptic Links: Bimanual Haptics for Virtual Reality Using Variable Stiffness Actuation (2018)Proceedings of ACM CHI
-
Conference PaperHaptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller (2018)Proceedings of ACM CHI
-
Conference PaperMRTouch: Adding Touch Input to Head-Mounted Mixed Reality (2018)IEEE Transactions on Visualization and Computer Graphics, Special Issue
-
Conference PaperHybrid HFR Depth: Fusing Commodity Depth and Color Cameras to Achieve High Frame Rate, Low Latency Depth Camera Interactions (2017)Proceedings of ACM CHI
-
Conference PaperMeetAlive: Room-Scale Omni-Directional Display System for Multi-User Content and Control Sharing (2017)Proceedings of ACM Interactive Surfaces and Spaces
-
Conference PaperSparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop (2017)Proceedings of ACM CHI
-
Conference PaperAugmenting the Field-of-View of Head-Mounted Displays with Sparse Peripheral Displays (2016)Proceedings of ACM CHI
-
Conference PaperHaptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences (2016)Proceedings of ACM CHI
-
Conference PaperNormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers (2016)Proceedings of ACM UIST
-
Conference PaperPre-Touch Sensing for Mobile Interaction (2016)Proceedings of ACM CHI
-
Conference PaperRoom2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment (2016)Proceedings of ACM Conference on Computer Supported Cooperative Work
-
Conference PaperSnapToReality: Aligning Augmented Reality to the Real World (2016)Proceedings of ACM CHI
-
Conference PaperFoveAR: Combining an Optically See-Through Near-Eye Display with Spatial Augmented Reality Projections (2015)Proceedings of ACM UIST
-
Conference PaperSensing Tablet Grasp + Micro-mobility for Active Reading (2015)Proceedings of ACM UIST
-
Conference PaperCrossMotion: Fusing Device and Image Motion for User Identification, Tracking and Device Association (2014)Proceedings of ACM International Conference on Multimodal Interaction
-
Conference PaperDyadic Projected Spatial Augmented Reality (2014)Proceedings of ACM UIST
-
Conference PaperRoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-Camera Units (2014)Proceedings of ACM UIST
-
Conference PaperSensing Techniques for Tablet+Stylus Interaction (2014)Proceedings of ACM UIST
-
Conference PaperTouchMover 2.0 - 3D Touchscreen with Haptic Feedback and Haptic Texture (2014)Proceedings of IEEE Haptics Symposium
-
Journal Article, Academic JournalForeword to the special section on touching the 3rd dimension (2013)Computers & Graphics, 37(3)
-
Conference PaperIllumiRoom: Peripheral Projected Illusions for Interactive Experiences (2013)Proceedings of ACM CHI
-
Conference PaperMotion and Context Sensing Techniques for Pen Computing (2013)Proceedings of Graphics Interface
-
Journal Article, Academic JournalThe Design of Organic User Interfaces: Shape, Sketching and Hypercontext (2013)Interacting with Computers, Oxford Journals, 25(2), pp. 133-142
-
Conference PaperTouchMover: Actuated 3D Touchscreen with Haptic Feedback (2013)Proceedings of ACM Interactive Tabletops and Surfaces
-
Conference PaperUnderstanding Touch Selection Accuracy on Flat and Hemispherical Deformable Surfaces (2013)Proceedings of Graphics Interface
-
Conference PaperLightGuide: Projected Visualizations for Hand Movement Guidance (2012)Proceedings of ACM CHI, pp. 179-188
-
Conference PaperMirageTable: Freehand Interaction on a Projected Augmented Reality Tabletop (2012)Proceedings of ACM CHI, pp. 199-208
-
Conference PaperSteerable Augmented Reality with the Beamatron (2012)Proceedings of ACM UIST, pp. 413-422
-
Technical ReportUnderstanding Mid-Air Hand Gestures: A Study of Human Preferences in Usage of Gesture Types for HCI (2012)Microsoft Research Technical Report MSR-TR-2012- 111
-
Conference Extended AbstractConstructing Virtual 3D Models with Physical Building Blocks (2011)CHI Extended Abstracts
-
Conference PaperEnhancing Naturalness of Pen-and-Tablet Drawing through Context Sensing (2011)Proceedings of ACM Interactive Tabletops and Surfaces, pp. 83-86
-
Conference PaperGrips and Gestures on a Multi-Touch Pen (2011)Proceedings of ACM CHI, pp. 1323-1332
-
Conference PaperOmniTouch: Wearable Multitouch Interaction Everywhere (2011)Proceedings of ACM UIST, pp. 441-450
-
Conference PaperPocketTouch: Through-Fabric Capacitive Touch Input (2011)Proceedings of ACM UIST, pp. 303-308
-
Conference PaperRock & Rails: Extending Multi-touch Interactions with Shape Gestures to Enable Precise Spatial Manipulations (2011)Proceedings of ACM CHI, pp. 1581-1590
-
Conference Extended AbstractSketchSpace: Designing Interactive Behaviors with Passive Materials (2011)CHI Extended Abstracts
-
Conference PaperCombining Multiple Depth Cameras and Projectors for Interactions On, Above, and Between Surfaces (2010)Proceedings of ACM UIST, pp. 273-282
-
Conference PaperDesign and Evaluation of Interaction Models for Multi-touch Mice (2010)Proceedings of Graphics Interface, pp. 253-260
-
Book, Chapter in Scholarly Book-NewImprecision, Inaccuracy, and Frustration: The Tale of Touch Input (2010)Tabletops - Horizontal Interactive Displays, ISBN/ISSN: 978-1-84996-112-7
-
Conference PaperManual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input (2010)ACM CHI Extended Abstracts
-
Conference PaperMulti-Point Interactions with Immersive Omnidirectional Visualizations in a Dome (2010)Proceedings of ACM Interactive Tabletops and Surfaces, pp. 19-28
-
Conference PaperPen + Touch = New Tools (2010)Proceedings of ACM UIST, pp. 27-36
-
Conference Extended AbstractPinch-the-Sky Dome: Freehand Multi-Point Interactions with Immersive Omni-Directional Data (2010)CHI Extended Abstracts
-
Conference PaperThree's Company: Understanding Communication Channels in Three-way Distributed Collaboration (2010)Proceedings of ACM CSCW, pp. 271-280
-
Book, Scholarly-NewVirtual Environments: Interactive 3D graphics and applications (Croatian translated title) (2010)ISBN/ISSN: 978-953-197- 606-0
-
Conference PaperAugmenting Interactive Tables with Mice & Keyboards (2009)Proceedings of ACM User Interface Software and Technology, pp. 149-152
-
Conference PaperBeyond Flat Surface Computing: Challenges of Depth-Aware and Curved Interfaces (2009)Proceedings of ACM MultiMedia
-
Technical ReportDepthTouch: Using Depth-Sensing Camera to Enable Freehand Interactions On and Above the Interactive Surface (2009)Microsoft Research Technical Report MSR-TR-2009-23
-
Conference PaperEnhancing Input On and Above the Interactive Surface with Muscle Sensing (2009)Proceedings of ACM Interactive Tabletops and Surfaces, pp. 93-100
-
Technical ReportInsights on Interactive Tabletops: A Survey of Researchers and Developers (2009)Microsoft Research Technical Report MSR-TR-2009-22
-
Conference PaperMouse 2.0: Multi-touch Meets the Mouse (2009)In Proceedings of ACM User Interface Software and Technology, pp. 33-42
-
Conference PaperRipples: Utilizing Per-Contact Visualizations to Improve User Interaction with Touch Displays (2009)Proceedings of ACM User Interface Software and Technology, pp. 3-12
-
Conference PaperSeparability of Spatial Manipulations in Multi-touch Interfaces (2009)Proceedings of Graphics Interface, pp. 175-182
-
Conference PaperShadowGuides: Visualizations for In-Situ Learning of Multi-Touch and Whole-Hand Gestures (2009)Proceedings of ACM Interactive Tabletops and Surfaces, pp. 165-172
-
Conference PaperSphere: Multi-Touch Interactions on a Spherical Display (2008)Proceedings of ACM User Interface Software and Technology, pp. 77-86
-
Conference PaperBalloon Selection: A Multi-Finger Technique for Accurate Low-Fatigue 3D Selections (2007)Proceedings of IEEE Symposium on 3D User Interfaces, pp. 79-86
-
Conference PaperPointer Warping in Heterogeneous Multi-Monitor Environments (2007)Proceedings of Graphics Interface '07, pp. 111-117
-
Conference PaperPrecise Selection Techniques for Multi- Touch Screens (2006)Proceedings of ACM CHI 2006, pp. 1263-1272
-
Conference PaperCross-Dimensional Gestural Interaction Techniques for Hybrid Immersive Environments (2005)Proceedings of IEEE Virtual Reality, pp. 209-116
-
Conference Extended AbstractMulti-Monitor Mouse (2005)ACM CHI Extended Abstracts, pp. 1208-1211
-
Conference PaperCollaborative Mixed Reality Visualization of an Archaeological Excavation (2004)Proceedings of The International Symposium on Mixed and Augmented Reality, pp. 132-140
-
Journal Article, Academic JournalConstructionist Design Methodology for Interactive Intelligences (2004)AI Magazine, 25(4), pp. 77-90
-
Technical ReportCross-Dimensional Gestural Interaction Techniques for Hybrid Immersive Environments (2004)Technical Report CUCS-029-04
-
Conference PosterDigitally Modeling, Visualizing and Preserving Archaeological Sites (2004)Proc. Joint Conference on Digital Libraries, pp. 389
-
Conference PaperSeeing into the Past: Creating a 3D Modeling Pipeline for Archaeological Visualization (2004)Proceedings of International Symposium on 3D Data Processing Visualization and Transmission, pp. 751-758
-
Conference PaperMutual Disambiguation of 3D Multimodal Interaction in Augmented and Virtual Reality (2003)Proceedings of The Fifth International Conference on Multimodal Interfaces, pp. 12-19
-
Conference PaperSenseShapes: Using Statistical Geometry for Object Selection in a Multimodal Augmented Reality System (2003)Proceedings of The Second International Symposium on Mixed and Augmented Reality, pp. 300-301
-
Journal Article, Academic Journale-Ireland: Europe's New Internet Hub (2001)Perspectives on Business and Economics, 19(Unknown Issue), pp. 39-48
-
Conference PaperPictionaire: Supporting Collaborative Design Work by Integrating Physical and Digital ArtifactsProceedings of ACM CSCW, pp. 421-424
Presentations
-
Fabricate It or Render It? Digital Fabrication vs. Virtual Reality for Creating Objects Instantly
(2022)
ACM CHI - New Orleans, LA
-
The Future of Mixed Reality Interactions
(2022)
HCI Lecture Series, Massachusetts Institute of Technology - Cambridge, MA
-
The Future of Mixed Reality Interactions
(2021)
International XR Workshop - Auckland, New Zealand
-
The Future of Mixed Reality Interactions
(2020)
ACM Interactive Surfaces and Spaces (ISS) - Lisbon, Portugal
-
The Future of Mixed Reality Interactions
(2020)
HCI Lecture Series, Stanford University - Stanford, CA
-
The Future of Mixed Reality Interactions
(2019)
IEEE AIVR - San Diego, California
-
The ABC of MR Interactions
(2018)
IEEE International Symposium on Mixed and Augmented Reality - Munich, Germany
-
VR Haptic Controllers: Multi-Purpose Handheld Haptics Beyond Vibro-Tactors
(2018)
IEEE Haptics Symposium 2018 - San Francisco, CA
-
Perception Equals Reality
(2017)
University of British Columbia - Vancouver, B.C., Canada
-
Perception Equals Reality
(2017)
ACM Spatial User Interfaces - Brighton, U.K.
-
Perception Equals Reality
(2017)
Virtual Reality World Congress - Bristol, U.K.
-
Perception Equals Reality
(2017)
Simon Fraser University - Burnaby, B.C., Canada
-
Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences
(2016)
ACM ISS - Niagara Falls, Ontario, Canada
-
Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences
(2016)
ACM CHI - San Jose, CA
-
Projected Augmented Reality with the RoomAlive Toolkit
(2016)
ACM ISS - Niagara Falls, Canada
-
Interacting with Photons: Creating Interactive Projected Augmented Reality Experiences
(2015)
TUX Sander's Distinguished Speaker Series - Toronto, ON
-
Interacting with Photons: Creating Interactive Projected Augmented Reality Experiences
(2015)
New York University - New York, NY
-
Interacting with Photons: Creating Interactive Projected Augmented Reality Experiences
(2015)
Columbia University - New York, NY
-
Beyond Flat Displays
(2013)
ACM CHI '13 Workshop on Displays Take New Shape - Paris, France
-
Beyond Flat Displays: Interactivity on Any Surface
(2013)
Stanford University, HCI Seminar - Stanford, CA
-
Beyond Flat Displays: Interactivity on Any Surface
(2013)
University of California Berkeley, SWARM Lab Seminar - Berkeley, CA
-
Displays take new shape: an agenda for future interactive surfaces
(2013)
CHI - Paris, France
-
IllumiRoom: Peripheral Projected Illusions for Interactive Experiences
(2013)
SIGGRAPH 2013 Emerging Technologies - Anaheim, CA
-
Istraživanja iz Microsoft Research-a na području proširene stvarnosti
(2013)
WinDays Conference - Umag, Croatia
-
Challenges of Multitouch Interaction on Deformable Surfaces
(2012)
Beyond Flat Displays Workshop, ACM ITS - Cambridge, Massachusetts
-
Rich Augmented Reality Interactions without Goggles, Gloves or 3D Trackers
(2012)
DUB Research Talk - Seattle, WA
-
The 3rd dimension of CHI (3DCHI): touching and designing 3D user interfaces
(2012)
CHI - Austin, Texas
-
High-fidelity Augmented Reality Interactions
(2011)
Microsoft Research Latin American Faculty Summit - Cartagena, Colombia
-
Multi-touch Interactions on Small Input Devices
(2011)
DUB Research Talk - Seattle, WA
-
Touching the 3rd dimension (T3D workshop)
(2011)
CHI - Vancouver, B.C.
-
Alternative Form Factors of Surface Computing
(2009)
Microsoft Research Faculty Summit - Redmond, WA
-
Design Challenges of Interactive Spherical User Interfaces
(2009)
Programming Reality Workshop at CHI'09 - Boston, MA
-
Riffing on Surface
(2009)
Department of Computer Science, University of California Santa Barbara - Santa Barbara, CA
-
DepthTouch: Using Depth-Sensing Camera to Enable Freehand Interactions On and Above the Interactive Surface
(2008)
IEEE Workshop on Tabletops and Interactive Surfaces - Amsterdam, the Netherlands
-
Non-Flat Surface Computing
(2008)
Microsoft Research Cambridge Lab - Cambridge, UK
-
Collaborative Mixed Reality Visualization of an Archaeological Excavation
(2005)
MIT Media Lab - Cambridge, MA
-
Development and Evaluation of Mixed Reality Interaction Techniques
(2005)
Workshop on 3D User Interfaces - Bonn, Germany
-
Creating Telepresence: 3D Modeling and Visualization of an Archaeological Excavation
(2004)
Ph.D. Seminar, Computer Science, Columbia University - New York, NY
-
VITA: Visual Interaction Tool for Archaeology
(2004)
The International Symposium on Mixed and Augmented Reality - Washington, D.C.
-
VITA: Visual Interaction Tool for Archaeology
(2004)
ACM SIGMM Effective - New York, N.Y.
-
Collaborative Visualization of an Archaeological Excavation
(2003)
Workshop on Collaborative Virtual Reality and Visualization - Lake Tahoe, CA
-
Mobile Augmented Reality Systems
(2002)
Living with the Genie: Governing Scientific and Technological Transformation in the 21st Century - New York, NY