Computer Science Department, University of Chicago
Our research is focused on the question: what if interfaces would share part of our body? Our group has materialized these ideas by creating interactive systems that intentionally borrow parts of the user’s body for input and output; allowing computers to be more directly interwoven in our bodily senses and actuators. We believe this might assist us in answering: what new interface paradigm comes after wearable devices?
One specific flavor of such devices that we have extensively explored are devices that borrow the user’s muscles by means of electrical muscle stimulation. These devices use part of the wearer’s body for output, i.e., the computer can output by actuating the user’s muscles with electrical impulses, causing it to move involuntarily. The wearer can sense the computer’s activity on their own body by means of their sense of proprioception. Our wearable systems have shown to: increase realism in VR, provide a novel way to access information through proprioception, and serve as a platform to experience and question the boundaries of our sense of agency, which includes investigations into neuroscience.
Tactile input generated by one’s own agency is generally attenuated. Conversely, externally caused tactile input is enhanced; e.g., during haptic exploration. We used functional magnetic resonance imaging (fMRI) to understand how the brain accomplishes this weighting. Our results suggest an agency-dependent somatosensory processing in the parietal operculum.
Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation Without Compromising Agency
Shunichi Kasahara, Jun Nishida and Pedro Lopes. In Proc. CHI’19, Paper 643 (full paper) and demonstration at SIGGRAPH'19 eTech.
We found out that it is possible to optimize the timing of haptic systems to accelerate human reaction time without fully compromising the user' sense of agency. This work was done in cooperation with Shunichi Kasahara from Sony CSL. Read more.
Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials
Lukas Gehrke, Sezen Akman, Pedro Lopes, Albert Chen, ..., Klaus, Gramann. In Proc. CHI’19, Paper 427. (full paper)
We detect visuo-haptic mismatches in VR by analyzing the user's event-related potentials (ERP). In our EEG study, participants touched VR objects and received either no haptics, vibration, or vibration and EMS. We found that the negativity component (prediction error) was more pronounced in unrealistic VR situations, indicating visuo-haptic mismatches.
Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation
Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. In Proc. CHI’18. (full paper)
Summary: We present a mobile system that enhances mixed reality experiences, displayed on a Microsoft HoloLens, with force feedback by means of electrical muscle stimulation (EMS). The benefit of our approach is that it adds physical forces while keeping the users’ hands free to interact unencumbered—not only with virtual objects, but also with physical objects, such as props and appliances that are an integral part of both virtual and real worlds.
Providing Haptics to Walls and Other Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation
Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. In Proc. CHI’17 (full paper) and demonstration at SIGGRAPH'17 studios
We explored how to add haptics to walls and other heavy objects in virtual reality. Our contribution is that we prevent the user’s hands from penetrating virtual objects by means of electrical muscle stimulation (EMS). As the shown user lifts a virtual cube, our system lets the user feel the weight and resistance of the cube. The heavier the cube and the harder the user presses the cube, the stronger a counterforce the system generates.
Muscle-plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output
Pedro Lopes, Doga Yueksel, François Guimbretière, and Patrick Baudisch. In Proc. UIST’16 (full paper).
We explore how to create more expressive EMS-based systems. Muscle-plotter achieves this by persisting EMS output, allowing the system to build up a larger whole. More specifically, it spreads out the 1D signal produced by EMS over a 2D surface by steering the user’s wrist. Rather than repeatedly updating a single value, this renders many values into curves.
Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation
Pedro Lopes, Alexandra Ion, and Patrick Baudisch. In Proc. UIST’15 (full paper). UIST best demo nomination
We present impacto, a device designed to render the haptic sensation of hitting and being hit in virtual reality. The key idea that allows the small and light impacto device to simulate a strong hit is that it decomposes the stimulus: it renders the tactile aspect of being hit by tapping the skin using a solenoid; it adds impulse to the hit by thrusting the user’s arm backwards using electrical muscle stimulation. The device is self-contained, wireless, and small enough for wearable use.
Affordance++: Allowing Objects to Communicate Dynamic Use
Pedro Lopes, Patrik Jonell, and Patrick Baudisch. In Proc. CHI’15 (full paper). CHI best paper award (top 1%)
We propose extending the affordance of objects by allowing them to communicate dynamic use, such as (1) motion (e.g., spray can shakes when touched), (2) multi-step processes (e.g., spray can sprays only after shaking), and (3) behaviors that change over time (e.g., empty spray can does not allow spraying anymore). Rather than enhancing objects directly, however, we implement this concept by enhancing the user with electrical muscle stimulation. We call this affordance++.
Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. In Proc. CHI’15 (full paper). CHI best talk award
We propose a new way of eyes-free interaction for wearables. It is based on the user’s proprioceptive sense, i.e., users feel the pose of their own body. We have implemented a wearable device, Pose-IO, that offers input and output based on proprioception. Users communicate with Pose-IO through the pose of their wrists. Users enter information by performing an input gesture by flexing their wrist, which the device senses using an accelerometer. Users receive output from Pose-IO by finding their wrist posed in an output gesture, which Pose-IO actuates using electrical muscle stimulation.
Muscle-propelled force feedback: bringing force feedback to mobile devices
Pedro Lopes and Patrick Baudisch. In Proc. CHI’13 (short paper). IEEE World Haptics, People’s Choice Nomination for Best Demo
Force feedback devices resist miniaturization, because they require physical motors and mechanics. We propose mobile force feedback by eliminating motors and instead actuating the user’s muscles using electrical stimulation. Without the motors, we obtain substantially smaller and more energy-efficient devices. Our prototype fits on the back of a mobile phone. It actuates users’ forearm muscles via four electrodes, which causes users’ muscles to contract involuntarily, so that they tilt the device sideways. As users resist this motion using their other arm, they perceive force feedback.
Synopsis: An introduction to the field of Human Computer Interaction (HCI), with a particular emphasis in understanding and designing user-facing software and hardware systems. This class covers the core concepts of HCI: affordance and mental models, input techniques (cursors, touch, text entry, voice, etc.), output techniques (visual menus and widgets, sound, haptics), conducting user studies, and so forth. It also includes a project in which students design, build and study a user-facing interactive system. (Premiers in Fall'19)
Synopsis: In this class we build I/O devices, typically wearable or haptic devices. These are user-facing hardware devices engineered to enable new ways to interact with computers. In order for you to be successful in building your own I/O device we will: (1) study and program 8 bit microntrollers, (2) explore different analog and digital sensors and actuators, (3) write control loops and filters, (4) explore stretchable and fabric based electronics, (5) learn how to approach invention, and (6) apply I/O devices to novel contexts such as Virtual Reality. See here for class website.
Synopsis: In this class, we examine emergent technologies that might impact the future generations of computing interfaces, these include: physiological I/O (e.g., brain and muscle computer interfaces), tangible computing (giving shape and form to interfaces), wearable computing (I/O devices closer to the user's body), rendering new realities (e.g., virtual and augmented reality) and haptics (giving computers the ability to generate touch and forces). See here for class website.
20 May 2019
Best of CHI'19 selection by Pedro & Jun at our lab meeting!
19 May 2019
Lab outing to Mitsuwa: we might have gone overboard with japanese snacks! Kudos to Jas for organizing.
3 May 2019
Pedro received an Best Video Communication award at CHI 2019 for his work Trussformer which he co-authored with Robert Kovacs (primary investigator, HPI).
Lab just started! This is Jas' first day (and Pedro's too).
We are always looking for exceptional students at the intersection of Human Computer Interaction, Electrical Engineering, Materials Science and Mechanical Engineering.
If you are considering applying for our lab:
1. Send us an email with your portfolio and CV.
2. Your portfolio (which preferably should be a website) must show documentation of the projects you are most proud (video documentation is ideal). We are especially looking for technical projects that involve: circuitry, signal processing, wearables or other interactive devices.
2. Your CV should state also which level of expertise you have with the areas that are crucial for our lab: have you built your own circuits, do you write control loops, do you do more hardware than software, etc.
4. Our lab is most suited for folks with some experience in HCI, EE, materials or ME.
5. Read our research interests carefully, if you are unsure about the fit, send a quick email first before applying.
6. If you are applying for an internship you mush have substantial previous experience in our areas of work.
Our lab is a welcoming environment that does not discriminate. We are a LGBTQ+ ally lab.
Our lab is supported by the following sponsor organizations: