Human Computer Integration Lab

Computer Science Department, University of Chicago

Our research is focused understanding how to integrate computer interfaces with the human body—we believe this is the interface paradigm that supersedes wearable computing. We explore this by engineering interactive systems that intentionally borrow parts of the user’s body for input and output.

We have used our wearable muscle stimulation devices, for example, to: make a user's muscles properly manipulate a tool they never used before, computationally accelerate a user's reaction time so they are able to take a photograph of a high-speed moving target, read and write information without using a screen, and transform someone's arm into a plotter so they can solve computationally complex problems with just a pen and paper.

We think these types of interactive devices are beneficial because they afford new emotional and physical modes of reasoning with computers, which goes beyond just symbolic thinking (reasoning by typing and reading language on a screen). While this physical integration between human and computer is beneficial in many ways, it also requires tackling a series of new philosophical challenges, such as the question of agency: how do I feel when my body is integrated with an interface, do I feel in control? We explore these questions, together with neuroscientists, by measuring and improving how our brain encodes the feeling of agency under this new kind of integrated interfaces.

The Human Computer Integration research lab is led by Prof. Pedro Lopes at the Computer Science Department of the University of Chicago.

Team


Pedro Lopes

Assist. Prof.

Jun Nishida

Postdoc Scholar

Jas Brooks

PhD student

Shan-Yuan Teng

PhD student

Jasmine Lu

PhD student

Alex Mazursky

PhD student

Arresh Amleshi

Research Assistant

Medha Goyal

Undergraduate

Pengyu Li

Intern

Zoe Liu

Predoctoral student

Jingxuan Wen

Predoctoral student

See here for our lovely alumni.

Our lab is a welcoming environment that does not discriminate. We are a LGBTQ+ ally lab. If you are emailing lab members, do not assume pronoums, just ask before.

Publications

Aero-plane: a Handheld Force-Feedback Device that Renders Weight Motion Illusion

Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Xing-Dong Yang, Pedro Lopes, Andrea Bianchi. In Proc. UIST’19 (full paper)

We engineered Aero-plane, a force-feedback handheld controller based on two miniature jet-propellers that can render shifting weights of up to 14 N within 0.3 seconds. Unlike other ungrounded haptic devices, our prototype realistically simulates weight changes over 2D surfaces. This work was a collaboration and was led by Andrea Bianchi, who runs the MAKinteract group at KAIST.

UIST'19 paper video

Action-dependent processing of touch in the human parietal operculum

Jakub Limanowski, Pedro Lopes, Janis Keck, Patrick Baudisch, Karl Friston, and Felix Blankenburg. In Cerebral Cortex (journal), to appear.

Tactile input generated by one’s own agency is generally attenuated. Conversely, externally caused tactile input is enhanced; e.g., during haptic exploration. We used functional magnetic resonance imaging (fMRI) to understand how the brain accomplishes this weighting. Our results suggest an agency-dependent somatosensory processing in the parietal operculum.

Cerebral Cortex paper

Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation Without Compromising Agency

Shunichi Kasahara, Jun Nishida and Pedro Lopes. In Proc. CHI’19, Paper 643 (full paper) and demonstration at SIGGRAPH'19 eTech.

Grand Prize, awarded by Laval Virtual in partnership with SIGGRAPH'19 eTech.

We found out that it is possible to optimize the timing of haptic systems to accelerate human reaction time without fully compromising the user' sense of agency. This work was done in cooperation with Shunichi Kasahara from Sony CSL. Read more.

CHI'19 paper video SIGGRAPH'19 etech (soon) CHI'19 talk (slides) CHI talk video

Detecting Visuo-Haptic Mismatches in Virtual Reality using the Prediction Error Negativity of Event-Related Brain Potentials

Lukas Gehrke, Sezen Akman, Pedro Lopes, Albert Chen, ..., Klaus, Gramann. In Proc. CHI’19, Paper 427. (full paper)

We detect visuo-haptic mismatches in VR by analyzing the user's event-related potentials (ERP). In our EEG study, participants touched VR objects and received either no haptics, vibration, or vibration and EMS. We found that the negativity component (prediction error) was more pronounced in unrealistic VR situations, indicating visuo-haptic mismatches.

CHI'19 paper CHI'19 talk (slides) CHI'19 talk video

Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation

Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. In Proc. CHI’18. (full paper)

Summary: We present a mobile system that enhances mixed reality experiences, displayed on a Microsoft HoloLens, with force feedback by means of electrical muscle stimulation (EMS). The benefit of our approach is that it adds physical forces while keeping the users’ hands free to interact unencumbered—not only with virtual objects, but also with physical objects, such as props and appliances that are an integral part of both virtual and real worlds.

video CHI'18 paper code

Providing Haptics to Walls and Other Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation

Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. In Proc. CHI’17 (full paper) and demonstration at SIGGRAPH'17 studios

We explored how to add haptics to walls and other heavy objects in virtual reality. Our contribution is that we prevent the user’s hands from penetrating virtual objects by means of electrical muscle stimulation (EMS). As the shown user lifts a virtual cube, our system lets the user feel the weight and resistance of the cube. The heavier the cube and the harder the user presses the cube, the stronger a counterforce the system generates.

video CHI'17 paper

Muscle-plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output

Pedro Lopes, Doga Yueksel, François Guimbretière, and Patrick Baudisch. In Proc. UIST’16 (full paper).

We explore how to create more expressive EMS-based systems. Muscle-plotter achieves this by persisting EMS output, allowing the system to build up a larger whole. More specifically, it spreads out the 1D signal produced by EMS over a 2D surface by steering the user’s wrist. Rather than repeatedly updating a single value, this renders many values into curves.

video UIST'16 paper code

Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation

Pedro Lopes, Alexandra Ion, and Patrick Baudisch. In Proc. UIST’15 (full paper). UIST best demo nomination

We present impacto, a device designed to render the haptic sensation of hitting and being hit in virtual reality. The key idea that allows the small and light impacto device to simulate a strong hit is that it decomposes the stimulus: it renders the tactile aspect of being hit by tapping the skin using a solenoid; it adds impulse to the hit by thrusting the user’s arm backwards using electrical muscle stimulation. The device is self-contained, wireless, and small enough for wearable use.

video UIST'15 paper talk video

Affordance++: Allowing Objects to Communicate Dynamic Use

Pedro Lopes, Patrik Jonell, and Patrick Baudisch. In Proc. CHI’15 (full paper). CHI best paper award (top 1%)

We propose extending the affordance of objects by allowing them to communicate dynamic use, such as (1) motion (e.g., spray can shakes when touched), (2) multi-step processes (e.g., spray can sprays only after shaking), and (3) behaviors that change over time (e.g., empty spray can does not allow spraying anymore). Rather than enhancing objects directly, however, we implement this concept by enhancing the user with electrical muscle stimulation. We call this affordance++.

video CHI'15 paper talk video code

Proprioceptive Interaction

Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. In Proc. CHI’15 (full paper). CHI best talk award

We propose a new way of eyes-free interaction for wearables. It is based on the user’s proprioceptive sense, i.e., users feel the pose of their own body. We have implemented a wearable device, Pose-IO, that offers input and output based on proprioception. Users communicate with Pose-IO through the pose of their wrists. Users enter information by performing an input gesture by flexing their wrist, which the device senses using an accelerometer. Users receive output from Pose-IO by finding their wrist posed in an output gesture, which Pose-IO actuates using electrical muscle stimulation.

video CHI'15 paper talk video

Muscle-propelled force feedback: bringing force feedback to mobile devices

Pedro Lopes and Patrick Baudisch. In Proc. CHI’13 (short paper). IEEE World Haptics, People’s Choice Nomination for Best Demo

Force feedback devices resist miniaturization, because they require physical motors and mechanics. We propose mobile force feedback by eliminating motors and instead actuating the user’s muscles using electrical stimulation. Without the motors, we obtain substantially smaller and more energy-efficient devices. Our prototype fits on the back of a mobile phone. It actuates users’ forearm muscles via four electrodes, which causes users’ muscles to contract involuntarily, so that they tilt the device sideways. As users resist this motion using their other arm, they perceive force feedback.

video CHI'13 paper talk video

The publications above are core to our lab's mission. If you are interested more of Pedro's publications in other topics, see here.

Teaching

1. Introduction to Human-Computer Interaction (CMSC 20300; Fall quarter)

Synopsis: An introduction to the field of Human Computer Interaction (HCI), with a particular emphasis in understanding and designing user-facing software and hardware systems. This class covers the core concepts of HCI: affordance and mental models, input techniques (cursors, touch, text entry, voice, etc.), output techniques (visual menus and widgets, sound, haptics), conducting user studies, and so forth. It also includes a project in which students design, build and study a user-facing interactive system. (Premiers in Fall'19)

This class is full, please use from now on.

2. Inventing, Engineering and Understanding Interactive Devices (CMSC 23220; Spring quarter)

Synopsis: In this class we build I/O devices, typically wearable or haptic devices. These are user-facing hardware devices engineered to enable new ways to interact with computers. In order for you to be successful in building your own I/O device we will: (1) study and program 8 bit microntrollers, (2) explore different analog and digital sensors and actuators, (3) write control loops and filters, (4) explore stretchable and fabric based electronics, (5) learn how to approach invention, and (6) apply I/O devices to novel contexts such as Virtual Reality. See here for class website.

3. Emerging Interface Technologies (CMS 33231; winter quarter)

Synopsis: In this class, we examine emergent technologies that might impact the future generations of computing interfaces, these include: physiological I/O (e.g., brain and muscle computer interfaces), tangible computing (giving shape and form to interfaces), wearable computing (I/O devices closer to the user's body), rendering new realities (e.g., virtual and augmented reality) and haptics (giving computers the ability to generate touch and forces). See here for class website.

News

  • 30th Sep 2019

    Joint lab meeting with Sihong Wang's Lab

  • 27th Sep 2019

    Pedro serves as jury for Futures Design Challenge 2019

  • 26th Sep 2019

    Pedro gives a talk for REPLICA (Berlin's art & science project)

  • 25th Sep 2019

    Jaeyeon Lee visited the lab (remotely!)

  • 24th Sep 2019

    Jun and Pedro demo at CNBC!

  • 23rd Sep 2019

    Yang Zhang (CMU) visited the Lab!

  • 23rd Sep 2019

    Jingxuan Wen (predoctoral student) arrived in the lab! Welcome!

  • 20th Sep 2019

    Celebrate CHI submissions!

  • 6th Sep 2019

    Medha's (fake) last day in lab! (fake) sadness! She's still with us, hooray!

  • 3rd Sep 2019

    Zoe Liu (predoctoral student) arrived in the lab! Welcome!

  • 1st Sep 2019

    We had to say goodbye to Romain! New PhD students Jasmine and Alex arrived in Chicago.

  • 27th August 2019

    Aaron last day! We made a custom made board game for Aaron's goodbye (also our first board game night!)

  • 23th August 2019

    Mayhem, laser cutter arrived and lab construction is underway.

  • 20th August 2019

    Hang out and work session with Ka Moamoa (Josiah's group at Northwestern) and Nabil!

  • 19th August 2019

    Japanese TV came all the way to interview Jun!

  • 16th August 2019

    Home made lunch (by Marco and Pedro) for Michael's last day! Happy Michael's day.

  • 9th August 2019

    We are sad to see Lydia and Eva go, but they will rock the CS World!

  • 2nd August 2019

    We got the Grand Prize for Revolution Research at SIGGRAPH 2019 (eTech), for our work on Preemptive Action, which was awarded by Laval virtual in partnership with SIGGRAPH.

  • 1st August 2019

    We are sad to see Elliot Kahn go, but he will be having lots of fun overseas! Thanks for all the help! In the meantime, we welcome Pengyu Li and Zhehao Li to the lab!

  • 31th July 2019

    Jun and Daniel are at SIGGRAPH demo'ing Preemptive Action (together with our collaborator Shunichi Kasahara from Sony CSL)!

  • 26th July 2019

    Gizmodo wrote this article about our work on agency, in particular Preemptive Action (CHI'19).

  • 26th July 2019

    UChicago News wrote a piece about our Cerebral Cortex and CHI papers.

  • 19th July 2019

    IEEE Spectrum wrote a piece about our Preemptive Action paper (CHI'19).

  • 10th July 2019

    Lab outing for (dumpling) dinner with Fred Chong's EPIQ lab!

  • 9th July 2019

    CDAC Internships are all underway! Welcome Medha too!

  • 6th July 2019

    Our Preemptive Action paper (CHI'19) was selected to the Best of CHI'19 event in Mumbai, organized by the Indian HCI community.

  • 1st July 2019

    Medha Goyal (CDAC intern) joins the lab for this summer! Welcome!

  • 19th June 2019

    Pedro receives best PhD dissertation award at University of Potsdam, Germany. Award handed by Oliver Günther (President of the University) and Jann Jakobs (former-mayor of Potsdam).

  • 21st June 2019

    Shan-Yuan Teng's first day in the lab. Followed by ping pong celebration and dinner.

  • 19th June 2019

    Pedro and Jun invited to the Neuro ICU by Dr. Fernando (UChicago), thanks for the visit!

  • 17th June 2019

    Romain Nith (McGill) and Ted Kim (UChicago, CDAC) joined the lab!

  • 7th June 2019

    Simeon and Aaron defend their practicum with style and fantastic live demos.

  • 5th June 2019

    Peggy Mason visits the lab!

  • 5th June 2019

    Sam Volchenboum and Josiah Hester visit the lab!

  • 31 May 2019

    Marco and Pedro invited to Museum of Science and Industy's after hours event (current exhibit is on wearables, go hck it out!).

  • 30 May 2019

    Jas Brooks gives a lab tour to 5th graders from Day Star Academy!

  • 22 May 2019

    Akifumi Takahashi gives a talk a the lab about EMS!

  • 21 May 2019

    Fernando Goldenberg visits the lab!

  • 20 May 2019

    Best of CHI'19 selection by Pedro & Jun at our lab meeting!

  • 19 may 2019

    lab outing to mitsuwa: we might have gone overboard with japanese snacks! kudos to jas for organizing.

  • 16 may 2019

    Angelika Zissimopoulos visits the lab! Also, Marco and Pedro got a tour of Logan's Kiln by Amber Ginsburg!

  • 15 May 2019

    Pedro was a respondent at the lovely meeting Automatic Recognition and Analysis of American Sign Language organized by Diane Brentari (UChicago) and Karen Livescu (TTIC).

  • 3 May 2019

    Pedro received an Best Video Communication award at CHI 2019 for his work Trussformer which he co-authored with Robert Kovacs (primary investigator, HPI).

  • 7 May 2019

    Jun received an Best Paper Award Honorable Mention at CHI 2019 for his work at th University of Tsukuba that allows you to experience the view of a smaller person, with Kenji Suzuki.

  • 6 May 2019

    We just presented two CHI papers (and Jun presents his tomorrow as well!). Lukas presented our EEG/EMS paper and Shun presented our agency/EMS paper!

  • 3 May 2019

    Pedro received the Outstanding Dissertation Award 2019 by the University of Potsdam for his PhD thesis at the Hasso Plattner Institute.

  • 2 May 2019

    Two CHI'19 papers accepted, presentations ready for Glasgow! Also Jun will present this paper that he authored while at the University of Tsukuba.

  • 1 May 2019

    11 amazing VR demos in class, students rocked!

  • 27 Apr 2019

    Pedro teaches signal processing to over 100 middle school girls that take part in CompileHer 2019.

  • 19 Apr 2019

    SIGGRAPH eTech demo accepted, demo ready for LA!

  • 19 Apr 2019

    First sad day in the lab, Steven's visit is over. Lab made a home cooked meal and hand made zine for Steven! See you in Belgium!

  • 15 Apr 2019

    Three new PhD students next fall! Introducing: Jasmine (CS, Duke) and Shan-Yuan (CS, NTU) and Alex Mazursky (ME, Miami, OH).

  • 15 Apr 2019

    Michelle Carr visited the lab.

  • 15 Apr 2019

    Sihong Wang visited the lab.

  • 8 Apr 2019

    Lab outing to Thalia Hall/Dusek with Gierad Laput who visited UChicago.

  • 3 Apr 2019

    We started a new class: CMSC 23220: Inventing, Engineering and Understanding Interactive Devices!

  • 21 Mar 2019

    Lab outing to the Museum of Science and Indusry; opening of the wearables exhibit. We met Ivan Poupyrev, Anthony Banks and Behnaz Farahi.

  • 15 Mar 2019

    Aaron, Jake, Lydia and Simeon joined the lab.

  • 15 Mar 2019

    We organized our own PhD Visit Day for Alex Mazursky!.

  • 14 Mar 2019

    Ted Kim is visiting UChicago.

  • 13 Mar 2019

    Ludwig Wall is visiting our lab and gives a talk on his work on Laser Stacker and Metamaterials.

  • 13 Mar 2019

    Ed Wang is visiting UChicago.

  • 12 Mar 2019

    We are doing a stretchable and soft electronics hands-on workshop at our graduate class! First MADD and CS joint workshop.

  • 9 Mar 2019

    HCI lab hangout. Lunch at the plant and SAIC's MA closing show.

  • 8 Mar 2019

    PhD visit day at UCHicago CS! Jasmine and Shan-Yuan visited the lab. Many thanks to our colleague Ben Zhao for an amazing visit day.

  • 5 Mar 2019

    Garrett Johnson visited the lab.

  • 1 Mar 2019

    We met Kyle Jamieson who was visiting UChicago.

  • 27 Feb 2019

    David Freedman visited the lab.

  • 25 Feb 2019

    Steven and Pedro went to visit Josiah Hester at Northwestern.

  • 22 Feb 2019

    Andrey Rzhetsky visited the lab.

  • 14 Feb 2019

    We met Marshini Chetty who was visiting UChicago.

  • 11 Feb 2019

    Howard Nusbaum visited the lab.

  • 8 Feb 2019

    Leslie Kay visited the lab.

  • 28-29 Jan

    Pedro speaks at MIT Media Lab (host by Pattie Maes and Michelle Carr)

  • 25 Jan

    IEEE VR PC meeting.

  • 10 Jan

    Folks from CAT visited the lab and our department.

  • 8 Jan 2019

    Hooray! This is Steven's first day at the lab. Nitesh also started working with us!

  • 8 Jan 2019

    We just started our new class Emergent Interface Technology (graduate level).

  • 7 Jan 2019

    Lab just started! This is Jas' first day (and Pedro's too).

Alumni


Steven Nagels

Visiting PhD student
Winter 2019
Now: UHasselt

Romain Nith

Visiting Ugrad
Summer 2019
Now: McGill

Aaron Tang

MsC Practicum
Spring Summer 2019
Now: Sandia National Labs

Zhehao Li

Visiting Ugrad
Summer 2019
Now: USTC

Marco Kaisth

Undergraduate
2019
Now: UWaseda

Ted Kim

Undergraduate
CDAC Summer 2019
Now: UChicago

Nitesh Nath

MsC practicum
Winter 2019
Now: DRW Trading

Simeon Markind

MsC practicum
Spring 2019
Now: UChicago Booth

Jake Grayson

Undergraduate
Spring 2019
Now: CERN

Daniel Steinberg

Undergraduate
Spring 2019
Now: UChicago

Michael LeMay

Undergraduate
Summer 19
Now: Facebook

Eva Tuecke

High schooler
CDAC Summer 2019
Now: IMSE

Apply

We are always looking for exceptional students at the intersection of Human Computer Interaction, Electrical Engineering, Materials Science and Mechanical Engineering.

If you are considering applying for our lab:

  • 1. Send us an email with your portfolio and CV.
  • 2. Your portfolio (which preferably should be a website) must show documentation of the projects you are most proud (video documentation is ideal). We are especially looking for technical projects that involve: circuitry, signal processing, wearables or other interactive devices.
  • 2. Your CV should state also which level of expertise you have with the areas that are crucial for our lab: have you built your own circuits, do you write control loops, do you do more hardware than software, etc.
  • 4. Our lab is most suited for folks with some experience in HCI, EE, materials or ME.
  • 5. Read our research interests carefully, if you are unsure about the fit, send a quick email first before applying.
  • 6. If you are applying for an internship you mush have substantial previous experience in our areas of work.
  •  

Supported by

Our lab is supported by the following sponsor organizations:


NSF

Jas Brooks and Jasmine
Lu supported by GRFP

JSPS

Jun Nishida
is supported by JSPS

CDAC

Supports our project with
Prof. Dr. Andrey Rzhetsky

Big Ideas Generator

Supports our research
on wearable haptics

Contact us

hci@uchicago.edu
  • John Crerar Library, 292
  • Computer Science Department
  • 5730 S. Ellis Avenue
  • Chicago, IL 60637, USA.