2023 Dr. Jacob A. George

William R. and Erlyn J. Gould Distinguished Lecture on Technology and the Quality of Life 
Jacob A. George, Ph.D., University of Utah
Wednesday, September 13, 2023
11:00 A.M. - Noon
J. Willard Marriott Library, Gould Auditorium, Level 1
University of Utah
The Lecture is free & open to the public
Parking & Directions

The Force is with U: Turning Luke Skywalker’s Bionic Arm from Sci-Fi Dream to Real-World Impact

In 2019, Dr. George and fellow researchers at the U made waves with the "LUKE Arm" (named after the robotic hand given to Luke Skywalker in The Empire Strikes Back), a prosthetic arm that has the ability to feel objects by transmitting the appropriate signals to the brain. Not only can the arm feel, but it can be controlled by the user's thoughts. On September 13, George will highlight this technology and the advances that have taken place since their breakthrough technology was first announced – including translation into bionic exoskeletons to aid individuals with paralysis and into smartwatches to allow anyone to seamlessly control smart devices and augmented reality by thought.

Clinical Need

Our hands represent a large part of what we can do and who we are. For individuals suffering from limb loss or paralysis, loss of hand function often results in physical disability and non-autonomy. This is often compounded by potential loss of employment, anxiety and depression. Assistive technologies—prosthetic hands, robotic exoskeletons and smart-home devices—have the potential to drastically increase patients’ independence and quality of life. Unfortunately, methods for controlling these assistive technologies remain primitive. For example, many amputees still use civil-war-era hooks that open and close using cables attached to their opposite shoulder. More advanced bionic arms have the potential to recreate the dexterous movements of the human hand, but they are often controlled using unintuitive strategies, like tilt-switches mounted to your shoes. Due to these limitations and others, up to 50 percent of upper-limb amputees simply abandon their prosthesis, never to use it again.

A New Hope: The “LUKE” Arm

At the University of Utah, George’s team uses electrical interfaces with the arm nerves and muscles to create a bidirectional link to an individual’s nervous system. Using this brain-computer-interface technology, George and colleagues enabled amputees to control a dexterous prosthetic arm entirely by thought. The advanced prosthetic arm was also endowed with a natural sense of touch, allowing users to have a more delicate touch and manipulate fragile objects without breaking them.

How it Works

“It might sound like science fiction, but it’s real,” describes George. “We control our hands with our thoughts all the time. When we think about moving our hands, information starts in the brain as a series of electrical impulses. Those electrical signals then travel down the spinal cord to the peripheral nerves where they cause the muscles to contract, which then causes the hand to move.” 

By tapping into those electrical signals, George’s team can determine what an individual is trying to do with their hands. “Even after an amputation, or after a stroke where the hand is too weak to physically move, we can record the electrical signals, amplify them on a computer and then use machine learning and artificial intelligence to decode what a person is trying to do,” continues George. 

But George’s technology does more than just allow individuals to control devices with their thoughts. The technology also provides users with a natural sense of touch. “When our hands make contact with objects, mechanical forces are converted into electrical signals that propagate up the nerves to the brain, where the brain interprets them as touch,” describes George. “If we do our jobs right, and recreate those electrical signals exactly, then the brain wouldn’t be able to tell the difference between real sensations and our ‘artificial’ sensations.” 

Enter: Virtual and Augmented Reality

George is now extending this technology into the commercial market, targeting applications in virtual and augmented reality. George notes that interactions with digital objects in virtual and augmented reality suffer from the same problems as prosthetic hands: “Without a sense of touch, it’s difficult to perform dexterous tasks like manipulating delicate objects or picking up objects without looking at them. But more importantly, without the ability to feel, interactions with objects are lifeless and you can no longer feel the comforting touch of a loved one.”

George isn’t the only one to realize this limitation of virtual and augmented reality. Meta Reality Lab’s (formerly Facebook) has been investing heavily in virtual and augmented reality. Meta recently unveiled a haptic glove that allows users to feel objects in virtual reality. But George argues that glove-based technology is limited to virtual reality and will have limited use in augmented reality, or mixed reality environments, where digital and physical objects both exist. “Imagine cooking in your kitchen and interacting with a digital cookbook. A sense of touch could help you feel and flip through the digital pages, but you can’t wear a glove in the kitchen while washing your hands, cutting food and getting hot items out of your oven,” explains George.

George’s team is integrating the technology behind the “LUKE Arm” into a wristband to provide users with a more elegant form of touch in augmented reality. The same way they create a sensation coming from a missing hand by stimulating the hand nerves, George’s team stimulates the hand nerves through the skin at the wrist to create sensations on the hand. Their technology is unique in that it provides a sense of touch on the hands while keeping the hands free for daily tasks. George has a prototype of this technology in the form of a wristband, and one day envisions it being integrated into common smartwatches like the Apple Watch. For these contributions to science and technology, George was recognized as one of North America’s brightest young leaders and entrepreneurs and named one of Forbes “30 under 30” in 2022.

Creating Inclusive Interfaces with Technology

In another line of research, George is using the “LUKE Arm” technology to develop a similar wristband to not only add a sense of touch, but also to detect a user’s intent. This technology could enable individuals to control and interact with digital objects in virtual or augmented reality without the need for instrumented gloves or external cameras. Individuals could also use this technology to interact with smart-home devices simply with their thoughts. The idea of telepathically controlling devices has captured a lot of interest; George’s team has secured research funding to pursue the technology from the Veteran’s Affairs, the University of Utah PIVOT Center and even Meta Reality Labs.

But George says the real benefit of this technology will be for the millions of individuals in the world living with life-altering neuromuscular disabilities. His team has demonstrated that their wristband can be used by paralyzed individuals with limited hand function to control virtual hands and smart-home devices. They have also demonstrated their wristband can evoke sensations on the hands of individuals with partial hand amputation. George says they “envision a world, an inclusive world, in which everyone can seamlessly interact with the technology around them, regardless of their physical capabilities.”

Augmenting Human Performance with Bionic Exoskeletons

The technology behind the “LUKE Arm” can be used to determine an individual’s intended motion even when the user is paralyzed or too weak to move their physical body. George also has ambitions to use the technology to restore mobility and dexterity to individuals who have experienced a stroke or spinal cord injury. Stroke is a leading cause of disability worldwide and 80% of stroke survivors suffer from upper-limb hemiparesis (i.e., paralysis of the hand) that limits activities of daily living, participation in society and prospects of returning to work.

In 2020, George worked with Dr. Tommaso Lenzi, last year’s Distinguished Gould Lecturer, to explore how the “LUKE Arm” technology could provide more intuitive and dexterous control of Lenzi’s lower-limb hip exoskeleton. The duo published their findings in Frontiers in Neurorobotics, paving the way for more dexterous bionic exoskeletons. Now, George is working on bringing this approach to upper-limb exoskeletons in partnership with Myomo Inc., the lead manufacturer of upper-limb orthotic devices for stroke patients, and with support from the prestigious NIH Director’s Early Independence Award. George is the only individual at the University of Utah, and the entire intermountain west region, to have received this prestigious award from the National Institutes of Health.

Research and Development Background

Dr. George’s work at the University of Utah builds upon the success of many University of Utah faculty prior to him. The realization of Luke Skywalker’s bionic arm at the University of Utah dates back to as early as 1989, when bioengineering professor Kenneth Horch and colleagues developed one of the first technologies for directly interfacing with human peripheral nerves. In 2005, with the help of orthopedic surgeon, Douglas Hutchinson, they provided proof-of-concept that the peripheral nerves retain sensory and motor function after an amputation, and that the residual nerves can be used to provide bi-directional control of a prosthetic hand. 

Today, researchers at the University of Utah use a different technology to interface with the human peripheral nerves known as the Utah Slanted Electrode Array (USEA). The USEA was originally developed in 2001 by bioengineering professor Richard Normann, and later commercialized by electrical & computer engineering professor Florian Solzbacher. The USEA is manufactured and produced in Salt Lake City by University of Utah spin-off company, Blackrock Neurotech.

In 2004, bioengineering professor Gregory Clark began a long-term multidisciplinary effort to use the USEA to make Luke Skywalker’s bionic arm a reality. This effort involved Hutchinson, bioengineering professors David Warren, Tomasz Petelenz and Phil Triolo and electrical & computer engineering professors Loren Rieth and V. John Matthews. Other key players included Tyler Davis (neurosurgery), Christopher Duncan (Physical Medicine & Rehabilitation), and several graduate and undergraduate students. George is among the graduate students who contributed to the project from 2016 to 2020. Support for the project also came externally from the Defense Advanced Research Projects Agency, the National Science Foundation, academic partners (University of Chicago and Cleveland Clinic) and industry partners (Blackrock Neurotech, Ripple Neuro, and DEKA). Both Blackrock Neurotech and Ripple Neuro are local companies spun out from the University of Utah.

Today, George stands on the shoulders of many before him as he continues to advance the technology forward. George says the future of this technology is best summarized by prior research participant, Keven Walgamott, who lost his arm 17 years prior to the research study. After using the technology, Walgamott summarized his experience by stating, “…Soon it will be just like Luke Skywalker, and then everyone will want one.”

George’s team seeks to achieve just that. They have partnered with Biologic Input Output Systems Inc., who licensed key patents from the University of Utah, to bring the world’s most advanced prosthesis to market. They are now preparing for large-scale clinical trials to demonstrate the safety and efficacy of the prostheses in patients’ own homes.

Dr. Jacob A. George, Ph.D. 

Dr. Jacob A. George is an Assistant Professor in the Department of Electrical & Computer Engineering and the Department of Physical Medicine & Rehabilitation at the University of Utah. He is the director of the Utah NeuroRobotics Lab and a foundational researcher in the Craig H. Neilsen Rehabilitation Hospital. Dr. George received his B.S. in Biomedical Engineering and a Certificate in Computational Science and Engineering from The University of Texas at Austin in 2016. He then received his M.S. and Ph.D. in Biomedical Engineering from the University of Utah in 2018 and 2020, respectively. Dr. George has been the recipient of several high-profile awards, including the NIH Director’s Early Independence Award and Forbes 30 Under 30.

About the Gould Lecture

The William R. and Erlyn J. Gould Distinguished Lecture on Technology and the Quality of Life was inaugurated October 7, 1992, at the University of Utah Marriott Library. In establishing the lecture series, Bill and Erlyn Gould both recognized the critical need for continuing public education about issues regarding modern technology and its impact on our daily lives. Inherent to the advantage of technology is the importance of understanding the ramifications and responsibilities that accompany modern scientific discovery. Only through continuing public education can scientific fact and social philosophy be successfully merged. The lecture series is intended to provide a forum for the discussion of problems, issues, experiences, and successful case histories of the regeneration and preservation of our communities through the application of modern technology.

It is hoped that an increased awareness of obligation in the public trust will emerge among practitioners of technology as they address the very important environmental and life-deteriorating problems facing society today.