AI student research conducted while adhering to COVID-19 protocols

Under the tutelage of artificial intelligence expert Travis Mandel, assistant professor of computer science, the interdisciplinary “Human-in-the-Loop Research Experience” was supported by the National Science Foundation.

By Susan Enright

Group of students and professor stand under a large tree, everyone is wearing masks and six feet apart.
Travis Mandel (front) and his students, who participated in an innovative research program over the summer, pose for photo while wearing masks and practicing social distancing. From left, Emily Risley, Sebastian Carter, Mark Jimenez, Travis Mandel, James Boyd, Kostadin Devedzhiev, and Edward Cashman. Courtesy photo, click to enlarge.

Six students at the University of Hawai‘i at Hilo participated in an innovative research program this summer where they worked on developing artificial intelligence (AI) systems to better support scientists in fields such as psychology, ecology, and marine science. The work was done on campus with appropriate coronavirus safety measures including proper social distancing both inside and outside the classroom.

Travis Mandel
Travis Mandel

Under the tutelage of Travis Mandel, assistant professor of computer science, the eight-week interdisciplinary data science research experience was supported by the National Science Foundation, and students taking the “Human-in-the-Loop Research Experience” were paid a stipend to participate. Mandel, an expert in AI, explains this wasn’t a typical course, but rather students were learning in a “hands-on manner while working on their cutting-edge research projects,” made all the more atypical by being held when students needed to practice social distancing.

The students were investigating three separate but related research projects. “These projects studied how humans and AI systems can best work together to solve real-world scientific challenges,” says Mandel. He explains the three projects:

  1. How can AI help scientists collect better data in domains like psychology and ecology? Most data is collected in a fairly uniform manner, however ideally an AI system could target data collection to places that are more “interesting.”  But how does it know what is “interesting”?  Well, it needs to work together with a scientist to iteratively improve the data collection process as more data is gathered. This type of thing is really challenging, but if successful has a huge potential to improve scientific data collection across a wide range of fields. (Students James Boyd and Sebastian Carter worked on this project.)

    Graph with a gold star to mark interesting point.
    Project 1: An new interface for scientists to view collected data and provide feedback to the AI system. Here, an interesting point is marked with a star. Project conducted by James Boyd and Sebastian Carter.
  2. Getting AI to work well requires lots of labeled data, especially when trying to identify new invasive plants or animal species. A very large amount of human effort is needed to do all this annotation upfront. We developed an interface where the AI system can work as a teammate with the human, both helping them and learning from them. There’s a lot of new challenges in this area, which we call “symbiotic learning,” but if they can be solved there is a huge potential to make the process more efficient. (Students Ed Cashman and Kosta Devedzhiev worked on this project.)

    Overhead map of forest with predictions outlined in orange.
    Project 2: An new annotation interface, where humans and AI work together to locate an invasive species (Miconia calvescens). Project by Ed Cashman and Kosta Devedzhiev. Imagery collected by Ryan Perroy and the SDAV lab.
  3. Tracking animals and objects as they move through space is something that is typically easy for a human and hard for a computer. State-of-the-art tracking systems are pretty good at tracking cars and pedestrians, but that’s only because they have a huge amount of data. What about situations where we don’t have nearly as much data, such as tracking a particular species of fish off Hilo Bay? We investigated tracking systems that can cope with these challenging limited-data scenarios, which are particularly important for marine science research. (Students Mark Jimenez and Emily Risley worked on this project.)

    Underwater image with fish; colored squares mark fish.
    Project 3: A screenshot of a video taken off Hilo Bay, showing AI fish tracking system in operation on several fish, including the Achilles Tang. Project by Mark Jimenez and Emily Risley. Imagery collected by Bobbie Suarez.

Solving these problems involved understanding a lot of diverse topics, from Javascript programming to multithreaded user interfaces to Bayesian statistics.

But research problems are really challenging, and asking students to solve them on their own from remote isolated locations as they hunker down safely during the pandemic can be daunting. “That’s why I really pushed hard to hold this in-person, because of the importance of each student being able to work closely together with myself as well as with a partner,” says Mandel.

Hands-on research in the age of COVID-19

The “work closely together” aspect of things presents a lot of challenges during a pandemic when everyone is required to practice social distancing.

Francis Cristobal
Francis Cristobal

“For instance, looking at code on a person’s screen is very difficult from six-plus feet away, and having me rapidly switch among groups isn’t easy with Zoom or other technologies,” explains Mandel. “As such, specialist Francis Cristobal and myself worked to create a unique room layout that would allow all the necessary types of interactions to take place.”

The idea was that the center of the room was a “movement area,” says Mandel, where he could freely move around, switching quickly between teams to help the students. Students were spaced out six-feet apart near the edges of the room, and were not allowed to move from their assigned seats. They could talk with the person sitting closest to them allowing for teams of two.

Diagram shows spacing of desks and work areas in classroom.
A diagram of the classroom arrangement. Courtesy image.

For students to see each other’s screens, they could project on the CyberCANOE (acronym for Cyber Enabled Collaboration Analysis Navigation and Observation Environment), a large screen format with ultra high resolution and 3D-enabled flat-screen displays connected to a high-performance computer. Or students could screen share over Zoom for the far group.

“A key aspect was that I needed to see the students’ screens to help them with their projects,” says Mandel.  “Looking over their shoulder wasn’t an option due to the six-foot distancing rule. Therefore, we came up with a unique setup where student’s monitors were mirrored to the monitor on the opposite side that was facing me in the middle of the room [so] I could easily see what the student was working on. We even connected the mice and keyboards up so that I could point to things or write to things on the students’ screen to allow me to better assist students while maintaining social distance.”

Students and professor were wearing masks at all times. They also sanitized workstations and door handles daily.

Seven people stands an arms length away from each other, arms spread out. All are wearing masks.
Travis Mandel (center) stands with students in his summer research program. From left, Sebastian Carter, Emily Risley, Mark Jimenez, Travis Mandel, James Boyd, Kostadin Devedzhiev, and Edward Cashman. The class practiced the strictest of coronavirus protocols to keep everyone safe from possible exposure. Courtesy photo.

The research experience

All the coronavirus protocols allowed the students to have a top-notch research experience. “I had an amazing group of students who really went above and beyond,” says the professor.

For example, one student was working on mathematical proofs for a large chunk of the summer. He started with the question, “How are you meant to prove something?” and ended up putting an involved proof together by the end of the eight weeks.

Another student had very limited Python experience at the beginning of the summer, but was able to rise to the challenge of dramatically optimizing some rather complex Python code.

“The immersive and intimate nature of the experience meant that I could really make sure each student was getting exactly what they needed,” explains Mandel. “If someone was finding something too easy or not interesting, I could switch them to a harder task, while if they were finding something too difficult, I could switch them to an easier task and have either myself or their partner take the harder one. This type of thing doesn’t happen as much in a typical class, which is usually one-size-fits-all as everyone goes through the same assignment at the same time.”

Mandel says he is really pleased with the experience and proud of the students.

“I am very grateful to the National Science Foundation for giving me the opportunity to hold this, as well as UH Hilo administrators for giving me permission to hold this in person,” he says. “I think it was a successful way to integrate research and education. Students really learned a lot of new skills, while at the same time helping solve some really challenging and important research questions involving AI and its impact on science. And we had a lot of fun along the way.”


Story by Susan Enright, a public information specialist for the Office of the Chancellor and editor of UH Hilo Stories. She received her bachelor of arts in English and certificate in women’s studies from UH Hilo.


Related post

UH Hilo computer scientist receives prestigious NSF grant to research artificial intelligence