In federally-funded research, UH Hilo students investigate problems in human-in-the-loop AI
The students have contributed significantly to important artificial intelligence research projects while gaining valuable skills in working with large codebases, problem-solving, critical thinking, collaborative research, and artificial intelligence.
By Susan Enright/UH Hilo Stories.
This summer, eight students at the University of Hawaiʻi at Hilo participated in a federally-funded data science program to investigate various research problems in human-in-the-loop (HITL) artificial intelligence and its connections with natural science. The program was led by Associate Professor of Computer Science Travis Mandel, an expert in HITL AI, a data-driven process that improves models and algorithms through human intervention and contribution to create better and more accurate AI (learn more about Mandel’s research).
“I am very proud of each and every student that participated in this year’s research experience,” says Mandel. “All eight students contributed significantly to important AI research projects while gaining valuable skills in working with large codebases, problem-solving, critical thinking, collaborative research, and artificial intelligence. These skills will help the students in their future careers, whether in data science or computer science.”
“I plan to continue to work with students to develop these projects into publications that share the important findings with the scientific community,” he adds.
Students who participated in the summer program and their majors are:
- Murphy Bierman, Data Science Certificate only
- Sebastian Carter, Data Science and Math
- Sam Chiu, Data Science and Marine Science
- Josh Kralewski, Data Science
- Marianne Martinez, Data Science
- Nathaniel McComas, Computer Science and Math
- Ian Scarth, Data Science
- Peyton Taylor, Computer Science
The eight-week interdisciplinary program, which has taken place over the past several summers, is sponsored by the National Science Foundation through two grants awarded to Mandel: a Faculty Early Career Development Program or CAREER grant (#HCC-1942229), and an EPSCoR Change Hawaiʻi award focused on the intersection of climate science and data science (#ORS-2149133). The EPSCoR Change Hawaiʻi grant is part of a University of Hawaiʻi statewide program — Established Program to Stimulate Competitive Research — to fund research, education, and capacity building in the sciences.
The funding for Mandel’s ongoing summer research programs has always provided the students stipends for their work. One change this summer was that in addition to the stipends, the grants also provided funds for equipment such as new laptops and displays to allow for smooth collaboration between the students and Mandel with a consistent and dependable environment.
UH Hilo Data Science Program
Several of the students in the AI summer program are majoring in data science, a newly established bachelor of science degree at UH Hilo, the first in the 10-campus UH System.
“It’s great to see that, despite the fact that this is the first summer the data science major has existed, the majority (of students in the summer research program) — five of the eight students — are data science majors, with two of the others going for a data science certificate,” says Mandel, who serves as coordinator of the degree program.
The academic program is designed to be interdisciplinary, building an in-depth skillset in artificial intelligence, machine learning, and statistics. Students can choose one four tracks to specialize further: astronomy, business, statistics, or computational. The data science program also offers a certificate to students majoring in any field.
“I think the excellent outcome of the (summer) research experience really shows the strength of UH Hilo’s unique data science program and our outstanding students,” says Mandel.
Student research projects
On August 1, the students presented their summer research projects at an open event at Mookini Library.
Here are summaries of the projects:
Project 1: Increasing the Effectiveness of AI Assistance
Presenters: Marianne Martinez and Sebastian Carter
Abstract:
AI systems can be very helpful, but many times it seems like their “assistance” is hurting more than it helps. In data annotation as an example, object detector models designed to help you annotate images can be extremely helpful but will occasionally show you predictions like this!
AI coding assistants like github copilot typically always show code completion snippets after a set amount of time, which can lead to frustrating situations where autocompletion options can actually make code slower and frustrate the user! Overcoming this in a robust way is a very challenging problem.
This summer, we have made large improvements to an algorithm for adapting AI assistance, and implemented it in multiple different settings, from computer vision with various detection models to coding. Our algorithm adapts the AI assistance to your personality, saving you precious time and overall increasing the efficiency of your AI assistant!
Project 2: Uncertainty Evaluation of Spatial Interpolators to Guide Scientists in New Data Collection
Presenters: Ian Scarth and Sam Chiu
Abstract:
This project combines cutting edge artificial intelligence and statistical methods with computer vision, up-to-date Hawaiʻi climate-science data, and novel means of evaluating algorithm uncertainty. When scientists are working with statistical or artificial intelligence models, understanding the uncertainty of the model is key to determining its usefulness. This is particularly relevant in the field of spatial interpolation, as it can guide researchers to the most valuable locations for collecting new data.
Our datasets include satellite imagery, live camera feeds, and ecosystem maps from leading climate scientists. These datasets were processed using custom machine learning models to recognize patterns in soil moisture, fog, and carbon sequestration across the Big Island. Existing literature has compared only a limited set of interpolators based on their results, while generally neglecting to evaluate their uncertainty estimates. To address this gap, we have prepared a user study in which participants will rank visualizations of uncertainty for each interpolator based on their intuitive usefulness.
Project 3: Real World AI Assisted Fish Tracking
Presenters: Nathaniel McComas and Peyton Taylor
Computer Vision systems for analyzing images and videos have been making huge leaps in performance but there are many problems that have yet to benefit from these improvements. For example, underwater divers have difficult tasks that involve identifying and keeping track of fish. It’s challenging to create systems that can function underwater, let alone perform difficult tasks such as identifying and tracking objects. Our goal is to study how people can work most effectively together with these systems, including developing metrics that better capture how well the AI will help the user.
This summer, we made several steps toward this goal. Specifically, we improved a simulation of users working with an AI system in real-time, ensuring that the code can also run on Android phones which can be taken underwater. We integrated recent metrics, AI algorithms, custom scripts, and large datasets to improve and measure performance. We identified overlooked problems with existing metrics and current visualizations that would hurt performance in a real-world setting. The next step is to launch our software into real-world experiments which will involve providing underwater divers with our application to run on phones, and assist in tracking fish.
Project 4: It Can Sing, It Can Dance, It Can Act: Talented AI Systems To Look Out For
Presenters: Murphy Bierman and Josh Kralewski
There is growing interest in multi-modal AI; that is, AI systems that can take in and produce different kinds of data: Images, text, structured data, code, etc. One real-world example is that in many scientific problems, scientists need to mark up (annotate) imagery to train an AI to recognize a species or pattern of interest. However, scientists may wish to use different styles of annotation to train the AI and explore what kind of data causes the AI to perform the best. Most existing computer vision systems are designed only for one type of annotation. It still is unclear how to create a system that efficiently processes all the different possible types of input.
This summer, we researched and developed different kinds of multi-modal AI systems. Systems that can take in and produce markups for images in text, outline, highlight and bounding box formats at the same time. This includes systems that manage multiple different AI’s at the same time, a system that forwards data to a multi-modal LLM, and a system that uses a custom neural network. We streamlined our testing infrastructure to improve our experiments.
Finally, we explored how these work in different domains, integrating different climate datasets and models into our annotation pipeline.
Story by Susan Enright, public information specialist for the Office of the Chancellor and editor of UH Hilo Stories. She received her bachelor of arts in English and certificate in women’s studies from UH Hilo.
