AI scientists at UH Hilo receive prestigious editor’s choice award for paper on their fish tracking software
Computer scientist Travis Mandel, director of UH Hilo’s interdisciplinary data science program, is working with team of students, alumni, and faculty to develop the FISHTRAC software.
Above, performance on a video from the FISHTRAC software test set, developed by UH Hilo data scientist Travis Mandel and his research team. Despite the challenging camera motion and background, Robust Confidence Tracking (RCT) is able to keep track of many fish in this case. (Video from the study)
A University of Hawai‘i at Hilo computer scientist’s paper about his development of artificial intelligence applications for fish tracking has won an editor’s choice award. The paper was co-authored by UH Hilo students and alumni.
Associate Professor of Computer Science Travis Mandel, director of UH Hilo’s interdisciplinary data science program, is working with a team of students, alumni, and faculty to develop the FISHTRAC software. The project focuses on building an AI program that can reliably track individual fish through video footage and photographs for scientific purposes, saving trackers countless hours spent reviewing images often difficult to discern.
“My research is always collaborative,” says Mandel about the interdisciplinary team effort between computer scientists and ecologists. “I’m a computer scientist but I don’t just study computer science problems in isolation.”
The paper, “Detection confidence driven multi-object tracking to recover reliable tracks from unreliable detections” is published in Pattern Recognition, Volume 135, March 2023.
The research team
Mandel’s areas of research are human-in-the-loop AI and computer vision. His doctor of philosophy in computer science and engineering (2017) is from the University of Washington, and his bachelor of science in computer science is from Carnegie Mellon University (2011).
Co-authors on the paper with Mandel are UH Hilo alumni Mark Jimenez (computer science, 2022, now a production engineer working in the software industry), Emily Risley (computer science, 2020, currently a software engineer working in industry), Taishi Nammoto (physics, 2020, now a software developer working in industry), and Rebekka Williams (mathematics, 2020, now a graduate student in communicology at UH Mānoa).
Co-authors also include current UH Hilo students Meynard Ballesteros (computer science undergraduate) and Bobbie Suarez (tropical conservation biology and environmental science master’s program).
Also co-authoring the paper is Max Panoff, a doctoral student in electrical and computer engineering at the University of Florida.
The research
The central question the research is addressing: How can AI and machine learning systems work with humans to solve real problems?
Initially, Mandel was called on by environmental scientists at UH Hilo to help with computer vision issues, or the ability of software to recognize objects consistently in photographs or videos. The process of teaching an AI engine to learn is complex, and projects such as these are on the cutting edge of computer science and environmental science today.
Although Mandel’s training was not in computer vision, the pressing need for research within this field quickly presented itself.
“A lot of people started reaching out to me, faculty members and grad students in different disciplines, saying hey, can you help us with our computer vision problems?” says Mandel. The video identification would serve as an alternative to catch and release tagging, a process which is quite invasive to the fish.
The editor’s choice award from Pattern Recognition was unexpected for Mandel and his team.
“This journal is one of the top journals in computer vision, so we were already excited,” says Mandel. “[The research] is not quite the typical work you see in the field because most work in computer vision is on very standard data sets. For instance, in the problem we were trying to solve, it’s mostly on pedestrians and cars.”
Mandel says the research team received a mixed reception in the past.
“People in Hawai’i are really excited about fish, but the rest of the computer vision community, it doesn’t really match their expectations,” he says.
The issue of data is vital in computer vision, as the network relies on data to train itself to recognize objects. Because of this gap in knowledge, Mandel and his team trained their computer vision algorithm on footage of reef fish in the Hilo area.
Challenges and successes
Due to its unusual nature, the project was initially met with some pushback. Mandel’s vision and work is unexpected in the field because most data scientists are really interested in deep learning approaches, approaches that are so called “end to end,” and those programs need huge amount of labeled data. “We don’t have that much labeled data,” says Mandel, “[so] that was a real battle, because unfortunately, this happens a lot in science, in some areas people have their preconceived notions.”
Despite these challenges, the paper received the prestigious editor’s choice award.
“What’s nice about this award,” says Mandel, “is it shows the recognition of all the hard work put in by myself and all these undergraduate students that worked with me on this paper to try to solve important problems facing Hawai‘i using computer vision techniques.”
Now, the FISHTRAC software with its Robust Confidence Tracking (RCT) is able to reliably track different kinds of fish on video.
-
- Take a look at the publicly available FISHTRAC codebase. This codebase is an important contribution to the field, as there do not exist codebases that allow one to compare a wide variety of multi-object tracking or MOT trackers on new datasets.
“It certainly works better than all the other algorithms we compared it to in the paper, which was a very large number,” says Mandel. It’s still not perfect, he says, but the reality is the AI allows for much less human effort and time in terms of someone sitting and watching a hundred videos and then drawing boxes around every frame, “because that takes forever.”
FISHTRAC’s effectiveness at tracking fish is a great example of how AI and machine learning networks can work effectively with humans as part of a team. The application of these technologies to environmental science has promising implications, and programs like this will make many aspects of environmental science and conservation more efficient and less invasive in the future.
Related stories
UH Hilo data scientist Travis Mandel finds intellectual freedom in tenure
Story by Evangeline Lemieux, who is double majoring in English and medical anthropology at UH Hilo.
