UH Hilo computer scientist Travis Mandel and his students explore artificial intelligence and natural science

Posted on October 2, 2023November 15, 2023 by Staff

A UH Hilo summer class explored research problems in human-in-the-loop artificial intelligence and its connections with natural science. Professor who taught class will be giving a related talk on Oct. 9.

AI class stands for photo in from of Computer Science and Engineering Department. — Over the summer, a National Science Foundation-supported class worked on research problems in human-in-the-loop artificial intelligence and its connections with natural science. Associate Professor of Computer Science Travis Mandel (at center right above), who instructed the class, will be giving a talk Oct. 9 about developing computer vision algorithms for Hawai‘i Island datasets, details below. In photo above, from left, students Sharmin Zaman, Gus Coffey, Kalani Perry, Rodel Tagalicud, Assoc. Prof. Mandel, and students Marcelino Bautista Jr., Ryp Ring, and Ryan Foley.(Courtesy photo)

By Susan Enright.

Over the summer, Associate Professor of Computer Science Travis Mandel and seven of his students worked to investigate various research problems in Human-in-the-Loop Artificial Intelligence and its connections with natural science. The course was sponsored by the National Science Foundation.

Below are descriptions of the projects followed by information about an upcoming talk by Associate Professor Mandel, “Exploring and Developing Computer Vision Algorithms for Hawai‘i Island Datasets,” scheduled in-person and online Oct. 9.

Here are the summer projects:

Project 1: Modernizing a Human-in-the-Loop Annotation Interface

Presenters: Ryp Ring and Rodel Tagalicud

Abstract:

Most modern computer vision systems can eventually perform complex image analysis tasks automatically, but only once they are handed thousands of examples (annotations) tediously created by humans. With the intention of speeding up that process, we are developing an AI-powered interface that adapts to the user in real time. Before the summer, our system had many unknowns and weaknesses. With the very rapidly developing world of AI, the biggest unknown was that we did not know if human-in-loop annotation was still a good fit for more recent object detection systems, as the object detection system that we currently use was developed 6 years ago. Over the summer, we looked at 2 different object detection systems, one based on transformers (like ChatGPT), and the other which was just developed within the past year. We also used various analysis tools to identify the weaknesses in our system to make it better optimize the annotation experience while taking into account the strengths of both the object detector and the user.

Project 2: Analyzing the Efficacy and Performance of AI-Assisted Data Collection

Presenters: Gus Coffey and Sharmin Zaman

Abstract:

In the realm of climate science, the demand for accurate and comprehensive data is very high, with the goal often being to feed this data into AI/ML systems for analysis. However, much less work has investigated how AI can help climate scientists better collect the data in the first place, which is important since data collection is costly. This summer, we worked to extend an AI framework to handle the problem of collecting land use data to estimate carbon sequestration, an important problem in climate science. This involved creating specialized models to convert raw measurements into carbon sequestration estimates, and consuming annotated imagery to build a simulator of the process. We also studied the critical problem of computing and visualizing uncertainty, developing new methods including Bayesian statistical approaches and comparing them with deep machine learning methods. We anticipate this work will lay the foundation for future human-AI collaboration in the domain of climate science.

Project 3: Towards Assessing Multi-Object Tracking Efficiency in Real-World Settings

Presenters: Marcy Bautista and Kalani Perry

Abstract:

Computer Vision and AI are often evaluated using artificial score functions, however, it is unclear whether these scores match what people actually want from these systems in a real-world task. Multi-object tracking (MOT), for example, is a fundamental problem where an AI system keeps track of objects as they move through video, by putting boxes around them. Scores usually relate to the accurate placement of the boxes, but these scores may or may not correlate to a real-world task such as following multiple fish while diving off the coast of Hawaii Island. To help answer these questions, we continued the development of an application that measures human performance in the presence of automatic MOT assistance. This summer, we significantly improved speed on three different platforms (Windows, Mac, and Android), and added elements that allow us to evaluate users completing realistic marine science tasks with and without AI assistance. This will allow us to run future user studies that spur the development of MOT systems to better assist humans.

Project 4: Foundations of AI-Assistance for Human Annotation Selection

Presenter: Ryan Foley

Abstract:

A large contribution to the success of computer vision algorithms is a significant amount of valuable human time and effort spent marking images to develop these models. For instance, humans might spend time answering questions about the images, creating boxes around areas of interest, or drawing tight outlines around objects in an image. These different types of annotation come with different tradeoffs in terms of time and accuracy. Current work assumes researchers perfectly select the best annotation type for their problem and budget. The goal of this project is to integrate an AI-assistant for the Label Studio interface which automatically adjusts the annotation type to be most effective for given climate science data and to best utilize the user’s time. This summer, we adapted the Label Studio interface to work with an AI to cycle annotation styles and integrated algorithms for estimating carbon sequestration, an important topic in climate science. There is still much work to be done, but vital steps were made to progress this project towards its future development.

Upcoming talk

Associate Professor Mandel will be giving a talk titled, “Exploring and Developing Computer Vision Algorithms for Hawai‘i Island Datasets,” on Monday, Oct. 9, 2023. at 4:00 p.m. at Wentworth Hall room 1 and online via Zoom (Meeting ID: 965 4702 3084 with Passcode: TCBES). The public is invited. The event is sponsored by UH Hilo’s tropical conservation biology and environmental science graduate program.