Amir Sadovnik

In this project, we expand on the easy-VQA dataset to create a new dataset of simple, short-length videos and corresponding questions and answers about them. We then implement two models to perform the VQA task. The first model answers questions about videos that contain a single shape and the second model performs VQA on videos that contain two shapes using attention.

This project is motivated by the tendency for visual question answering (VQA) models to struggle with counting questions. Another goal is to provide a small, simple dataset for quick experimentation. Our code is based on easy-VQA-keras, but adapted for counting. We hope this code will provide a starting point to investigate more challenging counting problems.

This online tool to help integrate computer science and literacy education for elementary school students was designed as part of the NAF grant: "CS for Appalachia: A research-practice partnership for integrating computer science into East Tennessee schools" (NSF Grant No. 1923509).