Addressing Processing Ambiguity

This project works towards a first author paper for the leading AI@UNC member heading the project, and co-authorships for significantly contributing members. This project currently focuses on detecting when a language model perceives a question as ambiguous by analyzing the entropy at the logit level. The current hypothesis is that ambiguous questions should lead to higher entropy in the model's predictions due to increased uncertainty, while unambiguous questions should result in lower entropy. The project currently involves sourcing ambiguous and unambiguous questions from established datasets, such as AmbigNQ and repurposed data from other research on grammatical and syntactical ambiguities. Members will then pass these questions through various language models to calculate entropy using different probability methods (e.g., first token, minimum, full sequence, and average probabilities). The ultimate goal is to uncover patterns that correlate question ambiguity with model uncertainty, improving the understanding of how language models handle ambiguous inputs.