LMC: Language Model Calibration

We are a social science research lab in the Haas School of Business. We explore research questions related to overconfidence.

We plan to evaluate language model uncertainty on a variety of tasks. We will assess GPT-3, UnifiedQA, and other public language models on reading comprehension, reasoning, and additional challenge datasets. Model outputs on these tasks will be assessed for calibration and accuracy. A comparison to human errors on the same tasks will illuminate differences in human and AI reasoning around uncertainty, an important topic for AI safety.