On the Calibration and Uncertainty with Pólya-Gamma Augmentation for Dialog Retrieval Models

Tong Ye, Shijing Si, Jianzong Wang, Ning Cheng, Zhitao Li, Jing Xiao
October 2023
PDF Cite arXiv AAAI
An illustration of PG-DRR prediction models for dialog response retrieval

Abstract

Deep neural retrieval models have amply demonstrated their power but estimating the reliability of their predictions remains challenging. Most dialog response retrieval models output a single score for a response on how relevant it is to a given question. However, the bad calibration of deep neural network results in various uncertainty for the single score such that the unreliable predictions always misinform user decisions. To investigate these issues, we present an efficient calibration and uncertainty estimation framework PG-DRR for dialog response retrieval models which adds a Gaussian Process layer to a deterministic deep neural network and recovers conjugacy for tractable posterior inference by Pólya-Gamma augmentation. Finally, PG-DRR achieves the lowest empirical calibration error (ECE) in the in-domain datasets and the distributional shift task while keeping R10@1 and MAP performance.

Type
1
Publication
In Thirty-Seventh AAAI Conference on Artificial Intelligence
Click the Cite button above to demo the feature to enable visitors to import publication metadata into their reference management software.
LLM
Shijing Si
Shijing Si
Researcher
Jianzong Wang
Jianzong Wang
Honorary Director

[email protected]