publications

2026

1. Confidence Reasoning Graphs: Structured Confidence Estimation for LLM Agents

   Brendan King, Farima Fatahi Bayat, Jean-Flavien Bussotti, and 2 more authors

   arXiv Pre-Print , 2026

   Code
2. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents

   Brendan King, and Jeffrey Flanigan

   arXiv Pre-Print , 2026

   Code

2025

1. In Search of the Lost Arch in Dialogue: A Dependency Dialogue Acts Corpus for Multi-Party Dialogues

   Jon Cai, Brendan King, Peyton Cameron, and 8 more authors

   In Findings of the Association for Computational Linguistics: ACL 2025, Jul 2025

   Abs

   Understanding the structure of multi-party conversation and the intentions and dialogue acts of each speaker remains a significant challenge in NLP. While a number of corpora annotated using theoretical frameworks of dialogue have been proposed, these typically focus on either utterance-level labeling of speaker intent, missing wider context, or the rhetorical structure of a dialogue, losing fine-grained intents captured in dialogue acts. Recently, the Dependency Dialogue Acts (DDA) framework has been proposed to for modeling both the fine-grained intents of each speaker and the structure of multi-party dialogues. However, there is not yet a corpus annotated with this framework available for the community to study. To address this gap, we introduce a new corpus of 33 dialogues and over 9,000 utterance units, densely annotated using the Dependency Dialogue Acts (DDA) framework.Our dataset spans four genres of multi-party conversations from different modalities: (1) physics classroom discussions, (2) engineering classroom discussions, (3) board game interactions, and (4) written online game chat logs. Each session is doubly annotated and adjudicated to ensure high-quality labeling. We present a description of the dataset and annotation process, an analysis of speaker dynamics enabled by our annotation, and a baseline evaluation of LLMs as DDA parsers. We discuss the implications of this dataset understanding dynamics between speakers and for developing more controllable dialogue agents.

2024

1. Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel

   Brendan King, and Jeffrey Flanigan

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

   Abs

   Training task-oriented dialogue systems typically requires turn-level annotations for interacting with their APIs: e.g. a dialogue state and the system actions taken at each step. These annotations can be costly to produce, error-prone, and require both domain and annotation expertise. With advances in LLMs, we hypothesize that unlabeled data and a schema definition are sufficient for building a working task-oriented dialogue system, completely unsupervised. We consider a novel unsupervised setting of only (1) a well-defined API schema (2) a set of unlabeled dialogues between a user and agent. We propose an innovative approach using expectation-maximization (EM) that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent. Evaluating our approach on the MultiWOZ benchmark, our method more than doubles the dialogue success rate of a strong GPT-3.5 baseline.

2023

1. Diverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking

   Brendan King, and Jeffrey Flanigan

   In Findings of the Association for Computational Linguistics: ACL 2023, Jul 2023

   Abs DOI

   There has been significant interest in zero and few-shot learning for dialogue state tracking (DST) due to the high cost of collecting and annotating task-oriented dialogues. Recent work has demonstrated that in-context learning requires very little data and zero parameter updates, and even outperforms trained methods in the few-shot setting. We propose RefPyDST, which advances the state of the art with three advancements to in-context learning for DST.First, we formulate DST as a Python programming task, explicitly modeling language coreference as variable reference in Python. Second, since in-context learning depends highly on the context examples, we propose a method to retrieve a diverse set of relevant examples to improve performance. Finally, we introduce a novel re-weighting method during decoding that takes into account probabilities of competing surface forms, and produces a more accurate dialogue state prediction. We evaluate our approach using MultiWOZ and achieve state-of-the-art multi-domain joint-goal accuracy in zero and few-shot settings.

2. Dependency Dialogue Acts — Annotation Scheme and Case Study

   Jon Cai, Brendan King, Margaret Perkoff, and 9 more authors

   In The 13th International Workshop on Spoken Dialogue Systems Technology, IWSDS 2023, Los Angeles, United States, Feb 2023

   PDF

2018

1. ProbAnnoWeb and ProbAnnoPy: probabilistic annotation and gap-filling of metabolic reconstructions

   Brendan King, Terry Farrah, Matthew A. Richards, and 3 more authors

   Bioinformatics (Oxford, England), May 2018

   Abs DOI

   SUMMARY: Gap-filling is a necessary step to produce quality genome-scale metabolic reconstructions capable of flux-balance simulation. Most available gap-filling tools use an organism-agnostic approach, where reactions are selected from a database to fill gaps without consideration of the target organism. Conversely, our likelihood based gap-filling with probabilistic annotations selects candidate reactions based on a likelihood score derived specifically from the target organism’s genome. Here, we present two new implementations of probabilistic annotation and likelihood based gap-filling: a web service called ProbAnnoWeb, and a standalone python package called ProbAnnoPy. AVAILABILITY AND IMPLEMENTATION: Our tools are available as a web service with no installation needed (ProbAnnoWeb) at probannoweb.systemsbiology.net, and as a local python package implementation (ProbAnnoPy) at github.com/PriceLab/probannopy. CONTACT: evangelos.simeonidis@systemsbiology.org or nathan.price@systemsbiology.org. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

2016

1. Mighty morphing metabolic models: leveraging manual curations for automatic metabolic reconstruction of clades

   Matthew A. Richards, Brendan King, Evangelos Simeonidis, and 1 more author

   F1000Research, May 2016