Sampling from Language Models under Syntactic Constraints

Prelegent(ci)

Paweł Parys

Afiliacja

University of Warsaw

Język referatu

angielski

Termin

5 listopada 2025 14:15

Pokój

p. 5440

Tytuł w języku polskim

Sampling from Language Models under Syntactic Constraints

Seminarium

Seminarium „Teoria automatów”

---

In recent years, large language models (LLMs) have revolutionized code writing. Using LLMs, one can quickly arrive at solutions that are close to the correct answer for many tasks, though mistakes still occur frequently. However, in some cases, we have formal syntactic or even semantic constraints on the code to be generated. Can we account for these constraints while generating code from an LLM?

In this work, we treat the language model as a black-box, returning conditional probabilities of the next tokens. We will not discuss the internal construction of LLMs. Rather, this talk focuses on sampling from appropriate distributions and efficient parsing techniques.

This is joint work with Sairam Vaidya, Taylor Berg-Kirkpatrick, Loris D'Antoni, and others.

---