The dominant paradigm for neural text generation is left-to-right decoding from autoregressive language models. Constrained or controllable generation under complex lexical constraints, however, requires foresight to plan ahead for feasible future paths. Drawing inspiration from the A* search algorithm, we propose NEUROLOGIC AFesque, a decoding algorithm that incorporates heuristic estimates of future cost. We develop lookahead heuristics that are efficient for large-scale language models, making our method a drop-in replacement for common techniques such as beam search and top-k sampling. To enable constrained generation, we build on NEUROLOGIC decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with AFesque estimates of future constraint satisfaction. Our approach outperforms competitive baselines on five generation tasks, and achieves new state-of-the-art performance on table-to-text generation, constrained machine translation, and keyword-constrained generation. The improvements are particularly notable on tasks that require complex constraint satisfaction or in few-shot or zero-shot settings. NEUROLOGIC AFesque illustrates the power of decoding for improving and enabling new capabilities of large-scale language models.
|Title of host publication||NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics|
|Subtitle of host publication||Human Language Technologies, Proceedings of the Conference|
|Publisher||Association for Computational Linguistics (ACL)|
|Number of pages||20|
|Publication status||Published - 2022|
|Event||2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 - Seattle, United States|
Duration: 2022 Jul 10 → 2022 Jul 15
|Name||NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference|
|Conference||2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022|
|Period||22/7/10 → 22/7/15|
Bibliographical noteFunding Information:
This work was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), Google Cloud Compute, and Allen Institute for AI, Microsoft PhD Fellowship.
© 2022 Association for Computational Linguistics.
All Science Journal Classification (ASJC) codes
- Computer Networks and Communications
- Hardware and Architecture
- Information Systems