Towards a robust retrieval-based summarization system
S. Liu, J. Wu, J. Bao, W. Wang, M. Hovakimyan, C. G. Healey
arXiv:2403.19889, 2024
arXiv:2403.19889, 2024
View PDF
arXiv
Cite
Cite
APA
Click to copy
Liu, S., Wu, J., Bao, J., Wang, W., Hovakimyan, M., & Healey, C. G. (2024). Towards a robust retrieval-based summarization system. ArXiv:2403.19889.
Chicago/Turabian
Click to copy
Liu, S., J. Wu, J. Bao, W. Wang, M. Hovakimyan, and C. G. Healey. “Towards a Robust Retrieval-Based Summarization System.” arXiv:2403.19889 (2024).
MLA
Click to copy
Liu, S., et al. “Towards a Robust Retrieval-Based Summarization System.” ArXiv:2403.19889, https://arxiv.org/abs/2403.19889, 2024.
BibTeX Click to copy
@article{s2024a,
title = {Towards a robust retrieval-based summarization system},
year = {2024},
journal = {arXiv:2403.19889},
author = {Liu, S. and Wu, J. and Bao, J. and Wang, W. and Hovakimyan, M. and Healey, C. G.},
howpublished = {https://arxiv.org/abs/2403.19889}
}
This paper describes an investigation of the robustness of large language models (LLMs) for retrieval augmented generation (RAG)-based summarization tasks. While LLMs provide summarization capabilities, their performance in complex, real-world scenarios remains under-explored. Our first contribution is LogicSumm, an innovative evaluation framework incorporating realistic scenarios to assess LLM robustness during RAG-based summarization. Based on limitations identified by LogiSumm, we then developed SummRAG, a comprehensive system to create training dialogues and fine-tune a model to enhance robustness within LogicSumm's scenarios. SummRAG is an example of our goal of defining structured methods to test the capabilities of an LLM, rather than addressing issues in a one-off fashion. Experimental results confirm the power of SummRAG, showcasing improved logical coherence and summarization quality. Data, corresponding model weights, and Python code are available online.