Understanding omitted facts in transformer-based abstractive summarization

Abstract

Text summarization is a natural language processing task that generates concise document summaries. It can be extractive or abstractive. The former extracts pieces of the document, while the latter generates new concise sentences after identifying the critical information from the input text. Abstractive Summarization (AS) more closely represents how a human would summarize and is used in multiple missioncritical downstream tasks in domains such as law and finance. However, the existing state-of-the-art AS models are based on black-box deep learning models such as Transformers. Hence, the users of such systems cannot understand why some facts from the document have been included in the summary while some have been omitted. This paper proposes an algorithm to explain which facts have been omitted and why in Transformerbased AS. We leverage the Cross-Attention (CA) in transformers to identify words in the input passage with minimum influence in generating the summary. These identified words are then given to a Large Language Model along with the input passage and the generated summary to explain the omitted facts and the reasons for omissions. The experimental results using the state-of-the-art AS model show that CA can help provide valuable explanations for the model’s fact selection process.