GANDALF: A LLM-based approach to map bark beetle outbreaks in semantic stories of Sentinel-2 images

Huge spruce forest areas have been damaged by massive bark beetle outbreaks across Europe during the past few years. Hence, forest health management requires large-scale inventory of bark beetle outbreaks to plan actions for promptly mitigating forest tree dieback. Deep learning techniques have recently achieved amazing results in imagery semantic segmentation tasks by dominating the recent research for mapping bark beetle outbreaks in Sentinel-2 images of forest areas. In addition, due to the impressive performance of Large Language Models (LLMs) in natural language understanding and generation tasks, LLMs have started attracting attention in multiple fields. In this paper, we describe GANDALF: an approach that leverages the potential of LLMs for mapping bark beetle outbreaks in Sentinel-2 images of forest areas. Specifically, we take advantage of the rich context of textual data to transform Sentinel-2 images in smart data ready for boosting accurate semantic segmentation modeling. We use a foundation LLM model to account for the text encoding of the spectral-spatial imagery context information. We fine-tune the LLM model to perform the semantic segmentation of forest images and use the Integrated Gradients (IG) algorithm to explain how each spectral-spatial information has an effect on the bark beetle outbreak detection. We assess the effectiveness of the proposed approach in a case study regarding bark beetle outbreaks in Sentinel-2 images of forest scenes in Czech Republic.