Trust in machine learning applied for analysis of non-financial disclosures

This study is designed to address the question of how far we can trust machine learning models applied to the analysis of non-financial disclosures. More specifically, we show the possibilities of using machine learning algorithms to process climate-related information based on companies’ disclosures on their climate risk management, adaptation, and mitigation strategies. In a methodical context, we run two rounds of the annotations of the non-financial reports, to cover companies’ climate change-related disclosures. Based on this, we build a model for supervised learning. The annotated fragments (text corpora) along with the assigned climate risk classes, are then used as a source of for both binary and multi-class models. For the binary classification tasks, all annotated fragments (text corpora) will be considered as risk-related, while in the multi-class task, the specific climate risk categories will be considered. Our experiments have shown that the level of trust, expressed in terms of different quality metrics, i.e., balanced accuracy, weighted precision, weighted recall, and weighted F1-score, should be greater when we use ML models as compared to simple keywords-based text analysis. We have proved that the ML models can achieve statistically significantly higher metric values than the baseline model and thus their predictions can be trusted to a greater extent.