An Empirical Study of the Consistency between Protocols for Evaluating Explanations of Predicted Links in Knowledge Graphs

Since knowledge graphs are often incomplete, link prediction methods are adopted to predict missing facts. Although scalable embedding models are commonly used for this purpose, they lack comprehensibility, which may be crucial in several domains. Explanation methods address this issue by identifying pieces of knowledge that support the predicted facts. Regretfully, comparing quantitatively the resulting explanations is challenging because there are different protocols and no insights on their consistency when evaluating the same explanation method. Filling this important gap, we measure their consistency particularly as the correlation between the metrics resulting from evaluating the same explanation methods via different protocols. This requires evaluating the LP-X method CrossE in terms of a different protocol in addition to the ones introduced specifically for CrossE. We conduct experiments with different widely known knowledge graphs and embedding models. The outcomes suggest an overall consistency.