Assistant Professor Arkaprava Saha is working to make artificial intelligence systems more understandable. One way of doing this is by enhancing the technique of layer-wise relevance propagation with the addition of semiring annotations. By introducing semiring annotations, he shows how relevance is propagated through neural networks and how it connects several explanation techniques within a single framework. 

AI’s “Black Box” Problem

Modern artificial intelligence systems have advanced greatly during the past several years and can now create visual art, translate natural languages, and generate human-like text and speech. Yet just how these systems become more capable remains largely unknown. Many neural networks operate as “black boxes”: they produce accurate predictions, but how and why they do so remains obscure, their reasoning concealed behind millions or billions of small adjustments to their network connections. This lack of transparency raises questions concerning trust and safety, especially in areas such as medicine and autonomous systems like driverless cars. 

To tackle this problem, researchers in Explainable XAI have developed methods that shed light on how neural networks reach their decisions. One of the best-known approaches is Layer-wise Relevance Propagation (LRP), which identifies the input features relevant to the output choice. However, LRP is not typically formulated as a graph-theoretical method that explicitly treats neural networks as graph structures. Even so, the technique works by tracking a neural network’s output back through its layers and assigning a relevance score to each neuron. In classifying images, for example, LRP can highlight the pixels that contributed most to a prediction. A picture classified as a cat, for instance, might show strong relevance around the animal’s ears, eyes, and whiskers, which gives researchers insight into the network’s reasoning. 

Although LRP has proven useful, it has limitations. Typically, relevance is represented by simple numerical values. Each neuron or input feature receives a single score that indicates its contribution to the final output. While this is helpful, it cannot produce more complex or structured information about how nodes and features interact or how decisions are formed. 
Such limitations have motivated us to broaden LRP’s capabilities so that it provides richer forms of information. 

New Approach Enables More Flexible Interpretations of Neural Network Behavior

Instead of treating relevance as a single number, our new approach uses semiring annotations to represent the flow of information through a neural network. A semiring is an algebraic structure that defines how values can be combined and propagated. Put simply, a semiring can be thought of as a set of rules for how pieces of information are added together and passed along the network. Rather than sending a single numeric score through the network, each neuron now passes along an annotation—a package of data governed by the semiring rules. Depending on the semiring chosen, this annotation might represent sets of contributing features, hierarchical groupings, or other kinds of information. The result is a richer, more flexible framework for explaining how neural networks reach decisions. 

How Semiring Annotation Works

Our new method uses the same approach as standard LRP. During the forward pass or inference step, a neural network processes an input and produces a prediction. During the backward pass, relevance values are propagated from the output layer back to the input layer. But the key difference is that these values now belong to a semiring instead of just numbers. 

To make this possible, we introduced annotation functions. These functions translate the numerical activations and weights of a neural network into semiring elements. Once this is mapped out and defined, the network can propagate relevance backward using the semiring’s operations. As in standard LRP, the total relevance assigned to a layer should equal the relevance received from the layer above. This ensures that explanations remain consistent throughout the network.
Because the framework is flexible, different semirings can be used to produce different kinds of explanations. Some semirings behave similarly to standard numerical relevance, while others produce more structured attributions. 

Experiments and Demonstrations

We tested our approach on neural networks used for image classification by applying several semiring variants, and compared the resulting relevance maps. Each semiring produced a different pattern of highlighted regions, reflecting its underlying meaning. Some semirings generated fine-grained numerical attributions, while others emphasised broader structures or feature-based groupings. For instance, we examined the task of classifying the image “castle”. The input image is shown in Figure 1.  

Figure 1. Input image for the class “castle”. 
Groudiev, A. et al. 2025. Extending Layer-wise Relevance Propagation in Neural Networks using Semiring Annotations.
In Proceedings of the ProvenanceWeek 2025 (PW’ 25). Association for Computing Machinery, New York, NY, USA, 37–45. https://doi.org/10.1145/3736229.3736266. 

One semiring provides a more localized explanation and thus avoids including less relevant elements, such as the street light in the upper foreground. The corresponding relevance maps are shown in Figure 2. 

Figure 2. Relevance maps for the class “castle” using the counting semiring.
The counting semiring produces an explanation qualitatively similar to classical LRP. 
Limiting relevance propagation yields a more localized explanation that avoids including less relevant elements such as the street light near the top of the input image of the castle. 
Groudiev, A. et al. 2025. Extending Layer-wise Relevance Propagation in Neural Networks using Semiring Annotations. In Proceedings of the ProvenanceWeek 2025 (PW’ 25). Association for Computing Machinery, New York, NY, USA, 37–45. https://doi.org/10.1145/3736229.3736266. 

Testing also revealed practical applications. In one case, relevance information guided neural network pruning, helping us to remove less important components without significantly reducing performance.  

Toward a More Trustworthy AI 

This fresh approach gives us a broader view of how neural networks operate. Instead of treating an explanation as a single numerical process, the semiring approach provides a mathematical framework that unifies different styles of explanation. We can design semirings tailored to specific tasks or goals. For example, one semiring might emphasize causal relationships, while another highlights groups according to their features. 

This flexibility could be valuable in safety-critical applications, where different users require different explanations. Engineers may need detailed technical attributions, while end users may prefer more intuitive explanations. A semiring-based framework offers a way to accommodate these diverse needs within a single method. 

As artificial intelligence systems become more integrated into society, the demand for trustworthy models will only increase. Extending relevance propagation with semiring annotations is an important step toward that goal.  

Figure 2. Relevance maps for the class “castle” using the counting semiring.　　　　　　　　　　　　　　　　　　　　The counting semiring produces an explanation qualitatively similar to classical LRP. Limiting relevance propagation yields a more localized explanation that avoids including less relevant elements such as the street light near the top of the input image of the castle. 

Groudiev, A. et al. 2025. Extending Layer-wise Relevance Propagation in Neural Networks using Semiring Annotations. In Proceedings of the ProvenanceWeek 2025 (PW’ 25). Association for Computing Machinery, New York, NY, USA, 37–45. https://doi.org/10.1145/3736229.3736266. 

Testing also revealed practical applications. In one case, relevance information guided neural network pruning, helping us to remove less important components without significantly reducing performance.  

Toward a More Trustworthy AI 

This fresh approach gives us a broader view of how neural networks operate. Instead of treating an explanation as a single numerical process, the semiring approach provides a mathematical framework that unifies different styles of explanation. We can design semirings tailored to specific tasks or goals. For example, one semiring might emphasize causal relationships, while another highlights groups according to their features. 

This flexibility could be valuable in safety-critical applications, where different users require different explanations. Engineers may need detailed technical attributions, while end users may prefer more intuitive explanations. A semiring-based framework offers a way to accommodate these diverse needs within a single method. 

As artificial intelligence systems become more integrated into society, the demand for trustworthy models will only increase. Extending relevance propagation with semiring annotations is an important step toward that goal.  

(Date of interview: February 2, 2026)