How does it work?
We embrace the idea of integration from Integrated Gradients (IG) to create explanation maps that utilize both intermediate representations of the network and gradients. IIA diverges from IG in several aspects: First, IIA does not confine gradient computation to the input x. In fact, recent studies have suggested that gradients derived from internal activation maps can yield improved explanation maps. Secondly, IIA employs an iterated integral across multiple intermediate representations (such as activation or attention maps) generated during the network’s forward pass. This enables the iterative accumulation of gradients w.r.t. the representations of interest. Lastly, unlike IG, IIA does not restrict the integrand to plain gradients, but encompasses a function of the entire set of representations produced by the network and their gradients.
IIA is a generic approach that provides a solution for both CNN's and ViT's. On CNN's we use the activations of the network as the intermediate representations while on ViT we take the attention matrices.
While aggregating the activations is done easily with hadamard product on CNN's, on ViT we deal with multiple attention heads and blocks, thus we use an approach called Gradient Rollout (based on the fimiliar Attention Rollout) in order to aggregate the attention information with its gradients.
