Estimating Training Data Influence by Tracing Gradient Descent
Garima Pruthi,Frederick Liu,Satyen Kale,Mukund Sundararajan
We introduce a method called TracIn that computes the influence of a trainingexample on a prediction made by the model. The idea is to trace how the loss onthe test point changes during the training process whenever the training example ofinterest was utilized. We provide a scalable implementation of TracIn via: (a) afirst-order gradient approximation to the exact computation, (b) saved checkpointsof standard training procedures, and (c) cherry-picking layers of a deep neuralnetwork. In contrast with previously proposed methods, TracIn is simple toimplement; all it needs is the ability to work with gradients, checkpoints, and lossfunctions. The method is general. It applies to any machine learning model trainedusing stochastic gradient descent or a variant of it, agnostic of architecture, domainand task. We expect the method to be widely useful within processes that studyand improve training data.
