Remember What You Want to Forget — Reading Notes
Cite this work
Show citation formats
Kennon Stewart (2025). Remember What You Want to Forget — Reading Notes.@inproceedings{2025_08_unlearning_reading_2025,
title={Remember What You Want to Forget — Reading Notes},
author={Kennon Stewart},
year={2025},
}The Lab’s Interest:
This is one of the more interesting papers on machine unlearning. Not only is the paper well-written—with heavy proofs relegated to the appendix—but the problem itself is deeply interesting. In an era of privacy regulation and data governance laws, clients need to be able to unlearn data just as they learn.
Summary
Generalization + resources, not just ERM:
The paper shifts unlearning from minimizing training loss to retaining test performance under deletions, explicitly accounting for both computation and storage costs in the unlearning process. : :
Separation from DP:
For convex losses, their algorithm can unlearn up to O(n/d^0.25) points while maintaining test loss guarantees, versus O(n/√d) via differentially private learning—showing a quadratic gap in dimension between unlearning and DP baselines. :
Mechanics & definition:
They formalize (ε, δ)-unlearning, and give an efficient scheme that stores compact statistics (e.g., a Hessian at the ERM) to perform a Newton-style correction plus calibrated noise—achieving guarantees without storing the whole dataset. : :
Reading Notes
User privacy is one of the strongest arguments against ubiquitous AI usage. When models are trained on large amounts of human data, how do we respect users’ right to be forgotten? There are two options: (1) retrain the model from scratch without the offending data, or (2) unlearn the influence of particular points via the same math that learned them.
One of these solutions is much more efficient than the other, and it isn’t the full retrain. Not only does retraining consume engineer time, but model training is notoriously taxing on the resources required to power a data center. Sekhari et al.’s proposal is a simultaneous response to both concerns.
I won’t bore you with the elegant mathematics behind convex optimization or Newton’s method on the Lab blog, but this is what it looks like to deploy mathematics to create machine learning systems that respect and serve the users from whom they learn.
Go ahead and give it a read. Shoot us an email at hi@secondstreetlabs.io with your thoughts — we read every response.