paper:arxiv-1802-08232The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Original abstract (expand)
This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.
Similar preprints — Semantic Scholar
Cited by (2)
- From Memorization to Reasoning in the Spectrum of Loss Curvature
- Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
Larger models learn tasks smaller models never learn, and the cause is data-induced competition for neurons, not expressivity: small models spend capacity on frequent or simple tasks, and their update