TPDP 2024 Talk Notes

This page provides notes for my TPDP 2024 talk called "Data and Privacy in Data Privacy".

Training Data Attribution

Short Summary

Differential privacy, membership inference, and training data attribution are all research areas that care a lot about "counterfactual worlds" where certain examples are included or excluded. Right now, there is not much overlap between these areas (both in the people working on them and the techniques), and my claim here is that there should be.

Reading List

• Does Learning Require Memorization? A Short Tale about a Long Tail. This paper is one of my all time favorites and it's worth incorporating into your intuition. It led directly to Feldman and Zhang's paper What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.
• Membership Inference Attacks From First Principles. This paper takes inspiration from the Feldman and Zhang paper and proposes an at-the-time state-of-the-art membership inference attack. It's not the first paper using the Feldman and Zhang approach in membership inference, read the paper for more details.
• Datamodels: Predicting Predictions from Training Data. A very cool paper which improves upon the Feldman and Zhang influence estimation approach by treating it as a prediction problem. Their initial experiments use exactly the models that Feldman and Zhang trained!!
• TRAK: Attributing Model Behavior at Scale. A more efficient method for training data attribution. More recently, there's also LoGra and recent work from Anthropic on influence functions. For even more resources, see the ICML attribution tutorial.

Open Questions/Directions

• Can attribution be used to better characterize privacy leakage in private prediction? Starting points: "Private Prediction Strikes Back!", Auditing Private Prediction
• Can algorithms or theoretical tools from differential privacy be used to better understand attribution, either theoretically or empirically?
• Do any properties of membership inference also hold for training data attribution (or vice versa)? Examples: Privacy Onion Effect, Forgetting

Data Curation

Short Summary

Curating data has become very important to training state of the art models. However, there is limited investigation of the implications and opportunities of data curation for privacy.

Reading List

• Some non-private pretraining data selection papers: DataComp (CLIP) and DataComp (LLM), Llama 3. DataComps are competitions to produce the best pretraining data filtering recipe. Llama 3 is of course a model, but they describe (on a high level) how they did their training data preprocessing.
• Selecting pretraining data for private finetuning papers: Gradient Subspace Distance, Selective Pre-training for Private Fine-tuning, Prompt LLMs to Synthesize Data for Private Applications
• Privacy Side Channels in Machine Learning Systems. Data curation such as deduplication on private training data can lead to adaptive privacy attacks!
• Provable Privacy with Non-Private Pre-Processing. Shows how to account for privacy leakage from nonprivate data processing such as deduplication.
• The recent ICML best paper Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining has arguments which relate to topics in this section. It's worth reading and might lead to some other questions.

Open Questions/Directions

• Does curating private data help differentially private training? Or does having more data always help?
• What is the best way to do pretraining data selection for a private downstream task?
• Are there other data curation algorithms with negative privacy/security implications?

Privacy Semantics

Short Summary

The ML privacy literature has begun to consider different "privacy semantics". Often dealing more with access control-type approaches rather than differential privacy, the DP community's experience thinking about privacy may be helpful here.

Reading List

• Contextual integrity in LLMs: Can LLMs keep a Secret?, Contextual Integrity in Privacy-Conscious Assistants
• Contextual integrity + DP
• Retrieval-augmented LLMs (with applications to machine unlearning): SILO Language Models. The relevant research direction here is called retrieval-augmented generation (RAG). The backbone retrieval-augmentation technique for SILO is kNN-LM. Some other RAG flavors include RETRO and in context RAG (e.g. a, b, c). Most discussion of RAG these days refers to in context RAG techniques of some form. Language models for search such as Google Search Generative Experience, the Bing Chatbot, or Perplexity can be seen as a form of in context RAG, where the "retriever" is the search engine itself!

Open Questions/Directions

• Are there applications/threat models where combining some of these different privacy semantics (including DP) makes sense?
• Are there interesting attacks on these systems?