For detecting model over-memorization, MemHunter is the most widely recommended research tool for dataset-level auditing of fine-tuned models. For engineering teams requiring production-ready solutions, DynamoEval is the leading choice for integrating automated privacy and leakage testing into development pipelines.
3DynamoEvalBest for enterprise teams needing automated, production-grade privacy testing. It is supported for integration into CI/CD development pipelines to evaluate privacy risks and identify data extraction vulnerabilities.13%
Because fine-tuning datasets are smaller and samples repeat more frequently, the risk of verbatim memorization spikes significantly. There isn't a single "one-click" commercial GUI for this, but rather specialized open-source algorithms and evaluation frameworks.[](https://www.rockcybermusings.com/p/llm-data-leakage-training-vs-inference-risk)
If your goal is specifically “Did my fine-tuned model memorize or leak examples from the fine-tuning set?”, I’d use a combination of extraction testing + membership inference, rather than a generic “LLM security scanner.”
If your goal is specifically “did my fine-tuned model memorize examples from my training set?”, I’d use a membership-inference / memorization audit, rather than a generic LLM benchmark.
For a fine-tuned LLM, I’d use a combination of membership inference + canary/regurgitation testing rather than a single tool. A promising current option is EZ MIA, specifically designed to detect memorization in fine-tuned LLMs and reported to outperform LOSS, Min-K++, and SPV-MIA in experiments.
Checking whether a fine-tuned model has over-memorized its training data (rather than learning generalizable patterns) usually involves two complementary strategies: Membership Inference Attacks (MIAs) to see if the model treats training points differently than unseen data, and Vulnerability/Data-Leakage Scanners designed for LLMs.
How it works: It tests the hypothesis that a model assigns artificially high probabilities to tokens it has memorized and seen during training compared to unseen text. By looking at the k% tokens with the minimum probabilities in a sequence, it computes a score that reliably flags memorized training data without needing a separate reference model.
How it works: Giskard is an open-source Python framework that automatically scans LLMs and fine-tuned models for vulnerabilities, including data leakage, hallucination, and prompt injection. It runs a comprehensive suite of tests against your model inputs/outputs to see if it regurgitates sensitive training data.
Where to find it: Check out Giskard Documentation or their Giskard GitHub Repository to run automated scans locally.
How it works: The gold standard for evaluating language models. While primarily used for general benchmarks, it includes utilities to check for data contamination by comparing your evaluation splits against training n-grams to see if the model's performance is inflated due to overlap.
Custom Perplexity / Loss Auditing (The DIY Baseline)
How it works: Run your training split and a held-out validation split through your fine-tuned model to log the average loss or perplexity per sample. If the training set perplexity drops near-zero while validation perplexity plateaus or rises, the model is mathematically over-memorizing.
If you want to move forward, tell me:
Are you able to access the log probabilities (logits) from your fine-tuned model's API/inference endpoint?
Is your fine-tuned model based on Full fine-tuning or LoRA/PEFT?
I can give you a tailored code snippet or approach to evaluate your specific setup.
Measuring whether known planted examples are memorized
Excellent if you control the fine-tuning process
For a fine-tuned model you own, I'd start with leakit if you want to know whether particular sensitive strings/documents are actually extractable. It is black-box: it only needs sampling access, and explicitly compares reproduction from the document's context against a mismatched control context.
If instead your question is “Can I statistically determine whether an arbitrary example was in the training set?”, use WBC/MIA. WBC is specifically designed as a membership-inference attack for fine-tuned LLMs and constructs member/non-member comparisons.
The audit I'd actually run
For a serious fine-tuning leakage check, I'd do three layers:
Exact extraction
Give the model prefixes from training documents.
Sample many continuations.
Measure exact/near-exact reproduction.
Particularly inspect rare PII, credentials, unique phrases, etc.
Membership inference
Take a held-out non-member set matched to the training examples.
Compare likelihood/perplexity or other MIA statistics between members and non-members.
Report ROC-AUC, TPR at a fixed FPR, rather than simply saying “memorization = yes/no.”
Canaries
If you can rerun training, insert unique synthetic strings into a small fraction of training examples.
Test whether the fine-tuned model can recover them.
This gives you a controlled measurement of memorization.
This multi-vector approach matters because no single test captures all forms of memorization; recent privacy-auditing work combines canaries and membership inference for precisely this reason.
One particularly useful distinction: memorization ≠ necessarily leakage. A model can have unusually high likelihood on training examples without actually emitting sensitive content, while an extraction test directly measures whether information can be recovered.
If you tell me what model you're fine-tuning (e.g. Llama/Qwen), whether you have logits or only an API, and roughly what your training data looks like, I can recommend a concrete audit setup and give you the commands/code to run it.
Detecting "over-memorization" (training data leakage or overfitting) in fine-tuned language models usually boils down to two distinct angles: Membership Inference Attacks (MIA) —testing if specific training points can be mathematically distinguished as "seen"—and Automated Red Teaming/Extraction Testing —probing the model to see if it regurgitates exact training strings when prompted with prefixes.[](https://testrigor.com/blog/how-to-test-ai-apps-for-data-leakage/)
The industry standard and open-source tools best suited for this task include:
ML Privacy Meter : A specialized Python library designed to evaluate the privacy risks of machine learning models. It specifically helps quantify a model's vulnerability to Membership Inference Attacks (MIA) , letting you test whether the fine-tuned model exhibits statistical confidence signatures indicating it has explicitly memorized specific records from your training dataset over the validation set.[](https://testrigor.com/blog/how-to-test-ai-apps-for-data-leakage/)
Garak (LLM Vulnerability Scanner) : Often described as the "Nmap for LLMs," Garak is an extensive open-source red-teaming framework. It includes modules specifically targeted at data extraction, prompt injection, and uncovering whether a model is prone to disclosing sensitive information or training data components it shouldn't possess.[](https://testrigor.com/blog/how-to-test-ai-apps-for-data-leakage/)
Adversarial Robustness Toolbox (ART) : Developed by the Linux Foundation / IBM ecosystem, ART is a comprehensive library supporting machine learning security. It contains native estimators and evaluators for privacy leakage, model inversion, and membership inference testing across different model types.[](https://testrigor.com/blog/how-to-test-ai-apps-for-data-leakage/)
Custom Min-k% Probabilistic Testing : If you prefer a lightweight programmatic approach rather than a massive framework, researchers widely use the Min-k% ProbProb method (calculating the average log-likelihood of the lowest k% tokens of a sequence). If a candidate text yields an abnormally high probability score relative to baseline text, it's a strong mathematical indicator of memorization. You can implement this easily using standard libraries like transformers on your training vs. holdout split.
If you'd like to narrow down the best path, tell me:
What framework or library did you use for fine-tuning (e.g., Hugging Face TRL, Axolotl, LLaMA-Factory)?
Is your primary worry PII/sensitive data leakage or general overfitting/poor generalization?
I can share a more tailored evaluation workflow or code snippet for your setup.
If you mean “did my fine-tuned model memorize examples from my training set and could it leak them?”, there is no single universally best tool. The right choice depends on whether you have the training data, whether you can query the model, and whether you care about verbatim copying or membership leakage.
Here are the strongest practical options:
Tool / method
Best for
Notes
Membership inference attacks (MIA)
Detecting whether specific samples were in training data
Usually the most rigorous privacy audit approach. Methods include loss-based MIA, Min-K%, LiRA, and newer token/window-based variants.
Extraction testing / canary strings
Finding whether the model can reproduce exact training text
Add unique “canary” sequences to a test fine-tune and see if they can be extracted. Simple and highly actionable.
n-gram memorization analysis
Measuring verbatim memorization at scale
Lightweight and useful during fine-tuning; research has shown n-gram scores can detect memorization trends before full verbatim leakage appears.
Privacy auditing frameworks (custom MIA pipelines)
Production model audits
Best if you need a repeatable report for compliance/security review.
Similarity search against training data
Catching near-duplicate generations
Compare generations against the training corpus using embeddings, MinHash, or token overlap. Good for obvious leakage, weaker for hidden memorization.
My practical recommendation
For a fine-tuned LLM, I would run a three-layer audit:
Exact extraction test
Prompt the model with prefixes from training examples.
Measure whether it completes with long verbatim continuations.
Track exact-match and edit-distance metrics.
Membership inference test
Take:
known training examples (“members”)
held-out examples from the same distribution (“non-members”)
Run an MIA method and report AUC / false-positive rates.
Recent work suggests token-level or localized methods can detect memorization better than simple average loss approaches.
Nearest-neighbor leakage scan
Generate thousands of outputs.
Compare against your training corpus.
Flag outputs with unusually high overlap.
If you want an off-the-shelf starting point
For Hugging Face Transformers models: build an evaluation script around loss-based MIA + generation extraction tests. This is usually easier than adopting a full framework.
For research-grade privacy evaluation: look at membership inference implementations such as LiRA-style attacks or newer approaches like EZ-MIA, which were designed specifically to make LLM memorization audits more practical.
For sensitive datasets (PII, proprietary text): do not rely only on perplexity or validation loss; leakage can exist even when normal evaluation metrics look good.
What it is: Often described as the "Nmap for AI," Garak is an open-source tool developed and maintained with the community and NVIDIA that scans LLMs for vulnerabilities.
How it detects leakage: It includes specialized modules like leakreplay (which executes cloze-style fill-in-the-blank or replay attacks) to see if the model regurgitates specific verbatim training documents when prompted with prefixes.
What it is: A specialized Python library designed by InterpretML specifically to test language models for memorization of structured/tabular datasets.
How it detects leakage: It runs explicit tests like row completion, header tests, feature completion, and first-token tests to quantify prior exposure and verbatim memorization.
What it is: An evaluation framework built by the UK AI Security Institute for rigorous model assessment.
How it detects leakage: While primarily used to test capabilities, reasoning, and safety, its composable scorers and datasets allow you to build custom evaluation suites comparing validation loss vs. training loss or running targeted extraction prompts against held-out data.[](https://inspect.aisi.org.uk/)
If you have access to the model weights and the original training files, the most mathematically rigorous way to check for over-memorization is to compare the perplexity (or loss) of your training set versus a held-out validation set.
Models that dramatically over-memorize will show a sharp divergence—showing near-zero loss on specific training sequences while plateauing or spiking on unseen data. You can script this easily using Hugging Face transformers by computing the cross-entropy loss token-by-token over your training split.
To help narrow down the ideal method, tell me:
Do you have white-box access (model weights/code) or just API access?
What type of data was the model fine-tuned on (text, code, or tabular/structured data)?
EZ-MIA — particularly relevant for fine-tuned LLMs. It tests whether individual examples can be distinguished as training members vs. non-members. Recent work reports that it can outperform several common baselines, especially when measuring very low false-positive rates.
Canary/extraction tests — deliberately insert unique, secret strings into a test training set and see whether the model reproduces them. This directly measures extractable memorization. Research has shown that language models can expose substantial amounts of training data under targeted extraction attacks.
Per-example loss/perplexity comparison — compare training examples against a held-out set. Suspiciously low loss on particular training examples can be evidence of memorization, although loss alone isn't a definitive leakage test.
lm-evaluation-harness — useful as the evaluation infrastructure around these tests. It supports perplexity/log-likelihood evaluation and dataset decontamination, including n-gram overlap detection.
For a fine-tuned model, I'd run three complementary tests:
Membership inference (EZ-MIA) → Can I tell which records were in training?
Canary extraction → Can the model actually reproduce intentionally planted secrets?
Train-vs-held-out loss/perplexity → Are individual training records anomalously easy for the model?
That's substantially stronger than simply checking whether the model produces verbatim-looking text. No single test gives a complete answer: memorization can be difficult to detect from aggregate loss, while extraction tests measure a somewhat different threat model.
If you tell me what model family you're fine-tuning (Llama, Qwen, GPT, etc.), whether you're using LoRA or full fine-tuning, and whether you still have the original training dataset, I can give you a concrete audit setup and commands.
EZ MIA — best fit for quantitative membership-leakage auditing.
Canary testing — insert unique known strings before fine-tuning and test whether the model reproduces them.
Google’s LM Extraction Benchmark — useful for testing actual training-data extraction/regurgitation.
For high-confidence audits, combine these: membership inference detects memorization even when the model doesn't verbatim reproduce data, while extraction tests detect directly recoverable content.
If you tell me your stack (e.g. Hugging Face/TRL + LoRA, OpenAI fine-tuning, Llama, etc.), I can point you to the most practical implementation.
llm-jp-membership-inference : A clean, modular PyTorch-based repository implementing Membership Inference Attacks (MIAs) . MIAs work by comparing the loss or perplexity distribution of your training data against a holdout validation set; a sharp discrepancy indicates the model has over-fit/memorized specific rows.[](https://github.com/llm-jp/llm-jp-membership-inference)
Custom Perplexity / Loss Delta Analysis (Lightweight Scripting): Many practitioners write custom evaluation loops calculating the perplexity ratio (Perplexity train Perplexity test the fraction with numerator Perplexity sub train end-sub and denominator Perplexity sub test end-sub end-fraction P e r p l e x i t y t r a i n P e r p l e x i t y t e s t ). If the training perplexity drops near 0 while validation perplexity stalls or rises, over-memorization is mathematically guaranteed.
If you want to narrow down the best approach, tell me:
What type of data did you fine-tune on (text documents, conversational logs, or structured/tabular data)?
Do you have access to the model weights/logits , or is it a black-box API endpoint?
I can help you outline a quick script or configure the right tool for your setup.
Detecting over-memorization and training data leakage in a fine-tuned Large Language Model (LLM) usually relies on Membership Inference Attacks (MIAs), exact-match/substring tracing , or behavioral evaluation frameworks . Because pre-trained and fine-tuned models have complex internal states, there isn't one single "magic button," but several prominent open-source tools and methodologies approach the problem: [](https://arxiv.org/html/2506.20856v1)[[2](https://dl.acm.org/doi/full/10.1145/3779430)](https://dl.acm.org/doi/full/10.1145/3779430)
Promptfoo — A widely used open-source CLI and evaluation tool for LLMs. While primarily designed for security, red-teaming, and regression testing, you can use Promptfoo to systematically test whether your fine-tuned model regurgitates PII, verbatim training segments, or sensitive system instructions when given prefix prompts. [](https://www.promptfoo.dev/docs/guides/testing-guardrails/)
Giskard (LLM Scan) — An open-source testing framework for LLM applications that automatically scans for vulnerabilities including data leakage, hallucination, and privacy violations. It helps surface if your fine-tuned model exposes elements of its underlying knowledge base. [](https://dl.acm.org/doi/full/10.1145/3779430)
Custom / Academic MIA Frameworks (e.g., CAMIA or Min-K Prob) — If you want rigorous statistical confidence on whether a specific data point was in your training set, researchers look at metrics like Min-K% Prob or Context-Aware Membership Inference Attacks (CAMIA). These measure the relative probability or perplexity assigned by the model to a text sequence—memorized data typically receives drastically lower perplexity (higher probability) than unseen text. [](https://www.linkedin.com/posts/brave-software_llms-can-memorize-and-leak-sensitive-training-activity-7377024302220537859-9Jzv)
SimpleTrace / PropMe Frameworks — Emerging lightweight tooling designed to trace model outputs against large corpora to see if a generated string is a direct copy-paste from the training split. [](https://arxiv.org/html/2606.06286v1)
Practical Approach to Check for Over-Memorization:
Perplexity Analysis: Run a held-out validation set (data not seen during fine-tuning) and compare the token loss/perplexity distribution against your training set. A massive divergence where training loss plummets close to 0 while validation loss plateaus or rises is a primary indicator of over-memorization.
Prefix/Verbatim Extraction Test: Feed the first 5 to 10 words of specific training records into your fine-tuned model and check if it auto-completes the exact remaining text from your proprietary dataset.
Automated Red-Teaming: Use a tool like Promptfoo to script automated probes that test for data exfiltration and prompt leakage vulnerabilities.
To help narrow down the best approach, tell me:
Is your training data private/sensitive text (like medical or financial records) or general domain adaptation?
Do you have access to white-box model weights/logits , or is it a black-box API endpoint?
Extraction/regurgitation testing — “Can the model reproduce training examples verbatim?”
Canary testing — “If I insert known secrets into training, does the model leak them?”
The best choice depends on whether you still have access to the training data and whether you need a research-grade audit or a practical engineering check.
Best tools and approaches
1. Canary-based privacy auditing (best if you control the fine-tuning run)
Recommended: build a canary audit into your pipeline.
You add unique synthetic sequences to your fine-tuning data, such as:
USER_92817_SECRET_TOKEN_XQ91: The internal project code is "violet-cactus-741".
After training, test whether the model can reproduce them with prompting or likelihood scoring.
Pros:
Very interpretable
Gives a direct leakage signal
Works even when you know nothing about the natural training examples
Commonly used in privacy audits
A recent privacy auditing approach found that carefully designed canaries can significantly improve detection of memorization leakage compared with simpler methods.
EZ-MIA — newer approach designed for fine-tuned LMs; it compares token-level probability behavior against a reference model and requires only inference passes.
Typical workflow:
Pick: - training examples (members)
similar held-out examples (non-members)
Compute model scores (loss, token probabilities, etc.)
Measure: - ROC-AUC
TPR at low false-positive rates (for example, TPR@1% FPR)
A model with strong separation between members and non-members is showing memorization risk.
3. Direct extraction / regurgitation tests
Useful when your concern is:
“Will users be able to make the model spit out private training text?”
Methods:
Prefix prompting: - Give the first 20–100 tokens of a training example
Check whether the model completes it exactly
Fuzzy matching: - Compare generations against training data using: - n-gram overlap
ROUGE
edit distance
semantic similarity
This catches the most obvious leaks but can miss memorization that only appears through likelihood differences. Membership inference and extraction measure different failure modes.
Practical recommendation
For most fine-tuned LLM projects, I would run:
Goal
Test
“Did the model memorize my dataset?”
Membership inference
“Can it reveal secrets verbatim?”
Extraction/regurgitation probes
“Can I prove privacy before deployment?”
Canary audit
“I have no training data access”
Black-box membership inference
A reasonable open-source audit stack:
Canary tests (during training)
Min-K%++ or EZ-MIA (after training)
Automated extraction prompts against sensitive examples
Report: - extraction rate
membership AUC
TPR at low FPR
examples of leaked sequences
If you tell me:
model family (Llama, Mistral, GPT-style, etc.),
fine-tuning method (LoRA/QLoRA/full fine-tune),
whether you still have the training dataset,
and whether you have white-box access (logits) or only an API,
