Model distillation and quantization framework

If your goal is to turn a large, expensive model into a smaller, faster model, I'd start with Hugging Face's llm-compressor for quantization, and use Transformers or TRL for knowledge distillation and fine-tuning. If you need a more complete training pipeline, NVIDIA's NeMo is also worth evaluating.

The important distinction is that distillation and quantization solve different problems:

• Knowledge distillation trains a smaller student model to imitate a larger teacher. It can preserve useful behavior while reducing inference cost.
• Quantization reduces the precision of model weights (and sometimes activations), cutting memory use and often increasing inference speed.
• Combining them can yield a compact student model that is then quantized for deployment.

Best frameworks by use case

• 

1. Hugging Face TRL — best starting point for distillation

(My default pick)

Use its DistillationTrainer for on-policy knowledge distillation, where the student generates its own responses and learns from the teacher's token distributions. It integrates with the Hugging Face training ecosystem.huggingface.co+1

Best for: open-weight LLMs, experimentation, custom student models, Python workflows.

Documentation

---

• 

2. LLM Compressor — best for quantization

A strong choice when you want to compress a pretrained model without training a new architecture. Supports GPTQ, AWQ, INT8, FP8, low-bit weight formats, and additional compression techniques. It is designed to work with vLLM inference.docs.vllm.ai+1

Best for: lower GPU memory, production serving, throughput and latency optimization.

Documentation

---

• 

3. NVIDIA NeMo Framework — best for large-scale distillation

Provides configurable distillation recipes, including teacher/student logit matching and intermediate-state losses. More attractive when you have multi-GPU infrastructure and need a scalable training pipeline.docs.nvidia.com+1

Best for: large LLMs, multi-GPU training, enterprise-scale model development.

Distillation guide

What I'd actually implement

For a typical LLM project, I'd use a modular pipeline rather than look for one framework that does everything.

Teacher model

Strong model + representative prompts

Distill with TRL

Train a smaller student; evaluate task quality

Quantize with LLM Compressor

Test INT8, FP8 or 4-bit weights as appropriate

Benchmark with vLLM

Measure quality, latency, throughput and memory

This is a recommended combination, not a requirement that these tools always be used together.

One important caveat: quantization alone may be the better first step if your main problem is serving cost. You can often compress the existing model without paying the compute cost of distillation. If the model is still too slow or too large, distill a smaller student and quantize that.

How to choose the quantization level

LLM Compressor documents these formats and their hardware requirements; the fastest option depends on your GPU, workload, and serving configuration.github.com+2

Let me narrow this down to the best stack for you

The most useful recommendation depends on what you're compressing and where it'll run.

1. What kind of model?

Open-weight LLM (Llama, Qwen, etc.)

API-only teacher (closed-source model)

Vision or multimodal model

---

2. What's your main objective?

Lower inference cost

Lowest latency

Both cost and latency

Fit on limited VRAM or edge hardware

---

3. What hardware will you use?

NVIDIA GPU

AMD GPU

CPU or Apple Silicon

Cloud GPU, not decided

---

4. What's the teacher model size?

Under 7B parameters

7B–14B parameters

30B–70B parameters

Over 70B parameters

Unknown or proprietary

---

5. Do you have access to the teacher's weights?

Yes, I have the weights

No, API access only

Both weights and API

[Recommend my stack]