What AI says about OmniQuant

OmniQuant is a simple and powerful quantization technique for large language models (LLMs). It supports weight-only quantization (W4A16, W3A16, W2A16) and weight-activation quantization (W6A6, W4A4). It provides a pre-trained model zoo of quantized LLMs (LLaMA-1/2, LLaMA-2-Chat, OPT, Falcon, Mixtral-7Bx8) and out-of-the-box workflows, including using MLC-LLM to run LLaMa-2-Chat.