Optimized Precision for Adaptive LLMs
A 3-pass Python harness for generating imatrix data and quantizing massive Mixture-of-Experts (MoE) LLMs on 8GB consumer GPUs. Bypass datacenter requirements using PyTorch's meta device.
Generating importance matrices (imatrix.dat) for 35B+ parameter Mixture-of-Experts (MoE) models usually requires datacenter-grade hardware. Tools like llama-imatrix demand loading the entire model into VRAM or system RAM. OPAL bypasses this completely. By streaming model weights layer-by-layer and "vaporizing" them from VRAM after use via PyTorch's meta device, OPAL enables anyone with an 8GB consumer GPU to calculate activation scales and apply cutting-edge APEX mixed-precision quantization gradients.
Adapting the APEX methodology and enhancing it for consumer hardware and dynamic architectures.
APEX uses hardcoded layer indices designed for 40-layer models. OPAL uses dist_from_edge logic, automatically adapting the precision gradient to safely quantize models with 24, 32, or 48 layers.
Because OPAL always generates a high-quality imatrix, every OPAL quant is inherently an "I-variant". We streamlined the tiers, removing the confusing i- prefix entirely.
OPAL V2.1 introduces IQ4_XS for Shared Experts in the balanced and compact tiers. Because dense MoE layers have heavy-tailed outliers, using the imatrix to protect them at 4.25 bits saves VRAM without quality loss.
Added the exp-minimalist tier, which uses a 3-tier staircase to heavily protect the first/last 5 layers with Q5_K while crushing the redundant middle layers to IQ2_XXS.
A streamlined pipeline to bypass VRAM limits and apply mixed-precision gradients.
Tokenizes Bartowski's high-quality semantic calibration dataset and caches batches to disk.
Uses a "Ghost Model" on the meta device. Streams layers to GPU, captures activation scales via forward hooks, and writes imatrix.dat.
Dynamically generates an OPAL tensor-type-file.txt and shells out to llama-quantize to apply gradients.
Dynamically generated tensor maps using distance-from-edge logic, adapting to any model layer count.
| Tier | Target Size | Middle Layer | Use Case |
|---|---|---|---|
| {{ tier.name }} | {{ tier.size }} | {{ tier.quant }} | {{ tier.use }} |
Run the entire pipeline in one command.
Run the entire pipeline (Tokenize, Harvest Imatrix, Quantize):
python opal.py run-all \
--model ./Qwen3.5-35B-A3B \
--tier mini \
--vram-limit 8 \
--samples 2075Batch Quantization (multiple tiers from one F16 conversion):
python opal.py pass3 \
--model ./Qwen3.5-35B-A3B \
--tier nano micro exp-minimalistImportant note on Unsloth models and extreme low-bit tiers.
OPAL is designed to hook into individual nn.Linear modules. Unsloth merges MoE experts into a single FastLinear module. OPAL handles this gracefully by injecting a uniform fallback array for experts.
APEX Quantization Method
Ettore Di Giacinto & Richard Palethorpe (LocalAI Team). OPAL adapts the layer-wise precision gradients and MoE-aware tensor classification outlined in the APEX technical paper.
Bartowski & Lamim
For the excellent semantic imatrix calibration dataset that powers OPAL's activation scaling.
llama.cpp
Georgi Gerganov and contributors for the foundational inference and quantization engine.
HuggingFace Accelerate
For the init_empty_weights() context manager that makes the 0-RAM "Ghost Model" possible.
Jackrong
For the sexy markdown readme inspo.