LLM inference & evaluation
Building repeatable LLM workflows on LANTA and measuring what changes when a 30B model is quantized.
Measured total evaluation time for the Sarvam-30B baseline and AWQ 4-bit variant across the combined benchmark.
The problem
Large-model experiments need repeatable environments and predictable inference workflows. Quantization can improve efficiency, but an overall speed result alone does not show whether task performance remains acceptable.
What I built
I built LLM fine-tuning and inference pipelines using Apptainer, DeepSpeed, SLURM, vLLM, and Ollama for 4B–70B+ models on Thailand's LANTA HPC cluster. Modular SLURM templates and FastAPI/WebSocket inference APIs supported consistent experiments across multi-node clusters.
The Sarvam-30B benchmark
I compared the baseline and AWQ 4-bit variants across 4,683 MATH-500 and MedMCQA test cases. Total evaluation time fell from 145 to 116 minutes, approximately 1.25× faster.
- MATH-500 accuracy: 76.4% baseline → 72.0% AWQ, a 4.4 percentage-point drop.
- MedMCQA accuracy: 36.7% baseline → 17.8% AWQ, an 18.9 percentage-point drop.
What the results mean
The efficiency gain came with very different accuracy costs across the two tasks. These results support evaluating quantization against the intended workload rather than treating a faster aggregate evaluation as a universal improvement. The time measurement is evaluation runtime, not a claim about serving throughput or time to first token.