llama.cpp

Why it matters

When capable open models appeared, the question was who could actually run them. llama.cpp's answer — a dependency-light C++ codebase with aggressive quantization — put LLM inference on hardware people already owned. It bootstrapped the entire local-AI ecosystem: Ollama, LM Studio, and countless tools sit on its foundations.

Engineering notes

• Quantization as a first-class design. K-quants trade precision for memory with unusual care; the format work (GGUF) standardized what everyone else builds on.
• Portable by construction. CPU-first with optional GPU backends (CUDA, Metal, Vulkan) — the same binary adapts to the hardware it finds.
• The ggml tensor library. A minimal, dependency-free tensor core underneath — the project builds its own foundations rather than inheriting framework bloat.

Limitations

• It's infrastructure, not an application — expect to build your own serving, batching, and eval layers on top.
• The pace of model-architecture support means bleeding-edge models can lag by days or weeks.