llama.cpp

llama.cpp

Original author(s)	Georgi Gerganov
Developer(s)	Georgi Gerganov and community
Initial release	Alpha ( b1083 ) / August 26, 2023 (2023-08-26)
Written in	C++
License	MIT License
Website	github.com/ggerganov/llama.cpp

llama.cpp is an open source software library that performs inference on various Large Language Models such as LLaMA. It is written in C++.

History

llama.cpp began development by Georgi Gerganov to implement LLaMA in pure C++ with no dependencies. The advantage of this method was that it could run on more hardware compared to other inference libraries that depended on hardware dependent closed source libraries like CUDA. It is written in C++. It currently has 55 thousand stars on GitHub. Before llama.cpp, Gerganov worked on a similar library called whisper.cpp which implemented OpenAI's "whisper" speech to text model. llama.cpp gained traction from users who did not have specialized hardware as it could run on just a CPU including on Android devices.

Architecture

llama.cpp initially could only run on CPUs but now can run on GPUs using multiple different back-ends including Vulkan and SYCL. These back-ends make up the GGML tensor library which is used by the front-end model-specific llama.cpp code. llama.cpp has its own model format called GGUF (previously referred to as GMML format). llama.cpp supports ahead of time model quantization as opposed to on-the-fly quantization.