What Is GGUF? A Plain-English Guide to GGUF Models (and How to Load One in Layla)

If you've spent any time exploring local AI models on Hugging Face, you've probably noticed files ending in .gguf everywhere. So what is a GGUF model, and why does almost every offline AI app — Layla included — use this format?

This guide explains what GGUF means, how it works, and walks you through loading any custom GGUF model into Layla so you can run uncensored, role-playing, or specialized AI models directly on your Android phone — no internet, no subscription, no cloud.

What is GGUF?

GGUF is a file format for running large language models (LLMs) on consumer hardware like laptops, desktops, and phones. A single .gguf file contains everything needed to run an AI model — the model weights, the tokenizer, the prompt template, and the metadata — packed into one portable binary that can be loaded by any GGUF-compatible inference engine.

GGUF was introduced in August 2023 by the llama.cpp project, the same open-source inference engine that powers Layla. Before GGUF, the project used an older format called GGML, but that format required code changes every time someone wanted to add a new model architecture. GGUF replaced it with a structured metadata system, which is why it has since become the de facto standard for distributing locally-run LLMs.

If you've ever used Ollama, LM Studio, GPT4All, Jan, koboldcpp, or Layla — you've used GGUF, even if you didn't know it.

What does GGUF stand for?

GGUF stands for GGML Universal File. ("GGML" itself is the name of the underlying tensor library, named after its creator Georgi Gerganov.) You'll occasionally see it expanded as "GPT-Generated Unified Format" — both versions show up in the wild, but the llama.cpp project itself uses "GGML Universal File".

Why GGUF models matter for offline AI on mobile

GGUF's killer feature is quantization — a technique that shrinks a model's weights from 16-bit or 32-bit numbers down to 8, 4, or even 2 bits each. This makes the file dramatically smaller without destroying its intelligence, which is what makes it possible to run a 7-billion or 8-billion parameter model on a phone.

Concretely, this means GGUF lets you:

• Run a capable AI assistant entirely offline, with no internet connection

• Keep all your conversations private — nothing ever leaves your device

• Avoid subscriptions and rate limits forever

• Pick any model from the community, including ones fine-tuned for specific styles or with content filters removed

What you can do with custom GGUF models in Layla

The pre-built models Layla downloads on first launch are good general-purpose assistants. But Layla's real power comes from being able to load any GGUF model you want.

The open-source community has fine-tuned thousands of GGUF models for every imaginable use case:

• Uncensored / no-filter chat models that respond without the guardrails of mainstream chatbots

• Roleplay and creative writing models like Stheno, MythoMax, and Mahou — built for immersive long-form conversations

• Coding models specialized for programming languages

• Reasoning and math models for problem-solving

• Domain-specific models for medicine, law, language learning, and more

You can browse all GGUF models tested to work well with Layla at our Hugging Face page: huggingface.co/l3utterfly.

How to load a custom GGUF model into Layla

Here's the full walkthrough using the popular Stheno-Mahou roleplay model as an example.

Step 1 — Pick a model on Hugging Face

For this example we'll use Stheno-Mahou, a well-liked roleplay-focused fine-tune of Llama 3.

Step 2 — Open the "Files and versions" tab

This is where Hugging Face lists every downloadable variant of the model.

Step 3 — Pick the right quant for your phone

Each filename is annotated with a "Q" number, like Q2_K, Q4_K_M, Q6_K, or Q8_0. This is the quantization level — how aggressively the model has been compressed.

The rule is simple:

• Higher Q number = bigger file = better response quality, but needs more RAM and a faster phone.

• Lower Q number = smaller file = faster on weaker hardware, but slightly dumber responses.

A reasonable starting point on most phones is Q4_K_M. If it feels fast and responsive, try Q6 or Q8 for better quality. If it feels sluggish, drop down to Q3 or Q2.

You may also notice three special quants: Q4_0_4_4, Q4_0_4_8, and Q4_0_8_8. These are optimized for newer ARM phones with i8mm hardware acceleration — they can run noticeably faster on supported devices. Check our guide on Layla's i8mm hardware support to see if your phone qualifies.

Step 4 — Download the file

Tap the little download arrow next to the quant you picked. The .gguf file will save to your phone's Downloads folder (or wherever your browser drops files).

Step 5 — Add the model in Layla

Open Layla and go to Inference Settings → Add a custom model → Local file. Use the file picker to find the .gguf file you just downloaded.

In the file picker options, choose your model that you just downloaded.

Step 6 — Set the correct prompt format

This is the step people forget. Every model family expects prompts wrapped in a specific format (Llama 3 uses one, Mistral uses another, ChatML is a third, and so on). The model's Hugging Face page will tell you which format it expects. Set it in Layla's prompt format settings, and you're done — your custom GGUF model is now running fully offline on your phone.

Frequently asked questions about GGUF

What is a GGUF file?

A .gguf file is a single binary that packages an AI model's weights, tokenizer, and configuration together. It's the format llama.cpp and most other local-AI tools use to load and run language models.

What does GGUF mean in AI models?

When a model is listed as "GGUF" on Hugging Face, it means it's been converted into the GGUF format and is ready to run locally on consumer hardware via tools like Layla, llama.cpp, Ollama, or LM Studio — without needing a GPU server or cloud API.

Is GGUF better than safetensors or PyTorch?

They serve different purposes. PyTorch and safetensors are training/research formats — full precision, large files, GPU-oriented. GGUF is an inference format — quantized, compact, optimized for running on CPUs, phones, and modest GPUs. If you want to use a model rather than train it, GGUF is the better choice.

Can I run GGUF models on Android?

Yes — that's exactly what Layla does. Layla wraps llama.cpp on Android and lets you load any GGUF model from your device, or download new ones from Hugging Face.

What GGUF quant should I download?

Start with Q4_K_M. It's the most popular balance of size, speed, and quality. Move up to Q6 or Q8 if your phone handles it; move down to Q3 or Q2 if not.

Where can I find GGUF models?

The largest collection is on Hugging Face — search any model name with "GGUF" and you'll find quantized versions. For models tested to work well with Layla specifically, see huggingface.co/l3utterfly.