Unsloth vs Axolotl vs LLaMA-Factory

The fine-tuning stack nobody agrees on

🧭 Part 3 of the 📚 LLM Fundamentals course

You open GitHub to pick a fine-tuning framework and find three tools with tens of thousands of stars each, three Reddit threads that say three different things, and twelve Medium posts that contradict each other. One thread swears Unsloth is the only sane choice. Another says LLaMA-Factory’s web UI is the fastest path. A third insists Axolotl is the only one that works at scale. Three hours later you’ve closed all the tabs, poured a second coffee, and know less than when you started.

Your team already decided that fine-tuning is the right call: the base model hallucinates on your domain data, prompt engineering hit a wall at 72% accuracy, and RAG is overkill for your classification task. We already covered when fine-tuning makes sense a few weeks ago.

I’ve run QLoRA jobs on all three this year. Each one solved a different problem. None of them solved all three.

TL;DR

• Use Unsloth if you’re on a single GPU and speed matters. 2x faster training, 70% less VRAM. Pip install, open a notebook, train. Don’t use if you need multi-GPU distributed training (free tier is single-GPU only).

• Use Axolotl if your team runs multi-GPU clusters and needs FSDP/DeepSpeed with full RLHF pipelines. YAML config, Docker recommended, 30 minutes to first run. Don’t use if you want a point-and-click experience.

• Use LLaMA-Factory if you’re not sure. Broadest model support (100+ templates), zero-code web UI, and it can use Unsloth as a backend for speed. Don’t use if you need granular control over training internals.

• Torchtune earns an honorable mention for teams deep in the PyTorch ecosystem who want maximum control over training loops. We cover it at the end.

• The counterintuitive finding: LLaMA-Factory with Unsloth backend trains within 6% of native Unsloth speed. You often don’t have to choose between ease and performance.

What Are We Even Comparing?

Fine-tuning frameworks sit between you and the raw PyTorch training loop. They handle the plumbing: loading model weights, applying parameter-efficient methods like LoRA, managing GPU memory, formatting datasets, and exporting the result. The alternative is wiring this yourself with raw HuggingFace Transformers, PEFT, and TRL: 200+ lines of boilerplate per experiment, no built-in dataset formatting, no VRAM optimization, and every hyperparameter set by hand.

Scope framing: We’re comparing the top 3 open-source fine-tuning frameworks for local/self-hosted training. We’re NOT covering hosted fine-tuning APIs (OpenAI, Together AI, Anyscale), managed platforms (AWS SageMaker, Vertex AI), or the lower-level HuggingFace libraries (PEFT, TRL) that these frameworks build on. If you’re using an API provider’s fine-tuning endpoint, this comparison isn’t for you.

⚠️ Confusion Alert: “Fine-tuning framework” vs “fine-tuning method.” LoRA, QLoRA, and full fine-tuning are methods (how you update weights). Unsloth, Axolotl, and LLaMA-Factory are frameworks (the tools that implement those methods). Every framework here supports multiple methods. The framework choice is about workflow, speed, and scale. The method choice is about VRAM and quality tradeoffs.

Evaluation framework: When choosing a fine-tuning framework, the decision comes down to 4 factors:

1. Training speed (tokens per second, wall-clock time to completion)

2. VRAM efficiency (can you train the model you need on the GPU you have?)

3. Flexibility (model coverage, training methods, multi-GPU, multimodal)

4. Ease of use (time from zero to first training run)

GitHub stars, Reddit sentiment, and "who's using it" marketing pages don't make this list. Each tool below gets evaluated against these four dimensions.

Head-to-Head Breakdown

Tools are ordered simplest to most complex.

1. LLaMA-Factory

In one sentence: The framework that makes fine-tuning feel like filling out a form, with the broadest model coverage in the ecosystem.

• Speed ●●●○ (fast w/ Unsloth backend)

• VRAM ●●●○

• Flexibility ●●●● (most models)

• Ease ●●●●

👍 The good:

• Lowest barrier to entry in the category. LlamaBoard, the built-in web UI, lets you select a model, upload a dataset, pick a training method, and click “Start.” No YAML, no Python, no config files. The framework supports SFT, DPO, PPO, GRPO, reward modeling, and pre-training through that same interface, backed by an ACL 2024 paper.

• 100+ model templates out of the box. Llama 4, Qwen 3, DeepSeek, Gemma, GPT-OSS, Phi-4, Mistral, GLM: if a model exists on HuggingFace, LLaMA-Factory probably has a template for it. Each template handles the tokenizer quirks, chat formatting, and special tokens that would otherwise take you an afternoon to debug.

• Unsloth as an optional backend. LLaMA-Factory can use Unsloth’s optimized kernels as its training backend. In benchmarks, this brings LLaMA-Factory’s training time within 6% of native Unsloth (3.4 hours vs 3.2 hours for Llama-3.1 8B QLoRA on A100 40GB), while keeping all of LLaMA-Factory’s UI and model management features¹.

👎 The bad:

• Abstraction hides debugging. When training breaks, LLaMA-Factory's error messages point to internal framework code, not your configuration mistake. The web UI doesn't expose advanced hyperparameters like gradient accumulation steps or warmup ratios, so you end up editing config files anyway.

• Multi-GPU support exists but lags. FSDP and DeepSpeed are supported through CLI configs, but the web UI can't configure distributed training. Once you need sequence parallelism or tensor parallelism across 4+ GPUs, you're writing CLI commands that bypass LlamaBoard entirely.

Quickstart (3 commands to first training run):

pip install llamafactory
llamafactory-cli webui   # launches LlamaBoard at localhost:7860
# Select model → Upload dataset → Click "Start" → Done.

🎯 Best for: Solo practitioners, small teams, or anyone who values breadth of model support and speed to first training run over fine-grained control.

⚠️ The ceiling: Migrate away when you need custom training loops, advanced distributed training strategies (sequence parallelism, tensor parallelism), or when debugging the abstractions costs more time than they save.

📡 Practitioner signal: The LLaMA-Factory paper was accepted at ACL 2024 and the repo has been used to train models like NovaSky’s Sky-T1 reasoning model². It reported 25K+ stars and 3,000+ forks as of the paper’s publication and has grown since.

2. Unsloth

In one sentence: The speed demon that rewrites GPU kernels from scratch to make single-GPU fine-tuning absurdly fast and memory-efficient.

• Speed ●●●● (fastest)

• VRAM ●●●● (lowest)

• Flexibility ●●●○ (single-GPU focus)

• Ease ●●●○

👍 The good:

• 2x faster training, 70% less VRAM. Unsloth writes custom Triton kernels for attention, MLP layers, and RoPE embeddings that bypass the standard HuggingFace Transformers path entirely. On a benchmark with Llama-3.1 8B QLoRA (rank 32, A100 40GB), Unsloth finished in 3.2 hours. Axolotl took 5.8 hours on the same hardware³.

• Consumer GPUs become viable. The VRAM savings from those fused kernels mean a 7B model fits in QLoRA on a single RTX 4090 (24GB), and a 70B model fits on a single A100 80GB. Before Unsloth, both required multi-GPU setups or cloud instances with 160GB+ aggregate memory.

Not sure what separates an RTX 4090 from an A100? We broke that down in What Does NVIDIA Actually Do?

• GRPO for reasoning on consumer hardware. GRPO trains models to generate chain-of-thought reasoning by comparing multiple outputs against each other, instead of using PPO's separate critic network to score each one. No critic means roughly half the VRAM. Unsloth supports GRPO for models from 1B to 70B parameters.

• MoE fine-tuning. Mixture-of-Experts models route each input to a subset of specialist layers instead of activating every parameter. This makes them cheaper to run at inference but harder to train. Unsloth's custom MoE kernels cut that training cost with 7x speed improvement and 36% VRAM reduction on models like Qwen3-30B-A3B (NVIDIA B200)

👎 The bad:

Multi-GPU requires paying. Unsloth’s speed comes from custom Triton kernels written per-operation. Those kernels target single-GPU execution paths. Distributing them across multiple GPUs means rewriting the memory management and communication layers, which is why multi-GPU lives in the paid Unsloth Pro tier and Unsloth Studio, not the open-source repo. If you need FSDP across 8 GPUs, you’re looking at Axolotl.

New architectures arrive on a delay. Every model family has different attention patterns, activation functions, and tensor layouts. Unsloth’s kernels are hand-optimized per architecture, so a new model family means new kernel work. Mainstream models (Llama, Qwen, Mistral) land fast. Niche architectures with unusual routing or attention patterns can lag weeks behind LLaMA-Factory, which wraps stock HuggingFace Transformers and inherits model support automatically.

Quickstart (5 lines to trained model):

from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    "unsloth/Llama-3.1-8B-Instruct",
    max_seq_length=2048, load_in_4bit=True
)
# Add LoRA adapters, load dataset, train. Full notebook: unsloth.ai/docs

🎯Best for: Individual researchers, startups with limited GPU budgets, anyone who needs to train fast on one or two GPUs without renting a cluster.

⚠️ The ceiling: Migrate away (or complement with Axolotl) when you need multi-node distributed training, complex RLHF pipelines with reward models, or when your team needs reproducible training across different cluster configurations.

📡 Practitioner signal: NVIDIA features Unsloth as the go-to framework for RTX AI PCs and DGX Spark, and Red Hat ships it as the backend for their enterprise Training Hub product.

3. Axolotl

In one sentence: The YAML-configured powerhouse built for teams that need the full ML training pipeline: multi-GPU distribution, RLHF, multimodal, and sequence parallelism.

• Speed ●●○○ (slower single-GPU)

• VRAM ●●○○

• Flexibility ●●●● (most complete)

• Ease ●●○○

  

👍 The good:

• The most complete multi-GPU story. When a model doesn’t fit on one GPU, you need a strategy for splitting it across multiple. FSDP2 shards model parameters across GPUs so each one holds a fraction of the weights. DeepSpeed ZeRO does the same but also shards optimizer states and gradients, freeing even more memory. Sequence Parallelism splits long input sequences across GPUs, which is how Axolotl handles context lengths beyond 32K tokens. Axolotl supports all three, plus ND Parallelism that composes them within and across nodes.

• Full RLHF pipeline in one tool. Most fine-tuning projects start with SFT (supervised fine-tuning on examples), then add DPO or PPO to align the model with human preferences using ranked output pairs. Running SFT in one framework and DPO in another means converting checkpoints, reformatting datasets, and debugging two config systems. Axolotl runs SFT, DPO, PPO, GRPO, reward modeling, and process reward modeling in the same YAML config format, so the two-stage SFT-then-DPO workflow shares a single dataset pipeline and checkpoint format.

• Mature multimodal support. Vision-language models process both images and text, which means the fine-tuning framework needs to handle two different tokenization paths, align image patches with text tokens, and manage the extra VRAM from image encoders. Axolotl supports LLaMA-Vision, Qwen2-VL, Pixtral, LLaVA, Gemma 3n, InternVL 3.5, and the audio model Voxtral. VRAM usage: 38GB for QLoRA on Qwen2-VL 7B, 62GB for full fine-tuning, both on A100 80GB⁴.

• Sample packing for throughput. Axolotl packs multiple short training examples into a single sequence, reducing wasted padding tokens and increasing effective throughput. On datasets with variable-length examples (common in instruction tuning), this can cut training time by 20-30% compared to naive batching.

👎 The bad:

• Slower on single GPU. Axolotl wraps HuggingFace Transformers with additional abstraction layers for config parsing, dataset preprocessing, and distributed training setup. Those layers run even when you’re on a single GPU, adding overhead that Unsloth’s direct-to-kernel approach skips entirely. On the same Llama-3.1 8B QLoRA benchmark (A100 40GB), Axolotl took 5.8 hours to Unsloth’s 3.2 hours. On multi-GPU clusters the overhead is amortized across GPUs and barely matters.

• Configuration complexity. A full Axolotl RLHF config can run 100+ lines of YAML. The failure mode is silent misconfiguration: a misspelled field name doesn’t error, it gets ignored, and you train with default hyperparameters without realizing it. Debugging requires diffing your config against the framework’s default values in source code.

• Logging breaks inside tasks. Axolotl wraps training in Task objects that capture stdout. Standard print and logging calls inside those objects don’t always surface to your terminal, so you can’t see training progress or debug intermediate outputs without patching the logging setup.

Quickstart (YAML config to training run):

# lora-8b.yml
base_model: meta-llama/Llama-3.1-8B-Instruct
adapter: lora
lora_r: 32
lora_alpha: 16
datasets:
  - path: your_data.jsonl
    type: sharegpt
micro_batch_size: 2
num_epochs: 3
optimizer: adamw_bnb_8bit
axolotl train lora-8b.yml

🎯 Best for: Teams with multi-GPU clusters, projects requiring RLHF/DPO alignment pipelines, multimodal fine-tuning, or training with very long context windows.

⚠️ The ceiling: You probably don’t outgrow Axolotl. You outgrow the YAML config approach and write custom training scripts directly in PyTorch. That’s when Torchtune becomes interesting.

📡 Practitioner signal: Modal, the serverless GPU platform, maintains an official Axolotl integration and recommends it as the default for beginners in their fine-tuning guide: “If you are a beginner: Use Axolotl. If you have limited GPU resources: Use Unsloth.”⁵

The Decision Flowchart

The flowchart captures the two key decision points:

Decision 1: Single GPU or multi-GPU? This eliminates half the options immediately. If you’re on a single GPU (which is most people starting out), you’re choosing between Unsloth’s speed and LLaMA-Factory’s breadth. If you’re running a cluster, Axolotl and Torchtune are your contenders.

Decision 2: Do you need a web UI? / Do you need PyTorch-native control? Within each branch, this separates the tools by philosophy. LLaMA-Factory’s UI is for speed to first run. Unsloth’s notebook-first approach is for speed to trained model. Axolotl’s YAML is for team reproducibility. Torchtune’s recipes are for researchers who want to modify the loop.

The escape hatch at the bottom matters: if you’re using OpenAI or Anthropic’s API and don’t need to self-host, their hosted fine-tuning endpoints skip all of this entirely. No GPU management, no framework choice, no VRAM math. The tradeoff is cost at scale and less control over the training process.

Comparison Table

All benchmarks below use Llama-3.1 8B QLoRA (rank 32) on A100 40GB unless noted. Green badges mark the category leader.

The Honest Take

You don’t have to pick one. LLaMA-Factory with Unsloth backend gives you the broadest model coverage and web UI with Unsloth’s kernel-level speed. The training time penalty is only ~6% vs native Unsloth. For multi-stage projects, Unsloth handles SFT (compute-bound, benefits most from kernel speed), then Axolotl handles DPO/RLHF alignment (requires a reference model alongside the training model, doubling VRAM, which is where multi-GPU distribution pays off).

The category caught up to itself. In 2024, Unsloth didn’t support GRPO, Axolotl didn’t have sequence parallelism, and LLaMA-Factory couldn’t use Unsloth as a backend. Those gaps forced your framework choice. By 2026, all three support LoRA, QLoRA, full fine-tuning, GRPO, DPO, and vision models. What remains different is workflow: how you configure, debug, and distribute training.

Data quality dominates framework choice. Databricks fine-tuned a Llama 3.1 8B on internal code written by their employees and ran a live A/B test: 1.4x higher acceptance rate than GPT-4o on bug fixes, 2x lower inference latency⁶. The training data was interaction logs naturally generated by developers using the existing code assistant. High-quality, task-specific data produced those numbers. The framework was Mosaic AI, a managed service. Any of the three tools here would have produced similar results on the same data.

Task definition matters too. Stripe fine-tuned an LLM to generate code fixes. Latency was high, accuracy was low. They reframed the problem as classification: instead of generating a fix, the model picks from a set of known fix patterns. Latency dropped, accuracy rose. The bottleneck was the task framing, not the model or the framework.

Honorable Mention: Torchtune

Torchtune is Meta’s PyTorch-native fine-tuning library. It’s the right choice for teams that want maximum control over the training loop and are deeply invested in the PyTorch ecosystem.

Why it earns a mention: PyTorch 2.5 compile support gives roughly 20-24% speed improvements (4.7 hours on the same 8B benchmark, between Unsloth’s 3.2 and Axolotl’s 5.8). Multi-node training with FSDP2 works out of the box with PyTorch’s native distributed launcher, no third-party wrappers needed. DoRA (Direction-Optimized Rank Adaptation) saves 8% VRAM vs QLoRA with no perplexity degradation on Llama-3.1 8B. Every training component (optimizer, loss function, scheduler, data loader) is a separate composable module you can swap without touching the rest of the pipeline.

Why it’s not in the main comparison: Model coverage is narrower, focused on Meta models and a handful of others. Every config is a Python “recipe” file, not YAML, so you need to read and modify Python code to change training behavior. For most AI engineers, Axolotl provides similar multi-GPU capabilities with less code to write.

Use Torchtune if: You’re a PyTorch developer who wants to modify the training loop itself, not just configure it. You’re building research prototypes where you need to swap optimizers, loss functions, or schedulers at the code level. You’re fine-tuning exclusively Meta models and want the tightest integration.

The One Thing to Remember

The framework you pick today won’t be the framework you use in two years. Unsloth, Axolotl, and LLaMA-Factory all export standard HuggingFace checkpoints. Your trained model is portable. Your data pipeline is portable. The framework is the least permanent decision you’ll make in your fine-tuning stack.

Pick the one that gets you to your first trained model fastest. Then iterate.

🏗️ Engineering Lesson The real cost of fine-tuning isn’t compute. It’s iteration time. A framework that trains 2x faster doesn’t save you 3 hours on one run. It saves you 3 hours on every experiment in a grid search of 20 configurations. That’s 60 hours. Choose for speed of experimentation, not speed of one run.

Unsloth for speed, Axolotl for scale, LLaMA-Factory for breadth. Match the tool to your infrastructure.

What’s your fine-tuning setup right now? Running Unsloth on a single GPU? Axolotl across a cluster? Something else entirely?

Where to Next?

📖 Go Deeper: What is LoRA? covers the parameter-efficient method that all three frameworks optimize for. (coming soon)

🔗 Go Simpler: What is Fine-tuning? explains when fine-tuning is the right call vs. RAG vs. prompt engineering.

🔀 Go Adjacent: What is Quantization? covers the technique that makes QLoRA possible, the method that lets you train on consumer GPUs.

---

1. Spheron, Axolotl vs Unsloth vs TorchTune: Best LLM Fine-Tuning Frameworks in 2026 (Mar. 2026).

2. Zheng et al., LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (Mar. 2024).

3. HuggingFace, Make LLM Fine-tuning 2x faster with Unsloth and TRL (Jan. 2024).

4. Axolotl GitHub, Multimodal Training documentation (2025).

5. Modal, Best frameworks for fine-tuning LLMs in 2025 (2025).

6. Databricks, The Power of Fine-Tuning on Your Data (Apr. 2025).