LLMOps (Large Language Model Operations) is the specialized practice of building, deploying, evaluating, and maintaining systems powered by large language models. It extends MLOps with LLM-specific challenges: prompt engineering and versioning, retrieval-augmented generation, evaluation without ground-truth labels, and the unique cost-quality-latency trade-offs of foundation model APIs and self-hosted LLMs.
Overview
mindmap
root((LLMOps))
Prompt Engineering
Prompt design patterns
Chain-of-thought
Few-shot examples
Prompt versioning
System prompts
RAG Systems
Document ingestion
Chunking strategies
Vector stores
Retrieval algorithms
Context window management
LLM Evaluation
Reference-based metrics
LLM-as-judge
Human evaluation
Hallucination detection
Benchmark suites
Fine-Tuning
LoRA and QLoRA
Instruction tuning
RLHF
Dataset curation
When to fine-tune vs RAG
LLMOps vs Traditional MLOps
graph TD
subgraph Comparison[LLMOps vs MLOps]
subgraph MLOpsCol[Traditional MLOps]
M1[Train model from scratch]
M2[Structured tabular or image data]
M3[Clear metric: AUC F1 accuracy]
M4[Deterministic predictions]
M5[Minutes to hours to train]
end
subgraph LLMOpsCol[LLMOps]
L1[Adapt foundation model via prompt or fine-tune]
L2[Unstructured text, multimodal]
L3[Fuzzy metrics: coherence, factuality, helpfulness]
L4[Stochastic generation - temperature sampling]
L5[Billions of params - days or weeks to fine-tune]
end
end
Topics in This Section
| File | Topic | Key Concepts |
|---|---|---|
| 01_prompt_engineering.md | Prompt Engineering | Patterns, chain-of-thought, versioning |
| 02_rag_systems.md | RAG Systems | Retrieval, chunking, vector stores |
| 03_llm_evaluation.md | LLM Evaluation | LLM-as-judge, benchmarks, RAGAS |
| 04_fine_tuning.md | Fine-Tuning | LoRA, instruction tuning, RLHF |