Llama Chat Template — [INST] Format

What is a chat template?

A chat template is a stateless transformation rule that converts a list of (role, message) pairs into a single token sequence the model can process. Most models are trained as base causal language models — they learn to predict the next token given all previous tokens. To turn that into a conversational agent, the training data itself must encode the conversation structure, so the model learns which parts are system context, which are user input, and where to place its own response.

The template is not learned; it is hardcoded during training and must be applied identically at inference time. If your template at inference differs from the training template, the model is operating outside its distribution and will degrade. That is why HuggingFace transformers stores the template in the tokenizer config and provides apply_chat_template() to enforce consistency.

Why templates matter for model behavior

Chat templates are not a nice-to-have formatting layer. They are part of the model's prior knowledge. During SFT (supervised fine-tuning) on instruction data, every training example is templated the same way, so the model learns that specific token sequences signal ‘system’, 'user', or 'assistant' context. If you train on ‘User:’ but inference uses ‘[INST]’, the model has no learned association and will likely treat your input as continuation rather than instruction.

The stakes are high: a mismatched template can cause the model to ignore system prompts, refuse instructions it should accept, or produce outputs in the wrong format. Additionally, the template influences token efficiency. Some templates add padding or redundant role tokens; others compress conversation state tightly. For inference latency, this difference compounds across thousands of requests.

Llama 2: the [INST] format

Llama 2 (7B–70B, July 2023) introduced the [INST]/[/INST] tag convention. The structure is:

<s>[INST] <<SYS>>
system prompt here
<</SYS>>

user message [/INST] assistant response </s>

Breaking this down: <s> is the BOS (beginning-of-sequence) token. The system prompt lives inside [INST] blocks, wrapped in <<SYS>> markers. Each turn is a single [INST]...message...[/INST] response pair. Multi-turn conversations chain these pairs:

<s>[INST] <<SYS>>
system prompt
<</SYS>>
First user message [/INST] First response </s>
<s>[INST] Second user message [/INST] Second response </s>

The design is tag-based, making it human-readable but verbose. The [INST] and [/INST] tokens add ~4 tokens per turn, and the system prompt must be repeated in every multi-turn message, inflating context for long conversations.

Llama 3: role tokens and the new format

Llama 3 (8B–405B, April 2024) abandoned tags for a role-token approach. Special tokens now mark role boundaries instead of tags:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
system prompt
<|eot_id|><|start_header_id|>user<|end_header_id|>
user message
<|eot_id|><|start_header_id|>assistant<|end_header_id|>
assistant response
<|eot_id|>

The key differences: <|begin_of_text|> opens the sequence once (not per turn), <|start_header_id|> and <|end_header_id|> wrap the role name, and <|eot_id|> marks end-of-turn (assistant generation should terminate here). The system prompt appears once at the start, not per turn, saving tokens in long conversations.

Multi-turn chaining is cleaner because the role structure is explicit and repetitive. The tokenization is also more efficient: role tokens are single special IDs rather than multi-token tags like [INST], reducing overhead per message.

The Auto-apply pattern: tokenizer.apply_chat_template()

Manually templating is error-prone. HuggingFace Transformers solves this by storing the template in the tokenizer and providing apply_chat_template():

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-chat')
messages = [
    {'role': 'system', 'content': 'You are a helpful assistant.'},
    {'role': 'user', 'content': 'What is 2+2?'}
]
templated = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
# Returns the raw string with [INST] tags already in place

The tokenize=False flag returns the raw template string; add_generation_prompt=True appends the assistant header so the model knows to respond. The tokenizer automatically detects which template to use based on the model card, ensuring consistency across different Llama versions and families.

Message roles: system, user, assistant

Conversations are lists of (role, content) tuples. The standard roles are:

• system: Context, instructions, and guardrails applied to the entire conversation. Appears once at the start. Models weight system prompts heavily because they are meant to override default behavior.
• user: Human input, questions, or requests. Can appear multiple times in multi-turn.
• assistant: Model output. Stored in the conversation so the model can see its own prior responses when generating the next turn.

Llama 3 adds ipython and tool roles for extended reasoning and tool-use scenarios, but the core three are universal. Some inference engines (vLLM, llama.cpp) also support tool roles for structured output and function calling.

System prompts and their placement

System prompts are a high-leverage knob for steering model behavior. During training, the system message is seen once per conversation, so the model learns to treat it as a strong prior that shapes all downstream responses.

In Llama 2, repeating the system prompt in each turn was the safest approach because the model might 'forget' the system context in multi-turn exchanges. Llama 3's role structure makes this less necessary: the system message is part of the fixed context, not something that decays. However, in practice, repeating the system prompt every 10-20 turns is still a good heuristic for very long conversations, since the model's later tokens have access only to recent context in the KV cache (if using prefix caching or context windows smaller than the full conversation).

Empty or minimal system prompts (just 'You are a helpful assistant') are valid and often preferred for maximum flexibility. Overly specific or adversarial system prompts can constrain the model or cause generation to refuse benign requests.

Token efficiency and formatting overhead

Templates add overhead. Llama 2's [INST] tags consume 4 tokens per turn; role tokens in Llama 3 consume ~6–8 tokens per turn depending on the role name length. For a 1000-turn conversation, this is 4000–8000 tokens of pure framing.

Beyond the template, whitespace and newlines matter. A template with newlines before and after role markers:

<s>[INST]\n\nuser message\n\n[/INST]

tokenizes to more tokens than:

<s>[INST] user message [/INST]

At scale, this difference is measurable. Dense formatters prefer minimal whitespace; readability-first templates accept the cost. For production, profile the template on your dataset and choose based on the latency budget, not aesthetics.

Common pitfalls and debugging

Pitfall 1: Mismatched template at inference. You train on Llama 2 but decode with Llama 3 format. The model hallucinates or ignores your input because role markers are unrecognized. Solution: always call apply_chat_template() from the saved tokenizer.

Pitfall 2: Forgetting add_generation_prompt. Without it, the template ends after the user message, and the model must generate an 'assistant' marker from scratch. This destabilizes generation. Always set add_generation_prompt=True.

Pitfall 3: Not escaping user content. If user input contains special tokens (e.g., a code block with [INST]), they can poison the template. Transformers handles this automatically, but custom templating must escape or sanitize user strings.

Pitfall 4: System prompt mutation. Changing the system prompt between inference and training biases responses. A model trained with 'Be concise' will generate longer outputs if you remove that instruction at inference time.

Cross-model compatibility and ecosystem

Llama chat templates are not universal. Different families use different formats:

• Mistral uses [INST] like Llama 2, but without the <<SYS>> wrapper.
• Phi uses <|user|> and <|assistant|> tags (different from Llama).
• ChatGLM (Chinese) has its own role markers.
• Qwen uses yet another variant.

Trying to reuse a Llama 2 template on Phi 3.5 or vice versa will degrade performance. HuggingFace mitigates this by storing the correct template in each model's tokenizer_config.json and auto-detecting on AutoTokenizer.from_pretrained(). For custom or edge models without a template in the config, you must specify it manually or accept degraded outputs.

Designing custom templates

When you fine-tune a model or work with a non-standard architecture, you may need a custom template. Design principles:

1. Be consistent with training. If you fine-tune on <user>...</user><assistant>...</assistant>, use the same format at inference.

2. Separate roles clearly. Ambiguous boundaries cause context leakage. Use unambiguous, non-colliding special tokens.

3. Test thoroughly. Generate on a small held-out set and verify that the model respects role boundaries and system prompts.

4. Document in tokenizer_config.json. Store your custom template in the tokenizer config so collaborators and future deployments auto-detect it.

Example custom template in Jinja2 (HuggingFace format):

{%- if system_message %}
{{- '<SYSTEM>' + system_message + '</SYSTEM>' }}
{%- endif %}
{%- for message in messages %}
{{- '<' + message['role'].upper() + '>' + message['content'] + '</' + message['role'].upper() + '>' }}
{%- endfor %}
{{- '<ASSISTANT>' }}

Llama chat template in vLLM and llama.cpp

vLLM (a high-throughput inference engine) auto-detects templates from the tokenizer config and applies them correctly. You pass messages as Python dicts and let vLLM handle formatting:

from vllm import LLM, SamplingParams

llm = LLM(model='meta-llama/Llama-2-7b-chat-hf')
messages = [{'role': 'user', 'content': 'Hello'}]
prompt = llm.get_tokenizer().apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
outputs = llm.generate([prompt], SamplingParams(temperature=0.7))

llama.cpp (inference on CPU/mobile) also supports Llama chat templates but requires manual construction for edge cases. Always check the engine's version; older llama.cpp versions do not support Llama 3 role tokens.

Best practices and recommendations

1. Always use apply_chat_template(). Never construct templates by hand. Manual construction is error-prone and fragile.

2. Test template changes. If you switch models or update templates, run a small benchmark to check output quality. Template mismatches are subtle and can silently hurt performance.

3. Store the template in the tokenizer config. This ensures reproducibility and lets collaborators inherit the correct format without asking.

4. Log the applied template. For debugging, it is helpful to log the raw templated string before tokenization so you can spot mismatches quickly.

5. For long conversations, consider chat history pruning. Llama models have fixed context windows. Rather than exceed them, truncate old messages, but keep the system prompt and the most recent turns for continuity.

6. Llama 3 is the standard going forward. New models adopt role tokens, not tags. If you are designing a new model, use Llama 3's format for better ecosystem compatibility.