Tweaking Local Language Model Settings with Ollama

# Introduction

 
Language models continue to shape how machine learning practitioners and developers build applications. The advent of capable, compact small language models add an intriguing layer to the mix. By bypassing third-party APIs, running models locally guarantees complete data privacy, eliminates per-token API costs, and enables offline operation. Among the tools powering this revolution, Ollama has emerged as one of the standards for running local inference due to its lightweight Go-based engine, simple CLI, and robust Docker-like model management system.

However, simply pulling a model and running it with the default settings is rarely optimal. Default configurations are tuned for a broad, general-purpose audience, often prioritizing safe, conversational chat over performance, deterministic reasoning, or specialized system needs. If you are building a coding assistant, an automated ETL pipeline, or a multi-agent system, the default configurations will likely lead to high latency, context-window limitations, or random and unpredictable outputs.

To elevate your local AI applications, you need to understand how to tune both the model-level hyperparameters and the server-level runtime environments. In this article, we will go deep under the hood of Ollama’s configuration engine, exploring how to fine-tune local language model parameters using the Ollama Modelfile, optimize hardware performance with server environment variables, and format precise prompt flows using Go template syntax.

# 1. The Ollama Modelfile: Your Local Model Blueprint

 
Much like a Dockerfile defines how a container is built, an Ollama Modelfile is a declarative configuration file that defines how a local language model should behave. It lets you customize system instructions, adjust model parameters, and package these configurations into a new, reusable model variant that you can run with a single command.

A basic Modelfile consists of a base model reference (using the FROM directive), system-level guidelines (using SYSTEM), and parameter modifications (using the PARAMETER directive):

// Example: A Custom Developer Modelfile

# Use Llama 3.1 8B as the base model
FROM llama3.1:8b

# Set model-level parameters
PARAMETER temperature 0.2
PARAMETER num_ctx 8192
PARAMETER min_p 0.05

# Define system persona and behavioral guidelines
SYSTEM """You are an elite, highly precise software engineer. 
Provide concise, modular, and optimized code solutions. 
Do not include conversational filler unless explicitly asked."""

To compile and run your custom model, you use the ollama create command in your terminal:

# Create the model named 'dev-llama' from the Modelfile
ollama create dev-llama -f ./Modelfile

# Run the newly created model
ollama run dev-llama

By encapsulating these parameters directly into the model definition, you ensure that every application or API call querying dev-llama inherits these optimizations out-of-the-box, without needing to pass raw JSON parameter payloads in each API request.

# 2. Fine-Tuning the Sampling Parameters

 
When a model generates text, it doesn’t “know” words; it calculates a probability distribution over its vocabulary for the next most likely token. Sampling parameters dictate how the engine chooses the next token from this distribution. Tweaking these settings is the single most effective way to align the model’s creativity and precision with your specific use case.

// Temperature: The Randomness Dial

The temperature parameter controls the scaling of the token probability distribution. Mathematically, it divides the raw logits (pre-softmax scores) generated by the model before they are converted into probabilities:

• Low temperature (e.g., 0.1 to 0.2): Flattens low-probability options and amplifies high-probability ones. This results in highly deterministic, consistent, and logical completions. Ideal for code generation, mathematical reasoning, structured data extraction (JSON/YAML), and factual summarization.
• High temperature (e.g., 0.8 to 1.2): Flattens the differences between token probabilities, making less likely tokens more competitive. This introduces diversity, randomness, and “creativity” into the responses. Ideal for creative writing and brainstorming.

# Configure for highly deterministic, structured tasks
PARAMETER temperature 0.1

// Top-K, Top-P, and Min-P: Narrowing the Token Pool

Left unchecked, even at low temperatures, models can occasionally select highly inappropriate tokens from the tail end of the probability distribution. To prevent this, model engines filter the active token pool before selecting the final token.

1. Top-K (e.g. 40): Restricts the pool to the K most probable next tokens. Any token ranked lower than 40 is immediately discarded, regardless of its actual probability. This is a crude but effective way to prune highly erratic tokens.
2. Top-P / Nucleus Sampling (e.g. 0.90): Restricts the pool to a dynamic set of tokens whose cumulative probability exceeds the threshold P. For example, at 0.90, Ollama sorts all tokens from highest to lowest probability and keeps only the top group that makes up the first 90% of the distribution. If the model is highly confident, the pool might compress to just 2 or 3 tokens; if it is confused, the pool expands.
3. Min-P (e.g. 0.05 to 0.10): A modern, vastly superior alternative to Top-P. Instead of taking a static cumulative slice, min_p filters out tokens whose probability is lower than a dynamic threshold relative to the leading token’s probability. For example, if the top token has a probability of 0.80 and min_p is set to 0.05, the minimum threshold for any other token to be considered is 0.80 * 0.05 = 0.04. If the top token is highly certain (e.g. 0.99), all other tokens are aggressively pruned. If the top token is uncertain (e.g. 0.15), the threshold drops to 0.0075, keeping a wide pool of creative choices open.

# Establish robust sampling limits in the Modelfile
PARAMETER top_k 40
PARAMETER top_p 0.90
PARAMETER min_p 0.05

⚠️ When using min_p, you should generally leave top_p at its default (1.0) or set it highly (0.95+) so it doesn’t interfere with the superior, dynamic scaling behavior of min_p.

# 3. Stopping Loops and Repetitive Outputs

 
One of the most frustrating failures in local model deployment is the repetition loop, where a model begins generating the exact same sentence, phrase, or code block indefinitely. This is usually triggered by a combination of a small model size (e.g. 1.5B or 3B parameters) and a lack of penalty boundaries.

Ollama provides three key parameters to prevent and interrupt these looping states.

// Repetition and Presence Penalties

• Repetition penalty (repeat_penalty): Multiplies the raw logits of tokens that have already been generated, making them less likely to appear again. A value of 1.1 to 1.2 is usually sufficient to discourage looping without making the model avoid necessary grammar words (like “the” or “and”).
• Presence penalty (presence_penalty): Applies a flat, one-time penalty to any token that has appeared at least once in the generated text, encouraging the model to introduce completely new topics or vocabulary.
• Frequency penalty (frequency_penalty): Applies a penalty proportional to the number of times a token has appeared, steadily discouraging the overuse of specific terms.

# Discourage loops and encourage vocabulary variety
PARAMETER repeat_penalty 1.15
PARAMETER presence_penalty 0.05
PARAMETER frequency_penalty 0.05

// Halting Generation with Stop Sequences

Sometimes, the model doesn’t loop internally, but it fails to realize when it has finished its turn, continuing to hallucinate fake responses from the user. You can prevent this by defining explicit stop sequences (stop tokens). When the model generates a stop sequence, the engine immediately halts inference and returns the response.

Common stop tokens include chat markers like <|im_end|>, markdown section headers, or custom delimiters:

# Stop generating when ChatML tags or User lines are generated
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER stop "User:"

# 4. Managing Context Windows and Memory

 
Local hardware resources — specifically video RAM (VRAM) on your GPU — are highly constrained. Understanding how to size your model’s memory structures is vital for building robust local applications.

// Context Length (num_ctx)

The context length (num_ctx) defines the size of the attention window (in tokens) that the model can process at once. This includes both the input prompt (and system history) and the newly generated output tokens.

By default, Ollama initializes many models with a conservative context window of 2048 or 4096 tokens to prevent memory overflow on lower-end hardware. However, modern models like Llama 3.1 or Mistral support native context windows up to 128,000 tokens. If you are building a retrieval-augmented generation (RAG) system or importing large code files, 2048 tokens will result in silent prompt truncation, leading to loss of context and highly inaccurate completions.

You can explicitly increase this parameter in your Modelfile:

# Expand context window to 16,384 tokens
PARAMETER num_ctx 16384

⚠️ Attention computation scales quadratically ($O(N^2)$) with context length. Doubling your num_ctx will dramatically increase the VRAM required to store the model’s active state during generation. Be sure your hardware can handle the increased allocation.

// KV Cache Quantization (OLLAMA_KV_CACHE_TYPE)

To track relationships between tokens over a long conversation, the model stores an active key-value (KV) cache in VRAM. At large context lengths (like 32k or 128k), the size of the KV cache could exceed the weight size of the model itself, causing out-of-memory crashes.

To combat this, Ollama supports KV cache quantization. Much like model weights can be compressed from 16-bit floats to 4-bit integers, the KV cache can be quantized to lower precisions with minimal degradation in text quality:

• f16: Standard, uncompressed 16-bit floating-point cache (default)
• q8_0: Compresses the KV cache to 8-bit integers, saving roughly 50% of KV VRAM with virtually zero impact on output quality
• q4_0: Compresses the KV cache to 4-bit integers, saving 75% of KV VRAM, allowing massive context sizes on consumer hardware at the expense of a slight increase in model perplexity

This parameter is set via the OLLAMA_KV_CACHE_TYPE server environment variable (detailed in the next section).

# 5. Server-Level Tuning: Environment Variables

 
While Modelfile parameters adjust how a specific model operates, server environment variables customize the Ollama background daemon itself. These configurations dictate how Ollama interacts with your operating system, handles system memory, manages parallel processing, and utilizes your hardware acceleration layers.

How you set these variables depends on your host operating system:

• macOS: Set via terminal exports or modified inside your application environment files (or launched via launchctl for background services)
• Linux (Systemd): Configured via systemctl edit ollama.service to inject environment configurations
• Windows (WSL2 / System): Set in standard Windows System Environment Variables or in your WSL terminal profile

// The Essential Server Variables

// Example: Injecting Configurations on Linux (Systemd)

For practitioners running production services on Ubuntu/Debian, edit the service file to inject these environment variables:

# Open the systemd configuration editor for Ollama
sudo systemctl edit ollama.service

Inside the editor block, add the following configuration:

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=24h"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_FLASH_ATTENTION=1"

Save the file and restart the daemon to apply your hardware optimizations:

# Reload systemd definitions and restart the service
sudo systemctl daemon-reload
sudo systemctl restart ollama

# 6. Prompt Templating: Go Template Syntax

 
A language model does not natively understand chat histories, user queries, or system roles. Instead, they expect a single, continuous stream of raw text formatted with special tokens that separate the system persona, the user message, and the assistant response.

Ollama uses the Go text template engine to convert high-level chat histories (e.g. standard OpenAI-compatible role JSON arrays) into the exact text format expected by the model.

If your template is configured incorrectly, your system prompt will be completely ignored, the model might fail to identify your instructions, and inference performance will severely degrade.

// Understanding the Go Template Structure

The TEMPLATE directive in an Ollama Modelfile uses structured tags to parse instructions. Here is an example mapping to the popular ChatML format (often used by models like Qwen, Mistral-instruct, and Hermes):

# Define the message stream formatting
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""

Let’s break down the Go template logic in this block:

• {{ if .System }} ... {{ end }}: Checks if a system prompt has been defined. If it has, it prints the start block <|im_start|>system, injects the system prompt variable {{ .System }}, and closes it with <|im_end|>.
• {{ if .Prompt }} ... {{ end }}: Takes the incoming user query ({{ .Prompt }}) and wraps it inside the user tokens <|im_start|>user and <|im_end|>.
• <|im_start|>assistant \n {{ .Response }}<|im_end|>: Directs the model that it is now the assistant’s turn to generate text. The engine streams the incoming output into {{ .Response }} and appends the final end-of-text marker.

When creating a new model, it is important to inspect the source model’s documentation to identify its precise template structure (e.g. Llama uses special headers like <|start_header_id|>system<|end_header_id|>, whereas Mistral uses bracket-based sequences like [INST] and [/INST]). Matching the expected template guarantees the highest possible instruction-following fidelity.

# 7. Practitioner Reference Architectures

 
To help you immediately apply these parameters, here are three pre-configured Modelfiles tailored to specific common runtime scenarios:

// 1. The Precise JSON Parser (Structured Extraction / Coding)

Designed for ETL pipelines, JSON extraction, and high-accuracy software development. Minimizes temperature and leverages dynamic pruning to strip out erratic tokens.

FROM llama3.1:8b

# Deterministic and highly restricted parameters
PARAMETER temperature 0.0
PARAMETER min_p 0.05
PARAMETER top_p 0.95
PARAMETER top_k 10

# Discourage loops
PARAMETER repeat_penalty 1.1

# Explicit stop markers
PARAMETER stop "<|im_end|>"
PARAMETER stop "User:"

// 2. The Creative Writer (Brainstorming / Interactive Agent)

Designed for conversational interfaces, dynamic agent workflows, and story generation. Elevates temperature while preventing vocabulary stagnation.

FROM llama3.1:8b

# Highly expressive and diverse parameters
PARAMETER temperature 0.9
PARAMETER min_p 0.08
PARAMETER top_p 0.98
PARAMETER top_k 60

# Stronger penalties to prevent loops and repetitiveness
PARAMETER repeat_penalty 1.20
PARAMETER presence_penalty 0.15
PARAMETER frequency_penalty 0.10

// 3. The RAG Powerhouse (Large Context / High Memory)

Designed for reading long PDF manuals, querying local databases, or processing multi-file workspaces. Maximizes context length and optimizes memory footprints.

FROM llama3.1:8b

# Large context allocation
PARAMETER num_ctx 32768
PARAMETER temperature 0.3
PARAMETER min_p 0.05

# Prevent looping on large prompts
PARAMETER repeat_penalty 1.15

# Wrapping Up

 
Local language model engineering is a delicate balance between quality of output and the realities of physical hardware constraints. Deploying a model using defaults leaves substantial performance, throughput, and accuracy on the table.

By taking control of sampling parameters like temperature and min_p, you can force models to be highly precise or creatively engaging. Implementing repetition penalties and stop sequences keeps your local models from falling into endless loops. At the same time, scaling up the context length while optimizing VRAM through KV cache quantization and flash attention allows you to tackle complex retrieval tasks on consumer GPUs.

By mastering the Ollama Modelfile and configuring server environment variables, you begin your transition from a passive consumer of AI tools to a systems engineer who designs high-performance, private, and beautifully optimized local intelligent pipelines. Keep your parameters tuned, keep your memory footprint lean, and let your local agents build.
 
 

Matthew Mayo (@mattmayo13) holds a master’s degree in computer science and a graduate diploma in data mining. As managing editor of KDnuggets & Statology, and contributing editor at Machine Learning Mastery, Matthew aims to make complex data science concepts accessible. His professional interests include natural language processing, language models, machine learning algorithms, and exploring emerging AI. He is driven by a mission to democratize knowledge in the data science community. Matthew has been coding since he was 6 years old.