Local inference, sized with evidence.

Estimate the whole footprint

Weights, quantisation, context length, cache and concurrent requests all affect memory. Leave headroom for the operating system and runtime. CPU inference and GPU inference have different cost and throughput profiles; a standard VPS does not imply GPU access.

Keep the model API private

A runtime such as Ollama should not be exposed to the internet without an appropriate security layer. Bind to loopback or a private interface where practical, and put authenticated TLS access in front of any remote route.

Download with provenance

Review the model licence, source, checksum where supplied and intended use. Untrusted model formats or surrounding loader code can expand the attack surface. Keep the runtime and dependencies maintained.

Benchmark a real request

Measure first-token latency, tokens per second, memory pressure and concurrent behaviour with the context size your application uses. Add request size limits, queueing and timeouts so one large prompt cannot exhaust the machine.

Prove the API is not public

Local Ollama API access does not require authentication. If only your own application calls it, keep the listener on loopback. For remote use, add an authenticated TLS gateway or a private tunnel with request and concurrency limits. A cloud API credential does not automatically protect the local listener. Test from a device without access and require rejection.

• Inspect the listening address
• Try an unauthenticated request from outside the permitted network
• Apply limits to context size and concurrent generations

ss -lntp
curl --fail http://127.0.0.1:11434/api/tags

Official documentation: Ollama authentication model

Keep a small benchmark notebook

Record the exact model variant, quantisation, context length, prompt and number of parallel requests. Measure runtime memory rather than assuming it equals the model download size. Repeat the same request after configuration changes and keep quality checks as well as speed measurements. If a job swaps heavily or crowds out the main application, reduce concurrency or use a more suitable machine.

Budget model memory before increasing context

Use parameter count multiplied by quantisation bits divided by eight as a rough weight-only byte estimate, then add runtime, context/KV cache, concurrency and operating-system headroom. It is not a capacity guarantee. In Ollama, parallel requests and larger context increase memory use, while queued work can make an apparently healthy service slow. Start with one model and one request, record peak memory, and raise concurrency only after the workload passes its latency target.

• Record model tag, runtime version, quantisation, prompt length and output limit with every benchmark.
• Separate cold-start loading from warm-request performance; they answer different sizing questions.
• If the host swaps or a process is killed, lower concurrency/context and inspect memory events before buying CPU cores.
• Review loaded models and cache growth when the first request is fast but later requests stall.

# Inspect the model runner and host without changing its configuration.
ollama ps
free -h
df -h

Official documentation: Ollama: memory, concurrency and network configuration

Test embeddings as their own workload

An embedding model turns input text into vectors for retrieval; it is not the same resource profile as a long generated answer. Use one model and preprocessing strategy consistently when indexing and querying. Ollama’s embed endpoint accepts text or batches and can reject overlong input when truncate is false. Choose an intentional chunking policy so documents are not silently cut at an inconvenient boundary.

• Benchmark an indexing batch and an interactive query separately, including storage and vector-database cost.
• Keep model name and embedding dimensions with the index. Changing them usually requires a deliberate reindexing plan.
• Use a small retrieval evaluation set: known questions, expected documents and cases that should return no relevant result.
• Treat prompts, indexed documents and access logs as separate data stores with their own access and deletion rules.

Official documentation: Ollama: embeddings API