Canonical page: https://koalavps.com/guides-self-host-an-llm-on-a-vps/ Language: English. Source snapshot: 2026-10-21. Generated: 2026-10-21T13:31:28.274Z. [Home](https://koalavps.com/) / [Explore](https://koalavps.com/resources/) / The KoalaVPS field notes The KoalaVPS field notes # Local inference, sized with evidence. A model that fits on disk may still need much more memory at runtime. Start with a representative benchmark. [Find your VPS](https://koalavps.com/vps/) ## IN THIS DOCUMENT [Estimate the whole footprint](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-1)[Keep the model API private](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-2)[Download with provenance](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-3)[Benchmark a real request](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-4)[Prove the API is not public](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-5)[Keep a small benchmark notebook](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-6)[Budget model memory before increasing context](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-7)[Test embeddings as their own workload](https://koalavps.com/guides-self-host-an-llm-on-a-vps/#section-8)[Knowledge base](https://koalavps.com/resources/) ## Estimate the whole footprint Weights, quantisation, context length, cache and concurrent requests all affect memory. Leave headroom for the operating system and runtime. CPU inference and GPU inference have different cost and throughput profiles; a standard VPS does not imply GPU access. ## Keep the model API private A runtime such as Ollama should not be exposed to the internet without an appropriate security layer. Bind to loopback or a private interface where practical, and put authenticated TLS access in front of any remote route. ## Download with provenance Review the model licence, source, checksum where supplied and intended use. Untrusted model formats or surrounding loader code can expand the attack surface. Keep the runtime and dependencies maintained. ## Benchmark a real request Measure first-token latency, tokens per second, memory pressure and concurrent behaviour with the context size your application uses. Add request size limits, queueing and timeouts so one large prompt cannot exhaust the machine. ## Prove the API is not public Local Ollama API access does not require authentication. If only your own application calls it, keep the listener on loopback. For remote use, add an authenticated TLS gateway or a private tunnel with request and concurrency limits. A cloud API credential does not automatically protect the local listener. Test from a device without access and require rejection. - Inspect the listening address - Try an unauthenticated request from outside the permitted network - Apply limits to context size and concurrent generations ``` ss -lntp curl --fail http://127.0.0.1:11434/api/tags ``` Official documentation: [Ollama authentication model](https://docs.ollama.com/api/authentication) ## Keep a small benchmark notebook Record the exact model variant, quantisation, context length, prompt and number of parallel requests. Measure runtime memory rather than assuming it equals the model download size. Repeat the same request after configuration changes and keep quality checks as well as speed measurements. If a job swaps heavily or crowds out the main application, reduce concurrency or use a more suitable machine. ## Budget model memory before increasing context Use parameter count multiplied by quantisation bits divided by eight as a rough weight-only byte estimate, then add runtime, context/KV cache, concurrency and operating-system headroom. It is not a capacity guarantee. In Ollama, parallel requests and larger context increase memory use, while queued work can make an apparently healthy service slow. Start with one model and one request, record peak memory, and raise concurrency only after the workload passes its latency target. - Record model tag, runtime version, quantisation, prompt length and output limit with every benchmark. - Separate cold-start loading from warm-request performance; they answer different sizing questions. - If the host swaps or a process is killed, lower concurrency/context and inspect memory events before buying CPU cores. - Review loaded models and cache growth when the first request is fast but later requests stall. ``` # Inspect the model runner and host without changing its configuration. ollama ps free -h df -h ``` Official documentation: [Ollama: memory, concurrency and network configuration](https://docs.ollama.com/faq) ## Test embeddings as their own workload An embedding model turns input text into vectors for retrieval; it is not the same resource profile as a long generated answer. Use one model and preprocessing strategy consistently when indexing and querying. Ollama’s embed endpoint accepts text or batches and can reject overlong input when truncate is false. Choose an intentional chunking policy so documents are not silently cut at an inconvenient boundary. - Benchmark an indexing batch and an interactive query separately, including storage and vector-database cost. - Keep model name and embedding dimensions with the index. Changing them usually requires a deliberate reindexing plan. - Use a small retrieval evaluation set: known questions, expected documents and cases that should return no relevant result. - Treat prompts, indexed documents and access logs as separate data stores with their own access and deletion rules. Official documentation: [Ollama: embeddings API](https://docs.ollama.com/api/embed) KEEP EXPLORING ## Continue reading. [Products ### Keep the intelligence. Own the runtime. Host the services around your AI workflow: retrieval, document processing, tool execution, webhooks and observability. Build a sys…](https://koalavps.com/ai-agent-hosting/)[Guides ### Buy for the workload you have. Choose an initial configuration from evidence, then leave a measured path to grow. A realistic estimate is more useful than the la…](https://koalavps.com/choose-vps/)[Guides ### Keep the machine yours. Security is an ongoing routine: reduce exposed surfaces, limit privileges, patch deliberately and make recovery possible.…](https://koalavps.com/server-security/) CONFIGURE YOUR INFRASTRUCTURE ## Take control. Deploy on your terms. No identity paperwork. Full administrator access. Crypto-native billing. [Select a configuration](https://koalavps.com/vps/)