AI Model Serving Explained: From Notebook to Production

Model serving is the work of turning a trained AI model into a service that people and applications can call reliably: an address they send a request to and get an answer from, quickly, at any hour, with many requests arriving at once. A model that works in a developer's notebook is a demonstration. A served model has an API, handles concurrent users, starts itself after a reboot, is monitored, is secured and can be updated without anyone noticing. Most of the effort in an AI project sits in that gap, and most of it is ordinary IT discipline applied to an unusual workload.

What changes between a notebook and production?

What are the layers of a serving stack?

1. Hardware. The GPU system, with enough memory for the model and its users.
2. Drivers and runtime. GPU drivers and a container runtime, so the software environment is repeatable. See Docker vs Kubernetes for AI.
3. Inference engine. The software that loads the model and generates output efficiently, such as vLLM, TensorRT-LLM or a packaged NIM. See TensorRT-LLM vs vLLM.
4. API layer. A standard endpoint that applications call. Using the OpenAI-compatible format means almost any tool can connect and the model behind it can be swapped.
5. Gateway. Authentication, per-user keys, rate limits, routing between models and usage accounting.
6. Applications. Chat interfaces, retrieval systems and agents. See Open WebUI vs AnythingLLM.
7. Observability. Metrics, logs and quality checks across all of the above.

A small deployment may run all seven on one machine. The layers still exist, and naming them makes it clear what is missing.

Which metrics define good serving?

Averages hide problems. A service that is fast on average but stalls for one request in twenty will be remembered for the stalls. Our guide to monitoring with DCGM, Prometheus and Grafana shows how to collect the hardware side.

What makes serving language models different?

• Requests vary enormously. One user sends a sentence, another a 50-page document. Engines use continuous batching to cope.
• Memory grows with use. Each active conversation holds its own KV cache, so memory, not compute, often sets the user limit.
• Answers stream. The service has to hold many open connections and deliver tokens steadily.
• Models are huge. Loading takes time, so restarts and model switches need planning. See data locality.
• Quality can drift without an error. A model can return fluent, wrong answers with a perfect uptime record, so output needs checking, not just the server.

A model is an asset. A served model is a system. The difference is everything that happens when the developer who built it is on leave.

A production readiness checklist

1. It restarts on its own. After a power cut or reboot, the service comes back without manual steps.
2. Access is controlled. Per-user or per-application keys; nothing exposed directly to the internet. See our security hardening checklist.
3. Limits are set. Maximum context, maximum output and maximum concurrent requests, so one heavy user can't take the service down.
4. It has been load-tested. At your real peak concurrency, with your real prompt lengths.
5. It is monitored. With alerts for memory, temperature, errors and latency.
6. Versions are pinned. Model, engine and driver versions are recorded, and upgrades are tested first.
7. Quality is tracked. A fixed set of test questions is run after every change.
8. There is a recovery plan. Backups of configuration and data, and a decision on what happens if the hardware fails. See the single point of failure problem.
9. Someone owns it. A named person, with a deputy.

How do you size hardware for serving?

Work from the requirement back to the machine: which model and precision, how much context per request, and how many requests at the same moment at peak. Memory must hold the weights once plus KV cache for every concurrent request. Speed per user depends on the memory bandwidth of the system. We work through examples in 20 employees on one GB10 and our model sizing table.

Buying in Singapore

For a first production service for a small team, the NVIDIA DGX Spark 4TB (S$9,450) or HP ZGX Nano G1n 4TB (S$11,499). For higher concurrency, the NVIDIA RTX PRO 6000 Blackwell Workstation Edition (S$29,100) or, in a server, the Server Edition (S$32,999). To protect the network path, the Fortinet FortiGate 70G (S$1,668), and for configuration and data backup, the Synology ActiveProtect DP340 (S$6,526.80) (listed 11 Nov 2026; prices change; ask for a current quotation). DGX Station GB300 systems such as the MSI XpertStation WS300 are price on request. To start, use Request for Quotation on the product page, call (65) 6978 3502, email Sales@sourceit.com.sg or use our contact page.

Frequently asked questions

What is AI model serving? Model serving is making a trained AI model available as a reliable service with an API, so applications and users can send requests and receive answers at any time, with many requests handled at once.

What is the difference between training and serving? Training creates or adjusts a model and happens occasionally. Serving runs the finished model for users continuously, and is judged on latency, throughput, reliability and security.

Which metrics matter for LLM serving? Time to first token, tokens per second per user, total throughput, tail latency, error rate and GPU memory headroom.

What software is used to serve LLMs? Common choices are vLLM, NVIDIA TensorRT-LLM and NIM, and for small setups Ollama, usually behind an OpenAI-compatible API with a gateway for authentication and limits.