Integrating large language models (LLMs) into production environments presents challenges beyond merely spinning up a container. Managing GPU drivers, model caching, secure endpoint exposure, and scalability requires native Kubernetes orchestration. NVIDIA NIM, combined with the NIM Operator, provides an abstraction layer that allows platform teams to declare the desired inference state without manually managing every Kubernetes resource. In this article, we explore a technical and business-oriented approach to deploying NIM on Kubernetes clusters, assuming the reader has basic experience with Helm, kubectl, and cloud or on-premise GPUs. Additionally, we show how this architecture integrates with custom software, artificial intelligence, and other solutions that Q2BSTUDIO implements for its clients.
Before starting, it is essential to validate that the cluster meets the minimum requirements: Kubernetes 1.26 or later, Helm 3, administrative access, nodes with compatible NVIDIA GPUs (e.g., A100, H100, L40S), and an active NVIDIA AI Enterprise subscription or NGC Developer Program access. Installing the GPU Operator is a mandatory prerequisite, as it handles drivers, container runtime integration, and exposes GPU resources as Kubernetes extended resources. Without this foundation, the NIM Operator cannot schedule pods correctly. Once nodes with allocatable GPUs are confirmed (kubectl get nodes -o custom-columns='NODE:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu'), we can install the GPU Operator using Helm: helm repo add nvidia and helm upgrade --install gpu-operator nvidia/gpu-operator --namespace gpu-operator --wait. The recommended version at the time of writing is v26.3.3. Confirm that the CUDA validator completes without errors before proceeding.
With the GPU Operator running, we install the NIM Operator version 3.1.1. Create the nim-operator namespace and deploy: helm upgrade --install nim-operator nvidia/k8s-nim-operator --namespace nim-operator --version 3.1.1 --wait. Verify that the NIMCache and NIMService Custom Resource Definitions (CRDs) are registered. Next, configure NGC authentication: we need two secrets in the workload namespace. The first is a Docker registry secret (ngc-secret) so that kubelet can pull container images from nvcr.io. The second is a generic secret (ngc-api-secret) storing the NGC API key, used by the cache job to download model artifacts. Importantly, these secrets only authenticate image and model downloads; they do not protect the inference endpoint. For application traffic security, implement TLS, OIDC authentication, API Gateway, or a service mesh. At Q2BSTUDIO, when working with clients requiring cybersecurity in their AI deployments, we recommend keeping the service internal (ClusterIP) and using a controlled ingress with authentication policies and rate limiting.
The next step is to design the persistent model cache. The NIMCache resource launches a Kubernetes Job that downloads compatible model profiles into a persistent volume. This avoids each inference pod having to download the full model on startup, drastically reducing initialization time. A typical configuration for a model like Llama 3.2 1B includes an 80 GiB PVC with ReadWriteOnce access mode. In production environments requiring multiple replicas on different nodes, evaluate whether the storage system supports ReadWriteMany or if a shared cache via NFS can be used. The operator also allows filtering engine profiles (e.g., tensorrt_llm) to avoid downloading unnecessary variants. Once the NIMCache reaches Ready state, we can deploy the NIMService. This resource creates a Deployment, a Service (default ClusterIP), and configures health probes (liveness, readiness, startup) with default values allowing up to 20 minutes for model initialization. Do not disable these probes; instead, adjust the startupProbe if the model takes longer to load.
To test the endpoint, use port-forwarding: kubectl port-forward service/ 8000:8000. Then verify the /v1/health/live, /v1/health/ready, and /v1/models endpoints. A chat completions request should return a valid JSON response. If readiness fails but liveness works, the model is still loading. In production, external exposure should be done via Ingress with TLS and authentication. The NIM Operator can automatically generate an Ingress if configured under spec.expose.router.ingress. However, client authentication remains the platform operator’s responsibility. We recommend implementing NetworkPolicy to restrict east-west traffic, allowing only labeled namespaces to access port 8000 of the NIM Service.
Horizontal scaling is enabled via the spec.scale.hpa field, which creates a HorizontalPodAutoscaler based on custom metrics like vllm:num_requests_waiting. For this to work, the cluster must have a Prometheus Adapter configured and the NIM Operator’s ServiceMonitor correctly labeled. Before enabling autoscaling, verify that enough GPUs are available and that the cache storage allows concurrent mounts on multiple nodes. Updating the operator and the model follows a careful protocol: before updating the NIM Operator, export current values and custom resources. Model upgrades involve creating a new NIMCache with the new image tag, waiting for it to become ready, then updating the NIMService to point to the new cache and image. Keep the old cache until rollback is safe. Operator rollback uses helm rollback, but CRD compatibility must be checked because Helm does not revert CRDs automatically.
Common failures include ImagePullBackOff due to incorrect NGC credentials, NIMCache stuck in Pending due to unavailable StorageClass, or pods in CrashLoopBackOff due to incompatible model profile with the GPU. The quick diagnostic sequence is: kubectl get nimcache,nimservice -n , then events and logs from the cache job and the inference pod. At Q2BSTUDIO, we combine these deployments with BI / Power BI solutions to monitor model performance in real time, and with AI agents that automate predictive maintenance tasks. The described architecture also allows integrating cloud services such as AWS or Azure for persistent storage and scalability, as offered in our cloud services. In conclusion, deploying NIM with the NIM Operator on Kubernetes is not just a technical exercise but a strategic decision that accelerates production-grade AI model deployment with security, observability, and scalability guarantees. The key is to build a solid foundation (GPU Operator + persistent cache + access controls) and then add capabilities incrementally. With the support of an experienced team like Q2BSTUDIO, organizations can transform artificial intelligence into a reliable operational asset.
