Contents - overview of RTX 5090 GPU specifications - driver download and installation - NVIDIA GPU integration with Docker - disable IPv6 - install Ollama and Open WEBUI with Docker Compose - model installation
Introduction In this article, I share the experience of installing and configuring an LLM environment on a custom-built PC designed for production use and to support a medical team in diagnostic analysis and integration with an SIMRS. The goal is to have a reliable environment to run large models, orchestrate them with Docker, and expose secure APIs. This work is carried out by Q2BSTUDIO, a custom software development company specializing in artificial intelligence, cybersecurity, AWS and Azure cloud services, business intelligence services, AI for enterprises, AI agents, and Power BI, as well as custom software solutions for clients needing tailored applications and advanced artificial intelligence services.
PC Specifications Operating system Ubuntu 24.04.2 LTS x86_64 Motherboard ASUS ROG CROSSHAIR X870E HERO CPU AMD Ryzen 9 9950X3D Memory approximately 62 GB Integrated AMD GPU Dedicated NVIDIA RTX 5090 GPU
Initial Check To identify the hardware and confirm GPUs, I used neofetch and dmidecode -t baseboard for the motherboard. After installing the driver, you can check the GPU status with nvidia-smi.
Install NVIDIA Driver The RTX 5090 is a very recent model. After several attempts, the driver version that worked on my system was 570.172.08. Key steps 1 Disable secure boot in BIOS by pressing F2 to enter Advanced Mode, then Boot, Secure Boot, and change OS Type to Other OS and Secure Boot Mode to Custom 2 Check recommended drivers with ubuntu-drivers devices and try ubuntu-drivers autoinstall if applicable 3 If there are conflicts, remove previous installations with apt purge ^nvidia-.* --yes apt autoremove 4 Clean up remnants with rm -rf /usr/local/cuda* rm -rf /etc/X11/xorg.conf rm -rf /lib/modules/$(uname -r)/kernel/drivers/video/nvidia* update-grub reboot 5 Download the official installer wget https://us.download.nvidia.com/XFree86/Linux-x86_64/570.172.08/NVIDIA-Linux-x86_64-570.172.08.run 6 Make it executable and install chmod +x NVIDIA-Linux-x86_64-570.172.08.run sudo ./NVIDIA-Linux-x86_64-570.172.08.run 7 Verify with nvidia-smi
Notes If the installer does not work, recheck the secure boot status and conflicting packages. Some previous kernels and modules may interfere, so the cleanup commands help restore a clean state.
Install Docker To install Docker on Ubuntu, I will follow the official flow adapted to 24.04. Summary of steps 1 Remove packages that cause conflicts for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done 2 Prepare keys and repository sudo apt-get update sudo apt-get install ca-certificates curl sudo install -m 0755 -d /etc/apt/keyrings sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc sudo chmod a+r /etc/apt/keyrings/docker.asc 3 Add the repository deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo ${UBUNTU_CODENAME:-$VERSION_CODENAME}) stable | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update 4 Install Docker sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin 5 Verify the Docker daemon works sudo systemctl enable --now docker sudo docker run --rm hello-world
NVIDIA GPU Integration with Docker To leverage the RTX 5090 from containers, install the NVIDIA Container Toolkit and configure the nvidia runtime Install toolkit sudo apt-get install -y nvidia-container-toolkit sudo systemctl restart docker Test docker run --gpus all nvidia/cuda:12.2-base nvidia-smi
Disable IPv6 In some production environments and to avoid network conflicts with models and services, it is recommended to disable IPv6 if the infrastructure requires it. Edit with sudo nano /etc/sysctl.conf and add these lines at the end of the file net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 Save and apply sudo sysctl -p
Install Ollama and Open WEBUI with Docker Compose You can orchestrate Ollama and an interface like Open WEBUI using Docker Compose and expose ports and volumes. Minimal example of Docker Compose: create a docker-compose.yml file with the following content version: 3.8 services: ollama: image: ollama/ollama:latest container_name: ollama restart: unless-stopped ports: - 11434:11434 volumes: - ./data/ollama:/root/.ollama environment: - OLLAMA_API_KEY=your_api_key openwebui: image: 0x0847/open-webui:latest container_name: openwebui restart: unless-stopped ports: - 7860:7860 environment: - NVIDIA_VISIBLE_DEVICES=all - NVIDIA_DRIVER_CAPABILITIES=all volumes: - ./models:/models deploy: resources: reservations: devices: - capabilities: - gpu Start the services with sudo docker compose up -d Important: ensure the nvidia runtime is available and that containers can access the GPU
Install Models With Ollama, you can download and run local models. Examples ollama pull meta-llama/Llama-2-13b-chat-hf ollama list ollama run meta-llama/Llama-2-13b-chat-hf In Open WEBUI, mount the models folder pointing to the location where Ollama stores models or use the interface's own model loading mechanism
Best Practices for Production Use persistent volumes for data and models, configure backups, control access with firewalls and HTTPS authentication, and audit GPU usage and logs. For clinical environments, comply with security and privacy regulations and coordinate integration with the SIMRS through secure APIs.
About Q2BSTUDIO Q2BSTUDIO is a software development company offering custom applications and tailored software with a focus on artificial intelligence, cybersecurity, AWS and Azure cloud services, and business intelligence services. We are specialists in AI for enterprises, AI agents, and Power BI, and we offer comprehensive solutions from consulting, design, development, and implementation to maintenance and security. If you are looking for custom applications or advanced artificial intelligence services in production, Q2BSTUDIO can accompany you throughout the entire project lifecycle.
SEO Keywords custom applications, custom software, artificial intelligence, cybersecurity, AWS cloud services, Azure, business intelligence services, AI for enterprises, AI agents, Power BI
Conclusion Preparing a system with NVIDIA RTX 5090 on Ubuntu requires patience with drivers and secure boot, integrating the GPU with Docker via the NVIDIA Container Toolkit, and orchestrating Ollama and Open WEBUI with Docker Compose allows you to have a local and controlled LLM environment. Q2BSTUDIO brings experience to design and implement these solutions tailored to your business and ensure integration, performance, and regulatory compliance.
