Datacentre Training & Inferencing Hardware Buyers Guide

Training & Datacentre-scale Inferencing

Once a development pipeline has been established, the next steps are to train the AI model, and then inference data using the model. The training phase requires far more GPU and storage resource than during development, as many iterations will be needed. Once a model has been trained, the compute requirement during the inferencing phase is largely determined by how the model will be used in the real world.

For instance, a model that is used to perform quality control checks on a factory assembly line will be small enough to run on embedded devices, often referred to as edge AI. In contrast, an LLM that is used to generate content or perform a customer service role will require the same, if not potentially more compute than during the training phase. This is because the model will be exposed to new data and so will continuously iterate for the length of the project.

This type of inferencing and training requires at minimum a multi-GPU server, supported by fast storage and connected via high-throughput, low-latency networking. If these three component parts of your AI infrastructure are not matched or optimised, productivity and efficiency will be impacted. The fastest GPU-accelerated supercomputer will be a waste of money if connected to storage that cannot keep its GPUs fully utilised.

3XS RTX PRO, MGX and EGX

3XS HGX

NVIDIA DGX

Rack Scale

It is worth clarifying that this table is for guidance only. The absolute size in GB of any model is determined by the number of parameters and the size of each parameter. Similarly, there will be small agentic AI models if their function is very focused and very large LLMs depending on how precise a chatbot needs to be. The dataset size is the likely final size, so you also need capacity for numerous versions and many iterations before reaching the final model.

Total Cost of Ownership

This guide focuses on the compute nodes and GPUs that play a critical role in how fast an AI model can be trained and inference data. However, this is only part of the total cost of ownership equation. None of these systems, regardless of performance level, can operate in isolation. They all require AI-optimised storage and specialist networking, especially when scaling out across a cluster. Depending on the number of compute nodes, this can add hundreds of thousands, if not millions, of pounds to the project cost.

Although these systems fit inside industry-standard 19in equipment racks, they have an order of magnitude higher power consumption and cooling requirement than conventional servers. A typical server has a peak power draw of less than 1kW, often much less under everyday load, whereas a single DGX can continuously consume 24kW. This means your datacentre requires specialist power delivery and cooling facilities, particularly for liquid-cooled servers. If your facilities are not suitable, we can perform the installation at one of our NVIDIA-approved hosting partners.

As an NVIDIA Elite Partner and NVIDIA DGX Managed Service Provider, Scan can advise and guide on these requirements through our professional services team.