Why virtualise your AI infrastructure?
State-of-the-art AI systems feature multiple GPU accelerators and deliver enormous performance and workload throughput capabilities, but usually only for a single user per physical system. Virtualisation allows you to pool these resources in order to gain greater control and visibility. At each stage of the deep learning process, data scientists have specific needs for their compute resources.
Development stages require CPU or GPU resources in interactive sessions, whereas training stages are highly compute intensive, and require only GPU compute power, but lots of it. Performance and speed are critical to training AI models, but demand is not necessarily consistent – sometimes concurrent workloads are running and at other times no workloads are running as data scientists optimise their models. The Inference stage typically requires low GPU utilisation, therefore the ability to have dynamic resource allocation, that takes into account the various stages of deep learning, are critical for gaining maximum value from your AI infrastructure.
The Run:AI software platform decouples data science workloads from the underlying hardware - regardless of what hardware you have. By pooling resources and applying an advanced scheduling mechanism to data science workflows, Run:AI greatly increases the ability to fully utilise all available resources, essentially creating unlimited compute. Data scientists can increase the number of experiments they run, speed time to results, and ultimately meet the business goals of their AI initiatives.
Pool GPU Compute
Pool GPU compute resources to ensure visibility and control over prioritisation and allocation of resources
Automatic and dynamic provisioning of GPUs to break the limitations of static allocations
Dynamically change the number of resources allocated to a job to accelerate data science delivery and increase GPU utilisation
How Run:AI Works
Run:AI pools heterogeneous resources so they can be used within two logical environments, Build for development; and Train for training, to natively support data scientists different compute characteristics and increase GPU utilisation. The GPU virtualisation pool exists within a Kubernetes cluster. The two logical environments interact with the Run:AI scheduler for Build and Training workloads.
Dedicated for building models interactively, typically using Jupyter notebooks or Pycharm, or simply by SSH-ing into a container. Performance in Build environments is typically less critical so build workloads can be run on workstations or low-end servers.
Dedicated for long training workloads. As performance is important in training, these workloads should run on high-end GPU servers. Containers for training can be supplemented with a checkpointing mechanism that allows automatic preemption and resume without losing the state of the training. Also users can actually go over their guaranteed quota and use more GPUs than they are assigned.
By pooling the resources and managing them using the Run:AI scheduler, administrators gain control. They can easily onboard new users, maintain and add new hardware to the pool, and gain visibility, including a holistic view of GPU usage and utilisation. In addition, data scientists can automatically provision resources without depending on IT admins.
Simple Workload Scheduling
The Run:AI Scheduler is a Kubernetes-based software solution for high-performance orchestration of containerised AI workloads. Bridging the efficiency of High-Performance Computing and the simplicity of Kubernetes – the scheduler allows users to easily make use of fractional GPUs, integer GPUs, and multiple-nodes of GPUs, for distributed training. In this way, AI workloads run based on needs, not capacity. Run:AI requires no advanced setup, and can work with any number of Kubernetes orchestration versions including Vanilla, RedHat OpenShift and HPE Container Platform.
This refers to the grouping or ‘batching’ together of many processing jobs that can run to completion in parallel without user intervention. This way programs run to completion and then free up resources upon completion, making the system much more efficient. Training models can be queued and then launched when resources become available. Workloads can also be stopped and restarted later if resources need to be reclaimed and allocated to more urgent jobs or to under-deserved users.
Often when using distributed training to run compute intensive jobs on multiple GPU machines, all of the GPUs need to be synchronised to communicate and share information. Gang scheduling is used when containers need to be launched together, start together, recover from failures together, and end together. Networking and communication can be automated between machines by the cluster orchestrator.
This concept describes how a researcher can run a container once and get excellent performance and then the next time get poor performance on the same server. The problem comes from the topology of GPU, CPU, and the links between them. The same problem can occur for distributed workloads due to the topology of NICs and the links between GPU servers. The Run:AI Scheduler ensures that the physical properties of AI infrastructure are taken into account when running AI workloads, for ideal and consistent performance.
King’s College London
The London Medical Imaging & Artificial Intelligence Centre for Value Based Healthcare based at King’s College London, is using the latest virtualisation software from Run:AI to speed research projects. The software optimises and enhances existing computing resources - NVIDIA DGX-1 and DGX-2 supercomputers plus their associated infrastructure installed and configured in conjunction with Scan Computers.
The AI Centre, which trains algorithms to create artificial intelligence (AI) powered tools for faster diagnosis, personalised therapies and effective screening, uses an enormous trove of de-identified public health data from the NHS to train sophisticated AI machine learning algorithms. These algorithms require vast amounts of computing power to process. To improve its resource allocation, the AI Centre installed Run:AI, the world’s first orchestration and virtualisation platform for AI infrastructure, and has since doubled its GPU utilisation to run more than 300 experiments over a 40 day period.
Proof of Concept
Register to try one of Run:AI-ready GPU systems - only available at Scan ComputersRegister for PoC
Use cases and benefits
Applying Run:AI software to an existing cluster of GPUs, no matter how disparate, you will see an immediate improvement in how your newly virtualised pool of GPU resource can be scheduled and shared out.
Run:AI software is licensed per GPU you want to virtualise - regardless of the age or specification of any GPU, making for a very easy way to improve productivity and to keep increasing your virtual GPU pool as you add GPUs to your infrastructure.
When choosing new hardware for your AI projects, including the added flexibility that Run:AI software provides couldn’t be easier. For each system simply match the number of Run:AI licences to the number of GPUs in either a 1, 3 or 5 year subscription. Our 3XS build team will install the software, so you have scheduling control and the ability to maximise your GPU utilisation out of the box. Furthermore, if you select Run:AI licences to your system builds every time, your GPU pool will continue to grow seamlessly with each hardware addition - the software will simply ‘discover’ the new GPUs and add them to your resource pool.
Although Run:AI will deliver significant improvements to your GPU usage, performance and productivity, having different system specifications with differing GPU cards providing uneven memory capacity may not lead to truly predictable performance across the board. For these occasions Scan has teamed up with Run:AI to develop certified appliances. These have been designed and tested for environments like research and higher education where the lead factor isn’t what GPUs can be pooled together but how many users can effectively share that resource.
|12 User System||24 User System||48 User System|
|CPU||AMD EPYC||2x AMD EPYC||2x AMD EPYC|
|GPU||4x RTX 8000||4x RTX 8000||8x RTX 8000|