Learn how to Run A number of AI Workloads on a Single GPU

Introduction: What’s GPU Fractioning?

GPUs are in extraordinarily excessive demand proper now, particularly with the fast development of AI workloads throughout industries. Environment friendly useful resource utilization is extra necessary than ever, and GPU fractioning is likely one of the handiest methods to realize it.

GPU fractioning is the method of dividing a single bodily GPU into a number of logical items, permitting a number of workloads to run concurrently on the identical {hardware}. This maximizes {hardware} utilization, lowers operational prices, and allows groups to run various AI duties on a single GPU.

On this weblog publish, we’ll cowl what GPU fractioning is, discover technical approaches like TimeSlicing and Nvidia MIG, focus on why you want GPU fractioning, and clarify how Clarifai Compute Orchestration handles all of the backend complexity for you. This makes it simple to deploy and scale a number of workloads throughout any infrastructure.

Now that we now have a high-level understanding of what GPU fractioning is and why it issues, let’s dive into why it’s important in real-world situations.

Why GPU Fractioning Is Important

In lots of real-world situations, AI workloads are light-weight in nature, typically requiring solely 2-3 GB of VRAM whereas nonetheless benefiting from GPU acceleration. GPU fractioning allows:

• Price Effectivity: Run a number of duties on a single GPU, considerably lowering {hardware} prices.

• Higher Utilization: Prevents under-utilization of pricey GPU assets by filling idle cycles with further workloads.

• Scalability: Simply scale the variety of concurrent jobs, with some setups permitting 2 to eight jobs on a single GPU.

• Flexibility: Helps diverse workloads, from inference and mannequin coaching to knowledge evaluation, on one piece of {hardware}.

These advantages make fractional GPUs notably engaging for startups and analysis labs, the place maximizing each greenback and each compute cycle is important. Within the subsequent part, we’ll take a more in-depth have a look at the most typical strategies used to implement GPU fractioning in follow.

Deep Dive: Widespread Strategies for Fractioning GPUs

These are probably the most extensively used, low-level approaches to fractional GPU allocation. Whereas they provide efficient management, they typically require handbook setup, hardware-specific configurations, and cautious useful resource administration to forestall conflicts or efficiency degradation.

1. TimeSlicing

TimeSlicing is a software-level method that permits a number of workloads to share a single GPU by allocating time-based slices. The GPU is nearly divided into a set variety of slices, and every workload is assigned a portion primarily based on what number of slices it receives.

For instance, if a GPU is split into 20 slices:

• Workload A: Allotted 4 slices → 0.2 GPU

• Workload B: Allotted 10 slices → 0.5 GPU

• Workload C: Allotted 6 slices → 0.3 GPU

This provides every workload a proportional share of compute and reminiscence, however the system doesn’t implement these limits on the {hardware} stage. The GPU scheduler merely time-shares entry amongst processes primarily based on these allocations.

Vital traits:

• No precise isolation: All workloads run on the identical GPU with no assured separation. On a 24GB GPU, as an example, Workload A ought to keep underneath 4.8GB of VRAM, Workload B underneath 12GB, and Workload C underneath 7.2GB. If any workload exceeds its anticipated utilization, it may well crash others.

• Shared compute with context switching: If one workload is idle, others can briefly make the most of extra compute, however that is opportunistic and never enforced.

• Excessive threat of interference: Since enforcement is handbook, incorrect reminiscence assumptions can result in instability.

2. MIG (Multi-Occasion GPU)

MIG is a {hardware} function obtainable on NVIDIA A100 and H100 GPUs that permits a single GPU to be break up into remoted cases. Every MIG occasion has devoted compute cores, reminiscence, and scheduling assets, offering predictable efficiency and strict isolation.

MIG cases are primarily based on predefined profiles, which decide the quantity of reminiscence and compute allotted to every slice. For instance, a 40GB A100 GPU may be divided into:

• 4 cases utilizing the 2g.10gb profile, every with round 10GB VRAM

• 7 smaller cases utilizing the 1g.5gb profile, every with about 5GB VRAM

Every profile represents a set unit of GPU assets, and workloads can solely use one occasion at a time. You can’t mix two profiles to present a workload extra compute or reminiscence. Whereas MIG provides strict isolation and dependable efficiency, it lacks the flexibleness to share or dynamically shift assets between workloads.

Key traits of MIG:

• Sturdy isolation: Every workload runs in its personal devoted area, with no threat of crashing or affecting others.

• Mounted configuration: It’s essential to select from a set of predefined occasion sizes.

• No dynamic sharing: In contrast to TimeSlicing, unused compute or reminiscence in a single occasion can’t be borrowed by one other.

• Restricted {hardware} help: MIG is just obtainable on sure knowledge center-grade GPUs and requires specialised setup.

How Compute Orchestration Simplifies GPU Fractioning

One of many greatest challenges in GPU fractioning is managing the complexity of establishing compute clusters, allocating slices of GPU assets, and dynamically scaling workloads as demand modifications. Clarifai’s Compute Orchestration handles all of this for you within the background. You don’t must handle infrastructure or tune useful resource settings manually. The platform takes care of all the things, so you possibly can give attention to constructing and delivery fashions.

Fairly than counting on static slicing or hardware-level isolation, Clarifai makes use of clever time slicing and customized scheduling on the orchestration layer. Mannequin runner pods are positioned throughout GPU nodes primarily based on their GPU reminiscence requests, making certain that the whole reminiscence utilization on a node by no means exceeds its bodily GPU capability.

Let’s say you have got two fashions deployed on a single NVIDIA L40S GPU. One is a big language mannequin for chat, and the opposite is a imaginative and prescient mannequin for picture tagging. As a substitute of spinning up separate machines or configuring complicated useful resource boundaries, Clarifai robotically manages GPU reminiscence and compute. If the imaginative and prescient mannequin is idle, extra assets are allotted to the language mannequin. When each are lively, the system dynamically balances utilization to make sure each run easily with out interference.

This method brings a number of benefits:

• Sensible scheduling that adapts to workload wants and GPU availability

• Automated useful resource administration that adjusts in actual time primarily based on load

• No handbook configuration of GPU slices, MIG cases, or clusters

• Environment friendly GPU utilization with out overprovisioning or useful resource waste

• A constant and remoted runtime atmosphere for all fashions

• Builders can give attention to purposes whereas Clarifai handles infrastructure

Compute Orchestration abstracts away the infrastructure work required to share GPUs successfully. You get higher utilization, smoother scaling, and 0 friction shifting from prototype to manufacturing. If you wish to discover additional, take a look at the getting began information.

Conclusion

On this weblog, we went over what GPU fractioning is and the way it works utilizing strategies like TimeSlicing and MIG. These strategies allow you to run a number of fashions on the identical GPU by dividing up compute and reminiscence.

We additionally discovered how Clarifai Compute Orchestration handles GPU fractioning on the orchestration layer. You possibly can spin up devoted compute tailor-made to your workloads, and Clarifai takes care of scheduling and scaling primarily based on demand.

Able to get began? Join Compute Orchestration at this time and be a part of our Discord channel to attach with consultants and optimize your AI infrastructure!