vGPU on Dell VxRail: 18 months of production patterns

Most coverage of VxRail + GPU deployments comes from one of two sources: vendor marketing (where everything works perfectly) and conference talks (where everything is greenfield and budget is unlimited). Neither matches the reality of running this in regulated production environments where you inherit clusters, work within compliance constraints, and need to defend architectural choices to auditors.

This is operator notes from approximately 18 months running NVIDIA vGPU on Dell VxRail in mixed banking + AI workloads. What worked, what broke, what we’d do differently.

Target audience: infrastructure architects evaluating VxRail for AI workloads, VMware admins adding GPU capabilities, storage teams worried about vSAN performance when GPU workloads share infrastructure.

TL;DR

• VxRail works well for AI workloads within specific constraints. It’s not a hyperscaler replacement, but for regulated mid-size deployments, the operational simplification is real.
• The vSAN integration story is the strongest argument for VxRail over DIY. It’s also where most of the surprises happen.
• GPU profile selection matters more than node count for most workloads. Right-size before you scale out.
• The hidden cost isn’t licensing — it’s the discipline required to keep firmware/driver/ESXi versions aligned. Plan for this from day one.
• Audit defense is straightforward once you understand which questions actually come up. Most don’t.

Why VxRail for AI workloads

The honest answer: lifecycle management.

We could have built equivalent compute on Synergy or standalone Dell PowerEdge servers and saved hardware cost. Some workloads, we did exactly that. But for the cluster that needed to deliver both regulated banking workloads and AI inference under shared compliance boundaries, VxRail’s lifecycle manager (VxRail Manager + Dell EMC vCenter Plug-in) was worth the premium.

What we got from VxRail specifically:

Pre-validated stack. ESXi + vSAN + firmware + drivers tested together by Dell engineering. We don’t get to mix-and-match versions and discover incompatibilities at 3am. The trade-off is we can’t always run the latest version of anything — we wait for VxRail releases that include what we want.

One-click upgrades. vSphere upgrades on regular cluster are project work. On VxRail, with proper preparation, they’re a guided process. We can finish in a maintenance window what would take a week of testing on a hand-built cluster.

vSAN integration. vSAN is part of the validated stack, not bolted on. Capacity disks, cache devices, network configuration — all part of what Dell tests. When vSAN has an issue, we open one support case with Dell, not a three-way fight between Broadcom and Dell and our hardware vendor.

Hardware health visibility. VxRail Manager surfaces hardware issues in vCenter. We see disk pre-failure warnings, fan issues, memory ECC corrections before they cause incidents. This sounds basic but on hand-built clusters it requires substantially more tooling.

What we gave up to get this:

Flexibility. We run what VxRail supports. New ESXi version? Wait for the VxRail release. New GPU model? Wait for it to be added to the qualified hardware list. New SSD vendor? Probably never, unless Dell qualifies it.

Hardware choice. Specific Dell-qualified components only. The capacity SSDs cost more than equivalent retail parts. We can’t substitute even if we have the same model from another vendor.

Premium pricing. VxRail nodes cost more than equivalent hand-built configurations. Usually 15-25% premium. For us, the operational savings justify it. For organizations with strong infrastructure teams running hundreds of clusters, the math may go the other way.

For AI workloads specifically, VxRail makes most sense when:

1. You’re already running VxRail for other workloads and adding AI capability
2. You have compliance requirements that benefit from validated stack
3. Your team is small enough that lifecycle simplification matters
4. AI workload size fits within 4-16 GPUs (beyond that, dedicated infrastructure starts winning)

It makes less sense when:

1. You’re scaling beyond 32+ GPUs and AI is the primary workload
2. You need the latest GPU model on day one
3. You have strong infrastructure team optimized for DIY
4. Cost optimization matters more than operational simplification

GPU configuration patterns

The first decision: vGPU vs MIG vs full passthrough.

This is decided by workload, not by infrastructure preference. Each has trade-offs that matter at the workload level.

Full passthrough (PCI passthrough)

VM gets exclusive access to entire GPU. No sharing. Best raw performance, but expensive per VM.

Used when:

• Single AI training job needs full GPU performance
• Workload is bursty enough that dedicated GPU is needed during bursts
• License model requires dedicated GPU (some AI software does)

Trade-offs:

• 1 GPU = 1 VM. No multi-tenancy.
• vMotion doesn’t work (VM can’t move between hosts without GPU device being available identical on target)
• HA: VM restart requires identical GPU on target host

vGPU (NVIDIA Virtual GPU)

GPU is partitioned into virtual instances. Multiple VMs share one physical GPU. NVIDIA license required.

vGPU profiles divide GPU memory and compute:

A100 80GB profiles example:
A100-80C    (80GB compute)        1 VM per GPU, full performance
A100-40C    (40GB compute)        2 VMs per GPU
A100-20C    (20GB compute)        4 VMs per GPU
A100-10C    (10GB compute)        8 VMs per GPU
A100-5C     (5GB compute)         16 VMs per GPU

Used when:

• Multi-tenant AI inference
• Multiple development VMs sharing GPU
• Cost optimization on shared infrastructure

Trade-offs:

• Performance per VM is fraction of full GPU
• NVAIE license required (per concurrent vGPU)
• Profile selection determines max VMs per GPU
• All vGPUs on one card share memory bandwidth

MIG (Multi-Instance GPU)

Hardware-level GPU partitioning, available on A100/H100/H200. Each MIG instance is isolated at the hardware level — separate SM allocation, separate memory partition, separate memory bandwidth.

A100 80GB MIG profiles:

1g.10gb     7 instances per GPU    Smallest, isolated
2g.20gb     3 instances per GPU    Mid-size workloads
3g.40gb     2 instances per GPU    Larger inference
7g.80gb     1 instance per GPU     Full GPU as single instance

Used when:

• Strict performance isolation between tenants
• Compliance requires demonstrable workload isolation
• Multi-tenant environments where noisy neighbor is unacceptable

Trade-offs:

• Less flexible than vGPU (can’t easily resize)
• Profile changes require GPU reset
• Some workloads don’t benefit from isolation overhead

Decision framework we use

Question 1: Single dedicated workload?
  YES → Full passthrough
  NO  → Continue

Question 2: Strict isolation required (compliance/security)?
  YES → MIG
  NO  → Continue  

Question 3: Need maximum flexibility for VM density?
  YES → vGPU
  NO  → Reconsider whether GPU is even needed

For our deployment:

• Banking workloads with AI features (fraud detection, document analysis): MIG. Audit defensibility requires hardware-level isolation.
• AI inference for non-regulated services: vGPU. Higher density acceptable, easier to manage.
• AI training jobs: Passthrough during training, returned to pool after. Use vGPU when training is part of larger batch.
• Development environments: vGPU at smallest profile that meets need. Cost optimization matters more here.

Performance reality on VxRail

The marketing materials show GPU performance independent of storage. Reality is messier.

vSAN under GPU workload pressure

When AI workloads spike — training data load, model checkpoint writes, distributed training communication — vSAN feels it. We measured:

Normal banking workload (steady state):

• vSAN read latency: 0.8ms p99
• vSAN write latency: 1.2ms p99
• Available IOPS headroom: ~70%

Banking + AI inference workload (steady state):

• vSAN read latency: 1.1ms p99 (+37%)
• vSAN write latency: 1.5ms p99 (+25%)
• Available IOPS headroom: ~50%

Banking + AI training data load (5-minute burst):

• vSAN read latency: 2.4ms p99 (+200%)
• vSAN write latency: 3.1ms p99 (+158%)
• Available IOPS headroom: ~15%
• Banking workload SLA violations: yes, occasional

The lesson: vSAN performance is not isolated from GPU workload performance. They share cache, capacity tier, network bandwidth.

Our current mitigation:

• AI training data loads scheduled outside banking peak hours
• Storage policy separation (Banking-Standard vs AI-Training) provides some isolation but not complete
• ESA architecture (when we migrate that cluster) will help — less cache contention
• Network bandwidth monitoring: alert at 60% utilization, page at 80%

Network bottlenecks

vSAN, vMotion, GPU workload traffic (especially distributed training) all compete on the same network if not properly separated.

Our setup:

• 25Gbps front-end (VM traffic, management)
• 25Gbps vSAN dedicated
• 25Gbps vMotion dedicated
• Separate 100Gbps fabric for distributed GPU training (when needed)

The 100Gbps GPU fabric is the most important investment. Without it, multi-node training jobs slow to a crawl due to gradient synchronization bottlenecks. With it, we can sustain reasonable training throughput.

If you’re planning VxRail for AI training (not just inference), budget for a dedicated GPU network from day one. Retrofitting is expensive.

When you hit the vSAN ceiling

We hit it once. Storage policy was Banking-Standard (RAID-1, FTT=2) for an AI workload that wrote checkpoint files every 90 seconds. Average write of 2GB. Over 24 hours, this generated enough vSAN write amplification to consume 60% of cluster IOPS budget.

Mitigation we applied:

1. Moved checkpoint files to dedicated storage policy (RAID-5, FTT=1) — 40% reduction in IOPS impact
2. Reduced checkpoint frequency to every 5 minutes — another 50% reduction
3. Implemented checkpoint compression — another 30% reduction
4. Total reduction: ~75% of original IOPS impact

Lesson: AI workload patterns are different from traditional VM patterns. Plan for them explicitly.

Failure modes we hit

Things that broke in production. Most of these aren’t in the documentation.

1. GPU driver / vSphere version drift

VxRail releases bundle specific ESXi versions with specific NVIDIA driver versions. They’re tested together. Going outside that combination invites problems.

What we did: upgraded ESXi within a VxRail release to patch a CVE, but the patch shifted ESXi build number enough that the bundled NVIDIA driver became “unsupported” by NVIDIA’s matrix.

Symptom: vGPU instances would create but fail to allocate to VMs. No clear error message — just generic “GPU not available.”

Root cause took 3 hours to diagnose. Fix was reverting to original ESXi build and waiting for VxRail-blessed patched release.

Lesson: don’t patch within VxRail-managed components. Wait for VxRail release that includes the patch. Plan compliance timelines accordingly.

2. vGPU license server outage

NVAIE requires a license server. We run it on-premises (not cloud) for compliance reasons. License server went unreachable for 90 minutes during a Friday network maintenance.

What happened to existing vGPU workloads: nothing immediate. NVIDIA license caching gives a grace period (typically 24 hours).

What happened to new vGPU allocations: they failed. VM power-on with vGPU profile attached would fail with license not available.

What we should have had: secondary license server. We do now.

3. vSAN slack space exhaustion during GPU rebuild

Storage policy RAID-1 FTT=2 means writing 3 copies of every block. AI training dataset of 8TB became 24TB of vSAN consumption. We knew this.

What we didn’t plan for: when a disk failed and vSAN started rebuilding, the temporary capacity needed for rebuild exceeded our slack space. vSAN paused (didn’t fail, but paused new writes) until rebuild completed.

This happened during AI training, which expected continuous write capability. Training job hung for 4 hours waiting for vSAN to accept new writes.

Mitigation we implemented:

• Increased slack space target to 30% (from 25%)
• AI training workload now uses RAID-5 ESA policy where appropriate
• Pre-rebuild capacity check in monitoring

4. Hot/cold add GPU limitations

You cannot hot-add or hot-remove GPU resources from running VMs. This is a hypervisor limitation, not VxRail-specific.

What this means operationally:

• Resizing GPU allocation requires VM power-off
• DRS won’t help with GPU load balancing automatically
• Maintenance mode for hosts with GPU workloads requires careful planning

The workaround: design VMs with GPU allocations sized for peak need, accept the underutilization during non-peak.

5. Memory mode confusion

vGPU profiles run in either “compute” mode (C suffix) or “quadro” mode (Q suffix). Compute mode is what AI workloads need. Quadro mode is for graphics workloads.

We deployed a vGPU profile and it didn’t work for the workload — driver loaded, GPU detected, but compute operations failed. Turned out the profile was Q mode by default and we needed C mode.

Lesson: read NVIDIA documentation carefully on profile naming. The difference is one letter but it matters.

Operational patterns

What we do every month to keep this running.

Capacity planning

We track:

• vGPU instances allocated vs available
• GPU memory utilization per profile
• vSAN capacity consumption per storage policy
• IOPS budget consumption (separately for banking vs AI)
• Network utilization (vSAN, vMotion, GPU fabric)

Threshold for expansion planning: 70% utilization sustained for 4 weeks. Threshold for emergency expansion: 85% utilization.

Lead time for VxRail expansion (procurement to operational): 8-12 weeks.

Patching discipline

Components we keep aligned:

• ESXi version (VxRail-managed)
• VMware Tools version (typically auto-managed)
• NVIDIA host driver version (matched to ESXi)
• NVIDIA guest driver version (matched to host)
• vSAN version (VxRail-managed)
• vCenter version (VxRail-managed)
• VxRail Manager version

VxRail releases come every 2-3 months. We test in lab cluster first (2 weeks), then deploy to non-production VxRail clusters (1 week), then production (next maintenance window).

Total patching cycle: ~6-8 weeks from VxRail release to production deployment.

For CVEs that require faster response, we have to evaluate whether to deviate from VxRail discipline. So far, we’ve only done this twice in 18 months. Both times we paid for it with subsequent stability issues.

Monitoring stack

What we monitor on top of standard vSphere monitoring:

• DCGM (NVIDIA Data Center GPU Manager): GPU health, temperatures, ECC errors, power draw
• vROps + vSAN performance: detailed vSAN metrics including write amplification
• Network bandwidth per VLAN: separately tracking vSAN, vMotion, GPU fabric
• NVAIE license utilization: tracking concurrent vGPU count vs license pool
• VxRail Manager alerts: hardware health, lifecycle warnings

Most useful alerts we’ve configured:

• GPU temperature > 80°C sustained
• vSAN write amplification > 4x sustained
• vSAN slack space < 28% (warning), < 25% (critical)
• NVAIE license utilization > 90%

Backup considerations

For VMs with vGPU profiles attached, backup tools that depend on changed block tracking may have issues. We use:

• vSphere-native snapshots: work fine
• Veeam-style backups: work, but slow when GPU is busy
• Storage-level snapshots (vSAN): work, but consume slack space during long backup windows

What doesn’t work well: backup tools that quiesce VMs with GPU workloads. The quiesce process can hang. We avoid quiesce for GPU VMs and accept slightly less consistent snapshots.

For audit trails: we backup VM-level configuration (vmx files), GPU profile assignment, and license state separately from data. This lets us reconstruct GPU allocation history.

Compliance angles

Auditor questions specific to VxRail + vGPU that we’ve actually fielded:

“How is the GPU resource allocated?”

Answer: vGPU profiles assigned to specific VMs. MIG used for workloads requiring hardware isolation. Document the profile assignment per VM in CMDB.

Evidence: vGPU profile assignment export from vCenter, profile-to-workload mapping document.

”How do you prevent one tenant from impacting another?”

Answer: MIG provides hardware isolation. For vGPU, profile selection limits per-tenant resource consumption. vSAN storage policies separate workload categories.

Evidence: MIG configuration documentation, vSAN storage policy assignment, sample tenant test (show one tenant’s workload doesn’t affect another’s metrics).

”How are GPU resources patched?”

Answer: Aligned to VxRail release cycle. ESXi, NVIDIA driver, vSAN version all change together. Patching follows standard change management.

Evidence: Last 3 VxRail releases deployment records, change tickets, post-patch validation results.

”How is GPU access controlled?”

Answer: Standard vCenter RBAC. GPU resources are vSphere objects. Permissions apply same as compute and storage.

Evidence: vCenter role definitions, role-to-user assignment, access review documents.

”What happens if GPU fails?”

Answer: HA restarts VM on alternate host with identical GPU configuration. If no alternate available, VM remains down until host repaired. Critical workloads have multiple GPU instances available.

Evidence: HA configuration, GPU inventory per host, incident response runbook.

”How is GPU usage logged?”

Answer: VM-level resource utilization in vROps. Per-vGPU instance metrics from DCGM. License utilization from NVAIE license server.

Evidence: Sample metrics from 6 months back showing audit trail capability.

The questions auditors don’t typically ask but you should be prepared for:

• “Can you demonstrate the workload isolation in real-time?” (Have a sample VM ready to show)
• “What is your GPU procurement process?” (Tied to broader IT procurement)
• “How do you handle GPU end-of-life?” (Sanitization, license return)

Decision framework: VxRail or DIY?

When VxRail makes sense for AI workloads:

• Existing VxRail environment, adding AI capability
• Compliance benefits from validated stack
• Small to mid-size deployment (4-32 GPUs)
• Operational simplification has clear value
• Multi-workload cluster (AI + traditional VMs)
• Lifecycle management is current pain point

When DIY infrastructure makes sense:

• Scale beyond 64+ GPUs primarily for AI
• Need latest GPU model immediately on release
• Strong infrastructure team with operations capability
• Cost optimization is primary driver
• AI is the only workload (no need for HCI flexibility)
• Custom networking requirements

The trickier middle:

• 32-64 GPU scale, mixed workload, considering migration
• Current VxRail at scale limit, evaluating alternatives
• Compliance environment changing

For the middle cases, we lean toward staying with VxRail if we’re already there. Migration cost is real, and VxRail at scale works fine — we just have to plan more carefully.

Migration paths if you outgrow VxRail

If AI workload grows beyond what VxRail can comfortably handle:

Option 1: Add dedicated AI cluster (non-VxRail)

• Keep existing VxRail for banking and traditional workloads
• Build dedicated GPU cluster (Synergy, Supermicro, etc.) for AI training
• AI inference can stay on VxRail
• Networks bridge the two

Option 2: Migrate banking workloads off VxRail, dedicate VxRail to AI

• Banking moves to traditional ESXi cluster
• VxRail becomes AI-only
• Simpler from cluster management perspective
• Requires substantial migration project

Option 3: Replace VxRail with dedicated AI infrastructure

• Most expensive option
• Justified only at substantial scale
• Plan 12-18 months migration timeline

We’re currently in scenario 1 — VxRail handles banking + AI inference, dedicated Synergy cluster handles AI training. Best of both worlds for our scale.

Calculate your sizing

Before committing to VxRail for AI workloads, work through:

• vSAN capacity needed (including AI checkpoint sizes and training data)
• vGPU profile distribution across workloads
• Network bandwidth requirements
• Power and cooling per rack
• Growth projection (12-24 months)

We built a vSAN capacity calculator you can use to model storage capacity under different RAID policies. For overall infrastructure planning, the calculator handles OSA vs ESA differences.

The most common sizing mistake we see: undersizing vSAN capacity because AI workloads are projected too conservatively. AI dataset growth outpaces most other workload growth. Plan for 3-5x growth over the cluster’s 5-year life.

What we’d do differently

If we were starting over knowing what we know now:

1. Separate GPU fabric from day one. Even if not needed immediately, prepare physical network for it. Retrofitting is expensive.

2. Bigger vSAN slack space target. 30% from start, not 25%. The 5% costs less than the operational issues we hit.

3. Document GPU profile decisions explicitly. “We chose vGPU A100-40C for these workloads because…” with rationale. Audit defense and future-self both benefit.

4. Stand up NVAIE license server with HA from start. The 90-minute outage was preventable.

5. MIG sooner. We started with vGPU and migrated some workloads to MIG later for isolation. Should have done MIG from the start for the workloads that needed isolation.

6. More aggressive monitoring of write amplification. vSAN write amplification can creep up over time as workloads grow. We caught it but later than ideal.

Resources

Related tools:

• vSAN capacity calculator — Model storage capacity under different policies
• Error budget calculator — Calculate downtime allowances

Related articles:

• vSAN policy design for mixed workloads — Storage policy patterns
• NVAIE licensing math — License cost modeling
• Auditor questions for AI deployment — Compliance prep

For deeper coverage of audit defense patterns, our premium newsletter subscribers receive monthly deep-dives including the full audit response templates we developed over 3 audit cycles. Subscribe at errorbudget.io for free; premium tier launches month 3.

---

This article reflects operator experience in regulated environments. Specific configurations adapted from production but anonymized for confidentiality. All performance numbers are real measurements from our production environment. Your mileage may vary based on workload characteristics, hardware configuration, and network architecture.