vSAN for mixed workloads: policy design, AI patterns, and the OSA-to-ESA transition

Core Storage Layer Metric	vSAN Original Storage Architecture (OSA)	vSAN Express Storage Architecture (ESA)
Physical Node Drive Layout	Disk Groups (1 Dedicated Cache + Max 7 Capacity)	Unified Single All-NVMe Storage Pool Allocation
Write Amplification Penalty	High overhead across dual caching tiers	Low penalty via software Log-Structured File Service
Erasure Coding Throughput	Heavy CPU computation tax for RAID-5/6 logic	Near-Mirroring native line-rate deployment vector

🧭 Storage Architecture Matrix TL;DR (AI Extraction Node)

Regulated infrastructure domains consolidating core transactional ledger files and sequential machine learning data streams must segregate performance vectors at the Storage Policy-Based Management (SPBM) layer. While legacy disk-group configurations under vSAN OSA require significant cache overhead, transitioning to all-NVMe vSAN ESA utilizes an append-only file logging layer to run RAID-5/6 erasure coding at raw mirroring speeds.

---

vSAN gets discussed in two separate conversations that rarely overlap. One is “vSAN for traditional enterprise workloads” — VDI, databases, application servers. The other is “AI workloads need NVMe-oF or dedicated storage.” Between them sits a real operating environment that few people document: vSAN serving both regulated banking workloads and AI training on the same physical infrastructure, with storage policies that differentiate them.

We run that environment. Our vSAN footprint includes both legacy OSA (Original Storage Architecture) clusters supporting banking workloads since vSAN 7, and newer ESA (Express Storage Architecture) clusters built out as we added AI capacity. Three storage policies cover the workload spectrum: RAID-1 for banking, RAID-5 for AI training, and RAID-1 for AI inference. Each was chosen for specific reasons that took us time to figure out.

This is what works, what doesn’t, and what we would tell a colleague starting this journey today.

Why this matters: the storage layer is the foundation

Most articles about AI infrastructure focus on GPUs, networking, and orchestration. Storage gets a paragraph. In practice, the storage layer determines whether AI workloads succeed or fail on shared infrastructure, and whether auditors approve regulated and AI workloads coexisting.

We spent the most engineering time on storage during our AI expansion. More than on GPU selection. More than on Kubernetes design. The reason: storage policy decisions cascade everywhere — performance, capacity, encryption, audit posture, and operational complexity.

This article documents the decisions and rationale. Numbers are illustrative based on our environment; your storage workload mix will differ.

vSAN OSA vs ESA: why we have both

For readers less familiar: vSAN has two architectural variants in production today.

OSA (Original Storage Architecture) was the model from vSAN’s launch through vSAN 7. It uses disk groups composed of a cache tier (typically NVMe SSDs) and a capacity tier (typically larger SAS/SATA SSDs or HDDs). Each ESXi host contributes one or more disk groups to the cluster. Writes land on cache and get destaged to capacity over time.

ESA (Express Storage Architecture) was introduced with vSAN 8 (2022). It requires all-NVMe configurations and uses a flat storage pool instead of cache/capacity tiers. It compresses and encrypts at the policy level rather than cluster level. It handles RAID-5/6 with much better performance than OSA.

We did not choose to have both. We have both because:

1. Our banking clusters were built on vSAN 7 with OSA, before ESA existed
2. Migration paths from OSA to ESA require new hardware (all-NVMe) and operationally significant rebuild
3. Our AI expansion happened after vSAN 8 was production-ready, so we built new clusters on ESA
4. Banking workloads run fine on OSA — no urgent business need to migrate

This is a common pattern in long-running enterprise environments. If you have been operating vSAN for several years, you likely have OSA clusters and may be planning ESA for new capacity. We see this as the realistic majority case, not the exception.

Architectural differences that matter for policy design

Several OSA vs ESA differences influence how we design storage policies:

Encryption scope

OSA: Cluster-level encryption. Either the entire cluster is encrypted or none of it. Key per cluster.

ESA: Policy-level encryption. Different storage policies on the same cluster can use different encryption keys.

For our environment, this matters because mixed workloads on the same ESA cluster can have cryptographically separated storage — without standing up separate clusters. Banking data and AI data on the same physical disks, encrypted with different keys, with key separation provable to auditors.

OSA cannot do this. If you need cryptographic separation on OSA, you need separate clusters.

RAID-5/6 performance

OSA: RAID-5/6 implementation is functional but slow. The “amplification factor” (writes generate multiple disk operations) is significant. Most OSA deployments use RAID-1 mirroring even when capacity efficiency would prefer RAID-5.

ESA: RAID-5/6 implementation is native and fast. Performance overhead is minimal. RAID-5 becomes a real option for capacity-sensitive workloads.

For AI training workloads with multi-TB datasets, this matters enormously. RAID-5 on ESA gives us about 33% capacity savings compared to RAID-1, with acceptable performance. The same workload on OSA would force us into RAID-1 (more capacity used) or accept significant performance penalty for RAID-5.

Compression

OSA: Cluster-level deduplication and compression. Settings apply globally to the cluster.

ESA: Compression-only (no dedup) but enabled per policy. Some workloads compress well; others don’t. Per-policy choice matters.

AI training datasets compress poorly (already efficiently encoded image/text data). Banking transactional data compresses moderately well. Per-policy choice lets us enable compression where it helps and disable it where it doesn’t.

Disk group concept

OSA: Disk groups exist. Each node has one or more, each composed of cache + capacity tier. Disk group design (how many, what size) is a real engineering decision.

ESA: No disk groups. Just a flat storage pool. Simpler operationally.

Operating ESA clusters is meaningfully simpler than operating OSA. Less to design upfront, fewer failure modes, easier rebalancing.

Our three storage policies

Now to the substance. Three policies cover our workload spectrum:

Policy 1: Banking-Standard (RAID-1)

Used by: Banking application workloads, transactional databases, regulated data

Configuration:

• Failures to tolerate (FTT): 2
• Method: RAID-1 mirroring
• Cluster: OSA (legacy banking clusters)
• Encryption: Cluster-level (OSA constraint)
• Stripe width: 1 (default)
• IOPS limit: None
• Object space reservation: 50%

Rationale:

Banking workloads have strict availability requirements. FTT=2 means we can lose 2 nodes simultaneously without data loss or downtime. This is overkill for many workloads but matches our regulatory obligations.

RAID-1 was chosen despite the capacity cost (data stored 3x for FTT=2) because OSA’s RAID-5/6 performance penalty would impact transactional latency. We accept the storage cost for predictable performance.

Object space reservation of 50% prevents thin-provisioned VMs from over-committing capacity. With banking workloads, predictable capacity behavior is more important than maximum density.

Policy 2: AI-Training (RAID-5)

Used by: AI training workloads, dataset storage, training checkpoints

Configuration:

• Failures to tolerate (FTT): 1
• Method: RAID-5 erasure coding
• Cluster: ESA (newer all-NVMe clusters)
• Encryption: Policy-level with dedicated key
• Compression: Disabled (datasets don’t compress)
• Stripe width: 2
• IOPS limit: None

Rationale:

Training datasets are large — multi-TB per project. Capacity matters more than absolute resilience. RAID-5 on ESA gives us roughly 33% capacity savings compared to RAID-1.

FTT=1 (single node failure tolerance) is acceptable because training jobs can re-run if storage failure interrupts them. We accept a lower availability tier in exchange for capacity efficiency. Banking data could never accept this trade-off.

Policy-level encryption uses a different key than banking. Auditors can verify cryptographic separation despite shared physical infrastructure. This was the deciding factor in choosing ESA for AI clusters.

Stripe width of 2 spreads I/O across more nodes, helping with large sequential reads typical of training workloads.

Compression disabled because measured compression ratios on our datasets are minimal (~1.05x). Not worth the CPU overhead.

Policy 3: AI-Inference (RAID-1)

Used by: Model serving artifacts, inference VMs, cache layers for hot models

Configuration:

• Failures to tolerate (FTT): 1
• Method: RAID-1 mirroring
• Cluster: ESA (newer all-NVMe clusters)
• Encryption: Policy-level with dedicated key (same as AI-Training)
• Compression: Disabled
• Stripe width: 1
• IOPS limit: None
• Object space reservation: 25%

Rationale:

Inference services need higher availability than training (live customer-facing) but lower than banking. FTT=1 is appropriate.

RAID-1 chosen over RAID-5 because inference reads are random-pattern (different models accessed in different orders), where RAID-1 outperforms RAID-5 in practice. Training reads are largely sequential, which is RAID-5’s strength.

This is the policy nuance that took us months to figure out. Two AI workloads, same physical cluster, but different storage policies because the access patterns differ. AI-Training and AI-Inference are not interchangeable from a storage perspective.

Same encryption key as AI-Training keeps key management simple within the AI workload class.

Disk group design for OSA clusters (legacy concerns)

OSA disk group design is where most legacy banking deployments have specific decisions to document. For new readers operating ESA only, skip this section.

Our OSA banking clusters use:

• 2 disk groups per host
• Each disk group: 1x NVMe cache (~1.6 TB write-intensive) + 4x SAS SSDs (~3.84 TB capacity each)
• Result: ~30 TB raw per node, ~12-15 TB usable after RAID-1 + slack
• Stripe width and FTT applied at policy level

Why 2 disk groups instead of 1 larger or 3+ smaller:

• Failure domain isolation: cache failure affects only that disk group
• Performance: parallel paths to cache improve concurrent workload handling
• Recovery: rebuild time per disk group is manageable
• Cluster-wide: matches our 8-node cluster sizing well

Disk group design choices are sticky. Once a cluster is deployed, changing disk group size or composition requires sequential node maintenance with full rebuilds. Plan disk groups carefully before initial deployment.

ESA storage pool design for AI clusters

ESA simplifies the design because there are no disk groups. Each node contributes its NVMe storage to a flat pool. Configuration is mostly about hardware sizing.

Our ESA AI clusters use:

• All-NVMe configuration (ESA requirement)
• Each node: 8x NVMe SSDs (~7.68 TB each, mixed use)
• Result: ~60 TB raw per node, with RAID-5 on policies achieving roughly 40 TB usable per node
• No cache/capacity distinction — all NVMe acts as both

This is meaningfully more capacity than our OSA banking nodes despite using fewer drives. NVMe density and ESA’s pool model both contribute.

Cluster sizing for AI:

• 6-8 nodes per cluster
• Compute and storage co-located (compute modules running ESA)
• Network: 100Gb redundant vSAN data network (jumbo frames enabled)
• Witness: dedicated witness VM for stretch protection (where applicable)

Performance reality: numbers from production

Synthetic benchmarks tell you what hardware can do. Production workloads tell you what it actually does. Here are representative numbers from our environment.

Banking workload patterns (OSA, RAID-1)

Transactional database I/O during business hours:

• Read IOPS: 25,000-40,000 sustained, peaks 80,000
• Write IOPS: 10,000-15,000 sustained
• Average latency: 0.8-1.2 ms reads, 1.5-2.5 ms writes
• Throughput: 800 MB/s read, 400 MB/s write at peak

OSA handles this comfortably. Cache tier absorbs writes; capacity tier serves reads from working set after warm-up. No surprises after 5+ years of operation.

AI training workload patterns (ESA, RAID-5)

Image classification training, multi-GPU job:

• Sequential read throughput: 3-5 GB/s per node during dataset loading
• Random read IOPS during shuffled batches: 80,000-120,000
• Write throughput: 1-2 GB/s during checkpoint operations
• Latency: 0.5-0.8 ms reads, 0.8-1.5 ms writes

ESA handles training I/O patterns well. RAID-5 doesn’t penalize sequential reads (which dominate dataset loading). Random reads scale across the storage pool. Checkpoint writes are bursty but manageable.

Compared to a dedicated NVMe-oF array we benchmarked separately:

• Dedicated NVMe-oF: 8-12 GB/s sequential reads, 0.2-0.3 ms latency
• ESA vSAN: ~40-50% of NVMe-oF peak throughput

We accept this trade-off because vSAN integrates with our existing operations. For workloads that need maximum performance, we use NVMe-oF; for most AI workloads, vSAN is sufficient.

AI inference workload patterns (ESA, RAID-1)

Live model serving for production inference:

• Read IOPS: 5,000-15,000 (varies by model popularity)
• Write IOPS: minimal (model updates, not steady state)
• Latency: 0.4-0.7 ms reads (consistent, low variance)
• Throughput: 500 MB/s - 2 GB/s depending on traffic

The latency consistency matters more than peak throughput for inference. RAID-1 delivers predictable latency for random-pattern reads. RAID-5 would add variability we don’t want for customer-facing services.

Capacity planning across mixed workloads

Capacity planning becomes interesting when policies have different efficiency ratios.

Our planning model:

Workload	Policy	Raw-to-usable ratio	Notes
Banking transactional	RAID-1, FTT=2	3:1 (33% efficient)	Data stored 3x
AI training	RAID-5, FTT=1	1.33:1 (75% efficient)	4+1 parity
AI inference	RAID-1, FTT=1	2:1 (50% efficient)	Data stored 2x

For 100 TB of usable banking capacity, we provision 300 TB raw. For 100 TB of usable AI training capacity, we provision 133 TB raw. The ratio matters when forecasting hardware needs.

---

💡 Storage Sizing Automation Tool Notice: Avoid capacity allocation errors when mapping your raw-to-usable storage boundaries or slack space parameters. Before adjusting cluster SPBM rules, use our live, browser-based vSAN ESA Capacity Calculator to instantly model RAID structures and FTT overhead. To evaluate underlying disk array bottlenecks, access the RAID Storage Amplification Modeler. Both run 100% locally and offline without diagnostics harvesting trackers.

---

We maintain capacity dashboards per policy showing:

• Raw capacity allocated
• Usable capacity after policy overhead
• Currently consumed
• Slack space (vSAN recommends 20-25% slack for performance)
• Forecast exhaustion date based on growth trend

Capacity forecasting for AI workloads is harder than banking. Banking grows predictably (transaction volume, customer count). AI growth is bursty — a new model project can add 30 TB overnight when training data lands.

Encryption and key management

Encryption is where ESA’s per-policy capability matters most for compliance.

OSA: cluster-level encryption

Banking OSA cluster has one encryption key, applied to all data on the cluster. Key managed via external HSM (Hardware Security Module) per our security architecture.

For audit, this is straightforward to explain: “All banking data on this cluster is encrypted with key X, managed in HSM Y, accessible only by these specific operators per these audit-logged procedures.”

Limitation: if we needed to put another data class on the same cluster (e.g., test data, or another regulated data type), encryption separation would require a separate cluster.

ESA: policy-level encryption

AI ESA cluster has two encryption keys: one for AI-Training policy, one for AI-Inference policy. Both managed via the same HSM but with different access controls.

For audit, this enables separate access policies: training data has different operator access than inference data, even on shared physical infrastructure. Auditors verified key separation by:

1. Reviewing HSM key inventory
2. Reviewing access logs for each key
3. Verifying vSAN policy assignments to data stores
4. Testing key revocation scenarios

The verification took meaningful time but resulted in stronger audit posture than cluster-level encryption could achieve.

Key rotation

We rotate vSAN encryption keys annually as part of broader cryptographic hygiene. OSA rotation is cluster-wide and takes hours; ESA rotation is per-policy and faster.

Rotation procedure documented in our operations runbook. Auditors verify rotation evidence quarterly.

Audit considerations specific to mixed workload vSAN

The audit conversation for shared vSAN clusters is more involved than for dedicated storage.

Questions auditors asked

1. “How are workload classes prevented from accessing each other’s data?”

   • Answer: Storage policies enforce VM-to-datastore mapping; vCenter RBAC controls policy assignment; network isolation segregates traffic paths
   • Evidence: vCenter audit logs, storage policy assignments, network configuration

2. “On ESA clusters with multiple encryption keys, how is key separation enforced?”

   • Answer: External HSM holds keys; vCenter holds key references not actual keys; key access requires specific roles in HSM AD integration
   • Evidence: HSM access logs, vCenter role definitions, sample access traces

3. “What happens to data when a workload class is decommissioned?”

   • Answer: Storage policy deletion triggers data cleanup; encryption key destruction completes the data lifecycle
   • Evidence: Documented decommissioning procedure with verification checkpoints

4. “How is capacity capped to prevent one workload from starving another?”

   • Answer: IOPS limits available at policy level (not currently used); space reservations prevent over-commitment; capacity dashboards alert on thresholds
   • Evidence: Capacity history showing no starvation incidents

5. “What is the recovery procedure if storage policies become corrupted or inconsistent?”

   • Answer: vSAN configuration backup procedures; documented recovery runbook; tested annually in DR exercises
   • Evidence: Recovery test results, runbook version history

6. “How are storage performance and availability monitored to detect issues early?”

   • Answer: vCenter integrated with our SIEM; custom dashboards per policy; alert thresholds based on baseline behavior
   • Evidence: Monitoring screenshots, alert history, incident records

Documentation we maintain

Beyond standard vSAN documentation, we keep:

• Policy definition document (with rationale for each setting)
• Encryption key inventory mapped to policies
• HSM access logs retained per compliance requirements
• Workload-to-policy mapping (updated on every workload change)
• Capacity history per policy
• Quarterly policy review meeting minutes
• Annual penetration test reports focused on storage layer
• DR exercise results testing storage recovery

The documentation burden is real. We invested significant operations time setting this up properly. The investment pays back at audit time — questions get answered with prepared evidence rather than ad-hoc investigation.

The storage layer is one piece of a larger platform. The same infrastructure thinking carries into running private AI on VMware and GPU expansion with HPE Synergy — compute and storage decisions are tightly coupled in practice, and the policies above only hold up when the surrounding platform is designed with them in mind.

When this approach works (and when it doesn’t)

vSAN with policy-differentiated mixed workloads works when:

• Organization already operates vSAN for some workload class
• AI workload performance requirements fit within vSAN capabilities
• Compliance allows shared infrastructure with proper isolation controls
• Operations team can manage per-policy configurations
• Long-term roadmap includes ESA adoption for new capacity

It doesn’t work when:

• AI workloads need maximum storage performance (use NVMe-oF instead)
• Compliance requires physically separated infrastructure
• Organization lacks vSAN operational maturity
• Workload mix doesn’t justify policy diversity (single workload class is simpler)
• All-flash hardware budget doesn’t accommodate ESA requirements

We have made the trade-off in favor of vSAN for our environment. Specific high-performance AI training workloads still use NVMe-oF — but that is the exception, not the default.

What we would do differently

Looking back across multiple years of vSAN operation:

1. Document policy rationale from the start

Our first policies were created without documented rationale. When auditors or new team members asked “why FTT=2?” we had to reconstruct the reasoning. Now every policy has documented rationale in a controlled document.

2. Plan ESA migration earlier

Banking OSA clusters work fine, but ESA adoption would let us consolidate workload classes onto fewer clusters with policy separation. We are now planning OSA-to-ESA migration as part of the next hardware refresh cycle, but we could have started this conversation 18 months ago.

3. Standardize disk group designs across OSA clusters

We built early OSA clusters with slightly different disk group configurations as we learned. Now they all behave similarly enough that this isn’t a real problem, but standardizing earlier would have simplified operations.

4. Build policy assignment automation

Manual policy assignment to new VMs led to occasional mistakes (banking VM assigned AI policy, etc.). Now policy assignment is automated through tags and infrastructure-as-code, with validation that catches mismatches.

5. Establish capacity forecasting per policy

Initial capacity forecasting was cluster-wide. With multiple policies having different efficiency ratios, per-policy forecasting is more useful. We rebuilt our capacity dashboards to show usable capacity per policy with appropriate trend analysis.

6. Coordinate vSAN version planning with vSphere planning

vSAN and vSphere versions need to align. We have had moments where our vSphere upgrade plans didn’t account for vSAN compatibility constraints. Now we treat them as a single planning exercise.

OSA-to-ESA migration considerations

For organizations planning OSA-to-ESA migration, lessons from our analysis:

Hardware requirements

ESA requires all-NVMe. If your OSA clusters use SAS/SATA capacity tiers, you cannot migrate in place — you need new hardware.

Cost implications:

• All-NVMe density is higher per drive but capacity per dollar may be lower for cold tiers
• Cluster sizing may differ (ESA has different optimal node counts)
• Network requirements stay similar but increased throughput may stress 25Gb fabrics

Operational changes

ESA introduces operational differences worth planning for:

• Per-policy encryption requires new key management workflows
• Storage pool concept replaces disk group thinking — retrain operations team
• Performance characteristics differ — re-baseline monitoring thresholds
• Some VAAI/storage features differ between OSA and ESA

Data migration

vSphere Storage vMotion handles data migration between OSA and ESA clusters. For our planning, we model migration as:

• New ESA cluster built alongside existing OSA cluster
• VMs migrated workload-by-workload via Storage vMotion
• Old OSA cluster decommissioned after migration validated
• Encryption transition planned during migration (new keys on ESA)

Realistic timeline: 6-12 months for a banking-scale cluster migration including all planning, validation, and migration windows.

When to migrate vs stay

Reasons to migrate:

• Need policy-level encryption for shared workloads
• Need RAID-5/6 performance for capacity-sensitive workloads
• All-NVMe hardware refresh coming anyway
• Operations team ready to retrain

Reasons to stay on OSA:

• Existing clusters working well, no urgent need
• Hardware refresh not budgeted
• Operations team capacity limited
• No need for ESA-specific features

We are in “preparing to migrate” mode — building ESA expertise on AI clusters first, then migrating banking clusters during next hardware refresh.

Operational notes for running both architectures

Things we have learned from running OSA and ESA simultaneously:

Skills overlap but aren’t identical

A vSAN OSA operator can learn ESA fairly quickly, but the workflows are different enough that we treat them as separate competencies for our runbooks. Operators certified on both feel more confident, but training each architecture separately.

Monitoring needs adaptation

Standard vSAN monitoring (vCenter integration) works for both, but the meaningful metrics differ. OSA cache utilization is critical; ESA doesn’t have a cache concept. Adjust dashboards per architecture.

Vendor support

HPE support for both architectures is mature, but cases often need explicit architecture flag. We standardize case templates to include “OSA cluster” or “ESA cluster” upfront.

Documentation duplication

Our runbooks have parallel sections for OSA and ESA procedures where they differ. Maintenance overhead. Worth it for accuracy, but a real cost.

Closing notes

vSAN serving mixed regulated and AI workloads on shared infrastructure is operationally feasible but requires deliberate engineering. Policy design is where most of the thinking happens. Encryption strategy matters as much as performance tuning. Audit posture depends on documentation that you build before you need it.

OSA and ESA are both valid choices for current production. New deployments lean toward ESA because of policy-level features. Existing OSA deployments continue to operate reliably. Many organizations like ours will operate both for years before completing migration.

The three-policy approach we use (RAID-1 banking, RAID-5 AI training, RAID-1 inference) reflects specific trade-offs around availability, capacity efficiency, and access patterns. Your workload mix will lead to different policy choices. The framework — design policies around workload requirements rather than treating storage as commodity — should generalize.

Future articles will cover the specific operational playbook for vSAN ESA migration (when our cycle starts), the monitoring stack we use for vSAN (Prometheus integration patterns), and the encryption key management workflows that make audit defense efficient. Subscribe to the newsletter to follow along.

---

Operating notes from running vSAN clusters with mixed banking and AI workloads. Numbers reflect production patterns from our environment; your workload mix will differ. Verify all configuration recommendations against current VMware documentation and your own validation testing. I am an architect, not a VMware reseller — this is operator perspective.