Inside the lab follow MDrepairs on YouTube, TikTok & Instagram - visit our YouTube channel ↗

Data Recovery Guide

VMware ESXi Datastore Failed: VM Recovery Walkthrough

Updated November 18, 2026
In this article

If your ESXi datastore is showing as inaccessible, unmounted, or your VMs are reporting "Invalid configuration for device" errors, do not let anyone format the LUN or recreate the VMFS volume. The VMFS filesystem holds your VMDKs, their snapshots, and the descriptor files that point to them. Even when ESXi says the datastore is gone, the underlying data is usually still there, just unreachable through the normal mount path. A reformat throws all of that away permanently.

Datastore-level failures are different from single-VM VMDK corruption. When one VMDK breaks, you can usually restore from backup and move on. When the datastore itself fails, you've potentially lost every VM that lived on that LUN: domain controllers, file servers, the SQL VM, the Exchange VM, everything. Possible causes include the following: the underlying RAID or SAN failed, VMFS metadata got corrupted by a half-finished operation, the LUN was disconnected mid-write, or someone re-signatured a snapshot and tried to mount it the wrong way. The recovery path varies by cause.

What VMFS Actually Is

VMFS (Virtual Machine File System) is VMware's clustered filesystem designed for ESXi hosts to share access to the same underlying storage. It sits on a LUN (a logical block device exposed by a SAN, iSCSI target, or local RAID controller) and presents a filesystem on top of it. Inside that filesystem you have directories per VM, each containing one or more VMDK files (virtual disks), VMX descriptor files (VM configuration), NVRAM files, swap files, and snapshot delta files.

The thing that makes VMFS different from a normal filesystem is the metadata structure. It uses on-disk locks so multiple hosts can read and write concurrently without stepping on each other. It has heartbeat blocks that hosts use to coordinate. It maintains a journal so a host crash doesn't leave the filesystem in a half-written state. All of this metadata sits in well-known offsets on the LUN.

When VMFS goes bad, it's usually one of those metadata structures: the partition table (GPT or MBR), the superblock at the start of VMFS, the file descriptor (FDC) area, or the heartbeat region. Lose any of these, and ESXi can't make sense of the volume even though your VMDKs are still sitting there, perfectly intact, in their data blocks.

How a Datastore Fails

Datastore failures break down into a few categories at the lab. Each has a different recovery approach.

1. Underlying Storage Failure

The disks beneath the LUN failed. RAID-5 with two dead drives. RAID-6 with three. A SAN that lost an entire shelf. A failed RAID controller that wrote garbage during a rebuild. ESXi sees an unreadable or partially readable LUN and marks the datastore inaccessible. The VMFS itself was healthy. The disks under it stopped delivering valid blocks. This overlaps with RAID-5 recovery work.

2. LUN Disconnect Mid-Write

The SAN went away briefly. iSCSI lost connectivity. The fiber channel link dropped. ESXi was writing to the LUN at the moment of disconnect. VMFS journal entries are incomplete. When the LUN comes back, ESXi sees inconsistent metadata and refuses to mount. Often recoverable with careful journal replay or rollback to the last consistent state.

3. VMFS Metadata Corruption

Something corrupted the filesystem metadata directly. Common causes: a third-party tool wrote to the LUN directly while ESXi was using it. A LUN was used by another host with conflicting filesystem (XFS, NTFS) and that filesystem's metadata overwrote VMFS structures. A buggy storage driver wrote misaligned blocks. A failed dedupe or compression operation on the SAN. These are the trickiest because the corruption is often not in a single location.

4. Re-Signature Gone Wrong

Someone took a SAN snapshot, presented it back to ESXi, and re-signatured it without understanding what re-signaturing does. Worse, sometimes the "original" gets re-signatured by accident, making it look like a copy. ESXi then refuses to mount because the signature doesn't match what the host expects.

5. Accidental Delete or Format

A storage admin deleted the LUN from the SAN without realizing it was in use. A new ESXi host was installed and during setup the installer formatted the wrong LUN. A datastore was "removed" from vCenter, which by itself doesn't delete data, but a follow-up format does.

Symptoms in vCenter and ESXi

The symptom set depends on how the datastore broke. Common combinations:

  • Datastore shows "Inaccessible" in vCenter
  • VMs on the datastore are "Invalid" or "Orphaned"
  • esxcfg-volume -l doesn't list the volume even though the LUN appears in esxcfg-scsidevs -l
  • VMkernel log full of "WARNING: LVM: ... failed to find vmfs partition" entries
  • Attempting to mount shows "Cannot change the host configuration"
  • Snapshot warnings: "Resignaturing of VMFS volume is required"
  • partedUtil getptbl shows partition table mismatches or unreadable partitions
  • VM power-on returns "File system specific implementation of LookupAndOpen[file] failed"

The very first command worth running is esxcli storage filesystem list to see what the host actually sees, and esxcfg-volume -l to see what VMFS volumes ESXi knows about (mounted or not). Together those tell you whether the LUN is even reaching the host or whether the problem is one level lower.

VMFS3 vs VMFS5 vs VMFS6

The VMFS version matters for recovery. Different versions have different on-disk structures and tooling support.

VersionBlock Size / Max VMDKRecovery Notes
VMFS61MB block, up to 64TB VMDKCurrent default since ESXi 6.5. SE Sparse default snapshots.
VMFS51MB block, up to 62TB VMDKVery common, many production hosts still running this. Recovery toolchain is mature.
VMFS31, 2, 4, or 8MB block, up to 2TB VMDKLegacy. Out of support but still in the wild on old hosts.
VVOLStorage-side abstraction, not VMFSDifferent recovery model entirely. Storage array handles allocation.

When you call us, knowing your VMFS version helps us scope the job. Run vmkfstools -P -v 10 /vmfs/volumes/<datastore-name> on a working host (if you have any) to see the version. If the entire datastore is down, you may need to recall from documentation or by inspecting the volume header in lab.

What NOT to Do

Do NOT:
  • Format or reformat the LUN. Once you click "Create Datastore" on the same LUN, the new VMFS overwrites the old superblock and metadata. Recovery becomes much harder.
  • Run voma -m vmfs -f fix as a first response. The fix flag attempts repairs that can make corruption worse. Use -f check for read-only diagnosis first. We'll talk about this more below.
  • Delete and recreate the partition table. If the partition table is the only thing damaged, recovery is straightforward. Recreate it with wrong parameters and you have to find the right ones the hard way.
  • Run fsck or any Linux-side filesystem tool. VMFS is not ext4 or XFS. Tools that don't understand VMFS will silently rewrite blocks they shouldn't touch.
  • Reboot the SAN, RAID controller, or ESXi host repeatedly hoping the datastore comes back. Each reboot can trigger filesystem checks and metadata rewrites that erode the recoverable state.
  • Force-mount with re-signature unless you understand exactly what re-signaturing does. Re-signaturing changes the volume's UUID and updates VMX paths. If you re-signature the wrong copy, you orphan the right one.
  • Delete VMFS snapshots from inside vCenter if the datastore underneath is unstable. Snapshot consolidation under bad I/O can corrupt deltas and base disks simultaneously.

Safe Diagnostic Commands

These commands are read-only or near-read-only and tell you a lot without making things worse.

1. esxcli storage filesystem list

Shows mounted filesystems on the host. Look for your datastore here. If missing, the volume isn't mounted (different from being gone).

2. esxcfg-volume -l

Lists VMFS volumes the host knows about, mounted or not. Snapshot volumes will show here too. If the volume shows but isn't mounting, this is the entry point for diagnosing why.

3. esxcfg-scsidevs -l

Lists SCSI devices the host sees. Your LUN should appear here even if VMFS is broken. If the LUN is missing here, the problem is at the storage transport layer (SAN, iSCSI, FC) not VMFS.

4. partedUtil getptbl /vmfs/devices/disks/<naa.xxx>

Reads the partition table from the LUN. You should see one or more partitions of type vmfs. If this errors out or shows the wrong partition layout, the partition table is damaged.

5. voma -m vmfs -f check -d /vmfs/devices/disks/<naa.xxx>:1

VOMA (vSphere On-disk Metadata Analyzer) in check mode. Read-only. Reports VMFS metadata issues without modifying anything. Run this BEFORE any fix attempt.

The output of those five commands gives us enough to scope the job over the phone. Have it ready when you call.

When voma Helps and When It Hurts

VMware's voma tool is the official VMFS check and repair utility. It works. Sometimes. The catch is that voma's fix mode is destructive in cases it shouldn't be, and admins reach for it before reading the warnings in the VMware documentation.

The safe rule is: voma -f check is read-only. Run it freely. voma -f fix writes to the volume. Don't run it as a first step. If voma reports specific recoverable errors (heartbeat lock issues, single-resource conflicts), fix can clear them. If voma reports widespread corruption, fix can make things worse by zeroing structures that contained recoverable data.

The other gotcha: voma can only run on an unmounted volume. If your datastore is mounted but read-only or behaving weirdly, you'll need to unmount first. Unmounting a damaged datastore can itself fail or trigger more corruption. Don't unmount unless you're committed to the recovery process.

Lab rule: we never run voma fix on a customer's only copy of the data. We image the LUN first (bit-for-bit copy to lab-controlled storage), then work on the image. The customer's original is preserved no matter how the recovery goes. That's the level of caution this work demands and it's why DIY voma repairs sometimes destroy what was salvageable.

When the Storage Underneath Is the Failure

A RAID or SAN fault can appear as a VMFS problem when the underlying disks cannot deliver consistent reads.

Telltale signs the storage is the real issue:

  • RAID controller logs show one or more drives in failed or degraded state
  • The SAN management console shows array warnings, RAID rebuild stuck, dedupe errors
  • Some VMs on the datastore work, others don't, and there's no logical pattern (it's geographic on the LUN)
  • esxcli logs show repeated I/O errors on the LUN, retries, and eventual aborts
  • The datastore was fine yesterday and broke after a "routine" SAN firmware update or drive replacement

When this is the case, the recovery starts at the disk level. We image individual drives in our lab. We rebuild the RAID virtually in software. We carve out the VMFS partition from the rebuilt array. Only then do we even look at VMFS. Fixing the VMFS first without solving the storage is like patching the roof while the foundation is collapsing.

For storage rebuilds where multiple drives have failed beyond what the RAID level was designed for, the work gets significantly more involved. Modern RAID-5 with one drive failed is fixable in software. RAID-5 with two drives failed in the wrong order is a lab job. RAID-6 with three drives failed is also lab work. Our server data recovery overview covers this side in more depth.

Lab Recovery Process

Here's roughly what happens when an ESXi datastore lands at MDrepairs.

Imaging

If the storage hardware is failing, we image the drives with specialized hardware that handles bad sectors and timeout management gracefully. Imaging attempts to copy readable sectors onto lab storage. The customer's original disks go into a labeled box and aren't touched again unless we need a second image attempt for a specific bad area.

Virtual RAID Rebuild

From the disk images, we rebuild the RAID virtually. This requires knowing or determining the RAID level, stripe size, parity rotation, member order, and drive offset. Most of these we extract from on-disk metadata or RAID controller signatures. When we can't, we reverse-engineer them by looking for known filesystem patterns at expected positions.

VMFS Parsing

With the RAID virtually rebuilt, we have a clean block device exposing the VMFS partition. We parse the VMFS metadata directly: superblock, FDC, file resource manager, heartbeat region, journal. Damaged metadata gets reconstructed from redundant copies inside VMFS itself (the format includes some redundancy). Once VMFS is parseable, we walk the directory tree to locate VMDKs.

VMDK Extraction and Validation

For each VMDK, we extract it from VMFS into a flat file we can mount. We verify the VMDK descriptor matches the data blob. We mount it in our analysis environment and check that the guest filesystem inside (NTFS, ext4, XFS) is intact. We then either ship the VMDKs back to the customer for re-import into their ESXi environment, or extract specific files from inside the VMDKs if that's what the customer needs.

For a single VMDK extraction where the rest of the datastore is fine, our VMDK-focused recovery process covers a narrower path. But when the whole datastore is down, the work is at the VMFS layer and up.

The Broadcom Era and What's Changed

Since Broadcom acquired VMware in late 2023, a lot has shifted for VMware customers. Licensing changed dramatically. Free ESXi went away. Many smaller shops are running unpatched versions because the cost to renew exceeded their budget, or they're mid-migration to Proxmox, Nutanix, or Hyper-V. This affects recovery work in a few ways.

First, the host-side tools may be older but the file formats are the same.

Second, customers mid-migration to alternative hypervisors sometimes have a foot in each world when failure hits. They're moving VMs to Proxmox, the migration tool corrupts a VMDK on the source side, and now they need data off ESXi without a working host to mount the datastore. The recovery is the same: parse VMFS in lab, extract VMDKs, hand them back.

Third, support from VMware itself is harder to get than it used to be. Existing GSS tickets get answered slower. Smaller customers without enterprise contracts have fewer escalation paths.

Cost and Turnaround

Datastore recovery costs scale with the size of the LUN, the number of drives in the underlying RAID, the level of VMFS damage, and whether the underlying storage needs imaging.

ScenarioTypical RangeNotes
VMFS metadata repair, healthy underlying storageQuoted per caseWhen the LUN reads fine but VMFS structures are damaged
Single failed drive RAID-5/6 plus VMFS rebuildQuoted per caseRAID-aware imaging and VMFS reconstruction
Multi-drive failure RAID + VMFS recoveryQuoted per caseMore complex RAID rebuild, longer imaging time
SAN-level failure (shelf loss, controller fail)Quoted per caseMultiple LUNs, complex storage virtualization to unwind
Reformatted or accidentally deleted LUNQuoted per caseHeader reconstruction, deeper carving required

Our standard recovery estimate is 4–6 weeks; timing may vary by case. Confirm rush diagnostic availability with the lab; rush options advance the diagnostic start only and do not set a recovery completion date. The $100 diagnostic is credited toward a successful recovery and is non-refundable once diagnostics start. Donor drives or replacement parts you approve are non-refundable once ordered, even if recovery fails. You approve the estimate before recovery work starts. We also tell you when we don't think recovery is viable, which saves you time and money.

Preventing the Next Datastore Failure

Datastore failures don't happen at random. They follow patterns. Here's what works to avoid them.

Don't Run Datastores at More Than 80% Full

VMFS performance degrades steeply above 80% utilization, and metadata operations become more failure-prone. Allocate fresh datastores before existing ones hit 80%. Move VMs proactively.

Replace RAID Drives Before They Fail

SAN management software lets you alert on drive health. Pay attention to it.

Don't Use Snapshots as Backups

VMFS snapshots are delta files. They're operational tooling for short-term changes. They are not backups and they break the original VMDK when consolidation goes wrong. Use real backup software (Veeam, Nakivo, Veritas) for backups, not snapshots.

Test Restore from Backup Quarterly

Same rule as Exchange: a backup you've never restored from is a hope. Spin up a recovered VM from backup once a quarter. The first time will reveal something broken in your backup chain. Our disaster recovery planning guide goes deeper on this.

Document Your VMFS Version, Block Size, and Partition Layout

When the datastore is down, this information is gold. Run vmkfstools -P -v 10 on each datastore once a quarter and save the output somewhere outside the affected VMs. Future you will thank present you.

Patch ESXi Hosts on a Schedule

Some VMFS-related bugs have been fixed in patches. Running unpatched ESXi for years is asking for failures that already have public fixes. The catch in the Broadcom era is licensing makes patching access harder, but where you have access, use it.

Separate Production from Snapshot/Backup Storage

Putting backups on the same SAN as production VMs means a SAN-level failure takes out both. Use a separate target (different vendor if possible) for the backup destination.

FAQ

Can you recover from a VSAN failure too, not just VMFS?

Yes. VSAN is a different beast (object storage rather than filesystem) but recovery is possible. The work is significantly more complex because VSAN distributes data across multiple hosts. Reach out and we'll scope it specifically.

Our datastore is on iSCSI. Does that change anything?

The recovery process is the same once we have the LUN data. If the iSCSI target itself is fine and only ESXi can't mount the volume, we can often work over an iSCSI export from the customer's network. For severely damaged storage, we image the underlying disks in lab.

What if my VMs were thin-provisioned? Will recovery be partial?

Thin-provisioning doesn't change recoverability much. The VMDK descriptor tells us how much space is actually allocated. We recover what's allocated. Unallocated zones were always zeros anyway. The recovered VMDK will be the same size as it was when running.

Can you recover VMs that were running when the datastore failed?

Yes, including their memory snapshots if they exist. The VMX file plus VMDK plus any VMEM (memory state) lets the VM resume from where it was. Pure data inside the VMDK is always recoverable independent of running state.

How do I get the recovered VMDKs back into my environment?

We ship them as flat VMDK files or as OVF/OVA bundles depending on what you want. You can register them into a new datastore via vCenter, or import the OVA on a fresh host. For VVOL environments, we deliver as standard VMDKs that you can convert.

Will this work on Hyper-V VHDX or KVM qcow2 too?

That's a different recovery process but yes, we handle Hyper-V VHD/VHDX and KVM qcow2 corruption as well. Different file formats, similar approach: parse the format, extract the contents.

My ESXi host crashed and now the datastore won't mount even though the LUN looks fine. What happened?

Most likely the host left orphan locks on the heartbeat region or the journal is in a half-state. Sometimes this is fixable with voma check followed by a careful clear of stale heartbeats. Sometimes it's a sign of deeper VMFS damage. Diagnostic commands above will tell us.

How long until I get my VMs back online?

Our standard recovery estimate is 4–6 weeks; timing may vary by case. The queue, parts, device condition, imaging and delivery affect the schedule. Rush jobs can have an earlier diagnostic start for an additional fee. Confirm availability with the lab; rush service does not set a recovery completion date. More on timing expectations here.

What does the lab actually deliver?

For datastore recoveries, we typically deliver the entire recovered VMFS contents (every VMDK, every VMX) on lab-prepared external drives or via secure file transfer. You can re-create the datastore on new storage and copy the VMs in, or import them individually.

Do you work with VMware partners and MSPs?

Yes. Often the MSP knows the customer's environment best, and we handle the lab-level recovery. We don't try to take over the customer relationship. More on our services here.

MDrepairs is an independent data recovery lab serving businesses nationwide via mail-in service. We're not affiliated with VMware or Broadcom. Our specialty is recovering data when standard tools have failed.

Lost your data?
Let's assess your options.

Estimated turnaround: 4–6 weeks; timing may vary by case. Start with a free initial quote. The $100 lab diagnostic is credited on success; diagnostic and approved parts charges apply on failure.

Split the cost of your recovery with Affirm. Explore financing Subject to eligibility.

Devices recovered at our lab

These rows come from BenchKey, the ticket system we built for our own shop. Customer names, contact details and descriptions are left out.

Gloved hand holding a hard drive head stack at our Lincroft lab
A gloved hand holds a hard drive head stack at our Lincroft lab.
4.6 · 345 Google reviews$100 diagnostic, credited toward a successful recovery
Start your mail-in recovery