VMware ESX: Recovering a Creshed ESX server v3.0.1

Posted: December 21, 2008 in LINUX *NIX

Just think, your lab is  in a week’s time and you’ve lost all your virtual machines (pub, sub, unity etc.?). The ESX server doesn’t boot!! Hue.. Happy birthday!

Well, I put about 15 hours to figure out how to bring up the crushed VMWARE ESX server.

Root cause of server CRESH:

I tried mounting 1TB disk to VMWARE but VMware ESX server kernels and anaconda doesn’t have UCCI support for USB devices.
As a result, it crashed the VMWARE modules and I couldn’t recover it by using several methods, including – booting from redhat/fedora/centos disk/, esxcfg-boot -p -r stuffs. Nothing seemed to work.
I found that the /etc/pam.d files were gone when I chrooted the ESX server hard drive. Also in /etc/sysconfig/network-script/  -the folder was empty.


It was damn easy. I wasted a whole Saturday night trying to figure out the best method. The challenge was I had unity, epic, CCM 4 to 6 VMware images on the server. So building from the scratch wasn’t an option.

I tried everything but still no luck till 4am.  I took some fizzy drinks and had a quick nap of 20 minutes and then resumed what I was doing. A new idea came up in my head. Google didn’t tell me what to do and any help url. I popped in the VMware ESX 3.0.1 CD to my other laptop’s syndrome and started installing it.  I saw there is an option when you install a new server which says  tick on “do you want to keep existing .vmx/datastore“, just ticked there.  I was so relaxed after seeing that option.  I took that CD out of my laptop and popped into the ESX 3.0.1 monster server (ML580) and started installation. During installation there is an option, which says “keep your vmstore” and wipe out everything. Viola… Here you go, the mother of all trouble was  just sitting there in ESX portion option.

So I choose > install (not upgrade)>> then ticked on that keep old VMware-store in baton preparation step and continued the installation procedure.

In a nutshell, above exercise installed VMWARE ESX server and  also retained old partition which had my all VMware images.

The whole above process only took 40 minutes.

After installation finished, the server boot up with a flashy screen asking for a password. After I punched my password immediately I checked.vmx files and found they are there…

Sometimes it’s too easy task, but we seem not finding it right away. I hate vendor not documenting things like that. Yes VMware has poor documentation which wasted my Saturday night staying at home and doing nothing but playing with the VMware esx box.

Anyhow, it was a good fun and i really enjoyed as there was  a challenge and it was good to find out a solution by myself.

Cheers, Push

Leave a Reply , I will reply ASAP

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s