Making non-vss applications backups with netbackup

So, I have an application that that keeps writing even if a vss snapshot is running.

Wich could mean that the file that the application is writing to could be corrupt if restoring the vss image.

When using netbackup withc vmware backup, netbackup usesthe snapshot function in vmware to be sure that all files will be backuped, taking an image and leaving the disks read only, and all writes are written to a temporary location.

It only takes a second to change the writers to write to an alternate location, so if I have an application that have to be stopped och paused during backup, I only have to stop that application during the time when the snapshot is taken.
But how do I know when to stop the application and when to start it again?

1. Navigate to the folder where vmware tools are installed
2. Create a folder and rename it to backupScripts.d
3.Open "backupScripts.d" and create a script. Vmware calls all the scripts in that folder with %1, where %1 could be:

    freeze : before snapshot
    freezefail : when snapshot fails
    thaw : when snapshot is ready

To create an example script wich you can use to check your backup later if it really works:

if "%1" == "freeze" goto dofreeze
if "%1" == "freezeFail" goto dofreezeFail
if "%1" == "thaw" goto doThaw
goto EOF

:dofreeze
echo freeze > c:\freeze.txt
goto EOF

:dofreezeFail
echo freezefail > c:\freezefail.txt
goto EOF

:doThaw
echo thaw > c:\thaw.txt
goto EOF

:EOF

4. Name the script to whatever you want with .cmd or .bat.

5. Start a snapshot on the server, to emulate what netbackup are doing, check "Quiesce guest file system" and uncheck "Snapshot the virtual machine's memory"

Creating shared disks for virtual MS Cluster

Even though you have the hardware single point of failure problem solved with a vmware cluster, you still need to set up a traditional cluster for seamless upgrading of the virtual servers. As for filservers, application servers and so on.

1. Install two Windows Server 2008 R2 Enterprise servers.
2. Share out a quorum disk and if you need one, a filedisk, wich will be used for the fileserver files, from your disk array tou your ESX servers. Remember that the recommended solution is to only use a fibre channel SAN for this solution. (If you want to have the clustered virtual machines on 2 different esx hosts.
3. Add a new disk to your first server in the cluster, using Raw Device Mappings.

Make sure that you choose a new scsi-interface.

Select the newly created scsi-interface and choose Physical on SCSI Bus Sharing

On the second machine in the cluster:

Add a disk as you did on the first machine, but choose "Use existing virtual disk" and browse on the storage where you saved the configuration for the first disks.
remember to select a new scsci device as well

When done, add a new network card and assign it a private network adress on both machines and you are ready to set up the cluster..

The cons of it is that you can´t use vmotion or taking a snapshot of the machines, however, you can move your virtual machine when it is powered off.

Upgrading and patching hosts

You just gotta love it!

Patching host in vmware are just wonderful.
When patching physical host the old way you had to schedule it off working hours, the preparation for it took a mych longer time than the patching itself.

Before:

1. Reading about the latest patches
2. Testing them in an test enviroment on a "Common" server
3. Testing them in an test enviroment for the specific server in a test enviroment
4. Schedule with the administration for the server when to have the downtime and the patching
5. Prepare and upload all the patches to the server
6. Patch the server

What I'm talking about is when patching BIOS, firmware upgrades and driver updates, not windows updates and so on (Even if the procedure is quite the same)

So, what do we have today? Whe have a lot fewer physical server, that themself are a lot more critical, holding 20 or more virtual servers. Wich actually means that they are 20 or more times as critical than before.

With that said, you could believe that it would take a lot more time and preparations for patching hosts... It would take 20 times or more time to prepare for patching a host. Right?

No... its the opposite

We all have our virtual servers in a cluster, and when patching, simply put the host in maintenence mode and all of the virtual servers on the host will automaticly migrate to the other hosts in the same cluster.

Seamless using vmotion, you wont notice it on the virtual servers..

* You don´t have to shcedule a downtime with the administrators for the server
* You don´t have to test every single virtual server
* You can patch your vmware ESX-host during work hours, saving a lot of overtime expenses.

With that said, that only comply for esx host patches, not with patches on the virtual server itself, like windows update and things like that.

Snapshot differences between disks

There should be no time difference between the disk when converting a server, but only to be sure I have done a simple test.

Made a simple script which pipe out the current time:

time /T >>c:\time_CRoot.txt
time /T >>E:\time_ERoot.txt
time /T >>F:\time_FRoot.txt
time /T >>I:\time_IRoot.txt

Then I let the scheduler execute the script once per minute.
(Controlpanel -> Scheduled Tasks) or with command prompt
schtasks /create /tn timelog /tr c:\time.cmd /sc MINUTE

I then start the conversion with vmware converter and writhe down the time
In this experiment the conversion started 2:47 pm and ended at 3:31 pm

So, how does the timestamps looks on the disks on the converted machine?

Last timestamp's on the different disks are as follows:

C: 2:47 pm
E: 2:47 pm
F: 2:47 pm
I: 2:47 pm

The conclusion of that is that the first thing that the converter are doing at a windows machine is to take a snapshot of the volumes apparently with the help of VSS.

That is if you don´t check "Synchronize changes that occur to the source during cloning"

However, if I check that value...
Started 8:06pm and ended 8:55 pm (Took about 5 minutes longer)

C: 8:39 pm
E: 8:39 pm
F: 8:39 pm
I: 8:39 pm

Probably the second snapshot with VSS are taking about 15 minutes to complete of the disks.
So, you will miss some minutes even if you use the above settings, so make sure that you turn of databases and services to keep as much as possible static.

Online Converting

The best thing when virtualizing is of course to make a new virtual machine from scratch. However, time is money, and there is all kinds of configurations of servers out there, some are documented, some arent.

It is a good thing that you then are able to convert an existing physical server to virtual. You will get better performance and a more stable enviroment.

There are 3 ways to convert a physical machine, to make a ghost image and import it to vmware, make an online import with vmware converter and and offline conversion with vmware importer coldclone iso.

Mostly I use the online method, most servers that we have are application servers, that is, they are quite static. SQL servers we usually use to make new servers of, or move the database to our SQL hotel. There is also a license matter there, expecially for oracle.
Offline conversion is done on servers that are more dynamic or it is an DMZ server.

When doing an online conversion we do it in the following steps:

1. Make sure that you can login on the server that you should convert, and make sure you can do it with no network connected. (Cached AD Login works fine) . Also make sure that you have the network that the physical machine are using VLan:ed to the VMware enviroment.

2. Start importing (Mostly from vsphere "Import Machine" ) uncheck the option to install vmware tools and only choose to connect one network card.

3. When it is done, edit the virtual machine settings, deleting USB hubs, serial ports and deleting the network card and adding a new (VMXNET 3), usually don't edit the number of CPU settings, at least not from many to one single. also add a cdrom drive.

4. Boot up the virtual copy without network connected, log in and wait for the devices to be found.

5. Install vmware Tools and reboot.

6. Uninstall all hardware specific programs for the old hardware.

7. Start a command prompt and type "set devmgr_show_nonpresent_devices=1" and then sevmgmt.msc to start device manager.

8. Choose "Show hidden devices" in device manager and delet all the greyed out devices.

9. Set the network settings as they are in the physical machine, reboot the machine and make sure everything looks fine in the event viewer (Except the errors thats because youre not connected to a network).

10. Connect to the console of the physical machine (With ILO, Drac or any other tool),choose the network settings and select to disable the network. When it is disabled, you enable the network card on the converted virtual machine in the settings.

Voila!
You have successfully converted a server with a downtime of about 2-5 seconds.

Microsoft SQL licensing in VMWare

Got a question for a couple of days ago of how many licenses they would need to install a sql server in our virtual enviroment.
I checked it up on microsoft and found a very interesting document on microsoft, SQLServer2008_LicensingGuide.pdf .

Accourding to that, you only have to buy a CPU license for each physical CPU in the Cluster, and adding as many virtual CPU:s as you have cores+hyperthreading for each processor.
In our enviroment, that would mean that we could give a SQL server 12 virtual CPU:s and still only pay for one Standard SQL server License. However, if you add another virtual server you have to buy another license.

However, if you buy a SQL server Enterprise License the difference is that you can add as many virtual servers as you want on the dedicated host.

So a dedicated cluster, not for performance but of licensing purposes, should be considered.

Choppy mouse in vmware console

Got a couple of converted and newly installed servers with choppy mouse control in console mode.
The fix is to install the vmware graphic driver. I thought that that one would install when installing vmware tools, but it wont, even odder, it places the driver in a quite strange folder.

Start the device manager and expand "Display Adapter". If the "Faulty "Standard VGA Graphics Adapter" is shown you have to replace the driver. 














Right click it and select "Update Driver Software"

Select "Browse my computer for driver software"

Go to C:\program files\Common Files\VMware\Drivers\wddm_video

Press next -> close and restart the virtual machine.
You should now have the VMware SVGA 3D driver installed

VmWare and DMZ

We did sure consider if we should have a virtual solution for the dmz at all. However we came to the conclusion to have a vmware vsphere cluster of two esx servers, one in each server room.
In our cluster for the LAN we are using blade servers from HP, in DMZ we are using stand alone servers.

For easy management we have the service console in our LAN and all other networks in dmz, separated by physical network cards.
The networks in dmz are separated by VLAN ID.

The only problem is when we have to convert a physical server on the dmz to virtual. Because of that the service console is connected to the LAN, and we don´t want any server from the DMZ to talk on the LAN. (It is called DMZ for a reason). The solution to that is to only use offline converts of servers in dmz. To physically go down to the server room, shut down the server, reconnect the network card to a LAN port on the switch and boot up with the VMWare converter offline CD (coldclone.iso).

The cons of that is that we have a longer downtime comparing to online converting (Wich works perfectly if you have a quite static server, as web-servers, application servers and so on), the pros is that it is safer in a secure point of view.

However, if you are setting up a similar enviroment, make sure that you are using a gigabit connection to the service console.. It takes a looong time to convert a server on a 100 mbit connection...

Failed netbackup flashbackup

We have had a couple of redberries in netbackup for some time on one virtual converted server.

"Snapshot error encountered(156)"  is the only thing that netbackup are reporting.

So, I tried to take a manual flashbackup of the server to se what happened in vsphere tasks.

The job "Create virtual machine snapshot" ended with the error code
"Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine."

Tried to do a manual snapshot, and did not got any error, however, when checking "Quiesce guest file system(Needs VMWare Tools installed" it would create the same error.

It had to be something with vmware tools and vss so I simply tried to uninstall vmware tools, reboot, and install it again, however it did not work anyway.







Checked the event-viever on the server and noticed som error codes from vss in the application log:

"An internal inconsistency was detected in trying to contact shadow copy service writers." with event ID: 12302
















Googled around found a solution and ran the following commands to reregister vss:

cd /d %windir%\system32
Net stop vss
Net stop swprv
regsvr32 ole32.dll
regsvr32 oleaut32.dll
regsvr32 vss_ps.dll
vssvc /register
regsvr32 /i swprv.dll
regsvr32 /i eventcls.dll
regsvr32 es.dll
regsvr32 stdprov.dll
regsvr32 vssui.dll
regsvr32 msxml.dll
regsvr32 msxml3.dll
regsvr32 msxml4.dll


It worked perfectly after that.

The cause of the problem is however still unknown, but I suspect that some techie in our department have run the Microsoft Security Wizard and made some registry changes from that.

Converted a DELL PowerEdge 1950 today and got some problems from VMWare Converter saying "Operating system not found".
I suspected it could be the Raid Drivers that was missing and the converter could not see the disk.
I downloaded the drivers for DELL PERC 6/i Integrated from Dell Support executed the file wich extracted the driverfiles to c:\dell\drivers\R194151
Then add the driverfiles to the iso with the petool.exe:

"petool.exe -i coldclone.iso -d C:\dell\drivers\R194151"

Burned the iso to a blanc cd-r and booted up with the CD. Worked perfect and vmware converter found the disks.