Part 1: Networks, IP assignment, NAT and Firewall

The Edge Gateway feature in Vcloud Director 5.1 is the evolution of the Routed Organisation Network in Vcloud Director 1.5. The Edge Gateway is implemented using vCloud Networking and Security (formerly vShield) and is fully integrated into vCloud Director. I will be looking specifically at the vCloud implementation of the Edge Gateway and how to configure the various features for your virtual Data Centre. Features include:

DHCP server*
Network Address Translation (NAT)*
Firewall*
Load Balancer**
Static Routes
VPN*

*enhanced from vCD 1.5
** New feature in vCD 5.1

Notable enhancements from the vCD 1.5 implementation include the ability to have more than 2 interfaces on an Edge Gateway (up to 10, internal or external), a higher specified ‘full configuration’ and an HA configuration.

In this first post I will be looking at the types of networks available in vCloud Director, adding networks, assigning IP addresses to vApps and configuring NAT and firewall rules. In part 2 I will look at configuring the Load Balancer feature and VPN configuration.

Deployment of an Edge Gateway

As a vCloud user, it is necessary to get your vCloud Provider (such as StratoGen) to deploy the Edge Gateway into your organisation. You will need to provide some information such as how many external IP addresses you will require, what your internal network address ranges are and whether you will need the compact edition or the full edition.
The compact edition has 1 x vCPU and 256MB RAM and the Full edition has 2 x vCPU and 1GB RAM.
Bear in mind your provider may levy an additional charge for the increased resources used by the full edition.

External Networks, Org VDC Networks and vApp networks

Prior to deploying an Edge Gateway it is useful to have a bit of background on how networks are presented in vCloud Director. There are three main types: External, Organisation and vApp.

External Networks
These are configured in vCloud Director by the service provider. In essence they present a port group from the underlying vShere infrastructure to vCloud Director. The service provider configures the IP range or ranges available on the port group in vCloud Director. The IP range may be a shared range that multiple customers can utilise or the service provider may present a dedicated range for a specific customer.
The Edge gateway will need to connect to one or more external networks. Your service provider will provide details on what options are available and the number of public IPs you are entitled to.

Org VDC networks
Org VDC networks are configured in your virtual data centre (VDC). As a customer you can create two types of Org VDC network: isolated or routed.
Isolated networks are effectively internal networks that can only be accessed by vApps in your VDC. They can be shared between VDCs in the same organisation but cannot be used to access external networks. You still have to define a default gateway IP for an isolated network, and it still deploys an appliance, but the only configurable service on an isolated network is DHCP.
Routed networks are internal networks that can be shared between VDCs in the same organisation, but the major difference is that they also connect to an Edge Gateway’s internal interface. vApps connected to a routed network use the Edge Gateway’s internal IP address as their default gateway.

The service provider is also able to add a third type of network to your VDC. It is a direct connection to an External network (as described above). This may be to give you access to either a shared or dedicated IP address pool. For example the provider may present an Org VDC Network that connects directly to a shared public IP range configured as an External network in vCloud Director.

vApp Networks
vApp networks are configured within the vApp rather than at the Org VDC level. They can connect directly to an Org VDC network or via a routed connection to an Org VDC network. If a routed connection is created a compact Edge Gateway appliance is deployed for the vApp, which has a reduced feature set (DHCP, Static routing, NAT and Firewall) than an Organisation level Edge Gateway. We will cover vApp networks in another post.

Once you have decided on your network topology you can ask your service provider to deploy the Edge Gateway for you. For this article I will use a simple topology with 2 Internal networks and access to a single external network.

So now my provider has deployed an Edge Gateway, they also need to sub-allocate IP ranges to the gateway for me to use for my services. These are the public IP addresses I will use for NAT rules and VPN connections. The Gateway will have at least 1 IP address already on the External Network, and that IP can be allocated for use in NAT rules or VPN connections. My provider has also allocated 2 extra public IP addresses for me to use. These will be allocated to me and cannot be used by any other device on the External network. In my case the provider has attached my Edge Gateway to a shared IP range so other customers may have IPs on the same Public IP space. The other option would have been to request a dedicated subnet.

I am going to configure my network as follows:

vApp1 has a single Windows server which will offer FTP services publicly. I want to be able to access it via VPN from my other vCloud platform (details in Part 2!). I will use DHCP to allocate the Server an IP address.
vApp2 has 2 CentOS servers running apache. I want to use the load balancer feature to allow http access publicly (details in Part 2!) and also to allow SSH administration from the Internet. I will use IP pool address assignment for this vApp.

Configuring the Org VDC Networks

To create an Org VDC Network go to the Administration tab in your vCloud Director Organisation and choose the VDC you wish to create the network in. Click on the Org VDC Networks tab then the plus sign:

I want to create a routed network so I’ve selected that option and also selected my Edge Gateway:

Click next and you can then define the internal network. For my 2 internal network I have set the following settings:

For the Windows FTP server network:

For the CentOS apache servers network:

The DNS servers could be DNS servers provided by your vCloud service provider, public (open) DNS servers, or servers internal to your VDC – It will depend on your deployment. For testing I used StratoGen’s DNS servers.

When the Org VDC networks have finished deploying (in the background this means configuring one of the unused NICs on the Edge Gateway and configuring a new port-group in vSphere) they will appear in the OrgVDC Networks pane:

You can also see that the Edge Gateway has 3 used NICs, 1 External Network and 2 Organisation VDC Networks confiured:

Once both Org VDC networks have been created they can be made avilable to any vApps deployed in the VDC. In the vApp go to the Networking tab and lick the plus sign. We want to add an Organisation VDC network:

Select the Org VDC network you want to add to the vApp:

Now the Virtual machines can be attached to the networks. For the Windows VM I am using DHCP so will configure the NIC on the VM as follows:

For the CentOS servers I am going to use IP Pool assignment. This is a vCloud Director feature that is similar to DHCP but has few important differences. When we created the Org VDC network we specified a range of IPs for our pool. You either assign IP addresses automatically as per the screenshot or specify addresses manually.

Once an address has been assigned it will be unavailable for other VMs on the same network. There are a couple of prerequisites for IP Pool address assignments: Guest Customization must be enabled which in turn requires that VMware tools are installed on the Virtual Machine. If you are running a Guest OS that doesn’t support VMware tools then consider using DHCP for automatic address assignment. The assigned IP address will be applied to the VM when it is powered on. You can check assigned IPs by right clicking on the Org VDC Network and choosing IP Allocation:

Configuring DHCP

We want to assign a DHCP address to our Windows server so we need to configure this first on our edge gateway. Right click on the edge Gateway and choose Edge Gateway Services. The DHCP tab is the leftmost tab and appears first. Currently there is no configuration there as you can see:

Click on add and enter an interface to apply the DHCP scope on – we want it on our WindowsFTPNet interface. I’ve chosen a range that doesn’t overlap with the IP Pool configured on the WindowsFTPNet Org VDC network. I’ve also decided to keep the lease times at the default.

Once that’s done click OK then tick the Enable box.

Click OK and the Edge Gateway will initialise the DHCP server. I’m now going to power on my Windows vApp. Once this has powered on we can see that the network adapter has picked up the first IP in the DHCP scope. It has also picked up the correct default gateway and the DNS servers (trust me on that one!) I configured on my Org VDC network.

So DHCP is working correctly but we don’t have Internet access as yet as we haven’t defined any NAT rules or firewall rules. I’m going to keep things simple by allowing outbound access to the Internet from all my VMs, but you may well want to be more stringent in your security configuration.

Before adding NAT rules I’m going to make sure my CentOS VMs have also been assigned the correct IP configuration. As these VMs were deployed from my provider’s catalog, they were both preinstalled with VMware tools and Guest customization was enabled already. If you were deploying a VM from scratch you would need to install these manually which you can do from the vCloud Director interface. The installation is automatic in Windows, but for Linux VMs it only mounts the VMware tools installer – you must complete the installation manually from the command line. You will need Perl installed, and for CentOS/RedHat Linux once the Installer is mounted run the following:

1. mkdir /cdrom
2. mount /dev/cdrom /cdrom
3. cd /tmp
4. tar -xvf /cdrom/VMwareTools (tab to auto-complete)
5. cd vmware (tab to auto-complete)
6.  ./vmware-install.pl
7.  umount /cdrom

Looking at the vCloud Director Interface it looks like it has assigned IPs successfully to the VMs:

If we check one of the VMs from the console we can see this is the case by running ifconfig:

So all our VMs have IPs assigned correctly in the networks they are attached to. Next we want to give them all the ability to access the Internet. This means configuring two further parts of the Edge Gateway: NAT rules and Firewall rules.

Configuring NAT and Firewall rules

In vCloud Director 5.1 you have more granular control over the NAT configuration in the Edge Gateway. Specifically you can configure Source NAT (SNAT) rules, which control traffic leaving the Edge Gateway’s external interface(s), and Destination NAT (DNAT) rules, which control traffic arriving at the Edge Gateway’s external interface(s).

Adding SNAT rules

To allow our VMs to access the outside world we are going to have to add a NAT rule for each internal Org VDC network. This will translate internal private IP addresses to a public address or addresses. For outbound access there’s no particular reason why you can’t use the same external IP address for all your internal networks. Depending on your configuration you may want to use a public IP for each internal network or even for individual VMs on that network. I will be using a single IP for all networks. Currently we have no NAT rules configured:

To create the SNAT rule we go to the NAT tab in Edge Gateway Services and click add. The SNAT rule is to be applied on the external interface that we will be using. I only have one configured but if you have multiple external networks you would apply the rule on the interface you want the traffic to egress from. The original (Internal) source IP/range is the Internal Org VDC range and the Translated (External) source IP/Range is the public IP or Ip range on the external interface you want your internal traffic to appear as on the Internet. Make sure the enabled box is ticked!

Click OK and the rule will be added to the NAT rule table. As you can see I have added an additional rule for the CentOsOrgnetwork as well applied on the external interface and using the same public IP.

There is one thing preventing the VMs from accessing the outside world now – The Firewall!

Configuring the Firewall

By default the Edge Gateway denies all traffic and this is how you should always configure a production firewall. For testing purposes it does allow you to set the default action to allow, but please remember to change this before putting it into production!

The Firewall on the Edge Gateway offers granular options for allowing access to services on your vCloud infrastructure. We want to allow our internal Org VDC networks access to the Internet, so we will add a rule for each Org VDC network.
In Edge Gateway services go to Firewall and click add. First give the rule a descriptive name. The source will be the internal network range. Source ports are set to any as we want all traffic to be allowed outbound. The destination is set to external – you could set this to any but external is more specific as we have multiple internal networks. Destination port and protocol are both set to any to allow my VMs full outbound access. Make sure the enabled box is ticked, then click OK.

In a production environment and depending on what services you are running you will most probably want to be a lot stricter with your outbound traffic.
Once we have clicked OK and the Edge Gateway has finished updating we can see our Windows box can get to the outside world.

As you can see from the screenshot below I have added another similar rule for the CentOS network.

So all our VMs can access the Internet, but if we want to publish services on our VMs that can be accessed from the Internet we will need to add some extra rules. We need Destination NAT (DNAT) rules to forward traffic from our assigned Public IPs to our internal networks and also firewall rules to allow the traffic to pass.

First of all I want to allow Remote Desktop access to my windows VM and SSH access to my CentOS VMS.

Adding DNAT rules

As with SNAT rules, DNAT rules are applied on the external interface you are using for the NAT translation. We only have one external interface so that makes our choice pretty easy. I am going to add a rule that forwards traffic destined for tcp port 3389 (Remote Desktop Protocol) on the public IP address to port 3389 on the private IP address of the Windows server.

Click OK. We now need to add a firewall rule to allow the traffic that has been forwarded to tcp port 3389 (please note the destination is the public IP – not the prviate IP).

In a production environment you would probably lock the source IP down to your location. Once this rule has been applied I can use RDP to manage my Windows server.
It’s important to note that in the latest version of vCloud Director Firewall rules use match on original addresses i.e. the public IP that you specified in the corresponding DNAT rule rather than the internal address.
As you can see from the following screenshot I have added DNAT rules for the CentOS servers to forward SSH traffic. For the second server I have changed the original port – this allows me to manage both servers on a single IP address.

I’ve also added the necessary firewall rules to the edge Gateway:

It’s important to note that firewall rules are applied in order (top to bottom), and if you need to alter that order you can do this easily by dragging and dropping the individual rules to where you want them in the list.

So now we have our 2 vApps accessible from the Internet for management purposes, and they are also able to access the outside world. In the next part of this blog I will look at setting up the Edge Gateway load balancer, and configuring VPNs as well as the accompanying firewall configuration.

In the current revision of vCloud Director (5.1 and 5.1.1) there is an issue that may present itself as vCD disconnecting from vCenter at random times coupled with connection alerts from vCloud Director such as the email alert shown below.

Further information of the error can be seen in the log /opt/vmware/vcloud-director/logs/vcloud-container-info.log.
Look for the following error.  ORA-01013: user requested cancel of current operation.
You can do this as follows.
# less /opt/vmware/vcloud-director/logs/vcloud-container-info.log
Then press / and type in user requested cancel of current operation to go to the location in the log where this entry is recorded.

As detailed in my previous post you can change the connection time to get around it to allow vCD time to reconnect to the vCenter Server. However there is another work around available that involves modifying the vCloud Director SQL database to remove some null entries that keep on creeping up in value and trigger this disconnect in the first place.

To do this I suggest you stop your vCloud Director cells first and make sure to backup the SQL server database. These instructions are for Oracle.

1. Quiesce the services of the cells using the cell-management-tool and then stop the services with service vmware-vcd stop as described here.
2. Backup the Oracle server.
3. Open an SSH connection to the Oracle server and type sqlplus then provide the vcloud username and password.  (Hint username is vcloud)
4. Run the following commands at the SQL> prompt.

SQL> select count(*) from task_inv where (status = 2 OR status = 3) AND completion_date is null;

This will return a numerical value, probably in the tens or hundreds of thousands. What we need to do is to run a series of commands to reduce this number down. It is this number that is causing vCloud Director to time out during the synchronization process.
Run these commands to fix this.

A. Get list of all vc_ids in the setup.
SQL> select distinct vc_id from task_inv;

B. For each vCenter in the setup.
1. Get max managed object value for that vCenter. That is the vc_id obtained from the above query.
SQL> select substr(moref, 6) from (select * from task_inv where vc_id = vc_id order by to_number(substr(moref, 6)) desc) where rownum = 1;

This will give result_1. Next we need to do some basic maths. We will keep ‘top’ 1000 entries per VC and will delete rest of them.

2. result_1 minus 1000 = result_2

3. Using the above result_2, run the following.

SQL> delete from task_inv where (status = 2 OR status = 3) AND completion_date is null AND vc_id = vc_id AND to_number(substr(moref, 6)) < result_2;

4. Run a commit; command.

5. Finally run the original query to see if the number has gone down.
SQL> select count(*) from task_inv where (status = 2 OR status = 3) AND completion_date is null;

Don’t forget to restart the cell services on vCD.  service vmware-vmd start and tail the cell.log to watch the progress of restarting the cell service. /opt/vmware/vcloud director/logs/cell.log -f

It will continue to creep up until VMware fix this in an update currently due as release version 5.1.2 at the end of April 2013.

Constantly Syncing Inventory

A vCloud Director cell may fail to finish the synchronisation with a vCenter Server.  This is an issue where vCloud Director is constantly stating ‘syncing inventory’ in the vCenters section of the system>Manage & Monitor page.

You may find that a simple restart of the affected cell services may fix the issue.  If you are running a multi-cell environment you can do this by quiescing the currently active cell and then stopping and restarting the vCD services.

First disable the cell and pass the active jobs to the other cells.

Display the current state of the cell to view any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --status

Then Quiesce the active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --quiesce true

Confirm the cell isn’t processing any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --status

Now shut the cell down to prevent any other jobs from becoming active on the cell.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --shutdown

Then restart the services.
# service vmware-vcd restart

If you are not running multiple cells you can just restart the service but it will cause a loss of service during the restart.  A typical restart takes around 2-5 minutes.  You can monitor the progess of the restart by tailing the cell.log file.
# tail -f /opt/vmware/vcloud-director/logs/cell.log

Once it say’s 100%, it is done.

If restarting the services doesn’t help try rebooting the cell.  Use the same commands as above to pass active tasks over to the other cells first before rebooting.
When the cell restarts check and see if the cell will reconnect and finish the sync.  If not check the log /opt/vmware/vcloud-director/logs/vcloud-container-info.log.  Look for the following error.  ORA-01013: user requested cancel of current operation.

You can do this as follows.
# less /opt/vmware/vcloud-director/logs/vcloud-container-info.log

Then press / and type in “user requested cancel of current operation” to go to the location in the log where this entry is recorded.

The reason for ORA 1013: error can be:

• caused by the user – actually canceling the operation
• caused by a response to congruent errors
• the result of timeouts

When  vCloud Director sync is taking place, after processing the updates vCD performs database insertions.  Sometimes while persisting these updates vCloud Director will stop the sync and restart it, hence the constant sync.

Here is how to get around the issue.
1. Take a snapshot of the cell.
2. Quiesce the services of the cells using the cell-management-tool and then stop the services with service vmware-vcd stop as described above.
3. Open vi and add this line to /opt/vmware/vcloud-director/etc/global.properties
database.defaultQueryTimeout=300

# vi /opt/vmware/vcloud-director/etc/global.properties

4. Start the vCloud Director services again.
# service vmware-vcd start

If you do the above for all cells then the setting should be applied.

How Snapshots Work

A snapshot of a virtual machine is a point in time image of the current state and data.  The state is the virtual machines current power state, and the data is made up of all the files that make up the virtual machine including memory, disk, network cards, USB devices and so on.

A snapshot can be created simply through the use of the vSphere Client and the vSphere Web Client by right clicking on a virtual machine and selecting Snapshot>Create Snapshot.  You are then presented with the following options.

Name – Name for the snapshot.
Description – Description of the snapshot.
Snapshot the virtual machine’s memory – All the memory in active use on the virtual machine is written to a memory dump file (vmsn file) that is included in the snapshot.
Quiesce guest file system (Needs VMware Tools installed) – The quiescing process tells the operating system to write transactions out of the memory buffers and in-memory cache to the disk so that the virtual machine can have a consistent state that can be recovered from.

When the snapshot is created an additional disk is added to the virtual machine called a child disk or a delta disk which is labelled as <vm-name>-<number>.vmdk and  <vm-name>-<number>-delta.vmdk.
The <vm-name>-<number>-delta.vmdk file is a hidden file that will not show up in the datastore browser. You can however view this by connecting to the ESXi host either through SSH or through the vMA (vSphere Management Assistant). Here is an example of the same datastore location through a remote SSH connection.
Snapshot child disks are sparse disks that use a copy-on-write mechanism which means that only changed data is written to the child disks which allows for space saving by not replicating existing data.  The data is only written to the disk following a write.  This means that the child delta disks can save quite a bit of space.

In the illustration below the hashed blocks represent changed data blocks and the white blocks represent empty space due to the sparse layout of the disks.
Some additional files are created with the snapshot; the virtual machine snapshot database <vm-name>.vmsd and the virtual machine memory state file <vm-name>.vmsn.  The virtual machine snapshot database name file <vm-name>.vmsd contains the snapshot information and is where the snapshot manager gets its information from. It is a text readable file that can prove useful when trying to troubleshoot snapshot issues.

Here is an output of the snapshot .vmsd file associated with the example virtual machine.

.encoding = "UTF-8"
snapshot.lastUID = "1"
snapshot.current = "1"
snapshot0.uid = "1"
snapshot0.filename = "Demo-VM01-Snapshot1.vmsn"
snapshot0.displayName = "Demo-Snapshot01"
snapshot0.description = "Example Snapshot"
snapshot0.type = "1"
snapshot0.createTimeHigh = "316405"
snapshot0.createTimeLow = "-1275531695"
snapshot0.numDisks = "1"
snapshot0.disk0.fileName = "Demo-VM01.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot.numSnapshots = "1"

The snapshot options are controlled through the VMware API using the following options.

CreateSnapshot - Creates the snapshot. This is labelled as ‘Take Snapshot‘ in the vSphere Client.
RemoveSnapshot  - Remove the snapshot and delete the associated <vm-name>-<number>.vmdk and <vm-name>-<number>-delta.vmdk disks.  This is labelled as ‘Delete’ in Snapshot Manager in the vSphere Client.
RevertToSnapshot – This option takes the running state of the virtual machine back to the state of the last snapshot and changes made since are lost.  You can save the current state of the virtual machine by taking another snapshot should you need to revert back to the currently active state of the virtual machine.  This is labelled as ‘Go to‘ in Snapshot Manager in the vSphere Client.
RemoveAllSnapshots – This option removes all the snapshots by writing the active state of the child disk into the parent disk.  Pre-vSphere 4 Update 2 f there are multiple snapshots and thus multiple child disks, each child disk will write it’s contents into its parent disk all the way up the chain until the child disks have written all their changes into their parent disks.  At this point all the child disks are deleted.

If you think about what that means for a second, if you have lots of large snapshots then you will also need to ensure there is enough free space to accommodate these snapshots during the RemovalAllSnapshot process.

As an example lets say that your virtual machine has 4 snapshots on it which are left on there whilst carrying out some work on the server and these snapshots grow in size as follows.

Original disk – 100GB
Snapshot one – 10GB
Snapshot two – 20GB
Snapshot three – 10GB
Snapshot four – 20GB

When the RemoveAllSnapshots API is called the four snapshots would roll up, so four would roll into three, then three into two, then two into one and finally one into the original disk.  What was originally a 100GB virtual machine disk is suddenly a machine with a potential size requirement of 240GB!

Thankfully that is no longer the case with vSphere 4 Update 2 version or later.  The changes made were that the snapshots would roll up starting with the closest disk, so snapshot one would roll into the original disk, then two into the original disk, then three and finally four.  This means that not only is space saved during the RemoveAllSnapshots but also data is only written once rather than repeatedly during each snapshot roll up.
This is labelled as ‘Delete All‘ in Snapshot Manager in the vSphere Client.
Consolidate – The consolidate option was added in vSphere 5 and is there to allow you to write back the child disks that may have become disassociated from the Snapshot Manager due to a failed RemoveSnapshot or RemoveAllSnapshots command.  This failure can be caused by a time out during the write back of the child disks to the parent disks.

A virtual machine may show up in the vSphere Client as requiring consolidation with a Needs Consolidation alert on the summary tab of the virtual machine.
There is also a Needs Consolidation column in the virtual machines view from any higher level in vCenter, such as the cluster level.
Click the image for a larger view.

Orphaned Snapshots

What may happen is that the Snapshot Manager may think that the consolidation process is complete and so you do not get an error related to the virtual machine requiring consolidation in the vSphere Client but when you check the .vmx file or select the option to edit settings and view the location of the virtual machine disk files you may see that the disk is actually called <vm-name>-<number>.vmdk.  If this is the case look in the datastore browser and you will see the files <vm-name>-<number>.vmdk.

You can also open an SSH connection to the host  to view the  <vm-name>-<number>.vmdk and <vm-name>-<number>-delta.vmdk files by listing out the contents of the directory location of the virtual machine.  You can do this with the following commands.
#cd /vmfs/volumes/<datastorename>/<VirtualMachineName>
#ls -lah


Here you will see all the disk files including the hidden flat disks.  <vm-name>-<number>-flat.vmdk. The flat disks are the actual virtual machine disk files, the ‘plain’ .vmdk files are a configuration file pointing to the flat disk file.
If you see that the VM is running from a snapshot delta you have several options.

Option 1 – Clone the virtual machine.  A nice simple fix.  To ensure a consistent state of the virtual machine you will need to shut the machine down first before starting the clone, otherwise the cloned VM will be in the state the the original virtual machine was in during the initial snapshot taken at the start of the clone process.  Please note this snapshot state is a crash consistent snapshot; one without the option to quiesce the disk or snapshot the memory so any items on the virtual machine not committed to disk will be lost.
Option 2 – Take and delete a snapshot in the vSphere Client.  What will happen with this option is that the snapshot removal will also perform the consolidate action and rewrite the additional delta child disks back to the original parent disk.  Should you try this option and the snapshot removal doesn’t fix it either try shutting the virtual machine down first or selecting the option to Quiesce guest file system whilst taking the snapshot.
Option 3 - Take and delete a snapshot using an SSH connection to the host.  You may find that the snapshot removal still doesn’t work using the vSphere Client.  If so try the same process from the command line.  Use these steps as a guide.

Step 1 – List out the VMID of the virtual machines on the host
# vim-cmd vmsvc/getallvms

Alternatively use grep to list out just the virtual machine name you are looking for.  In my example I use
# vim-cmd vmsvc/getallvms | grep Demo*

Here is the output.
22     Demo-VM01    [EQL03-SHARED05] Demo-VM01/Demo-VM01.vmx
windows7Server64Guest       vmx-08

Step 2 – Verify if the snapshot exists
# vim-cmd vmsvc/snapshot.get [VMID]

Here is the output.
# vim-cmd vmsvc/snapshot.get 22
Get Snapshot:
|-ROOT
--Snapshot Name : Demo-Snapshot01
--Snapshot Id : 1
--Snapshot Desciption :
--Snapshot Created On : 2/1/2013 12:11:49
--Snapshot State : powered off

Step 3 – Create a new snapshot
# vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

Here is the output.
# vim-cmd vmsvc/snapshot.create 22 Demo-Snapshot02 "Snapshot Demo 2 Two" 0 0
Create Snapshot:

Step 4 – Remove all the snapshots  (Labelled as Delete all in Snapshot Manager)
# vim-cmd vmsvc/snapshot.removeall [VMID]

Here is the output.
# vim-cmd vmsvc/snapshot.removeall 22
Remove All Snapshots:

Run a directory list command ls -lah to confirm that the snapshots have all been removed.

You can also take and remove snapshots using the vSphere CLI or vSphere Management Assistant  (vMA) and PowerCLI.  The vSphere CLI and vMA uses the same commands as above, you just need to specify the remote server that you want to perform the checks against.

For example run this to take a snapshot of a virtual machine running on an ESXi host through vCenter Server.
> vmware-cmd -h <vCenter Server> -U <user_name> -P <password> createsnapshot <name> <description> quiesce [0|1] memory [0|1]

PowerCLI can use the following commands to take a snapshot.
> New-Snapshot [-Name] <Snapshot_Name> [-Description <Description_Of_Snapshot>] [-Memory] [-Quiesce] [-VM] <Virtual_Machine_Name> [-Server <vCenter_Server>]

Checking for virtual machine disk locks

Should any <vm-name>-<number>.vmdk delta disks remain the next step is to see if any virtual machine disks have locks on them.  For this you can use the vmkfstools command set and have a look at the current mode of the relevant .vmdk file.
A virtual machine disk can be in one of four modes.

mode 0 = no lock.
mode 1 = is an exclusive lock.  This will be the case if the virtual machine is powered on and in use.  A powered on virtual machine will also have an up to date modification date on the .vmdk file.
mode 2 = is a read-only lock.  This will be the case of the <vm-name>-flat.vmdk  of a running virtual machine with snapshots.
mode 3 = is a multi-writer lock.  This will be the mode of the vmdk if it is used for Microsoft clusters disks or fault tolerance virtual machines.

Ensure you are in the relevant virtual machine directory and use the following actions to perform these checks.

Step 1 – Check the mode state of the virtual machine flat disk file  (<vm-name>-flat.vmdk)
 # vmkfstools -D <vm-name>-<number>.vmdk

Here is the output of the demo VM with a snapshot in place.
# vmkfstools -D Demo-VM01-flat.vmdk

Lock [type 10c00001 offset 159152128 v 123, hb offset 3244032
gen 25, mode 2, owner 00000000-00000000-0000-000000000000 mtime 1190286 nHld 1 nOvf 0]
RO Owner[0] HB Offset 3244032 50b60d57-e9cb48dc-9d82-984be10fc230
Addr <4, 346, 95>, gen 106, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 42949672960, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

As you can see the base disk is in read only mode because all changes are currently being written to the snapshot delta disk.
If I run the same command on the snapshot delta disk I get the following.

# vmkfstools -D Demo-VM01-000001-delta.vmdk

Lock [type 10c00001 offset 262713344 v 152, hb offset 3244032
gen 25, mode 1, owner 50b60d57-e9cb48dc-9d82-984be10fc230 mtime 1190281 nHld 0 nOvf 0]
Addr <4, 598, 134>, gen 147, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 86016, nb 1 tbz 0, cow 0, newSinceEpoch 0, zla 1, bs 1048576

This disk is in exclusive lock mode because the virtual machine is switched on and is being used to write the changes to.   You can see which host has the lock on this virtual machine disk by looking at the MAC address given after the word, owner.

Step 2 – Shut the virtual machine down to see if the lock gets released
Here is the output following a shutdown of the virtual machine.

# vmkfstools -D Demo-VM01-flat.vmdk

Lock [type 10c00001 offset 159152128 v 124, hb offset 3244032
gen 25, mode 0, owner 00000000-00000000-0000-000000000000 mtime 1190723 nHld 0 nOvf 0]
Addr <4, 346, 95>, gen 106, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 42949672960, nb 0 tbz 0, cow 0, newSinceEpoch 0, zla 3, bs 1048576

As you can see the mode is 0 on the demonstration virtual machine meaning that the machine disk is not locked by another device.  Once the mode is 0 you should be able to take a snapshot and remove a snapshot successfully.

Step 3 – Forcefully remove the lock
If you find that the mode is anything other than 0 then another device is locking the disk.  This may be another host or depending on your backup software may be your backup server.  If the file is still locked you should see the MAC address of the owner.  If you find that it is your backup server that corresponds to the MAC address restarting the backup server should release the lock.  If it is another host then you will need to unregister the virtual machine from the current host and re-register it on the host with the corresponding MAC address.  Once you have registered it on the appropriate host try and power it on.  If it still fails check if the virtual machine still has a World ID assigned to it on the host identified as the owner of the lock.

# esxcli vm process list

Demo-VM01
World ID: 3657905
Process ID: 0
VMX Cartel ID: 3670192
UUID: 42 36 06 d4 0f 1b 35 61-17 aa f9 4b 8d 6c e1 78
Display Name: Demo-VM01
Config File: /vmfs/volumes/4fe306c8-b1c504a6-a734-984be10fb3e4/Demo-VM01/Demo-VM01.vmx

The world ID number (3657905) is the Virtual Machine Monitor (VMM) for vCPU 0.  Run the following command to force the virtual machine to stop by killing the process.

# esxcli vm process kill --type soft --world-id 3657905

Should you find that you are not able to see the virtual machine name when running this command this is because the virtual machine is not running on this host.
If this is the case or you are not able to kill the process you can restart the management agent or reboot the host to release the lock.

It is worth noting that you can use the k command in esxtop to kill a running virtual machine process. SSH to the host and perform the following.

Step 1 – Run esxtop by typing esxtop
Step 2 -Press c to switch to the CPU resource utilization screen (This is the default view)
Step 3 -Press Shift+f to display the list of fields
Step 4 -Press c to add the column for the Leader World ID
Step 5 -Identify the target virtual machine by its Name and Leader World ID (LWID)
Step 6 -Press k
Step 7 -At the World to kill prompt, type in the Leader World ID from step 5 and press Enter
Step 8  -Wait up to 30 seconds and validate that the process is no longer listed

January 2012 – USA vCloud Platform launched in Denver Colorado – due to increasing demand for USA based vCloud hosting, StratoGen decided to launch our service across the pond

June 2012 – StratoGen relocated to new offices at 37 Frederick Place in Brighton.  Due to continued successful growth more office space was required to house our UK based sales & support teams, so a new home was found.

July 2012 – StratoGen’s New York office was established at 380 Lexington Avenue to provide local support for our rapidly growing US customer base.

August 2012 – StratoGen attended VMworld San Francisco where we met with key customers, learned about key product developments, had the opportunity for an intimate audience with outgoing CEO Paul Maritz, and even got some sound bites on a video posted on the VMware vCloud Blog!

October 2012 – Stratogen’s New Jersey Datacentre launched – due to increasing demand for our US based vCloud Powered VMware Hosting, we brought online our East Coast facility at the impressive IO New Jersey modular datacentre, which survived the havoc wreaked by Frankenstorm, unlike many Manhattan based facilities including some that we evaluated!

November 2012 - StratoGen Upgrades all vCloud Datacentres to vSphere 5.1 & vCloud Director 5.1 becoming one of the first vCloud Powered providers to offer the great new features of 5.1 including Snapshot capability, Storage Profiles and enhanced networking with the vCloud Networking & Security suite

November 2012 – StratoGen Malaysia Datacentre launched in Kuala Lumpur making StratoGen one of the few VMware vCloud Powered hosting companies with true global coverage.

Looking forward to 2013, we’re excited to be growing the team in our US office, and we’re working hard to grow our well established based of ISV and Managed Service Provider customers.

There’s no question that cloud based services will continue to grow in 2013, as more workloads move to the cloud, and more software vendors look to make the shift from traditional on premise client-server deployments to Software As A Service based offerings.  VMware themselves are doing a lot of work in this area with ISV’s to ‘vApp’ their applications to simplify migration onto vCloud VMware hosting environments – ultimately deploying an application needs to become as simple as selecting it from a catalog as far as the end user is concerned.

We’d like to thank all of our existing customers for their continued support of StratoGen in 2013, and look forward to working with many more companies over the coming months to help them with their journey to the cloud.

This new facility is ideal for anyone experiencing network latency issues between USA or EMEA to the Asia Pac region.  It is also beneficial for Malaysian businesses who may have access to governement development funds, who need to host their infrastructure in country in order to qualify for these grants.

Our Malaysian vCloud platform is based in the Jaring datacenter in Technology Park Kuala Lumpur.   The platform is running the latest VMware vSphere 5.1 with vCloud Director 5.1, offering cutting edge cloud management features including VM Snapshots, Server Load Balancing and advanced vShield firewall tecghnology, all managed via the same vCloud Director login.

As with all StratoGen facilities, Jaring Kuala Lumpur is ISO27001 certified, ensuring the utlimate in security for customer data hosted in the facility.

The first customers are already up and running in the facility.  Any customers wishing to trial our Malaysian cloud can do so by visiting www.stratogen.net/trial and selecting the Asia region from the drop down menu.

vCloud Director 5.1 – we’re going to write a post all about this as we’re almost done with our upgrade, but in summary here’s the stuff that made it through that we love:

• Tiered Storage – ability to mount different volumes from different storage pools
• Snapshot – easy to create snapshot of a VM and revert to it
• vShield Edge – the new ‘Edge Gateway’ looks great – up to 10 NICs, VPNs, load balancing – a proper firewall, just what we wanted!

Site Recovery Manager – not really integrated with vCloud Director at all, so the rumours we heard were unfounded.  Still a great product though.

Project Octopus – Octopus has now fallen under the Horizon banner and has been named ‘Horizon Data’.  It will sit alongside Horizon App Manager and Horizon Desktop.  The suite is shaping up nicely but we still don’t know when we can get our hands on it – come on VMware!

VMware IaaS – AKA Project Zephyr, the VMware branded paid proof of concept offering has now been launched, and VMware are pushing this out as a way for organisations to have a play with vCloud Director without purchasing it or committing to a service provider.  This has ruffled a few feathers in the Service Provider community, but here at StratoGen we’re not too bothered about it.  The unofficial line from VMware is that Zephyr was borne out of frustration with service providers not embracing the Test Drive program and putting up decent trials of their own.  If only more of our counterparts had a full automated no obligation VMware Hosting Free Trial! We’ve already spoken with prospective clients who have tried the VMware offering and become frustrated with the lack of support and lack of functionality (eg no API!), so StratoGen’s fully functioning free trial is a far better option!

Have Bon Jovi Still got it? – Well Jon Bon Jovi and the Kings of Suburbia payed a great set to the VMworld Party, BUT – no Living on a Prayer!  We patiently waited thinking it would be his final encore, but we left disappointed.  You can’t have Bon Jovi without Living on Prayer!  Very disappointing.

In addition to the learning there was also plenty of socialising and networking – we got to meet outgoing CEO Paul Maritz at a cosy ‘vCloud Partner Appreciation Reception’ and I even managed to feature in a short video about why we love partnering with VMware.

There seem to be a lot of exciting developments in the pipeline, and there are a few things we hope to be enlightened on while we’re there.

• vCloud Director next major release 
  • When is it coming?
  • What features can we expect in GA?  We’re hoping to see:
    • Storage Tiering
    • Snapshot capability
    • More end user control of vShield Edge (load balanding configuration etc)

• Site Recovery Manager – there have been rumours of integration between SRM and vCloud Director – we’re intrigued to find out more…

• Project Octopus – much anticipated and discussed at last year’s VMworld – we want it, when can we have it?  Hopefully we’ll get an update and an idea of launch dates…

• VMware IaaS – there have been a few rumours flying around about VMware launching their own IaaS cloud (http://www.crn.com/news/cloud/240004995/vmware-poised-for-public-cloud-splash-but-partners-still-reeling.htm) giving some partners cause for concern – we want to know more!

• Have Bon Jovi still got it? You may ask ‘Did they ever have it?’ – They’re headlining the VMworld Party, so we’re going to find out!

Hopefully we’ll get the answers to these questions and more – we’ll post what we are allowed to next week!

Step 1 > Have your exported Virtual Machine files ready!

Remember, the StratoGen vCloud Platform will only allow you to upload files in the .OVF format. If you exported your virtual machine in the .OVA format, unfortunately you will need to redo the export process, ensuring you select OVF (Multiple Files option)

Step 2 > Log in to your StratoGen vCloud Director account

Using a supported browser, connect to your URL, as provided by your StratoGen representative at the time of sign up.  Enter your username and password to login to your account.

Step 3 > Select the ‘Catalogs’ tab

The initial homepage for your cloud is displayed. Now click on the ‘Catalogs’ tab.

Step 4> Create a new Organization Catalog

To add your OVFs the files into a catalog, you’ll first need to create one. Click on the green‘+’ to add a new catalog:

Give the catalog a name, and step through the Wizard.

Step 5 > Upload OVF Files

One you’ve created your catalog, open it up, click on the upload button and then browse to the location that you exported the VMware Image file to, which in my case was the desktop:

Select ONLY the .ovf file – the export process will have created a couple of other files but you don’t need to worry about these – the import process will pick them up:

Click the ‘upload’ button in the catalogue wizard, and then wait for the import to the catalog to complete:

The time taken to upload the Image will depend on the size, and the speed of your upload connection.

Step 6 > Deploy Virtual Machine from uploaded image

Once the upload has completed we are then able to deploy the .ovf into your vCloud Virtual Datacentre. In my demo account, I already have a couple of vApps set up, so I’m going to add the uploaded VM to one of my existing vApps.

To do this, open up the vApp, and then click on the ‘Add Virtual Machine’ button. The catalog wizard should pop up, defaulting to the catalog that you just created, and you should see your virtual machine in it. Select the VM, then click the ‘Add’ button and then click ‘Next’:

In the next step we need to give the VM a network connection – choose either the Direct Internet Connection for external connectivity, or import into an existing Internal Network structure – if applicable. Once the network has been added, click next through to the end of the wizard, and the VM Image will begin deploying. If I go back to my vApp diagram I can see the uploaded image is now deployed in my vApp alongside the other servers which were there previously:

That’s it! I can now power up the machine, and my VMware Image has been moved from my local cloud onto the StratoGen VMware Hosting platform.

Troubleshooting File Uploads

If you experience issues trying to upload .OVF files, they will most likely be caused by browser incompatibilities or Java related issues.

Browser Compatibility List (Correct for vCloud Director 1.5.1)

This table outlines support for browsers on Microsoft Windows operating systems:

This table outlines support for browsers on Linux operating systems:

Java Version 7 Issues

We have seen various issues with browsers using Java version 7. Typical problems include the ‘browse’ button not working when trying to load the file selector. If you experience a similar issue we recommend that you install and use Java JRE version 6 instead*. It is best to install the 32Bit and 64Bit versions. You can download the latest Java 6 from this URL:

http://www.oracle.com/technetwork/java/javase/downloads/jre-6u32-downloads-1594646.html

*Please check that this won’t impact any existing applications that you already have installed.

This is because the virtual machine has to wait for the amount of allocated vCPUs (logical CPUs) to become available so that the CPU instructions can be scheduled on multiple cores on the physical host.  The higher the number of vCPUs, the more cores that must be available for each execution.

When you migrate a virtual machine into vCloud Director from an existing physical server you may find that the server previously had many cores that may no longer be required.  If an application such as Microsoft SQL is using 8 cores on a physical server it is best practice to reduce the number of CPUs associated with the virtual machine to a lower number such as 2 vCPUs.  Each application will have different CPU demands and so it is best to use performance monitoring tools to determine how much of a bottleneck CPU is for your application.  You can do this in Microsoft Windows with Performance Monitor.

Once you have determined the appropriate amount of vCPUs that are required for your virtual machine reduce the amount of vCPUs using the steps below.

Note: The virtual machine must be powered off to perform these steps.

1. Right-click on the virtual machine and click Properties.
2. Click Hardware tab.
3. Use the Number of CPUs drop-down to lower the vCPU count by 1.
4. Click OK.
5. If your virtual machine still experiences performance issues, and if its kernel or HAL can handle switching to a single vCPU, lower the vCPU count to 1.
   
   
   Warning: If your virtual machine’s kernel or HAL cannot handle switching to a single vCPU, unexpected behaviour may occur.  This is typical of Windows Server Operating Systems so don’t lower multi-cored virtual machines below 2 vCPUs.