Constantly Syncing Inventory

When performing an upgrade of vCloud Director 1.5 to 5.1 we ran into this issue to do with synchronisation.

Constantly Syncing Inventory

A vCloud Director cell may fail to finish the synchronisation with a vCenter Server.  This is an issue where vCloud Director is constantly stating ‘syncing inventory’ in the vCenters section of the system>Manage & Monitor page.
Syncing Inventory
You may find that a simple restart of the affected cell services may fix the issue.  If you are running a multi-cell environment you can do this by quiescing the currently active cell and then stopping and restarting the vCD services.

First disable the cell and pass the active jobs to the other cells.

Display the current state of the cell to view any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --status

Then Quiesce the active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --quiesce true

Confirm the cell isn’t processing any active jobs.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --status

Now shut the cell down to prevent any other jobs from becoming active on the cell.
#  /opt/vmware/vcloud-director/bin/cell-management-tool -u <USERNAME> cell --shutdown

Then restart the services.
# service vmware-vcd restart

If you are not running multiple cells you can just restart the service but it will cause a loss of service during the restart.  A typical restart takes around 2-5 minutes.  You can monitor the progess of the restart by tailing the cell.log file.
# tail -f /opt/vmware/vcloud-director/logs/cell.log

Once it say’s 100%, it is done.

If restarting the services doesn’t help try rebooting the cell.  Use the same commands as above to pass active tasks over to the other cells first before rebooting.
When the cell restarts check and see if the cell will reconnect and finish the sync.  If not check the log /opt/vmware/vcloud-director/logs/vcloud-container-info.log.  Look for the following error.  ORA-01013: user requested cancel of current operation.

You can do this as follows.
# less /opt/vmware/vcloud-director/logs/vcloud-container-info.log

Then press / and type in “user requested cancel of current operation” to go to the location in the log where this entry is recorded.

The reason for ORA 1013: error can be:

  • caused by the user – actually canceling the operation
  • caused by a response to congruent errors
  • the result of timeouts

When  vCloud Director sync is taking place, after processing the updates vCD performs database insertions.  Sometimes while persisting these updates vCloud Director will stop the sync and restart it, hence the constant sync.

Here is how to get around the issue.
1. Take a snapshot of the cell.
2. Quiesce the services of the cells using the cell-management-tool and then stop the services with service vmware-vcd stop as described above.
3. Open vi and add this line to /opt/vmware/vcloud-director/etc/global.properties
database.defaultQueryTimeout=300

# vi /opt/vmware/vcloud-director/etc/global.properties

4. Start the vCloud Director services again.
# service vmware-vcd start

If you do the above for all cells then the setting should be applied.

About Simon Greaves

Lead virtualisation technologist and IT infrastructure guru at StratoGen. Specialist in VMware virtualisation datacenter and cloud technologies. --- VCAP5-DCD, VCAP4-DCD, VCAP4-DCA & MCSE.--- Love talking technology and hiking mountains.

2 Thoughts on “Constantly Syncing Inventory

  1. Tek mentor on 19/02/2013 at 7:13 am said:

    Hi Simon,

    i have the same issue.i tried with the one that you explained in this post.But its not working for me.Still showing “Syncing inventory”

  2. Simon Greaves on 19/04/2013 at 11:24 am said:

    Hi Tek mentor try this article instead vCenter Server Disconnecting From vCloud Director

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Post Navigation