VM migration Failure - Timeout Issues!

##### Action plan:
1. Disable Power Management on the host where the vdsm has to be restarted. This is to prevent fencing of that host by RHEV Manager.

2. SSH into destination host and add migration_listener_timeout value as 1500 to the existing values under [vars] in /etc/vdsm/vdsm.conf file followed by restarting the vdsmd service.

migration_listener_timeout = 1500

- Restart the vdsmd service: [ Please ensure 'Power Management' disabled on host before restarting vdsmd service as mentioned in above step 1. ]

# /etc/init.d/vdsmd restart

3. Now try migrating the vm.

Host In-consistency Issue - VM migrates from one host to another

Issue: Sometimes VMs starts migrating from one host to another due to host in-consistency!

Solution: Wait till All VMs are gone (migrated) to other Host. Double check and make sure no more VMs left on that host (May be it's good idea to migrate your VMs to SPM Host).
After migrations done:

1) Run Command : "service vdsmd restart"

a) Check the output, If you see errors (red lines) indicating that restart of that daemon was not successful.

b) In case you see errors (red output), Wait for another 90 seconds and check RHEVM console for host status. Make sure it's green before going it to next step.

c) Re-run "service vdsmd restart" (It should give you all green and proper restart of daemon)

Now you can bring back VMs. Host will be consistence at least for one month.


Take care!

IPA Replica Uninstall Error - Fixed

Issue

When removing a disconnected IPA replica server which had replication agreements with multiple IPA hosts, the meta-data for that replica is still contained on still running IPA servers. This does not allow you to remove the disconnected IPA server completely. There are special tasks you can use to remove this meta-data. For example:

Raw

Initial topology:  (simple triangle)
2 - 3
 \ /
  1

# on host1:
ipa-replica-manage disconnect host1 host2
ipa-replica-manage del host3  # not sure if this one is relevant here

# on host2:
ipa-server-install --uninstall -U

# on host1:
ipa-replica-prepare -p $ADMINPW --ip-address=$HOST2_IP $HOST2

# on host2:
sftp root@$HOST1:/var/lib/ipa/replica-info-$HOST2.gpg
ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$HOST2.gpg
...
The host qe-blade-04.testrelm.com already exists on the master server. Depending on your configuration, you may perform the following:

Remove the replication agreement, if any:
    % ipa-replica-manage del qe-blade-04.testrelm.com
Remove the host entry:
    % ipa host-del qe-blade-04.testrelm.com

# on host1:
ipa-replica-manage del $HOST2
'$HOST1' has no replication agreement for '$HOST2'

ipa host-del $HOST2
ipa: ERROR: invalid 'hostname': An IPA master host cannot be deleted or disabled

Resolution

Its recommended to take a full backup first before deleting any entries from LDAP database.

Now delete the orphaned IPA master references from LDAP database.

Raw

ldapdelete -r -x -D "cn=Directory Manager" -W 'cn=hostname.domain.com,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com'

My Case:

ldapdelete -r -x -D "cn=Directory Manager" -W 'cn=ipaserver2.infra.alphacloud.ae,cn=masters,cn=ipa,cn=etc,dc=infra,dc=alphacloud,dc=ae'

This bug is being worked upon by IPA engineering in the following BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=826677

Make sure dse.ldif does not contain any replication agreement for deleted host.

Root Cause

In a replicated environment, when you disconnect a master, the meta-data for that master is still contained in the other servers.

vdsmd restart to stabilize RHEV host

I am observing RHEV environment closely and found out that after several fixes, it runs like a charm. But, There is a certain activity that I have to perform every month to keep it stable. That activity is below command:


MAKE SURE OF NO VM is running on that host before running below command:

service vdsmd restart; sleep 90; service vdsmd restart

This command has some constraints; like:

1) There should be no VMs on this Host. VMs should not be migrating on or from this host
2) VMs should not be in a "?" state (if it's ? state VMs will reboot)
3) Run this command only if host shows inconsistent behavior

Please use this information to make it bug free.

RHEV-Mamanger (RHEVM) Console for maintenance

For maintenance purposes, you can use VNC console:

Find out vnc port of the VM via either virsh command or via virt-viewer file (from rhevm itself) and use tightVNC to get the console and do maintenance on RHEV Manager.
------------
[virt-viewer]
type=vnc
host=10.192.72.11
port=5903
password=bSl5QqDaF7IV

RHEV VM disk export and backup with best performance without east-west traffic

As per RedHat it is safest backup method for VM image  to reuse it elsewhere:
--
Choose one RHEV host with minimum VMs on it and place your VM with exports (NFS) data store on it. Also, make this host an SPM host and now run below steps:
--

1) Select a VM (Offline) and click on export

2) Check  Collapse snapshots to have only one VM disk. Wait to see confirmation dialogue!

3) exported VM can be accessed at NFS share as real disk image in images folder :)

All the best!

Virsh read commands to manage RHEV H - credentials

[root@dedroclp01 ~]# cat /etc/pki/vdsm/keys/libvirt_password
shibboleth

[root@dedroclp01 ~]# virsh list --all
Please enter your authentication name: vdsm@rhevh
Please enter your password:
 Id    Name                           State