RHEV host and Self hosted engine upgrade to RHEV 3.6

RHEL Host (RHEV) upgrade: ==================== Applying RHEV updates on Red Hat Enterprise Linux 6 Enable Repos: #subscription-manager repos --disable=* #subscription-manager repos --enable=rhel-6-server-rpms #subscription-manager repos --enable=rhel-6-server-optional-rpms #subscription-manager repos --enable=rhel-6-server-rhev-mgmt-agent-rpms #subscription-manager repos –enable=rhel-6-server-supplementary-rpms Set hosted-engine status to maintenance: #hosted-engine --vm-status #hosted-engine --set-maintenance --mode=global Before installing an update, make sure all previously released errata relevant to the system have been applied. To access updates when using Red Hat Enterprise Linux 6, launch the graphical update tool through System -> Administration -> Software Update, or from the command line via the following command: # gpk-update-viewer For a command line interface, use the following command to update the operating system: # yum update To install a specific package, such as vsftpd, use the following command: # yum install vsftpd  UPGRADING RED HAT ENTERPRISE VIRTUALIZATION MANAGER 3.6 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Upgrade_Guide/index.html#Upgrading_the_Self-Hosted_Engine1 The following procedure outlines the process for upgrading Red Hat Enterprise Virtualization Manager 3.5 to Red Hat Enterprise Virtualization Manager 3.6. This procedure assumes that the system on which the Manager is installed is subscribed to the entitlements for receiving Red Hat Enterprise Virtualization 3.5 packages at the start of the procedure. IMPORTANT If the upgrade fails, the engine-setup command will attempt to roll your Red Hat Enterprise Virtualization Manager installation back to its previous state. For this reason, the repositories required by Red Hat Enterprise Virtualization 3.5 must not be removed until after the upgrade is complete as outlined below. If the upgrade fails, detailed instructions display that explain how to restore your installation. Procedure 3.1. Upgrading to Red Hat Enterprise Virtualization Manager 3.6 Subscribe the system on which the Red Hat Enterprise Virtualization Manager is installed to the required entitlements for receiving Red Hat Enterprise Virtualization Manager 3.6 packages: With RHN Classic: # rhn-channel --add --channel=rhel-x86_64-server-6-rhevm-3.6 With Subscription Manager: # subscription-manager repos --enable=rhel-6-server-rhevm-3.6-rpms Update the rhevm setup packages: # yum update rhevm\*setup\* Run the following command and follow the prompts to upgrade the Red Hat Enterprise Virtualization Manager: # engine-setup Remove or disable the Red Hat Enterprise Virtualization Manager 3.5 channel to ensure the system does not use any Red Hat Enterprise Virtualization Manager 3.5 packages: With RHN Classic: # rhn-channel --remove --channel=rhel-x86_64-server-6-rhevm-3.5 With Subscription Manager: # subscription-manager repos --disable=rhel-6-server-rhevm-3.5-rpms Update the base operating system: # yum update

Migrating VMWARE Image (OVA) to RHEV 3.5 as a VM

I would like to share with you this secret way of migrating any VM from any hypervisor to RHEV. Red Hat has not mentioned this anywhere on the internet so I would recommend you to do this only on test environments:

1) Extract the OVA image using: tar xvf CFusion-server.ova

2) Convert the extracted VMDK disk file to raw lvm image using: qemu-img convert -f vmdk CFusion-server-disk1.vmdk -O raw CFusion-server-disk1.img

3) SCP the image (.img file) to you NFS exports server (also, Check the VM size)

4) In RHEVM, create a VM with same sized disk (preallocated)

5) Export that VM (to export NFS) and delete the newly created VM

6) Check the export VM image in images folder on NFS export location

7) Put export in maintenance mode and on nfs server, use dd command to write the main exported disk file under images folder

8) Now, activate the export again

9) Create a VM with THIN disk from exported image

You can see that VM is ready for its first run!

Procedure to update your hosted engine from 3.5 to 3.6

Procedure to update your hosted engine from 3.5 to 3.6 For going to 3.6 you need to have your host on 7.x, if your host are currently not on 7.x then you need to update your host first. For detailed procedure for updating your hosted engine to 3.6 please refer below document. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Topic :- 3.5. Upgrading the Self-Hosted Engine Link :- https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Upgrade_Guide/index.html#Upgrading_the_Self-Hosted_Engine1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

engine-image-uploader ERROR: not a gzip file!! ->> Creating an OVF Archive That is Compatible with the Image Uploader!

Creating an OVF Archive That is Compatible with the Image Uploader

1. Run the tar -zcvf my.ovf images/ master/ command to create the tar/gzip ovf archive.
2. Anyone you give the resulting ovf file to (in this example, called my.ovf) can import it to Red Hat Enterprise Virtualization Manager using the engine-image-uploader command.

You have created a compressed OVF image file that can be distributed. Anyone you give it to can use the engine-image-uploader command to upload your image into their Red Hat Enterprise Virtualization environment.

Host non responsive due to "not able to connect to storage Default"!

Found out (in logs) that FC storage latency was high.

By default RHEV Manger checks for 10ms. We changed that to 36ms earlier and 75ms later. Still, we  see it going to 89-200ms which seems danger to environment.

Escalated to storage team.

RHEV export storage domain can not be imported

Resolution

1) Find export storage domain metadata at the following location (on nfs server):

///dom_md/metadata

where is the root of NFS export used as the export storage domain
and is the UUID for storage domain

2) Edit storage domain metadata and replace POOL_UUID line with:

Raw

POOL_UUID=

3) Completely remove _SHA_CKSUM line

Example of edited metadata:

Raw

CLASS=Backup
DESCRIPTION=myexport
IOOPTIMEOUTSEC=1
LEASERETRIES=3
LEASETIMESEC=5
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=0
POOL_UUID=
REMOTE_PATH=10.10.1.20:/exports/myexport1
ROLE=Regular
SDUUID=88ff83d5-91c1-4067-a3ea-43d1ab6e867c
TYPE=NFS
VERSION=0

VM's shows status unknown and there are 2 libvirtd daemon running

This is the process to avoid any shutdown of VMs: 1) Disable power management for this host. Do this from the webUI by opening the Edit menu for the host, opening the Power Management tab, uncheck "Enable Power Management" and then click ok. 2) On the host, kill the two libvirt processes.

ps -ef|grep libvirt|grep listen 3) Start libvirtd again, note that this requires initctl instead of service on RHEL 6: # initctl start libvirtd 4) Wait about 60 seconds and then try the migration again.

RHEV Manager Alert Email Notification Implementation - RHEV 3.5

1) We need to create a .conf file like engine-notifier.conf at /etc/ovirt-engine/notifier/notifier.conf.d/

[root@rhevm ~]# ls -ltr /etc/ovirt-engine/notifier/notifier.conf.d/
total 12
-rw-r--r--. 1 root root 231 Jun  7  2015 README
-rw-r--r--. 1 root root  32 Mar  3 10:36 10-setup-jboss.conf
-rw-r--r--. 1 root root  66 Mar  3 10:39 engine-notifier.conf

[root@rhevm ~]# cat /etc/ovirt-engine/notifier/notifier.conf.d/engine-notifier.conf
MAIL_SERVER=10.100.2.6
MAIL_FROM=admin@rhevm.mydomain.com  #<- email="" p="" suits="" use="" whatever="" you="">
2) Make sure that "ovirt-engine-notifier" service is up and running to read this configuration.


3) Now, you need to specify email address to receive emails from RHEV manager Console at :
Syetsm > Users > Admin@Internal > Event notifier > Manage Events > Check all requited fields and specify recipient email id

You should be getting alerts now on.

That's All!

Recursive file permissions issue on / directory

One developer had run "sudo chown -R user ../../" from his home directory and created permission issues at / in one of the systems. Most of the directories at "/" level are with "user:root" permissions and it looks all bad. System has stopped responding to ssh connections. It says "connections refused" to new ssh connections. 

Solution:

This is a very bad condition but yes, we can recover it using the rpm database. Run the command as follows?

# for i in $(rpm -qa); do echo "Working on $i"; rpm --setugids $i; rpm --setperms $i; done

This will restore the user ownership, group ownership as well as set the permissions too. This will display lots and lots of messages on screen, as well as the errors, you can just ignore them out. And it will take much time to restore the permissions/ownership depending on the number of installed rpms.

Please note that only those file/directory permissions/ownership will be changed back to normal which are deployed by rpm. Other file permissions/ownerships won't be changed at all.

VM migration Failure - Timeout Issues!

##### Action plan:
1. Disable Power Management on the host where the vdsm has to be restarted. This is to prevent fencing of that host by RHEV Manager.

2. SSH into destination host and add migration_listener_timeout value as 1500 to the existing values under [vars] in /etc/vdsm/vdsm.conf file followed by restarting the vdsmd service.

migration_listener_timeout = 1500

- Restart the vdsmd service: [ Please ensure 'Power Management' disabled on host before restarting vdsmd service as mentioned in above step 1. ]

# /etc/init.d/vdsmd restart

3. Now try migrating the vm.

Host In-consistency Issue - VM migrates from one host to another

Issue: Sometimes VMs starts migrating from one host to another due to host in-consistency!

Solution: Wait till All VMs are gone (migrated) to other Host. Double check and make sure no more VMs left on that host (May be it's good idea to migrate your VMs to SPM Host).
After migrations done:

1) Run Command : "service vdsmd restart"

a) Check the output, If you see errors (red lines) indicating that restart of that daemon was not successful.

b) In case you see errors (red output), Wait for another 90 seconds and check RHEVM console for host status. Make sure it's green before going it to next step.

c) Re-run "service vdsmd restart" (It should give you all green and proper restart of daemon)

Now you can bring back VMs. Host will be consistence at least for one month.


Take care!

IPA Replica Uninstall Error - Fixed

Issue

When removing a disconnected IPA replica server which had replication agreements with multiple IPA hosts, the meta-data for that replica is still contained on still running IPA servers. This does not allow you to remove the disconnected IPA server completely. There are special tasks you can use to remove this meta-data. For example:

Raw

Initial topology:  (simple triangle)
2 - 3
 \ /
  1

# on host1:
ipa-replica-manage disconnect host1 host2
ipa-replica-manage del host3  # not sure if this one is relevant here

# on host2:
ipa-server-install --uninstall -U

# on host1:
ipa-replica-prepare -p $ADMINPW --ip-address=$HOST2_IP $HOST2

# on host2:
sftp root@$HOST1:/var/lib/ipa/replica-info-$HOST2.gpg
ipa-replica-install -U --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$HOST2.gpg
...
The host qe-blade-04.testrelm.com already exists on the master server. Depending on your configuration, you may perform the following:

Remove the replication agreement, if any:
    % ipa-replica-manage del qe-blade-04.testrelm.com
Remove the host entry:
    % ipa host-del qe-blade-04.testrelm.com

# on host1:
ipa-replica-manage del $HOST2
'$HOST1' has no replication agreement for '$HOST2'

ipa host-del $HOST2
ipa: ERROR: invalid 'hostname': An IPA master host cannot be deleted or disabled

Resolution

Its recommended to take a full backup first before deleting any entries from LDAP database.

Now delete the orphaned IPA master references from LDAP database.

Raw

ldapdelete -r -x -D "cn=Directory Manager" -W 'cn=hostname.domain.com,cn=masters,cn=ipa,cn=etc,dc=domain,dc=com'

My Case:

ldapdelete -r -x -D "cn=Directory Manager" -W 'cn=ipaserver2.infra.alphacloud.ae,cn=masters,cn=ipa,cn=etc,dc=infra,dc=alphacloud,dc=ae'

This bug is being worked upon by IPA engineering in the following BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=826677

Make sure dse.ldif does not contain any replication agreement for deleted host.

Root Cause

In a replicated environment, when you disconnect a master, the meta-data for that master is still contained in the other servers.

vdsmd restart to stabilize RHEV host

I am observing RHEV environment closely and found out that after several fixes, it runs like a charm. But, There is a certain activity that I have to perform every month to keep it stable. That activity is below command:


MAKE SURE OF NO VM is running on that host before running below command:

service vdsmd restart; sleep 90; service vdsmd restart

This command has some constraints; like:

1) There should be no VMs on this Host. VMs should not be migrating on or from this host
2) VMs should not be in a "?" state (if it's ? state VMs will reboot)
3) Run this command only if host shows inconsistent behavior

Please use this information to make it bug free.

RHEV-Mamanger (RHEVM) Console for maintenance

For maintenance purposes, you can use VNC console:

Find out vnc port of the VM via either virsh command or via virt-viewer file (from rhevm itself) and use tightVNC to get the console and do maintenance on RHEV Manager.
------------
[virt-viewer]
type=vnc
host=10.192.72.11
port=5903
password=bSl5QqDaF7IV

RHEV VM disk export and backup with best performance without east-west traffic

As per RedHat it is safest backup method for VM image  to reuse it elsewhere:
--
Choose one RHEV host with minimum VMs on it and place your VM with exports (NFS) data store on it. Also, make this host an SPM host and now run below steps:
--

1) Select a VM (Offline) and click on export

2) Check  Collapse snapshots to have only one VM disk. Wait to see confirmation dialogue!

3) exported VM can be accessed at NFS share as real disk image in images folder :)

All the best!

Virsh read commands to manage RHEV H - credentials

[root@dedroclp01 ~]# cat /etc/pki/vdsm/keys/libvirt_password
shibboleth

[root@dedroclp01 ~]# virsh list --all
Please enter your authentication name: vdsm@rhevh
Please enter your password:
 Id    Name                           State

rhevm network persistence issue

Today I observed a strange issue due to host non operational

>Tried service vdsmd restart but there was another issue with rhevm network which went down and was not coming up:

I had to recreate /etc/sysconfig/network-scripts/ifcfg-rhevm (it was trying dhcp for ip addressing) from another host.

Also I had to re create  bond0 at /var/lib/vdsm/persistence/netconf/bond and rhevm at /var/lib/vdsm/persistence/netconf/nets

--=====

[root@dedrhosp4 bin]# cd /var/lib/vdsm/persistence/netconf
[root@dedrhosp4 netconf]# pwd
/var/lib/vdsm/persistence/netconf
[root@dedrhosp4 netconf]# ll
total 8
drwxr-xr-x. 2 root root 4096 Jul 30 22:07 bonds
drwxr-xr-x. 2 root root 4096 Jul 30 22:07 nets
[root@dedrhosp4 netconf]# ll -R
.:
total 8
drwxr-xr-x. 2 root root 4096 Jul 30 22:07 bonds
drwxr-xr-x. 2 root root 4096 Jul 30 22:07 nets

./bonds:
total 24
-rw-r--r--. 1 root root 58 Jul 30 22:07 bond0
-rw-r--r--. 1 root root 58 Jun 10 15:17 bond1
-rw-r--r--. 1 root root 58 Jun 10 15:17 bond2
-rw-r--r--. 1 root root 58 Jun 10 15:17 bond3
-rw-r--r--. 1 root root 58 Jun 10 15:17 bond4
-rw-r--r--. 1 root root 60 Jun 10 15:17 bond5

./nets:
total 68
-rw-r--r--. 1 root root 121 Jun 10 15:17 intnet_655
-rw-r--r--. 1 root root 106 Jun 10 15:17 neutron
-rw-r--r--. 1 root root 224 Jul 30 22:07 rhevm
-rw-r--r--. 1 root root 120 Jun 10 15:17 vlan_25
-rw-r--r--. 1 root root 203 Jun 10 15:17 vlan_651
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_656
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_657
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_700
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_701
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_712
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_714
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_715
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_800
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_801
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_802
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_803
-rw-r--r--. 1 root root 121 Jun 10 15:17 vlan_804

Host Non-operational issue - Storage domain monitoring thread slows down

Implemented following change to resolve/avoid this issue:

1)      Check the current configuration on RHEV Manager using after switching to root:

engine-config -g MaxStorageVdsTimeoutCheckSec

2)      Change the current configuration (change value to 75):

rhevm-config -s MaxStorageVdsTimeoutCheckSec=75

Now 2^nd parameter  “vdsHeartbeatInSeconds”

3)      Check the current configuration:

engine-config -g vdsHeartbeatInSeconds

4)      Change the current configuration (change value to 20) if it’s not set to 20 already:

rhevm-config -s vdsHeartbeatInSeconds=20

5)      Restart the ovirt-engine service after this.

               /etc/init.d/ovirt-engine restart

Check the changed configurations:

engine-config -g MaxStorageVdsTimeoutCheckSec

engine-config -g vdsHeartbeatInSeconds

Data storage domain (FC storage) has some issue. it is getting unavailable to different RHEV hosts randomly.

We can fix this issue by setting following steps on RHEV manager:
---

#grep vdsHeartbeatInSeconds.type /etc/ovirt-engine/engine-config/engine-config.properties

#cp /etc/ovirt-engine/engine-config/engine-config.properties /etc/ovirt-engine/engine-config/engine-config.properties.backup

#echo vdsHeartbeatInSeconds.type=Integer >> /etc/ovirt-engine/engine-config/engine-config.properties

#engine-config -s vdsHeartbeatInSeconds=20

#service ovirt-engine restart

#service jbossas restart

#engine-config  -l vdsHeartbeatInSeconds

#engine-config  --help

#engine-config -g vdsHeartbeatInSeconds

IPA user unable to login - rare issue

/tmp permission could be an issue, check and change!