Host non responsive due to "not able to connect to storage Default"!

Found out (in logs) that FC storage latency was high.

By default RHEV Manger checks for 10ms. We changed that to 36ms earlier and 75ms later. Still, we  see it going to 89-200ms which seems danger to environment.

Escalated to storage team.

RHEV export storage domain can not be imported

Resolution

1) Find export storage domain metadata at the following location (on nfs server):

///dom_md/metadata

where is the root of NFS export used as the export storage domain
and is the UUID for storage domain

2) Edit storage domain metadata and replace POOL_UUID line with:

Raw

POOL_UUID=

3) Completely remove _SHA_CKSUM line

Example of edited metadata:

Raw

CLASS=Backup
DESCRIPTION=myexport
IOOPTIMEOUTSEC=1
LEASERETRIES=3
LEASETIMESEC=5
LOCKPOLICY=
LOCKRENEWALINTERVALSEC=5
MASTER_VERSION=0
POOL_UUID=
REMOTE_PATH=10.10.1.20:/exports/myexport1
ROLE=Regular
SDUUID=88ff83d5-91c1-4067-a3ea-43d1ab6e867c
TYPE=NFS
VERSION=0

VM's shows status unknown and there are 2 libvirtd daemon running

This is the process to avoid any shutdown of VMs: 1) Disable power management for this host. Do this from the webUI by opening the Edit menu for the host, opening the Power Management tab, uncheck "Enable Power Management" and then click ok. 2) On the host, kill the two libvirt processes.

ps -ef|grep libvirt|grep listen 3) Start libvirtd again, note that this requires initctl instead of service on RHEL 6: # initctl start libvirtd 4) Wait about 60 seconds and then try the migration again.

How to troubleshoot device-mapper: table: 253:7: multipath: error getting device？

version：redhat 6.4
problem：

[root@j1 ~]#dmesg
[Hardware Error]: Machine check events logged
[Hardware Error]: Machine check events logged
[Hardware Error]: Machine check events logged
[Hardware Error]: Machine check events logged
ISO 9660 Extensions: Microsoft Joliet Level 3
ISO 9660 Extensions: RRIP_1991A
device-mapper: table: 253:7: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:7: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:7: multipath: error getting device
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:7: multipath: error getting device
device-mapper: ioctl: error adding target to table

[root@j1 ~]# dmsetup info -C
Name             Maj Min Stat Open Targ Event  UUID
vg00-LogVol01    253   5 L--w    1    1      0 LVM-eJV2Rjx11AacJLzet6tcb2z2O9PTUku64U1mtUXXTcl72B6Iu6W5V6Hin7uTNOkw
vg00-LogVol00    253   0 L--w    1    1      0 LVM-eJV2Rjx11AacJLzet6tcb2z2O9PTUku6EJrCeqVbJoyKw44HELUYjF9N2dVlHc4Q
mpathd           253   4 L--w   33    1      1 mpath-36001438009b064580000400000710000
mpathc           253   3 L--w    9    1      1 mpath-36001438009b064580000400000650000
mpathb           253   2 L--w   63    1      1 mpath-36001438009b0645800004000006b0000
vg00-LogVol03    253   1 L--w    1    1      0 LVM-eJV2Rjx11AacJLzet6tcb2z2O9PTUku6HogLq442U3F5A2Kj4PDAOm56R1Krex2j
vg00-LogVol02    253   6 L--w    1    1      0 LVM-eJV2Rjx11AacJLzet6tcb2z2O9PTUku6wQS2Nd3n04mFad51Lx1Y8VBdqIxHq0KV

[root@j1 ~]# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg00   1   4   0 wz--n- 558.68g 369.23g
[root@j1 ~]# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sda2  vg00 lvm2 a--  558.68g 369.23g
[root@j1 ~]# lvs
  LV       VG   Attr      LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  LogVol00 vg00 -wi-ao--- 97.66g
  LogVol01 vg00 -wi-ao---  9.77g
  LogVol02 vg00 -wi-ao--- 19.53g
  LogVol03 vg00 -wi-ao--- 62.50g
[root@j1 ~]# lsmod |grep dm_multipath
dm_multipath           17756  4 dm_round_robin
dm_mod                 82839  22 dm_multipath,dm_mirror,dm_log

[root@blj1 ~]# dmsetup info -C|grep "253  51"
c5b99426--ae33--4e40--9196--46a9d8166824-9010fc21--3827--4cce--b5f8--8c843868cab2 253  51 L--w    1   10      0 LVM-WtianbDfgqXa3kjqvNu3Pz2PVO1Vdv8CA84E6FJbCyYfkkDQpGdPYSwrgnk1sYPZ

[root@blj1 ~]# lvs |grep 8c843868cab2
  9010fc21-3827-4cce-b5f8-8c843868cab2 c5b99426-ae33-4e40-9196-46a9d8166824 -wi-ao----  17.12g

root@j1 ~]# grep -v ^$ /etc/multipath.conf
defaults {
        udev_dir                /dev
        polling_interval        10
        path_selector           "round-robin 0"
        path_grouping_policy    failover
        getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        prio                    alua
        path_checker            tur
        rr_min_io               100
        rr_min_io_rq            1
        rr_weight               uniform
        failback                immediate
        no_path_retry           12
        user_friendly_names     yes
}
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]"
        devnode "^hd[a-z][[0-9]]"
        devnode "^cciss!c[0-9]d[0-9]"
        devnode "^sda[0-9]"
}
devices {
        device {
               vendor                   "HP"
               product                  "OPEN-."
               path_grouping_policy     multibus
               getuid_callout           "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
               path_selector            "round-robin 0"
               path_checker             tur
               features                 "0"
               hardware_handler         "0"
               prio                     const
               failback                 immediate
               rr_weight                uniform
               no_path_retry            queue
               rr_min_io                1000
               rr_min_io_rq             1
       }
}

Script to check IPA user expiration and send email to user

1) Configure Email relay for the host, it would require you to have an SMTP server IP and put that in sendmail or pistfix (MTA) config


2) Test Emails

3) Use below script for rest of the task:
---------------------------------

#!/bin/bash

#written by - Ram Nath (ram@mydomain.com)
# notifies people a set number of days before expiry, once via email
# notifies people a set number of days before expiry, once via email

# open a kerberos ticket using keytab authentication
# the following keytab file was made using ktutil with rc4-hmac

/usr/bin/kinit admin@INFRA.MYDOMAN -k -t /test/bin/admin.keytab

# how many days before expiry? at which point a single email should be sent out

cd /tmp
THENUMBEROFDAYS=30

#queries the ldap server for whatever group you want, or search parameters you want to use
# grepping memberUid for the group you want and piping to awk results in a list of users
USERLIST=$(ldapsearch -x -b cn=users,cn=accounts,dc=infra,dc=MYDOMAIN | grep "uid:" | awk '{print $2}')

# start the main loop
for USER in $USERLIST;
do
# gets todays date in the same format as ipa
TODAYSDATE=$(date +"%Y%m%d")
echo "Checking Expiry For $USER"

# gets date, removes time uses cut to get only first 8 characters of date
EXPIRYDATE=$(ipa user-show $USER --all | grep krbpasswordexpiration | awk '{print $2}' | cut -c 1-8)

# using date command to convert to a proper date format for the subtraction of days left
CALCEXPIRY=$(date -d "$EXPIRYDATE" +%j)
CALCTODAY=$(date -d "$TODAYSDATE" +%j)
DAYSLEFT=$(expr $CALCEXPIRY - $CALCTODAY)

echo "$USER has $DAYSLEFT left"

# send out an email if it matches the specified number of days left
if [ $DAYSLEFT -le $THENUMBEROFDAYS ];
then

# create the email content
echo "Hi There," >> $USER.temp
echo " " >> $USER.temp
echo "Password for Cloud User $USER is going to expire in $DAYSLEFT days." >> $USER.temp
echo "Kindly ask user to reset it." >> $USER.temp
echo " " >> $USER.temp
echo "Regards," >> $USER.temp
echo "IPA Admin" >> $USER.temp

# send the email out
mailx -r PasswordAlerts@ipaserver.infra.mydomain -s "IPA user $USER's password expires in $DAYSLEFT days!" ram@mydomain.com < $USER.temp
# delete content file
rm -rf $USER.temp
fi
done

RHEV Manager Alert Email Notification Implementation - RHEV 3.5

1) We need to create a .conf file like engine-notifier.conf at /etc/ovirt-engine/notifier/notifier.conf.d/

[root@rhevm ~]# ls -ltr /etc/ovirt-engine/notifier/notifier.conf.d/
total 12
-rw-r--r--. 1 root root 231 Jun  7  2015 README
-rw-r--r--. 1 root root  32 Mar  3 10:36 10-setup-jboss.conf
-rw-r--r--. 1 root root  66 Mar  3 10:39 engine-notifier.conf

[root@rhevm ~]# cat /etc/ovirt-engine/notifier/notifier.conf.d/engine-notifier.conf
MAIL_SERVER=10.100.2.6
MAIL_FROM=admin@rhevm.mydomain.com  #<- email="" p="" suits="" use="" whatever="" you="">
2) Make sure that "ovirt-engine-notifier" service is up and running to read this configuration.


3) Now, you need to specify email address to receive emails from RHEV manager Console at :
Syetsm > Users > Admin@Internal > Event notifier > Manage Events > Check all requited fields and specify recipient email id

You should be getting alerts now on.

That's All!