Openstack with ceph setup best practices, learnings

Jambo frames - first thing to do on entire setup including eth, switch and br

/etc/hosts plays an important role , also chrony is critical on all nodes, virtual controllers may be advantage is planned properly.

To Mount a an rbd device use
#rbd (options)

To add a ceph client
#ceph-deploy install client

Used below links but none of them is a complete guide for Luminous:
https://gist.github.com/vanduc95/a17f80d0636badd9aa002f2b493b777b
https://www.server-world.info/en/note?os=Ubuntu_16.04&p=ceph&f=1

root@cephstor1:~# ceph-volume lvm prepare --filestore --data cephstor1_vg/cephstor1_lv  --journal  cephstor1_vg/jn_lv

root@cephstor1:~#ceph-volume lvm activate --filestore  
FS id can be retrieved using:
root@cephstor1:~# ceph-volume lvm list

sudo systemctl start ceph.target       # start all daemons
sudo systemctl status ceph-osd@12  # start one osd 12 daemon

#ceph osd tree | grep -i down

ceph pg dump_stuck inactive

https://ceph.com/geen-categorie/admin-guide-replacing-a-failed-disk-in-a-ceph-cluster/


cephuser ~/ceph should be preserved and known for administration later
--
Ceph admin keyring should be replicated to all hosts and permission should be changed accordingly
--
When using Ceph, you should configure block device timeout on nova-compute.
In nova.conf on compute:

block_device_allocate_retries=600
block_device_allocate_retries_interval=1

The above setting will force nova to wait up to 600 seconds checking every second.
The default retries is set to 180 I think. Set it higher if needed than 600 if you want to support larger images as they can take more time to copy.

--
You should use rbd for glance as well and enable the copy-on-write feature in /etc/glance/glance-api.conf
change from False to True:

show_image_direct_url = True

restart the glance services and give it a go. If you don't it takes long as the cinder does the qemu-img convert from any type to raw which adds up to the deployment time.