ceph: Prepare the environment

The goal of this guide is to serve as a precursor to the installation of ceph nautilus on CentOS 7.  This process was completed on a virtual environment and has been reproduced several times before this writing.   Ceph is an open-source software storage platform that replicates data and makes it fault-tolerant. A good definition and intro to ceph may be found in the Ceph documentation.

Ceph Environment Overview

For the simplicity of this guide, we will have only one monitor and three osd servers.  The hardware specifications are fairly minimal as is the installation of CentOS 7, minimal with very few extras installed. Each of the hard drives are 32 GB each, should be much larger for a production environment.

fqdn public ip cluster ip disks mem cpu
mon1.it-megocollector.com 10.10.0.100/24 192.168.1.100/24 1 2 1
osd1.it-megocollector.com 10.10.0.101/24 192.168.1.101/24 4 (sdb sdc sdd) 2 1
osd2.it-megocollector.com 10.10.0.102/24 192.168.1.102/24 4 (sdb sdc sdd) 2 1
osd3.it-megocollector.com 10.10.0.103/24 192.168.1.103/24 4 (sdb sdc sdd) 2 1

When complete there should be four servers with fully qualified domain names ("fqdn") host names that are registered in your local DNS that each contain a user named ceph that is a member of sudoers and can ssh into each of the other servers without a password.

Nework

On each of the four servers, ensure that there are two network adapters.  One adapter for the the public IP and the other adapter for the cluster IP.  The public IP will need a gateway and point to a DNS server.  The cluster IP will only be an IP address with NO gateway and NO DNS server.

If no DNS server, edit the /etc/hosts.  This may do the trick, but that is not how this was tested.

cat << 'EOF' >> /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.0.100 mon1 mon1.it.megocollector.com
10.10.0.101 osd1 osd1.it.megocollector.com
10.10.0.102 osd2 osd2.it.megocollector.com
10.10.0.103 osd3 osd3.it.megocollector.com
EOF

Hostname

Set the hostname on each of the four servers, substituting the appropriate fqdn as seen in the example below.

hostnamectl set-hostname mon1.it.megocollector.com --static

SeLinux

Until there is a solution, permissive is better than disable.  The following two lines will put SeLinux into permissive mode.

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux

Firewall

For this example, let's disable it.  An ansible script from an article written earlier today will re-enable with appropriate ports open.

systemctl stop firewalld
systemctl disable firewalld

After executing the ansible-playbook, the firewall rules were added.  These are the firewall rules added to the firewall as found on a fresh install.

[ceph@mon1 ceph-ansible]$ sudo firewall-cmd --list-services
ceph ceph-mon dhcpv6-client ssh
[ceph@mon1 ceph-ansible]$ sudo firewall-cmd --list-ports
80/tcp 2003/tcp 4505/tcp 4506/tcp 6800-7300/tcp 6789/tcp 8443/tcp 9100/tcp 9283/tcp 3000/tcp 9092/tcp 9093/tcp 9094/tcp 9094/udp 8080/tcp

Create ceph user

This user must be able to sudo and access without password.

useradd ceph
passwd ceph
cat << 'EOF' > /etc/sudoers.d/ceph
ceph ALL=(ALL) NOPASSWD: ALL
EOF

Create Keys

Access keys so that ceph user can ssh without prompt for password.  For this to work, change user to ceph. The '-' will put the ceph user in the /home/ceph directory.  Then run the commands.

su - ceph
echo -e | ssh-keygen -t rsa -N ""
echo 'StrictHostKeyChecking no' > ~/.ssh/config
chmod 644 .ssh/config

Copy the ssh keys between each of the four servers.

sshpass -p ceph ssh-copy-id 10.10.0.100 -f
sshpass -p ceph ssh-copy-id 10.10.0.101 -f
sshpass -p ceph ssh-copy-id 10.10.0.102 -f
sshpass -p ceph ssh-copy-id 10.10.0.103 -f

Test

From each server, test to see if the ceph user can ssh into each of the other servers.

ssh ceph@10.10.10.100
ssh ceph@10.10.10.101
ssh ceph@10.10.10.102
ssh ceph@10.10.10.103

#or use the fqdn.  Good test to see if DNS is working.

ssh ceph@mon1.it.megocollector.com
ssh ceph@osd1.it.megocollector.com
ssh ceph@osd2.it.megocollector.com
ssh ceph@osd3.it.megocollector.com

So what about the hard drives? The /dev/sdb, /dev/sdc, and /dev/sdd drives will not be prepared in advance of ceph.  Ceph will handle all the preparation. Done.