Deploying a ceph cluster

Documentation
Name:	Deploying a ceph cluster
Description:	Steps to deploy a ceph cluster
Modification date :	21/11/2018
Owner:	dodger
Notify changes to:	Owner
Tags:	ceph, object storage
Scalate to:	Thefuckingbofh

THIS DOCUMENT IS MERELY INFORMATIVE, YOU MUST FOLLOW OFFICIAL DOCUMENTATION!

THIS DOCUMENT HAS OUT-OF-DATE PARTS

JUST DON'T COPY PASTE FROM THIS DOCUMENT UNLESS YOU KNOW WHAT YOU'RE DOING!!!

External documentation

Previous Requirements

Basic knowledge of:

Variables used in this documentation

Name	Description	Sample
`${THESERVER}`	Variable used as salt target, it can be a mask of serves (see sample)	export THESERVER="bvmlb-os-00"
`${LISTOFSERVERS}`	Variable used as `ceph-deploy` target	export LISTOFSERVERS="bvmlb-osm-001 bvmlb-osd-001"

Deploy vm's

[SCRIPT] Project CloneWars.sh

We use:

3 OSD's: disk servers
1 vm as admin
1 vm as monitoring
1 vm as gateway

For disk servers:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -d 40GB -m 20 -O -r 4096 -v 2 -o 2

For adm & monitoring servers:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -m 20 -O -r 2048 -v 1 -o 2

For gateway:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -m 20 -O -r 4096 -v 2 -o 4

Run salt basic states

Connect to salt-master
Run postgresql sls <code bash> salt “${THESERVER}” state.apply salt “${THESERVER}” state.apply nsupdate </code>

Additional steps with salt

Install yum-plugin-priorities

In all the servers:

salt "${THESERVER}" pkg.install yum-plugin-priorities

Install ceph-deploy

In the adm:

salt "${THESERVER}" pkg.install ceph-deploy

Add ceph user

In all the servers:

salt "${THESERVER}" user.add ceph 1002

Check:

salt "${THESERVER}" user.info ceph

Add ceph user to sudoers

In all the servers:

salt "${THESERVER}" file.write /etc/sudoers.d/ceph \
"ceph ALL = (root) NOPASSWD:ALL"

Check:

salt "${THESERVER}" cmd.run 'cat /etc/sudoers.d/ceph'
salt "${THESERVER}" cmd.run "sudo whoami" runas=ceph

Generate ssh keys

All the servers:

salt "${THESERVER}" cmd.run \
    "ssh-keygen -q -N '' -f /home/ceph/.ssh/id_rsa" \
    runas=ceph

Populate ssh keys

Get pub keys, all servers:

salt "${THESERVER}" cmd.run "cat /home/ceph/.ssh/id_rsa.pub" |egrep -v "^b" | sed 's/^[[:space:]]\{1,5\}//g' > auth_keys_oss.txt

Populate authorizedkeys in all servers: <code bash> salt “${THESERVER}” file.copy /home/ceph/.ssh/idrsa.pub /home/ceph/.ssh/authorizedkeys while read LINE ; do salt “${THESERVER}” file.append /home/ceph/.ssh/authorizedkeys “${LINE}” ; done < authkeys_oss.txt </code> ====== Deploy CEPH ======




ceph actions are run on the adm node, not in salt-master





===== Software install =====

ON admin node:
<code bash>
su - ceph
mkdir ~/ceph-deploy
cd ~/ceph-deploy
</code>

export LISTOFSERVERS variable, for example:
<code bash>
export LISTOFSERVERS=“bvmld-osadm-101 bvmld-osm-101 bvmld-osd-101 bvmld-osd-102 bvmld-osd-103”
</code>



gateway node is not included on this list!



hammer is the latest version available for RHEL6!!




And run install:
<code bash>
ceph-deploy install ${LISTOFSERVERS} –repo-url https://download.ceph.com/rpm-hammer/el7/
</code>

Nautilus is the latest for RHEL7:
<code bash>
ceph-deploy install ${LISTOFSERVERS} –repo-url https://download.ceph.com/rpm-nautilus/el7/
</code>







Wait for ceph-deploy to finish its jobs (will take some time).







===== Deploy monitoring node =====
Export THESERVER var:
<code bash>
##########################
# monitoring server
##########################
THESERVER=“bvmlb-osm-001”
</code>



Deploy it:
<code bash>
ceph-deploy new ${THESERVER}
ceph-deploy mon create-initial
ceph-deploy gatherkeys ${THESERVER}
</code>


Enable messenger v2 protocol:
<code bash>
sudo ceph mon enable-msgr2
</code>


===== Deploy manager node =====



Only for luminous+ version (version>12.x)





Export THESERVER var:
<code bash>
##########################
# monitoring server
##########################
LISTOFSERVERS=“bvmlb-osm-001 bvmlb-osm-002”
</code>



Deploy it:
<code bash>
for THESERVER in ${LISTOFSERVERS} ; do
    ceph-deploy mgr create ${THESERVER}
done
</code>

==== Dashboard plugin (for manager)  ====

[HOWTO] Setup Ceph Dasboard
==== Enable PG autoscale (plugin for manager)  ====

[HOWTO] Enable PG autoscale

===== Deploy disk nodes =====

export LISTOFSERVERS variable, for example:
<code bash>
export LISTOFSERVERS=“bvmlb-osd-001 bvmlb-osd-002 bvmlb-osd-003”
</code>




Check disks:
<code bash>
##########################
# STORAGE SERVERS
##########################
#check
for THESERVER in ${LISTOFSERVERS} ; do
    echo “${THESERVER}”
    ceph-deploy disk list “${THESERVER}”
done
</code>


==== new versions ====
For old versions (version>12.x):
Create ceph filesystems:
<code bash>
# deploy storage nodes
for THESERVER in ${LISTOFSERVERS} ; do
     echo “${THESERVER}”;

     echo “################### ${THESERVER}: creating disk” ;

     ceph-deploy osd create ${THESERVER} –data /dev/sdb;

     echo “################### ${THESERVER}: listing (check) disk” ;

     ceph-deploy osd list ${THESERVER};

done
</code>


==== Old versions ====
For old versions (version<12.x):
Create ceph filesystems:
<code bash>
# deploy storage nodes
for THESERVER in ${LISTOFSERVERS} ; do
    echo “${THESERVER}”
    ceph-deploy disk zap ${THESERVER} /dev/sdb
    ceph-deploy osd prepare ${THESERVER} /dev/sdb
    ceph-deploy osd activate ${THESERVER} /dev/sdb1
    ceph-deploy disk list ${THESERVER}
    echo “press enter to continue”
    read
done
</code>


===== [OPTIONAL] Deploy admin on all nodes (except gateway) =====
export LISTOFSERVERS variable, for example:
<code bash>
export LISTOFSERVERS=“bvmld-osadm-101 bvmld-osm-101 bvmld-osd-101 bvmld-osd-102 bvmld-osd-103”
</code>

And deploy it:
<code bash>
##########################
# Deploy admin on all the nodes

ceph-deploy admin ${LISTOFSERVERS}
</code>






Check keyring (this is a salt command if you don't noticed it):
<code bash>
salt “${THESERVER}”  file.check_perms /etc/ceph/ceph.client.admin.keyring '{}' root root 644
</code>

====== Deploy gateway ======
Doc:
  * http://docs.ceph.com/docs/mimic/install/install-ceph-gateway/

Deploy rados from ceph adm:
<code bash>
THESERVER=“bvmlb-osgw-001”

ceph-deploy install –rgw ${THESERVER} –repo-url https://download.ceph.com/rpm-hammer/el7/
ceph-deploy admin ${THESERVER}
ssh ${THESERVER} “sudo yum -y install ceph-radosgw”
ceph-deploy rgw create ${THESERVER}
</code>


Make 80 as the default port:
<code bash>
cat »ceph.conf«EOF
[client.rgw.${THESERVER}]
rgwfrontends = “civetweb port=80”
rgwthreadpoolsize = 100
EOF
ceph-deploy –overwrite-conf config push ${THESERVER}
</code>

Restart radosgw:
<code bash>
ssh ${THESERVER} “sudo systemctl restart ceph-radosgw”
ssh ${THESERVER} “sudo systemctl status ceph-radosgw”
</code>

check (gw node):
<code bash>
ssh ${THESERVER} “sudo radosgw-admin zone get”
</code>

Sample:
<code bash>
ceph@bvmlb-osadm-001 ~/ceph-deploy $ ssh ${THESERVER} “sudo radosgw-admin zone get”
{
    “domainroot”: “.rgw”,
    “controlpool”: “.rgw.control”,
    “gcpool”: “.rgw.gc”,
    “logpool”: “.log”,
    “intentlogpool”: “.intent-log”,
    “usagelogpool”: “.usage”,
    “userkeyspool”: “.users”,
    “useremailpool”: “.users.email”,
    “userswiftpool”: “.users.swift”,
    “useruidpool”: “.users.uid”,
    “systemkey”: {
        “accesskey”: “”,
        “secretkey”: “”
    },
    “placementpools”: [
        {
            “key”: “default-placement”,
            “val”: {
                “indexpool”: “.rgw.buckets.index”,
                “datapool”: “.rgw.buckets”,
                “dataextrapool”: “.rgw.buckets.extra”
            }
        }
    ]
}
</code>


====== Deploying MDS ======

<code bash>
for i in 1 3  ; do bash CloneWars.sh -F -c nuciberterminal -i 10.20.55.1${i} -v 4 -o 2  -r 4096 -O -m 20 -h AVMLP-OSFS-00${i}  ; done
for i in 2 4  ; do bash CloneWars.sh -F -c nuciberterminal2 -i 10.20.55.1${i} -v 4 -o 2  -r 4096 -O -m 20 -h AVMLP-OSFS-00${i}  ; done
</code>

<code bash>
export THESERVER=“avmlp-osfs.ciberterminal.net”
salt “${THESERVER}” state.apply 
salt “${THESERVER}” state.apply nsupdate
salt “${THESERVER}” pkg.install yum-plugin-priorities
salt “${THESERVER}” user.add ceph 1002
salt “${THESERVER}” file.write /etc/sudoers.d/ceph “ceph ALL = (root) NOPASSWD:ALL”
salt “${THESERVER}” cmd.run “cat /etc/sudoers.d/ceph”
salt “${THESERVER}” cmd.run “sudo whoami” runas=ceph
salt “${THESERVER}” cmd.run     “ssh-keygen -q -N '' -f /home/ceph/.ssh/idrsa”     runas=ceph
salt “${THESERVER}” cmd.run “cat /home/ceph/.ssh/idrsa.pub” |egrep -v “^a” | sed 's/^space{1,5}g' > authkeysoss.txt
export THESERVER=“avmlp-os*.ciberterminal.net”
salt “${THESERVER}” file.copy /home/ceph/.ssh/idrsa.pub /home/ceph/.ssh/authorizedkeys
while read LINE ; do salt “${THESERVER}” file.append /home/ceph/.ssh/authorizedkeys “${LINE}” ; done < authkeys_avmlp-os.txt
</code>

From osm-001:
<code bash>
for i in ${MDSSERVERS} ; do scp ceph.repo ${i}:/home/ceph/ ; ssh ${i} “sudo mv /home/ceph/ceph.repo /etc/yum.repos.d/”  ; done
export MDSSERVERS=“avmlp-osfs-002.ciberterminal.net avmlp-osfs-001.ciberterminal.net avmlp-osfs-004.ciberterminal.net avmlp-osfs-003.ciberterminal.net”
export LISTOFSERVERS=${MDSSERVERS}
ceph-deploy install ${LISTOFSERVERS}
ceph-deploy mds create ${LISTOFSERVERS}
</code>



mds information will appear after the creation of the cephfs





<code bash>
export POOLNAME=“cephfsdata-ftp”
ceph osd pool create ${POOLNAME} 128 128 replicated
ceph osd pool set ${POOLNAME} crushrule ciberterminalRule	 
ceph osd pool set ${POOLNAME} compressionalgorithm snappy
ceph osd pool set ${POOLNAME} compressionmode aggressive
ceph osd pool set ${POOLNAME} compressionminblobsize 10240
ceph osd pool set ${POOLNAME} compressionmaxblobsize 4194304
ceph osd pool set ${POOLNAME} pgautoscalemode on
export POOLNAME=“cephfsmetadata-ftp”

ceph osd pool create ${POOLNAME} 128 128 replicated
ceph osd pool set ${POOLNAME} crushrule ciberterminalRule	
ceph osd pool set ${POOLNAME} pgautoscalemode on

ceph fs new cephfs cephfsmetadata-ftp cephfsdata-ftp
ceph fs ls
ceph -s
ceph mds stat
</code>




====== Troubleshooting ======
===== Error deploying radosgw =====
[TROUBLESHOOT] Error deploying radosgw
===== Completely remove OSD from cluster =====
[HOWTO] Completely remove OSD from cluster

Ciberterminal Wiki

Table of Contents

Deploying a ceph cluster

External documentation

Previous Requirements

Variables used in this documentation

Deploy vm's

Run salt basic states

Additional steps with salt

Install yum-plugin-priorities

Install ceph-deploy

Add ceph user

Add ceph user to sudoers

Generate ssh keys

Populate ssh keys