User Tools

Site Tools


ceph:deploy_ceph_cluster_for_humans

This is an old revision of the document!


Deploying a ceph cluster

Documentation
Name: Deploying a ceph cluster
Description: Steps to deploy a ceph cluster
Modification date :21/11/2018
Owner:dodger@ciberterminal.net
Notify changes to:Owner
Tags:ceph, object storage

External documentation

Previous Requirements

Variables used in this documentation

Name Description Sample
${THESERVER} Variable used as salt target, it can be a mask of serves (see sample)
export THESERVER="demoenv-ceph*-00*"
${LISTOFSERVERS} Variable used as ceph-deploy target
export LISTOFSERVERS="demoenv-cephm-001 demoenv-cephd-001"

Deploy vm's

[SCRIPT] Project CloneWars.sh

We use:

  • 3 OSD's: disk servers
  • 1 vm as admin
  • 1 vm as monitoring
  • 1 vm as gateway

For disk servers:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -d 40GB -m 20 -O -r 4096 -v 2 -o 2


For adm & monitoring servers:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -m 20 -O -r 2048 -v 1 -o 2


For gateway:

bash CloneWars.sh -c nuclu -h ${THESERVER} -i ${THESERVERIP} -m 20 -O -r 4096 -v 2 -o 4

Run salt basic states

  • Connect to salt-master
  • Run postgresql sls <code bash> salt “${THESERVER}” state.apply salt “${THESERVER}” state.apply nsupdate </code>

Additional steps with salt

Install yum-plugin-priorities

In all the servers:

salt "${THESERVER}" pkg.install yum-plugin-priorities

Install ceph-deploy

In the adm:

salt "${THESERVER}" pkg.install ceph-deploy

Add ceph user

In all the servers:

salt "${THESERVER}" user.add ceph 1002

Check:

salt "${THESERVER}" user.info ceph

Add ceph user to sudoers

In all the servers:

salt "${THESERVER}" file.write /etc/sudoers.d/ceph \
"ceph ALL = (root) NOPASSWD:ALL"

Check:

salt "${THESERVER}" cmd.run 'cat /etc/sudoers.d/ceph'
salt "${THESERVER}" cmd.run "sudo whoami" runas=ceph

Generate ssh keys

All the servers:

salt "${THESERVER}" cmd.run \
    "ssh-keygen -q -N '' -f /home/ceph/.ssh/id_rsa" \
    runas=ceph

Populate ssh keys

Get pub keys, all servers:

salt "${THESERVER}" cmd.run "cat /home/ceph/.ssh/id_rsa.pub" |egrep -v "^b" | sed 's/^[[:space:]]\{1,5\}//g' > auth_keys_oss.txt

Populate authorizedkeys in all servers: <code bash> salt “${THESERVER}” file.copy /home/ceph/.ssh/idrsa.pub /home/ceph/.ssh/authorizedkeys while read LINE ; do salt “${THESERVER}” file.append /home/ceph/.ssh/authorizedkeys “${LINE}” ; done < authkeys_oss.txt </code> ====== Deploy CEPH ======

ceph actions are run on the adm node, not in salt-master

===== Software install ===== ON admin node: <code bash> su - ceph mkdir ~/ceph-deploy cd ~/ceph-deploy </code> export LISTOFSERVERS variable, for example: <code bash> export LISTOFSERVERS=“demoenv-cephadm-101 demoenv-cephm-101 demoenv-cephd-101 demoenv-cephd-102 demoenv-cephd-103” </code>

gateway node is not included on this list!

hammer is the latest version available for RHEL6!!

And run install: <code bash> ceph-deploy install ${LISTOFSERVERS} –repo-url https://download.ceph.com/rpm-hammer/el7/ </code> Nautilus is the latest for RHEL7: <code bash> ceph-deploy install ${LISTOFSERVERS} –repo-url https://download.ceph.com/rpm-nautilus/el7/ </code>

Wait for
ceph-deploy to finish its jobs (will take some time).

===== Deploy monitoring node ===== Export
THESERVER var: <code bash> ########################## # monitoring server ########################## THESERVER=“demoenv-cephm-001” </code>
Deploy it: <code bash> ceph-deploy new ${THESERVER} ceph-deploy mon create-initial ceph-deploy gatherkeys ${THESERVER} </code> Enable messenger v2 protocol: <code bash> sudo ceph mon enable-msgr2 </code> ===== Deploy manager node =====

Only for luminous+ version (version>12.x)

Export THESERVER var: <code bash> ########################## # monitoring server ########################## LISTOFSERVERS=“demoenv-cephm-001 demoenv-cephm-002” </code>
Deploy it: <code bash> for THESERVER in ${LISTOFSERVERS} ; do ceph-deploy mgr create ${THESERVER} done </code> ==== Dashboard plugin (for manager) ==== [HOWTO] Setup Ceph Dasboard ==== Enable PG autoscale (plugin for manager) ==== [HOWTO] Enable PG autoscale ===== Deploy disk nodes ===== export
LISTOFSERVERS variable, for example: <code bash> export LISTOFSERVERS=“demoenv-cephd-001 demoenv-cephd-002 demoenv-cephd-003” </code>
Check disks: <code bash> ########################## # STORAGE SERVERS ########################## #check for THESERVER in ${LISTOFSERVERS} ; do echo “${THESERVER}” ceph-deploy disk list “${THESERVER}” done </code> ==== new versions ==== For old versions (version>12.x): Create ceph filesystems: <code bash> # deploy storage nodes for THESERVER in ${LISTOFSERVERS} ; do echo “${THESERVER}”;
echo “################### ${THESERVER}: creating disk” ;
ceph-deploy osd create ${THESERVER} –data /dev/sdb;
echo “################### ${THESERVER}: listing (check) disk” ;
ceph-deploy osd list ${THESERVER};
done </code> ==== Old versions ==== For old versions (version<12.x): Create ceph filesystems: <code bash> # deploy storage nodes for THESERVER in ${LISTOFSERVERS} ; do echo “${THESERVER}” ceph-deploy disk zap ${THESERVER}:/dev/sdb ceph-deploy osd prepare ${THESERVER}:/dev/sdb ceph-deploy osd activate ${THESERVER}:/dev/sdb1 ceph-deploy disk list ${THESERVER} echo “press enter to continue” read done </code> ===== [OPTIONAL] Deploy admin on all nodes (except gateway) ===== export
LISTOFSERVERS variable, for example: <code bash> export LISTOFSERVERS=“demoenv-cephadm-101 demoenv-cephm-101 demoenv-cephd-101 demoenv-cephd-102 demoenv-cephd-103” </code> And deploy it: <code bash> ########################## # Deploy admin on all the nodes ceph-deploy admin ${LISTOFSERVERS} </code>

Check keyring (this is a salt command if you don't noticed it): <code bash> salt “${THESERVER}” file.check_perms /etc/ceph/ceph.client.admin.keyring '{}' root root 644 </code> ====== Deploy gateway ====== Doc: * http://docs.ceph.com/docs/mimic/install/install-ceph-gateway/ Deploy rados from ceph adm: <code bash> THESERVER=“demoenv-cephgw-001” ceph-deploy install –rgw ${THESERVER} –repo-url https://download.ceph.com/rpm-hammer/el7/ ceph-deploy admin ${THESERVER} ssh ${THESERVER} “sudo yum -y install ceph-radosgw” ceph-deploy rgw create ${THESERVER} </code> Make 80 as the default port: <code bash> cat »ceph.conf«EOF [client.rgw.${THESERVER}] rgwfrontends = “civetweb port=80” rgwthreadpoolsize = 100 EOF ceph-deploy –overwrite-conf config push ${THESERVER} </code> Restart radosgw: <code bash> ssh ${THESERVER} “sudo systemctl restart ceph-radosgw” ssh ${THESERVER} “sudo systemctl status ceph-radosgw” </code> check (gw node): <code bash> ssh ${THESERVER} “sudo radosgw-admin zone get” </code> Sample: <code bash> ceph@demoenv-cephadm-001 ~/ceph-deploy $ ssh ${THESERVER} “sudo radosgw-admin zone get” { “domainroot”: “.rgw”, “controlpool”: “.rgw.control”, “gcpool”: “.rgw.gc”, “logpool”: “.log”, “intentlogpool”: “.intent-log”, “usagelogpool”: “.usage”, “userkeyspool”: “.users”, “useremailpool”: “.users.email”, “userswiftpool”: “.users.swift”, “useruidpool”: “.users.uid”, “systemkey”: { “accesskey”: “”, “secretkey”: “” }, “placementpools”: [ { “key”: “default-placement”, “val”: { “indexpool”: “.rgw.buckets.index”, “datapool”: “.rgw.buckets”, “dataextrapool”: “.rgw.buckets.extra” } } ] } </code> ====== Deploying MDS ====== <code bash> for i in 1 3 ; do bash CloneWars.sh -F -c datacenter01 -i 10.20.55.1${i} -v 4 -o 2 -r 4096 -O -m 20 -h demoenv-cephFS-00${i} ; done for i in 2 4 ; do bash CloneWars.sh -F -c datacenter02 -i 10.20.55.1${i} -v 4 -o 2 -r 4096 -O -m 20 -h demoenv-cephFS-00${i} ; done </code> <code bash> export THESERVER=“demoenv-cephfs.ciberterminal.net” salt “${THESERVER}” state.apply salt “${THESERVER}” state.apply nsupdate salt “${THESERVER}” pkg.install yum-plugin-priorities salt “${THESERVER}” user.add ceph 1002 salt “${THESERVER}” file.write /etc/sudoers.d/ceph “ceph ALL = (root) NOPASSWD:ALL” salt “${THESERVER}” cmd.run “cat /etc/sudoers.d/ceph” salt “${THESERVER}” cmd.run “sudo whoami” runas=ceph salt “${THESERVER}” cmd.run “ssh-keygen -q -N '' -f /home/ceph/.ssh/idrsa” runas=ceph salt “${THESERVER}” cmd.run “cat /home/ceph/.ssh/idrsa.pub” |egrep -v “^a” | sed 's/^space{1,5}g' > authkeysoss.txt export THESERVER=“demoenv-ceph*.ciberterminal.net” salt “${THESERVER}” file.copy /home/ceph/.ssh/idrsa.pub /home/ceph/.ssh/authorizedkeys while read LINE ; do salt “${THESERVER}” file.append /home/ceph/.ssh/authorizedkeys “${LINE}” ; done < authkeys_demoenv-ceph.txt </code> From osm-001: <code bash> for i in ${MDSSERVERS} ; do scp ceph.repo ${i}:/home/ceph/ ; ssh ${i} “sudo mv /home/ceph/ceph.repo /etc/yum.repos.d/” ; done export MDSSERVERS=“demoenv-cephfs-002.ciberterminal.net demoenv-cephfs-001.ciberterminal.net demoenv-cephfs-004.ciberterminal.net demoenv-cephfs-003.ciberterminal.net” export LISTOFSERVERS=${MDSSERVERS} ceph-deploy install ${LISTOFSERVERS} ceph-deploy mds create ${LISTOFSERVERS} </code>

mds information will appear after the creation of the cephfs

<code bash> export POOLNAME=“cephfsdata-ftp” ceph osd pool create ${POOLNAME} 128 128 replicated ceph osd pool set ${POOLNAME} crushrule ciberterminalRule ceph osd pool set ${POOLNAME} compressionalgorithm snappy ceph osd pool set ${POOLNAME} compressionmode aggressive ceph osd pool set ${POOLNAME} compressionminblobsize 10240 ceph osd pool set ${POOLNAME} compressionmaxblobsize 4194304 ceph osd pool set ${POOLNAME} pgautoscalemode on export POOLNAME=“cephfsmetadata-ftp” ceph osd pool create ${POOLNAME} 128 128 replicated ceph osd pool set ${POOLNAME} crushrule ciberterminalRule ceph osd pool set ${POOLNAME} pgautoscalemode on ceph fs new cephfs cephfsmetadata-ftp cephfsdata-ftp ceph fs ls ceph -s ceph mds stat </code> ====== Troubleshooting ====== ===== Error deploying radosgw ===== [TROUBLESHOOT] Error deploying radosgw ===== Completely remove OSD from cluster ===== [HOWTO] Completely remove OSD from cluster

ceph/deploy_ceph_cluster_for_humans.1563868117.txt.gz · Last modified: 2019/07/23 07:48 by dodger