Differences

This shows you the differences between two versions of the page.

--- ceph:modifying_crush_map [2019/07/31 10:53] – dodger
+++ ceph:modifying_crush_map [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-====== Modifying CRUSH map ======
-^  Documentation  ^|
-^Name:| Modifying CRUSH map |
-^Description:| How to modify CRUSH map  |
-^Modification date :|11/06/2019|
-^Owner:|dodger@ciberterminal.net|
-^Notify changes to:|Owner|
-^Tags:| ceph, object storage|
-====== Information ======
-<WRAP center round alert 60%>
-The described process is a advanced process, take care.
-</WRAP>
-====== Pre-Requirements ======
-  * Knowledge of ceph management
-  * Understand of CRUSH
-  * Understand of CRUSH mapping options
-====== Instructions ======
-===== Review actual CRUSH tree =====
-<code bash>
-ceph osd crush tree
-</code>
-Example:
-<code>
-proceph-osm-001 ~ # ceph osd crush tree
-ID  CLASS WEIGHT   TYPE NAME
- -1       39.97986 root default
- -3        1.99899     host proceph-osd-001
-   hdd  1.99899         osd.0
- -5        1.99899     host proceph-osd-002
-   hdd  1.99899         osd.1
- -7        1.99899     host proceph-osd-003
-   hdd  1.99899         osd.2
- -9        1.99899     host proceph-osd-004
-   hdd  1.99899         osd.3
--11        1.99899     host proceph-osd-005
-   hdd  1.99899         osd.4
--13        1.99899     host proceph-osd-006
-   hdd  1.99899         osd.5
--15        1.99899     host proceph-osd-007
-   hdd  1.99899         osd.6
--17        1.99899     host proceph-osd-008
-   hdd  1.99899         osd.7
--19        1.99899     host proceph-osd-009
-   hdd  1.99899         osd.8
--21        1.99899     host proceph-osd-010
-   hdd  1.99899         osd.9
--23        1.99899     host proceph-osd-011
-   hdd  1.99899         osd.10
--25        1.99899     host proceph-osd-012
-   hdd  1.99899         osd.11
--27        1.99899     host proceph-osd-013
-   hdd  1.99899         osd.12
--29        1.99899     host proceph-osd-014
-   hdd  1.99899         osd.13
--31        1.99899     host proceph-osd-015
-   hdd  1.99899         osd.14
--33        1.99899     host proceph-osd-016
-   hdd  1.99899         osd.15
--35        1.99899     host proceph-osd-017
-   hdd  1.99899         osd.16
--37        1.99899     host proceph-osd-018
-   hdd  1.99899         osd.17
--39        1.99899     host proceph-osd-019
-   hdd  1.99899         osd.18
--41        1.99899     host proceph-osd-020
-   hdd  1.99899         osd.19
-</code>
-===== Get actual map =====
-This command export the map from the running cluster, the file is **binary**:
-<code bash>
-ceph osd getcrushmap  -o crushmap.bin
-</code>
-===== Convert the map to text =====
-<code bash>
-crushtool  -d crushmap.bin -o crushmap.txt
-</code>
-===== the map file =====
-The file has the hierarchy of the tree as follows:
-  * root
-  * datacenter (if defined)
-  * rack (if defined)
-  * host
-\\
-Each item is defined as this example:
-^ rack  | <code>
-root default {
-#        id -1           # do not change unnecessarily
-#        id -2 class hdd         # do not change unnecessarily
-        # weight 39.980
-        alg straw2
-        hash 0  # rjenkins1
-        item CPD1 weight 0.000
-        item CPD2 weight 0.000
-}
-</code> |
-^ datacenter | <code>
-datacenter CPD1 {
-#        id -44          # do not change unnecessarily
-#        id -46 class hdd                # do not change unnecessarily
-        # weight 0.000
-        alg straw2
-        hash 0  # rjenkins1
-        item ciberterminal weight 0.000
-}
-</code> |
-So each "node" define the child "nodes" with the ''item'' keyword.
-<WRAP center round important 60%>
-Negative id's from entities **MUST BE REMOVED!!!** \\
-That's why I've left commented:
-<code>
-#        id -44          # do not change unnecessarily
-</code>
-</WRAP>
-Sample CRUSH map:
-<file crusmap.txt>
-# begin crush map
-tunable choose_local_tries 0
-tunable choose_local_fallback_tries 0
-tunable choose_total_tries 50
-tunable chooseleaf_descend_once 1
-tunable chooseleaf_vary_r 1
-tunable chooseleaf_stable 1
-tunable straw_calc_version 1
-tunable allowed_bucket_algs 54
-# devices
-device 0 osd.0 class hdd
-device 1 osd.1 class hdd
-device 2 osd.2 class hdd
-device 3 osd.3 class hdd
-device 4 osd.4 class hdd
-device 5 osd.5 class hdd
-device 6 osd.6 class hdd
-device 7 osd.7 class hdd
-device 8 osd.8 class hdd
-device 9 osd.9 class hdd
-device 10 osd.10 class hdd
-device 11 osd.11 class hdd
-device 12 osd.12 class hdd
-device 13 osd.13 class hdd
-device 14 osd.14 class hdd
-device 15 osd.15 class hdd
-device 16 osd.16 class hdd
-device 17 osd.17 class hdd
-device 18 osd.18 class hdd
-device 19 osd.19 class hdd
-# types
-type 0 osd
-type 1 host
-type 2 chassis
-type 3 rack
-type 4 row
-type 5 pdu
-type 6 pod
-type 7 room
-type 8 datacenter
-type 9 zone
-type 10 region
-type 11 root
-# buckets
-host ciberterminal_cluster {
-        # weight 19.990
-        alg straw2
-        hash 0  # rjenkins1
-        item osd.0 weight 1.999
-        item osd.2 weight 1.999
-        item osd.4 weight 1.999
-        item osd.6 weight 1.999
-        item osd.8 weight 1.999
-        item osd.10 weight 1.999
-        item osd.12 weight 1.999
-        item osd.14 weight 1.999
-        item osd.16 weight 1.999
-        item osd.18 weight 1.999
-}
-host ciberterminal2_cluster {
-        # weight 19.990
-        alg straw2
-        hash 0  # rjenkins1
-        item osd.1 weight 1.999
-        item osd.3 weight 1.999
-        item osd.5 weight 1.999
-        item osd.7 weight 1.999
-        item osd.9 weight 1.999
-        item osd.11 weight 1.999
-        item osd.13 weight 1.999
-        item osd.15 weight 1.999
-        item osd.17 weight 1.999
-        item osd.19 weight 1.999
-}
-root default {
-        # weight 0.000
-        alg straw2
-        hash 0  # rjenkins1
-        item ciberterminal_cluster weight 1.000
-        item ciberterminal2_cluster weight 1.000
-}
-# rules
-rule replicated_rule {
-        id 0
-        type replicated
-        min_size 1
-        max_size 10
-        step take default
-        step chooseleaf firstn 3 type host
-        step emit
-}
-rule ciberterminalRule {
-        id 1
-        type replicated
-        min_size 1
-        max_size 10
-        step take default
-        step chooseleaf firstn 1 type host
-        step emit
-}
-# end crush map
-</file>
-===== Important CRUSH parameters =====
-  * ''step chooseleaf firstn'': What I've found is that this value is the DEEP of your CRUSH tree:
-    * ''0'' Single Node Cluster
-    * ''1'' for a multi node cluster in a single rack
-    * ''2'' for a multi node, multi chassis cluster with multiple hosts in a chassis
-    * ''3'' for a multi node cluster with hosts across racks, etc.
-  * ''id'': should be removed from "buckets", not from "rules"
-===== Compile the new map =====
-<code bash>
-crushtool -c crushmap.txt -o crushmap_new.bin
-</code>
-===== Checking map before applying =====
-<WRAP center round info 60%>
-This step is **MANDATORY**!!!!
-</WRAP>
-Perform a test of OSD utilization with ruleset ''--rule <ID>'' and number of replicas ''--num-rep=<NUM_REP>'':
-<code bash>
-crushtool --test -i crushmap_new.bin --show-utilization --rule <ID> --num-rep=<NUM_REP>
-</code>
-Sample:
-<code>
-# crushtool --test -i crushmap_new.map --show-utilization --rule 1 --num-rep=4
-rule 1 (ciberterminalRule), x = 0..1023, numrep = 4..4
-rule 1 (ciberterminalRule) num_rep 4 result size == 1:  1024/1024
-  device 0:              stored : 58     expected : 51.2
-  device 1:              stored : 45     expected : 51.2
-  device 2:              stored : 46     expected : 51.2
-  device 3:              stored : 62     expected : 51.2
-  device 4:              stored : 45     expected : 51.2
-  device 5:              stored : 40     expected : 51.2
-  device 6:              stored : 47     expected : 51.2
-  device 7:              stored : 53     expected : 51.2
-  device 8:              stored : 36     expected : 51.2
-  device 9:              stored : 46     expected : 51.2
-  device 10:             stored : 43     expected : 51.2
-  device 11:             stored : 68     expected : 51.2
-  device 12:             stored : 53     expected : 51.2
-  device 13:             stored : 46     expected : 51.2
-  device 14:             stored : 52     expected : 51.2
-  device 15:             stored : 60     expected : 51.2
-  device 16:             stored : 50     expected : 51.2
-  device 17:             stored : 59     expected : 51.2
-  device 18:             stored : 52     expected : 51.2
-  device 19:             stored : 63     expected : 51.2
-(reverse-i-search)`p': crushtool --test -i crushmap_new.map
-</code>
-Perform a CRUSH algorithm check displaying the necessary steps to allocate any object:
-<code bash>
-crushtool --test -i crushmap_new.bin --show-choose-tries --rule <ID> --num-rep=<NUM_REP>
-</code>
-Sample (in this sample, with rule "1" all the objects will need 0 steps to find a placement):
-<code>
-# crushtool --test -i crushmap_new.map --show-choose-tries --rule 1 --num-rep=4
-:      2048
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-</code>
-Sample (in this sample, with rule "0" CRUSH will need more than 1 loop trough the algorithm to find a placement):
-<code>
-proceph-osm-001 ~/crushmaps/crush_20190725 # crushtool --test -i crushmap_new.map --show-choose-tries --rule 0 --num-rep=4
-:      3580
-:       251
-:       134
-:        76
-:        28
-:        15
-:         7
-:         2
-:         1
-:         1
-:         0
-:         0
-:         1
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-:         0
-</code>
-===== Apply the new map =====
-<code bash>
-ceph osd setcrushmap -i crushmap.new
-</code>
-===== Review the Map =====
-<code bash>
-ceph osd crush tree
-</code>
-Sample:
-<code>
-proceph-osm-001 ~/crushmaps/crush_20190725 # ceph osd crush tree
-ID CLASS WEIGHT  TYPE NAME
--3       2.00000 root default
--2       1.00000     host ciberterminal2_cluster
-   hdd 1.99899         osd.1
-   hdd 1.99899         osd.3
-   hdd 1.99899         osd.5
-   hdd 1.99899         osd.7
-   hdd 1.99899         osd.9
-   hdd 1.99899         osd.11
-   hdd 1.99899         osd.13
-   hdd 1.99899         osd.15
-   hdd 1.99899         osd.17
-   hdd 1.99899         osd.19
--1       1.00000     host ciberterminal_cluster
-   hdd 1.99899         osd.0
-   hdd 1.99899         osd.2
-   hdd 1.99899         osd.4
-   hdd 1.99899         osd.6
-   hdd 1.99899         osd.8
-   hdd 1.99899         osd.10
-   hdd 1.99899         osd.12
-   hdd 1.99899         osd.14
-   hdd 1.99899         osd.16
-   hdd 1.99899         osd.18
-</code>
-In graphic mode:\\
-<graphviz>
-digraph G {
-        compound=true;
-	default  [shape=rectangle,style=filled,fillcolor=coral];
-	cluster_1 [shape=rectangle,style=filled,fillcolor=coral];
-	osd1 [shape=cylinder,style=filled,fillcolor=coral];
-	osd3 [shape=cylinder,style=filled,fillcolor=coral];
-	osd5 [shape=cylinder,style=filled,fillcolor=coral];
-	osd7 [shape=cylinder,style=filled,fillcolor=coral];
-	osd9 [shape=cylinder,style=filled,fillcolor=coral];
-	cluster_2 [shape=rectangle,style=filled,fillcolor=coral];
-	osd0 [shape=cylinder,style=filled,fillcolor=coral];
-	osd2 [shape=cylinder,style=filled,fillcolor=coral];
-	osd4 [shape=cylinder,style=filled,fillcolor=coral];
-	osd6 [shape=cylinder,style=filled,fillcolor=coral];
-	osd8 [shape=cylinder,style=filled,fillcolor=coral];
-	default->cluster_1;
-	default->cluster_2;
-	cluster_1->osd1;
-	cluster_1->osd3;
-	cluster_1->osd5;
-	cluster_1->osd7;
-	cluster_1->osd9;
-	cluster_2->osd0;
-	cluster_2->osd2;
-	cluster_2->osd4;
-	cluster_2->osd6;
-	cluster_2->osd8;
-}
-</graphviz>
-===== Check pg location =====
-This will show the placement of each PG in ''[OSDx,OSDy...]'':
-<code bash>
-ceph pg dump | egrep "^[0-9]" | awk '{print $17}'|less
-</code>
-====== External documentation ======
-  * [[http://www.sebastien-han.fr/blog/2012/12/07/ceph-2-speed-storage-with-crush/]]
-  * [[https://swamireddy.wordpress.com/2016/01/16/ceph-crush-map-for-host-and-rack/]]
-  * [[http://cephnotes.ksperis.com/blog/2015/02/02/crushmap-example-of-a-hierarchical-cluster-map/]]
-  * [[https://ceph.com/community/new-luminous-crush-device-classes/]]
-  * [[https://alanxelsys.com/ceph-hands-on-guide/]]