| |
| ceph:troubleshooting:troubleshooting_degraded_redundancy [2020/08/17 09:28] – created dodger | ceph:troubleshooting:troubleshooting_degraded_redundancy [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1 |
|---|
| ====== Degraded data redundancy: 2 pgs degraded, 8 pgs undersized ====== | |
| <WRAP center round alert 60%> | |
| **DON'T** use syntax highlight in the following block or will make this page terrible slow!!! | |
| </WRAP> | |
| |
| General troubleshoot: | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph -s | |
| cluster: | |
| id: a3a799ce-f1d3-4230-a915-06e988fee767 | |
| health: HEALTH_WARN | |
| Degraded data redundancy: 2 pgs degraded, 8 pgs undersized | |
| | |
| services: | |
| mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d) | |
| mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminal.net | |
| mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby | |
| osd: 20 osds: 20 up (since 7w), 20 in (since 12M) | |
| rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net) | |
| | |
| data: | |
| pools: 9 pools, 96 pgs | |
| objects: 1.26M objects, 102 GiB | |
| usage: 659 GiB used, 39 TiB / 40 TiB avail | |
| pgs: 88 active+clean | |
| 6 active+undersized | |
| 2 active+undersized+degraded | |
| | |
| io: | |
| client: 11 MiB/s rd, 36 KiB/s wr, 128 op/s rd, 0 op/s wr | |
| | |
| avmlp-osm-001 /home/ceph # ceph health detail | |
| HEALTH_WARN Degraded data redundancy: 2 pgs degraded, 8 pgs undersized | |
| PG_DEGRADED Degraded data redundancy: 2 pgs degraded, 8 pgs undersized | |
| pg 14.0 is stuck undersized for 510298.054479, current state active+undersized, last acting [5,12] | |
| pg 14.1 is stuck undersized for 510298.091712, current state active+undersized, last acting [18,7] | |
| pg 14.2 is stuck undersized for 510298.007891, current state active+undersized+degraded, last acting [7,18] | |
| pg 14.3 is stuck undersized for 510298.086409, current state active+undersized, last acting [8,5] | |
| pg 14.4 is stuck undersized for 510298.054479, current state active+undersized+degraded, last acting [5,18] | |
| pg 14.5 is stuck undersized for 510298.033776, current state active+undersized, last acting [16,1] | |
| pg 14.6 is stuck undersized for 510298.086409, current state active+undersized, last acting [8,3] | |
| pg 14.7 is stuck undersized for 510298.091649, current state active+undersized, last acting [18,3] | |
| | |
| </code> | |
| Why ''pg 14'' is the **only one** that is failing??? | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph tell 14.0 query | jq . | |
| { | |
| "state": "active+undersized", | |
| "snap_trimq": "[]", | |
| "snap_trimq_len": 0, | |
| "epoch": 3629, | |
| "up": [ | |
| 5, | |
| 12 | |
| ], | |
| "acting": [ | |
| 5, | |
| 12 | |
| ], | |
| "acting_recovery_backfill": [ | |
| "5", | |
| "12" | |
| ], | |
| "info": { | |
| "pgid": "14.0", | |
| "last_update": "0'0", | |
| "last_complete": "0'0", | |
| "log_tail": "0'0", | |
| "last_user_version": 0, | |
| "last_backfill": "MAX", | |
| "last_backfill_bitwise": 0, | |
| "purged_snaps": [], | |
| "history": { | |
| "epoch_created": 3627, | |
| "epoch_pool_created": 3627, | |
| "last_epoch_started": 3628, | |
| "last_interval_started": 3627, | |
| "last_epoch_clean": 0, | |
| "last_interval_clean": 0, | |
| "last_epoch_split": 0, | |
| "last_epoch_marked_full": 0, | |
| "same_up_since": 3627, | |
| "same_interval_since": 3627, | |
| "same_primary_since": 3627, | |
| "last_scrub": "0'0", | |
| "last_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_deep_scrub": "0'0", | |
| "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596" | |
| }, | |
| "stats": { | |
| "version": "0'0", | |
| "reported_seq": "9", | |
| "reported_epoch": "3629", | |
| "state": "active+undersized", | |
| "last_fresh": "2020-08-11 11:50:39.473996", | |
| "last_change": "2020-08-11 11:50:38.238074", | |
| "last_active": "2020-08-11 11:50:39.473996", | |
| "last_peered": "2020-08-11 11:50:39.473996", | |
| "last_clean": "2020-08-11 11:50:37.135596", | |
| "last_became_active": "2020-08-11 11:50:38.238074", | |
| "last_became_peered": "2020-08-11 11:50:38.238074", | |
| "last_unstale": "2020-08-11 11:50:39.473996", | |
| "last_undegraded": "2020-08-11 11:50:39.473996", | |
| "last_fullsized": "2020-08-11 11:50:38.233209", | |
| "mapping_epoch": 3627, | |
| "log_start": "0'0", | |
| "ondisk_log_start": "0'0", | |
| "created": 3627, | |
| "last_epoch_clean": 0, | |
| "parent": "0.0", | |
| "parent_split_bits": 0, | |
| "last_scrub": "0'0", | |
| "last_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_deep_scrub": "0'0", | |
| "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "log_size": 0, | |
| "ondisk_log_size": 0, | |
| "stats_invalid": false, | |
| "dirty_stats_invalid": false, | |
| "omap_stats_invalid": false, | |
| "hitset_stats_invalid": false, | |
| "hitset_bytes_stats_invalid": false, | |
| "pin_stats_invalid": false, | |
| "manifest_stats_invalid": false, | |
| "snaptrimq_len": 0, | |
| "stat_sum": { | |
| "num_bytes": 0, | |
| "num_objects": 0, | |
| "num_object_clones": 0, | |
| "num_object_copies": 0, | |
| "num_objects_missing_on_primary": 0, | |
| "num_objects_missing": 0, | |
| "num_objects_degraded": 0, | |
| "num_objects_misplaced": 0, | |
| "num_objects_unfound": 0, | |
| "num_objects_dirty": 0, | |
| "num_whiteouts": 0, | |
| "num_read": 0, | |
| "num_read_kb": 0, | |
| "num_write": 0, | |
| "num_write_kb": 0, | |
| "num_scrub_errors": 0, | |
| "num_shallow_scrub_errors": 0, | |
| "num_deep_scrub_errors": 0, | |
| "num_objects_recovered": 0, | |
| "num_bytes_recovered": 0, | |
| "num_keys_recovered": 0, | |
| "num_objects_omap": 0, | |
| "num_objects_hit_set_archive": 0, | |
| "num_bytes_hit_set_archive": 0, | |
| "num_flush": 0, | |
| "num_flush_kb": 0, | |
| "num_evict": 0, | |
| "num_evict_kb": 0, | |
| "num_promote": 0, | |
| "num_flush_mode_high": 0, | |
| "num_flush_mode_low": 0, | |
| "num_evict_mode_some": 0, | |
| "num_evict_mode_full": 0, | |
| "num_objects_pinned": 0, | |
| "num_legacy_snapsets": 0, | |
| "num_large_omap_objects": 0, | |
| "num_objects_manifest": 0, | |
| "num_omap_bytes": 0, | |
| "num_omap_keys": 0, | |
| "num_objects_repaired": 0 | |
| }, | |
| "up": [ | |
| 5, | |
| 12 | |
| ], | |
| "acting": [ | |
| 5, | |
| 12 | |
| ], | |
| "avail_no_missing": [ | |
| "5", | |
| "12" | |
| ], | |
| "object_location_counts": [], | |
| "blocked_by": [], | |
| "up_primary": 5, | |
| "acting_primary": 5, | |
| "purged_snaps": [] | |
| }, | |
| "empty": 1, | |
| "dne": 0, | |
| "incomplete": 0, | |
| "last_epoch_started": 3628, | |
| "hit_set_history": { | |
| "current_last_update": "0'0", | |
| "history": [] | |
| } | |
| }, | |
| "peer_info": [ | |
| { | |
| "peer": "12", | |
| "pgid": "14.0", | |
| "last_update": "0'0", | |
| "last_complete": "0'0", | |
| "log_tail": "0'0", | |
| "last_user_version": 0, | |
| "last_backfill": "MAX", | |
| "last_backfill_bitwise": 0, | |
| "purged_snaps": [], | |
| "history": { | |
| "epoch_created": 3627, | |
| "epoch_pool_created": 3627, | |
| "last_epoch_started": 3628, | |
| "last_interval_started": 3627, | |
| "last_epoch_clean": 0, | |
| "last_interval_clean": 0, | |
| "last_epoch_split": 0, | |
| "last_epoch_marked_full": 0, | |
| "same_up_since": 0, | |
| "same_interval_since": 0, | |
| "same_primary_since": 0, | |
| "last_scrub": "0'0", | |
| "last_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_deep_scrub": "0'0", | |
| "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596", | |
| "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596" | |
| }, | |
| "stats": { | |
| "version": "0'0", | |
| "reported_seq": "0", | |
| "reported_epoch": "0", | |
| "state": "unknown", | |
| "last_fresh": "0.000000", | |
| "last_change": "0.000000", | |
| "last_active": "0.000000", | |
| "last_peered": "0.000000", | |
| "last_clean": "0.000000", | |
| "last_became_active": "0.000000", | |
| "last_became_peered": "0.000000", | |
| "last_unstale": "0.000000", | |
| "last_undegraded": "0.000000", | |
| "last_fullsized": "0.000000", | |
| "mapping_epoch": 0, | |
| "log_start": "0'0", | |
| "ondisk_log_start": "0'0", | |
| "created": 0, | |
| "last_epoch_clean": 0, | |
| "parent": "0.0", | |
| "parent_split_bits": 0, | |
| "last_scrub": "0'0", | |
| "last_scrub_stamp": "0.000000", | |
| "last_deep_scrub": "0'0", | |
| "last_deep_scrub_stamp": "0.000000", | |
| "last_clean_scrub_stamp": "0.000000", | |
| "log_size": 0, | |
| "ondisk_log_size": 0, | |
| "stats_invalid": false, | |
| "dirty_stats_invalid": false, | |
| "omap_stats_invalid": false, | |
| "hitset_stats_invalid": false, | |
| "hitset_bytes_stats_invalid": false, | |
| "pin_stats_invalid": false, | |
| "manifest_stats_invalid": false, | |
| "snaptrimq_len": 0, | |
| "stat_sum": { | |
| "num_bytes": 0, | |
| "num_objects": 0, | |
| "num_object_clones": 0, | |
| "num_object_copies": 0, | |
| "num_objects_missing_on_primary": 0, | |
| "num_objects_missing": 0, | |
| "num_objects_degraded": 0, | |
| "num_objects_misplaced": 0, | |
| "num_objects_unfound": 0, | |
| "num_objects_dirty": 0, | |
| "num_whiteouts": 0, | |
| "num_read": 0, | |
| "num_read_kb": 0, | |
| "num_write": 0, | |
| "num_write_kb": 0, | |
| "num_scrub_errors": 0, | |
| "num_shallow_scrub_errors": 0, | |
| "num_deep_scrub_errors": 0, | |
| "num_objects_recovered": 0, | |
| "num_bytes_recovered": 0, | |
| "num_keys_recovered": 0, | |
| "num_objects_omap": 0, | |
| "num_objects_hit_set_archive": 0, | |
| "num_bytes_hit_set_archive": 0, | |
| "num_flush": 0, | |
| "num_flush_kb": 0, | |
| "num_evict": 0, | |
| "num_evict_kb": 0, | |
| "num_promote": 0, | |
| "num_flush_mode_high": 0, | |
| "num_flush_mode_low": 0, | |
| "num_evict_mode_some": 0, | |
| "num_evict_mode_full": 0, | |
| "num_objects_pinned": 0, | |
| "num_legacy_snapsets": 0, | |
| "num_large_omap_objects": 0, | |
| "num_objects_manifest": 0, | |
| "num_omap_bytes": 0, | |
| "num_omap_keys": 0, | |
| "num_objects_repaired": 0 | |
| }, | |
| "up": [], | |
| "acting": [], | |
| "avail_no_missing": [], | |
| "object_location_counts": [], | |
| "blocked_by": [], | |
| "up_primary": -1, | |
| "acting_primary": -1, | |
| "purged_snaps": [] | |
| }, | |
| "empty": 1, | |
| "dne": 0, | |
| "incomplete": 0, | |
| "last_epoch_started": 3628, | |
| "hit_set_history": { | |
| "current_last_update": "0'0", | |
| "history": [] | |
| } | |
| } | |
| ], | |
| "recovery_state": [ | |
| { | |
| "name": "Started/Primary/Active", | |
| "enter_time": "2020-08-11 11:50:38.233290", | |
| "might_have_unfound": [], | |
| "recovery_progress": { | |
| "backfill_targets": [], | |
| "waiting_on_backfill": [], | |
| "last_backfill_started": "MIN", | |
| "backfill_info": { | |
| "begin": "MIN", | |
| "end": "MIN", | |
| "objects": [] | |
| }, | |
| "peer_backfill_info": [], | |
| "backfills_in_flight": [], | |
| "recovering": [], | |
| "pg_backend": { | |
| "pull_from_peer": [], | |
| "pushing": [] | |
| } | |
| }, | |
| "scrub": { | |
| "scrubber.epoch_start": "0", | |
| "scrubber.active": false, | |
| "scrubber.state": "INACTIVE", | |
| "scrubber.start": "MIN", | |
| "scrubber.end": "MIN", | |
| "scrubber.max_end": "MIN", | |
| "scrubber.subset_last_update": "0'0", | |
| "scrubber.deep": false, | |
| "scrubber.waiting_on_whom": [] | |
| } | |
| }, | |
| { | |
| "name": "Started", | |
| "enter_time": "2020-08-11 11:50:37.502984" | |
| } | |
| ], | |
| "agent_state": {} | |
| } | |
| |
| </code> | |
| ''pg 14'' does not have data nor activity!!!! | |
| <code> | |
| |
| avmlp-osm-001 /home/ceph # ceph pg dump | |
| version 2063560 | |
| stamp 2020-08-17 09:46:06.557196 | |
| last_osdmap_epoch 0 | |
| last_pg_scan 0 | |
| PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN | |
| 12.1f 37234 0 0 0 0 2448717330 0 0 3074 3074 active+clean 2020-08-16 23:01:57.976378 3629'342755 3629:445975 [4,10,19,17] 4 [4,10,19,17] 4 3629'342725 2020-08-16 23:01:57.976326 3629'342409 2020-08-14 13:40:26.247849 0 | |
| 12.1e 36870 0 0 0 0 2452671420 0 0 3074 3074 active+clean 2020-08-16 05:41:59.725318 3629'342325 3629:605224 [2,14,5,1] 2 [2,14,5,1] 2 3629'342214 2020-08-16 05:41:59.725253 3629'342075 2020-08-15 01:28:05.207465 0 | |
| 12.1d 36549 0 0 0 0 2255784538 0 0 3065 3065 active+clean 2020-08-17 03:55:29.331054 3629'342855 3629:507767 [6,0,7,5] 6 [6,0,7,5] 6 3629'342842 2020-08-17 03:55:29.330992 3629'342585 2020-08-14 21:29:33.393053 0 | |
| 12.1c 36839 0 0 0 0 2492477640 0 0 3013 3013 active+clean 2020-08-17 07:42:13.522039 3629'343720 3629:445046 [3,1,6,0] 3 [3,1,6,0] 3 3629'343709 2020-08-17 07:42:13.521977 3629'343568 2020-08-15 22:22:48.911933 0 | |
| 12.1b 36940 0 0 0 0 2470626204 0 0 3048 3048 active+clean 2020-08-16 13:35:08.123034 3629'344727 3629:501600 [1,13,12,6] 1 [1,13,12,6] 1 3629'344655 2020-08-16 13:35:08.122944 3629'344655 2020-08-16 13:35:08.122944 0 | |
| 12.1a 37215 0 0 0 0 2632273586 0 0 3070 3070 active+clean 2020-08-17 07:44:07.798470 3629'351695 3629:494814 [5,15,14,6] 5 [5,15,14,6] 5 3629'351679 2020-08-17 07:44:07.798401 3629'351587 2020-08-16 04:51:26.455475 0 | |
| 12.19 37311 0 0 0 0 2597837981 0 0 3085 3085 active+clean 2020-08-16 07:01:40.593214 3629'346755 3629:474433 [12,16,17,1] 12 [12,16,17,1] 12 3629'346674 2020-08-16 07:01:40.593159 3629'346466 2020-08-15 04:48:07.588319 0 | |
| 12.18 36731 0 0 0 0 2458805578 0 0 3092 3092 active+clean 2020-08-16 03:13:11.863075 3629'342802 3629:468666 [14,2,13,3] 14 [14,2,13,3] 14 3629'342683 2020-08-16 03:13:11.863020 3629'342449 2020-08-14 19:08:54.971752 0 | |
| 12.17 37234 0 0 0 0 2411367145 0 0 3084 3084 active+clean 2020-08-17 08:47:51.571373 3629'344472 3629:496481 [5,9,8,14] 5 [5,9,8,14] 5 3629'344468 2020-08-17 08:47:51.571280 3629'343706 2020-08-13 01:47:55.688179 0 | |
| 12.16 37146 0 0 0 0 2517689370 0 0 3090 3090 active+clean 2020-08-16 13:44:35.341646 3629'343210 3629:472908 [7,9,8,4] 7 [7,9,8,4] 7 3629'343143 2020-08-16 13:44:35.341577 3629'342810 2020-08-14 00:43:42.538689 0 | |
| 12.15 36841 0 0 0 0 2550933263 0 0 3040 3040 active+clean 2020-08-17 03:52:55.184968 3629'345294 3629:502420 [1,15,18,4] 1 [1,15,18,4] 1 3629'345270 2020-08-17 03:52:55.184909 3629'345169 2020-08-16 02:39:11.303006 0 | |
| 12.14 36936 0 0 0 0 2436184387 0 0 3076 3076 active+clean 2020-08-16 15:16:55.057652 3629'334510 3629:491995 [8,12,17,15] 8 [8,12,17,15] 8 3629'334418 2020-08-16 15:16:55.057592 3629'334259 2020-08-15 08:21:42.290408 0 | |
| 12.13 36989 0 0 0 0 2452289072 0 0 3095 3095 active+clean 2020-08-17 06:11:39.821708 3629'330241 3629:459043 [4,8,7,5] 4 [4,8,7,5] 4 3629'330217 2020-08-17 06:11:39.821627 3629'330217 2020-08-17 06:11:39.821627 0 | |
| 12.12 36937 0 0 0 0 2454977301 0 0 3057 3057 active+clean 2020-08-16 16:35:25.451333 3629'344686 3629:473960 [17,13,0,10] 17 [17,13,0,10] 17 3629'344608 2020-08-16 16:35:25.451262 3629'344608 2020-08-16 16:35:25.451262 0 | |
| 12.11 37165 0 0 0 0 2639030037 0 0 3010 3010 active+clean 2020-08-16 12:21:25.240115 3629'344750 3629:504301 [2,18,1,15] 2 [2,18,1,15] 2 3629'344663 2020-08-16 12:21:25.239991 3629'343787 2020-08-12 10:45:52.034024 0 | |
| 12.10 37063 0 0 0 0 2584770038 0 0 3046 3046 active+clean 2020-08-17 08:51:07.093579 3629'340698 3629:498656 [5,19,12,10] 5 [5,19,12,10] 5 3629'340689 2020-08-17 08:51:07.093497 3629'339693 2020-08-12 03:45:50.319811 0 | |
| 12.a 37207 0 0 0 0 2594481924 0 0 3094 3094 active+clean 2020-08-16 16:26:16.525831 3629'345493 3629:347039 [12,0,17,19] 12 [12,0,17,19] 12 3629'345385 2020-08-16 16:26:16.525778 3629'345385 2020-08-16 16:26:16.525778 0 | |
| 7.1 11 0 0 0 0 1560 1416 8 3076 3076 active+clean 2020-08-16 13:47:36.599722 3620'6878 3629:438065 [6,14,1,3] 6 [6,14,1,3] 6 3620'6878 2020-08-16 13:47:36.599648 3620'6878 2020-08-16 13:47:36.599648 0 | |
| 5.3 1 0 0 0 0 805 0 0 1 1 active+clean 2020-08-16 13:52:05.527723 3605'3 3629:4525 [3,13,0,18] 3 [3,13,0,18] 3 3605'3 2020-08-16 13:52:05.527692 3605'3 2020-08-13 05:28:18.436690 0 | |
| 6.0 1 0 0 0 0 0 0 0 346 346 active+clean 2020-08-16 12:20:48.361592 3629'535 3629:26442589 [4,14,9,17] 4 [4,14,9,17] 4 3629'535 2020-08-16 12:20:48.361537 3629'535 2020-08-16 12:20:48.361537 0 | |
| 12.b 36624 0 0 0 0 2542103794 0 0 3055 3055 active+clean 2020-08-17 06:38:51.532937 3629'337923 3629:353370 [3,5,0,4] 3 [3,5,0,4] 3 3629'337893 2020-08-17 06:38:51.532865 3629'337089 2020-08-12 13:17:48.634712 0 | |
| 7.0 11 0 0 0 0 1658 2912 16 975 975 active+clean 2020-08-16 09:04:51.825199 3629'976 3629:272919 [16,14,7,17] 16 [16,14,7,17] 16 3629'972 2020-08-16 09:04:51.825161 3629'972 2020-08-16 09:04:51.825161 0 | |
| 5.2 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-16 09:24:09.871395 0'0 3629:4488 [1,15,18,10] 1 [1,15,18,10] 1 0'0 2020-08-16 09:24:09.871348 0'0 2020-08-16 09:24:09.871348 0 | |
| 6.1 1 0 0 0 0 0 0 0 341 341 active+clean 2020-08-17 09:32:47.922139 3629'613 3629:26496283 [6,12,13,3] 6 [6,12,13,3] 6 3629'613 2020-08-17 09:32:47.922094 3629'613 2020-08-16 01:49:21.114037 0 | |
| 12.8 36745 0 0 0 0 2453736305 0 0 3046 3046 active+clean 2020-08-16 08:20:55.871458 3629'341424 3629:352763 [19,5,14,12] 19 [19,5,14,12] 19 3629'341323 2020-08-16 08:20:55.871363 3626'340243 2020-08-11 00:28:58.843200 0 | |
| 7.3 6 0 0 0 0 1730 0 0 34 34 active+clean 2020-08-16 18:17:48.970705 3626'35 3629:226711 [17,15,6,12] 17 [17,15,6,12] 17 3626'35 2020-08-16 18:17:48.970621 3626'35 2020-08-14 08:40:31.413876 0 | |
| 6.2 0 0 0 0 0 0 0 0 4 4 active+clean 2020-08-16 14:09:46.658487 2078'251 3629:58666 [10,6,13,11] 10 [10,6,13,11] 10 2078'251 2020-08-16 14:09:46.658432 2078'251 2020-08-11 16:12:50.795649 0 | |
| 5.1 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-16 19:33:01.145513 0'0 3629:4511 [19,7,16,0] 19 [19,7,16,0] 19 0'0 2020-08-16 19:33:01.145470 0'0 2020-08-16 19:33:01.145470 0 | |
| 12.9 36762 0 0 0 0 2574357560 0 0 3088 3088 active+clean 2020-08-17 08:13:42.940483 3629'342464 3629:357530 [15,3,6,16] 15 [15,3,6,16] 15 3629'342450 2020-08-17 08:13:42.940428 3629'342179 2020-08-15 03:51:12.708397 0 | |
| 7.2 7 0 0 0 0 1857 1532 8 1858 1858 active+clean 2020-08-16 21:11:17.394904 3629'1862 3629:357192 [3,7,0,2] 3 [3,7,0,2] 3 3629'1847 2020-08-16 21:11:17.394835 3629'1481 2020-08-11 22:28:46.467550 0 | |
| 6.3 1 0 0 0 0 0 0 0 346 346 active+clean 2020-08-16 23:47:33.943659 3629'571 3629:26442615 [19,13,18,14] 19 [19,13,18,14] 19 3629'571 2020-08-16 23:47:33.943612 3629'571 2020-08-13 13:20:52.435651 0 | |
| 5.0 1 0 0 0 0 348 0 0 1 1 active+clean 2020-08-16 15:09:36.160089 3605'2 3629:4694 [7,9,4,0] 7 [7,9,4,0] 7 3605'2 2020-08-16 15:09:36.159945 3605'2 2020-08-16 15:09:36.159945 0 | |
| 12.d 36881 0 0 0 0 2473024184 0 0 3060 3060 active+clean 2020-08-16 22:07:39.858423 3629'343047 3629:358430 [9,7,10,6] 9 [9,7,10,6] 9 3629'343010 2020-08-16 22:07:39.858356 3626'341844 2020-08-10 15:03:18.641927 0 | |
| 6.7 1 0 0 0 0 0 0 0 354 354 active+clean 2020-08-16 07:35:25.498839 3629'579 3629:26322313 [14,4,9,5] 14 [14,4,9,5] 14 3629'579 2020-08-16 07:35:25.498808 3629'579 2020-08-16 07:35:25.498808 0 | |
| 5.4 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-17 06:52:48.025888 169'1 3629:2390 [12,0,5,9] 12 [12,0,5,9] 12 169'1 2020-08-17 06:52:48.025854 169'1 2020-08-13 17:30:10.364425 0 | |
| 7.6 15 0 0 0 0 3862 0 0 48 48 active+clean 2020-08-16 06:56:31.300185 3629'52 3629:913522 [13,1,4,16] 13 [13,1,4,16] 13 3629'52 2020-08-16 06:56:31.300134 3629'52 2020-08-13 21:44:05.147193 0 | |
| 12.c 37146 0 0 0 0 2542888837 0 0 3042 3042 active+clean 2020-08-17 01:47:38.880451 3629'341293 3629:356990 [17,9,0,6] 17 [17,9,0,6] 17 3629'341259 2020-08-17 01:47:38.880347 3629'340318 2020-08-11 18:13:24.694080 0 | |
| 6.6 1 0 0 0 0 0 0 0 334 334 active+clean 2020-08-16 09:44:42.924292 3629'581 3629:26238773 [12,16,13,1] 12 [12,16,13,1] 12 3629'581 2020-08-16 09:44:42.924234 3629'581 2020-08-12 14:42:47.482884 0 | |
| 5.5 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-16 23:24:09.575998 0'0 3629:1051 [15,5,0,4] 15 [15,5,0,4] 15 0'0 2020-08-16 23:24:09.575954 0'0 2020-08-16 23:24:09.575954 0 | |
| 7.7 6 0 0 0 0 427 704 4 3075 3075 active+clean 2020-08-16 21:21:50.448714 3629'18477 3629:135290250 [11,17,14,4] 11 [11,17,14,4] 11 3629'18424 2020-08-16 21:21:50.448676 3626'17489 2020-08-10 20:15:44.650232 0 | |
| 12.f 37170 0 0 0 0 2421662736 0 0 3088 3088 active+clean 2020-08-16 22:46:18.321473 3629'339853 3629:355474 [12,6,7,1] 12 [12,6,7,1] 12 3629'339826 2020-08-16 22:46:18.321426 3629'339234 2020-08-13 10:34:11.712596 0 | |
| 7.4 6 0 0 0 0 1066 0 0 27 27 active+clean 2020-08-16 02:18:10.503010 3626'28 3629:320651 [3,7,16,18] 3 [3,7,16,18] 3 3626'28 2020-08-16 02:18:10.502933 3626'28 2020-08-16 02:18:10.502933 0 | |
| 6.5 2 0 0 0 0 0 0 0 668 668 active+clean 2020-08-16 16:40:36.754757 3629'940 3629:52661388 [19,1,4,18] 19 [19,1,4,18] 19 3629'940 2020-08-16 16:40:36.754710 3626'904 2020-08-10 04:26:29.538486 0 | |
| 5.6 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-16 02:46:27.887880 0'0 3629:4516 [18,14,13,5] 18 [18,14,13,5] 18 0'0 2020-08-16 02:46:27.887837 0'0 2020-08-10 06:09:08.174831 0 | |
| 12.e 36977 0 0 0 0 2513728716 0 0 3089 3089 active+clean 2020-08-16 10:19:08.917759 3629'342525 3629:471605 [16,18,7,11] 16 [16,18,7,11] 16 3629'342435 2020-08-16 10:19:08.917672 3626'341192 2020-08-10 05:51:27.175188 0 | |
| 7.5 14 0 0 0 0 2214 4484 24 3004 3004 active+clean 2020-08-16 15:54:32.903243 3629'7806 3629:239698 [15,19,8,12] 15 [15,19,8,12] 15 3629'7789 2020-08-16 15:54:32.903170 3629'7641 2020-08-14 02:43:54.992932 0 | |
| 6.4 1 0 0 0 0 0 0 0 354 354 active+clean 2020-08-16 12:49:44.792839 3629'543 3629:26332861 [17,9,14,8] 17 [17,9,14,8] 17 3629'543 2020-08-16 12:49:44.792781 3626'525 2020-08-11 09:11:27.786481 0 | |
| 5.7 3 0 0 0 0 897 0 0 3 3 active+clean 2020-08-17 00:25:41.809330 3605'5 3629:3491 [5,15,14,18] 5 [5,15,14,18] 5 3605'5 2020-08-17 00:25:41.809266 3605'5 2020-08-13 05:37:52.184231 0 | |
| 14.6 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.208103 0'0 3628:8 [8,3] 8 [8,3] 8 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.5 580 0 0 0 0 20971680 292948688 634088 3091 3091 active+clean 2020-08-17 02:30:59.655119 3629'587509 3629:615647 [11,7,12,10] 11 [11,7,12,10] 11 3629'587291 2020-08-17 02:30:59.655047 3629'583707 2020-08-14 15:10:59.726903 0 | |
| 12.4 37167 0 0 0 0 2532685297 0 0 3062 3062 active+clean 2020-08-17 07:58:58.470571 3629'344228 3629:360198 [9,19,18,14] 9 [9,19,18,14] 9 3629'344222 2020-08-17 07:58:58.470487 3626'343061 2020-08-11 07:51:26.283423 0 | |
| 8.0 26 0 0 0 0 0 0 0 3098 3098 active+clean 2020-08-16 13:17:58.821382 3629'7807401 3629:11594756 [15,1,18,10] 15 [15,1,18,10] 15 3629'7789704 2020-08-16 13:17:58.821333 3629'7766649 2020-08-15 10:37:00.441663 0 | |
| 10.2 8709 0 0 0 0 3677669925 0 0 3066 3066 active+clean 2020-08-16 12:00:52.962103 3629'5527685 3629:33958241 [8,6,9,17] 8 [8,6,9,17] 8 3629'5527659 2020-08-16 12:00:52.962034 3629'5527659 2020-08-16 12:00:52.962034 0 | |
| 11.3 0 0 0 0 0 0 0 0 3011 3011 active+clean 2020-08-16 18:46:04.571928 3605'7544511 3629:7551165 [4,2,5,19] 4 [4,2,5,19] 4 3605'7544511 2020-08-16 18:46:04.571891 3605'7544511 2020-08-16 18:46:04.571891 0 | |
| 14.7 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.205413 0'0 3628:8 [18,3] 18 [18,3] 18 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.4 553 0 0 0 0 20972154 240751960 521108 3084 3084 active+clean 2020-08-16 08:22:47.112467 3629'852149 3629:888128 [9,11,18,16] 9 [9,11,18,16] 9 3629'850960 2020-08-16 08:22:47.112414 3629'850960 2020-08-16 08:22:47.112414 0 | |
| 12.5 37005 0 0 0 0 2461994487 0 0 3066 3066 active+clean 2020-08-16 05:29:22.304563 3629'343950 3629:359515 [17,11,8,2] 17 [17,11,8,2] 17 3629'343834 2020-08-16 05:29:22.304452 3629'343085 2020-08-12 21:39:51.184699 0 | |
| 8.1 32 0 0 0 0 0 0 0 3080 3080 active+clean 2020-08-16 18:52:17.365345 3629'9099181 3629:13533326 [18,10,7,13] 18 [18,10,7,13] 18 3629'9084191 2020-08-16 18:52:17.365270 3626'8917520 2020-08-09 21:24:16.115253 0 | |
| 10.3 8686 0 0 0 0 3736113818 0 0 3051 3051 active+clean 2020-08-16 14:27:13.422714 3629'4468071 3629:27962874 [12,6,13,1] 12 [12,6,13,1] 12 3629'4468045 2020-08-16 14:27:13.422638 3629'4467930 2020-08-14 10:56:15.273485 0 | |
| 11.2 4 0 0 0 0 0 977588 3472 3068 3068 active+clean 2020-08-16 07:39:36.931627 3629'3773773 3629:3781491 [9,11,10,12] 9 [9,11,10,12] 9 3629'3773611 2020-08-16 07:39:36.931571 3629'3773091 2020-08-13 19:56:13.839973 0 | |
| 14.4 0 0 0 0 0 0 0 0 7 7 active+undersized+degraded 2020-08-11 11:52:11.578155 3629'7 3629:21 [5,18] 5 [5,18] 5 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.7 559 0 0 0 0 25166390 290987928 629844 3070 3070 active+clean 2020-08-16 00:16:42.372445 3629'373238 3629:428637 [16,2,17,1] 16 [16,2,17,1] 16 3629'372495 2020-08-16 00:16:42.372384 3629'370804 2020-08-13 15:45:28.525122 0 | |
| 12.6 37092 0 0 0 0 2326680636 0 0 3021 3021 active+clean 2020-08-16 17:24:46.808260 3629'342824 3629:358166 [0,14,7,15] 0 [0,14,7,15] 0 3629'342734 2020-08-16 17:24:46.808204 3629'342111 2020-08-13 01:50:02.075255 0 | |
| 8.2 26 0 0 0 0 0 0 0 3100 3100 active+clean 2020-08-16 06:06:19.967891 3629'8514055 3629:12656747 [9,11,10,0] 9 [9,11,10,0] 9 3629'8487923 2020-08-16 06:06:19.967832 3629'8407817 2020-08-12 17:11:15.659708 0 | |
| 10.0 8727 0 0 0 0 3670473841 0 0 3059 3059 active+clean 2020-08-16 23:18:06.938023 3629'4319994 3629:27876556 [17,9,4,14] 17 [17,9,4,14] 17 3629'4319978 2020-08-16 23:18:06.937879 3626'4319557 2020-08-10 19:02:28.103491 0 | |
| 11.1 5 0 0 0 0 0 842872 2620 3090 3090 active+clean 2020-08-16 09:24:39.030517 3629'7553373 3629:7563678 [13,15,16,4] 13 [13,15,16,4] 13 3629'7553355 2020-08-16 09:24:39.030471 3629'7553347 2020-08-15 05:17:56.146025 0 | |
| 14.5 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.299784 0'0 3628:8 [16,1] 16 [16,1] 16 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.6 603 0 0 0 0 29360128 264796356 579740 3056 3056 active+clean 2020-08-16 22:00:12.848404 3629'1228207 3629:1267959 [10,6,15,9] 10 [10,6,15,9] 10 3629'1227293 2020-08-16 22:00:12.848337 3629'1220689 2020-08-14 07:04:49.213009 0 | |
| 12.7 36975 0 0 0 0 2496638778 0 0 3095 3095 active+clean 2020-08-16 11:28:41.495579 3629'340113 3629:352605 [17,19,12,18] 17 [17,19,12,18] 17 3629'339994 2020-08-16 11:28:41.495468 3629'339838 2020-08-15 10:54:22.325510 0 | |
| 8.3 22 0 0 0 0 0 0 0 3026 3026 active+clean 2020-08-16 08:10:35.990042 3629'6260906 3629:9249769 [5,7,8,0] 5 [5,7,8,0] 5 3629'6243224 2020-08-16 08:10:35.989978 3629'6224128 2020-08-15 04:47:58.354067 0 | |
| 10.1 8733 0 0 0 0 3662648021 0 0 3077 3077 active+clean 2020-08-16 07:56:03.255010 3629'5147909 3629:31740280 [8,12,17,3] 8 [8,12,17,3] 8 3629'5147877 2020-08-16 07:56:03.254946 3629'5147596 2020-08-12 14:38:09.703348 0 | |
| 11.0 1 0 0 0 0 0 175636 484 264 264 active+clean 2020-08-16 10:55:15.816476 3626'264 3629:5205 [5,13,0,10] 5 [5,13,0,10] 5 3626'264 2020-08-16 10:55:15.816425 3626'264 2020-08-11 19:47:25.967648 0 | |
| 14.2 0 0 0 0 0 0 0 0 7 7 active+undersized+degraded 2020-08-11 11:50:38.288292 3629'7 3629:21 [7,18] 7 [7,18] 7 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.1 578 0 0 0 0 41943040 332682568 720092 3055 3055 active+clean 2020-08-16 09:55:04.810989 3629'366770 3629:422291 [2,8,17,9] 2 [2,8,17,9] 2 3629'366748 2020-08-16 09:55:04.810883 3629'360522 2020-08-12 11:49:08.440021 0 | |
| 12.0 36724 0 0 0 0 2658760797 0 0 3019 3019 active+clean 2020-08-16 16:46:26.833362 3629'338198 3629:353649 [1,3,8,0] 1 [1,3,8,0] 1 3629'338111 2020-08-16 16:46:26.833264 3626'336976 2020-08-10 18:00:59.920643 0 | |
| 11.7 0 0 0 0 0 0 0 0 3041 3041 active+clean 2020-08-16 13:08:43.122360 3611'3775141 3629:3777308 [12,6,19,9] 12 [12,6,19,9] 12 3611'3775141 2020-08-16 13:08:43.122319 3611'3775141 2020-08-16 13:08:43.122319 0 | |
| 8.4 29 0 0 0 0 0 0 0 3026 3026 active+clean 2020-08-17 09:38:30.847123 3629'9980713 3629:14831818 [8,4,5,7] 8 [8,4,5,7] 8 3629'9980537 2020-08-17 09:38:30.847075 3629'9913610 2020-08-14 20:58:23.413276 0 | |
| 10.6 8722 0 0 0 0 3650908162 0 0 3041 3041 active+clean 2020-08-16 03:36:55.650758 3629'5111161 3629:33506602 [13,19,16,12] 13 [13,19,16,12] 13 3629'5111126 2020-08-16 03:36:55.650689 3629'5110776 2020-08-11 22:33:13.472993 0 | |
| 14.3 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.240473 0'0 3628:8 [8,5] 8 [8,5] 8 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.0 581 0 0 0 0 50331648 297297128 643500 3089 3089 active+clean 2020-08-16 22:22:54.073570 3629'332670 3629:358920 [17,19,2,12] 17 [17,19,2,12] 17 3629'332663 2020-08-16 22:22:54.073482 3629'331971 2020-08-14 05:44:32.910314 0 | |
| 12.1 36559 0 0 0 0 2693058955 0 0 3042 3042 active+clean 2020-08-16 19:50:56.588358 3629'340741 3629:356295 [6,0,17,13] 6 [6,0,17,13] 6 3629'340666 2020-08-16 19:50:56.588301 3629'340168 2020-08-13 07:29:49.033271 0 | |
| 11.6 2 0 0 0 0 0 16804608 59704 3067 3067 active+clean 2020-08-16 15:16:17.271048 3629'7579572 3629:7600332 [18,4,7,15] 18 [18,4,7,15] 18 3629'7579572 2020-08-16 15:16:17.271006 3629'7579526 2020-08-15 13:57:43.665912 0 | |
| 8.5 25 0 0 0 0 0 0 0 3088 3088 active+clean 2020-08-16 19:36:31.891091 3629'8629693 3629:12815670 [2,8,19,15] 2 [2,8,19,15] 2 3629'8616157 2020-08-16 19:36:31.891044 3629'8501363 2020-08-11 19:35:10.376406 0 | |
| 10.7 8623 0 0 0 0 3593613124 0 0 3053 3053 active+clean 2020-08-16 06:02:32.648106 3629'4433774 3629:25394677 [12,6,13,15] 12 [12,6,13,15] 12 3629'4433731 2020-08-16 06:02:32.648055 3629'4433683 2020-08-15 00:48:32.278486 0 | |
| 14.0 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.238074 0'0 3628:8 [5,12] 5 [5,12] 5 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.3 606 0 0 0 0 34047265 207078008 448224 3050 3050 active+clean 2020-08-17 04:22:26.814711 3629'397666 3629:456073 [17,3,0,14] 17 [17,3,0,14] 17 3629'397480 2020-08-17 04:22:26.814605 3629'396120 2020-08-15 23:05:06.296695 0 | |
| 12.2 36744 0 0 0 0 2550191501 0 0 3072 3072 active+clean 2020-08-16 03:08:42.253250 3629'347207 3629:362427 [11,5,6,0] 11 [11,5,6,0] 11 3629'347074 2020-08-16 03:08:42.253178 3629'347074 2020-08-16 03:08:42.253178 0 | |
| 8.6 24 0 0 0 0 0 0 0 3062 3062 active+clean 2020-08-16 12:37:44.415819 3629'7978201 3629:11874913 [5,15,2,16] 5 [5,15,2,16] 5 3629'7959487 2020-08-16 12:37:44.415756 3629'7881700 2020-08-12 20:39:57.383255 0 | |
| 10.4 8773 0 0 0 0 3672511308 0 0 3054 3054 active+clean 2020-08-17 06:16:12.995209 3629'4131987 3629:27722587 [7,9,18,6] 7 [7,9,18,6] 7 3629'4131972 2020-08-17 06:16:12.995135 3629'4131953 2020-08-15 20:25:05.273445 0 | |
| 11.5 1 0 0 0 0 0 7596 32 3024 3024 active+clean 2020-08-15 23:27:34.148382 3620'3774307 3629:3779273 [11,1,16,14] 11 [11,1,16,14] 11 3620'3774307 2020-08-15 23:27:34.148298 3620'3774307 2020-08-15 23:27:34.148298 0 | |
| 14.1 0 0 0 0 0 0 0 0 0 0 active+undersized 2020-08-11 11:50:38.283647 0'0 3628:8 [18,7] 18 [18,7] 18 0'0 2020-08-11 11:50:37.135596 0'0 2020-08-11 11:50:37.135596 0 | |
| 13.2 592 0 0 0 0 36805387 242490960 524872 3051 3051 active+clean 2020-08-16 12:34:53.339785 3629'360774 3629:410030 [11,7,12,6] 11 [11,7,12,6] 11 3629'359198 2020-08-16 12:34:53.339730 3629'358452 2020-08-15 03:14:11.641874 0 | |
| 12.3 36975 0 0 0 0 2452554919 0 0 3077 3077 active+clean 2020-08-16 00:06:59.430694 3629'340108 3629:355298 [5,19,2,16] 5 [5,19,2,16] 5 3629'339995 2020-08-16 00:06:59.430645 3629'339599 2020-08-13 16:13:15.935250 0 | |
| 8.7 23 0 0 0 0 0 0 0 3052 3052 active+clean 2020-08-16 13:49:17.117923 3629'7044994 3629:10436162 [0,16,9,19] 0 [0,16,9,19] 0 3629'7029459 2020-08-16 13:49:17.117870 3626'6924487 2020-08-10 23:04:58.110588 0 | |
| 10.5 8547 0 0 0 0 3634533507 0 0 3045 3045 active+clean 2020-08-16 16:27:34.848239 3629'4734275 3629:27342785 [18,14,7,15] 18 [18,14,7,15] 18 3629'4734243 2020-08-16 16:27:34.848142 3629'4734099 2020-08-14 01:02:12.320648 0 | |
| 11.4 2 0 0 0 0 0 56754276 190332 3095 3095 active+clean 2020-08-17 09:39:41.814938 3629'3871595 3629:3921886 [7,9,10,4] 7 [7,9,10,4] 7 3629'3871573 2020-08-17 09:39:41.814879 3629'3868263 2020-08-12 14:00:41.440724 0 | |
| | |
| 14 0 0 0 0 0 0 0 0 14 14 | |
| 13 4652 0 0 0 0 259597692 2169033596 4701468 24546 24546 | |
| 12 1182753 0 0 0 0 80144984316 0 0 98035 98035 | |
| 11 15 0 0 0 0 0 75562576 256644 21660 21660 | |
| 10 69520 0 0 0 0 29298471706 0 0 24446 24446 | |
| 5 5 0 0 0 0 2050 0 0 5 5 | |
| 6 8 0 0 0 0 0 0 0 2747 2747 | |
| 7 76 0 0 0 0 14374 11048 60 12097 12097 | |
| 8 207 0 0 0 0 0 0 0 24532 24532 | |
| | |
| sum 1257236 0 0 0 0 109703070138 2244607220 4958172 208082 208082 | |
| OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM | |
| 19 32 GiB 2.0 TiB 33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18] 17 4 | |
| 18 28 GiB 2.0 TiB 29 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19] 22 6 | |
| 17 48 GiB 2.0 TiB 49 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,19] 21 9 | |
| 16 20 GiB 2.0 TiB 21 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19] 17 4 | |
| 15 32 GiB 2.0 TiB 33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19] 19 4 | |
| 14 36 GiB 2.0 TiB 37 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,17,18,19] 20 2 | |
| 13 27 GiB 2.0 TiB 28 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,12,14,15,16,17,18,19] 17 3 | |
| 12 47 GiB 2.0 TiB 48 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,17,18,19] 23 8 | |
| 11 12 GiB 2.0 TiB 13 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,17,18,19] 11 5 | |
| 10 17 GiB 2.0 TiB 18 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,19] 14 2 | |
| 3 24 GiB 2.0 TiB 25 GiB 2.0 TiB [0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 14 5 | |
| 2 20 GiB 2.0 TiB 21 GiB 2.0 TiB [0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 12 4 | |
| 1 36 GiB 2.0 TiB 37 GiB 2.0 TiB [0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 18 4 | |
| 0 41 GiB 2.0 TiB 42 GiB 2.0 TiB [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 21 2 | |
| 4 24 GiB 2.0 TiB 25 GiB 2.0 TiB [0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 18 4 | |
| 5 40 GiB 2.0 TiB 42 GiB 2.0 TiB [0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,17,18,19] 23 10 | |
| 6 55 GiB 1.9 TiB 56 GiB 2.0 TiB [0,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19] 21 4 | |
| 7 35 GiB 2.0 TiB 36 GiB 2.0 TiB [0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,19] 23 5 | |
| 8 32 GiB 2.0 TiB 33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19] 16 6 | |
| 9 31 GiB 2.0 TiB 33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18,19] 21 5 | |
| sum 636 GiB 39 TiB 659 GiB 40 TiB | |
| |
| * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. | |
| |
| </code> | |
| ''pg 14'' is the only one that **don't have** a high availability (3 replicas or more)... why? | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph pg dump_stuck inactive | |
| ok | |
| avmlp-osm-001 /home/ceph # ceph pg dump_stuck stale | |
| ok | |
| avmlp-osm-001 /home/ceph # ceph pg dump_stuck undersized | |
| ok | |
| PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY | |
| 14.1 active+undersized [18,7] 18 [18,7] 18 | |
| 14.0 active+undersized [5,12] 5 [5,12] 5 | |
| 14.3 active+undersized [8,5] 8 [8,5] 8 | |
| 14.2 active+undersized+degraded [7,18] 7 [7,18] 7 | |
| 14.6 active+undersized [8,3] 8 [8,3] 8 | |
| 14.7 active+undersized [18,3] 18 [18,3] 18 | |
| 14.4 active+undersized+degraded [5,18] 5 [5,18] 5 | |
| 14.5 active+undersized [16,1] 16 [16,1] 16 | |
| avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.0 | |
| pg 14.0 doesn't require recovery; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.0 | |
| pg 14.0 doesn't require backfilling; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.4 | |
| instructing pg(s) [14.4] on osd.5 to force-recovery; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.4 | |
| instructing pg(s) [14.4] on osd.5 to force-backfill; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.2 | |
| instructing pg(s) [14.2] on osd.7 to force-recovery; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.2 | |
| instructing pg(s) [14.2] on osd.7 to force-backfill; | |
| |
| avmlp-osm-001 /home/ceph # ceph pg ls | |
| PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP | |
| 5.0 1 0 0 0 348 0 0 1 active+clean 18h 3605'2 3629:4694 [7,9,4,0]p7 [7,9,4,0]p7 2020-08-16 15:09:36.159945 2020-08-16 15:09:36.159945 | |
| 5.1 0 0 0 0 0 0 0 0 active+clean 14h 0'0 3629:4511 [19,7,16,0]p19 [19,7,16,0]p19 2020-08-16 19:33:01.145470 2020-08-16 19:33:01.145470 | |
| 5.2 0 0 0 0 0 0 0 0 active+clean 24h 0'0 3629:4488 [1,15,18,10]p1 [1,15,18,10]p1 2020-08-16 09:24:09.871348 2020-08-16 09:24:09.871348 | |
| 5.3 1 0 0 0 805 0 0 1 active+clean 20h 3605'3 3629:4525 [3,13,0,18]p3 [3,13,0,18]p3 2020-08-16 13:52:05.527692 2020-08-13 05:28:18.436690 | |
| 5.4 0 0 0 0 0 0 0 0 active+clean 3h 169'1 3629:2390 [12,0,5,9]p12 [12,0,5,9]p12 2020-08-17 06:52:48.025854 2020-08-13 17:30:10.364425 | |
| 5.5 0 0 0 0 0 0 0 0 active+clean 10h 0'0 3629:1051 [15,5,0,4]p15 [15,5,0,4]p15 2020-08-16 23:24:09.575954 2020-08-16 23:24:09.575954 | |
| 5.6 0 0 0 0 0 0 0 0 active+clean 31h 0'0 3629:4516 [18,14,13,5]p18 [18,14,13,5]p18 2020-08-16 02:46:27.887837 2020-08-10 06:09:08.174831 | |
| 5.7 3 0 0 0 897 0 0 3 active+clean 9h 3605'5 3629:3491 [5,15,14,18]p5 [5,15,14,18]p5 2020-08-17 00:25:41.809266 2020-08-13 05:37:52.184231 | |
| 6.0 1 0 0 0 0 0 0 346 active+clean 21h 3629'535 3629:26443405 [4,14,9,17]p4 [4,14,9,17]p4 2020-08-16 12:20:48.361537 2020-08-16 12:20:48.361537 | |
| 6.1 1 0 0 0 0 0 0 341 active+clean 30m 3629'613 3629:26497099 [6,12,13,3]p6 [6,12,13,3]p6 2020-08-17 09:32:47.922094 2020-08-16 01:49:21.114037 | |
| 6.2 0 0 0 0 0 0 0 4 active+clean 19h 2078'251 3629:58666 [10,6,13,11]p10 [10,6,13,11]p10 2020-08-16 14:09:46.658432 2020-08-11 16:12:50.795649 | |
| 6.3 1 0 0 0 0 0 0 346 active+clean 10h 3629'571 3629:26443431 [19,13,18,14]p19 [19,13,18,14]p19 2020-08-16 23:47:33.943612 2020-08-13 13:20:52.435651 | |
| 6.4 1 0 0 0 0 0 0 354 active+clean 21h 3629'543 3629:26333677 [17,9,14,8]p17 [17,9,14,8]p17 2020-08-16 12:49:44.792781 2020-08-11 09:11:27.786481 | |
| 6.5 2 0 0 0 0 0 0 668 active+clean 17h 3629'940 3629:52663020 [19,1,4,18]p19 [19,1,4,18]p19 2020-08-16 16:40:36.754710 2020-08-10 04:26:29.538486 | |
| 6.6 1 0 0 0 0 0 0 334 active+clean 24h 3629'581 3629:26239589 [12,16,13,1]p12 [12,16,13,1]p12 2020-08-16 09:44:42.924234 2020-08-12 14:42:47.482884 | |
| 6.7 1 0 0 0 0 0 0 354 active+clean 26h 3629'579 3629:26323129 [14,4,9,5]p14 [14,4,9,5]p14 2020-08-16 07:35:25.498808 2020-08-16 07:35:25.498808 | |
| 7.0 11 0 0 0 1658 2912 16 975 active+clean 24h 3629'976 3629:272928 [16,14,7,17]p16 [16,14,7,17]p16 2020-08-16 09:04:51.825161 2020-08-16 09:04:51.825161 | |
| 7.1 11 0 0 0 1560 1416 8 3076 active+clean 20h 3620'6878 3629:438089 [6,14,1,3]p6 [6,14,1,3]p6 2020-08-16 13:47:36.599648 2020-08-16 13:47:36.599648 | |
| 7.2 7 0 0 0 1857 1532 8 1858 active+clean 12h 3629'1862 3629:357200 [3,7,0,2]p3 [3,7,0,2]p3 2020-08-16 21:11:17.394835 2020-08-11 22:28:46.467550 | |
| 7.3 6 0 0 0 1730 0 0 34 active+clean 15h 3626'35 3629:226724 [17,15,6,12]p17 [17,15,6,12]p17 2020-08-16 18:17:48.970621 2020-08-14 08:40:31.413876 | |
| 7.4 6 0 0 0 1066 0 0 27 active+clean 31h 3626'28 3629:320679 [3,7,16,18]p3 [3,7,16,18]p3 2020-08-16 02:18:10.502933 2020-08-16 02:18:10.502933 | |
| 7.5 14 0 0 0 2214 4484 24 3009 active+clean 18h 3629'7811 3629:239723 [15,19,8,12]p15 [15,19,8,12]p15 2020-08-16 15:54:32.903170 2020-08-14 02:43:54.992932 | |
| 7.6 15 0 0 0 3862 0 0 48 active+clean 27h 3629'52 3629:913582 [13,1,4,16]p13 [13,1,4,16]p13 2020-08-16 06:56:31.300134 2020-08-13 21:44:05.147193 | |
| 7.7 6 0 0 0 427 704 4 3076 active+clean 12h 3629'18478 3629:135300003 [11,17,14,4]p11 [11,17,14,4]p11 2020-08-16 21:21:50.448676 2020-08-10 20:15:44.650232 | |
| 8.0 26 0 0 0 0 0 0 3050 active+clean 20h 3629'7807653 3629:11595132 [15,1,18,10]p15 [15,1,18,10]p15 2020-08-16 13:17:58.821333 2020-08-15 10:37:00.441663 | |
| 8.1 32 0 0 0 0 0 0 3076 active+clean 15h 3629'9099477 3629:13533768 [18,10,7,13]p18 [18,10,7,13]p18 2020-08-16 18:52:17.365270 2020-08-09 21:24:16.115253 | |
| 8.2 26 0 0 0 0 0 0 3081 active+clean 27h 3629'8514336 3629:12657167 [9,11,10,0]p9 [9,11,10,0]p9 2020-08-16 06:06:19.967832 2020-08-12 17:11:15.659708 | |
| 8.3 22 0 0 0 0 0 0 3030 active+clean 25h 3629'6261110 3629:9250072 [5,7,8,0]p5 [5,7,8,0]p5 2020-08-16 08:10:35.989978 2020-08-15 04:47:58.354067 | |
| 8.4 29 0 0 0 0 0 0 3048 active+clean 24m 3629'9981035 3629:14832299 [8,4,5,7]p8 [8,4,5,7]p8 2020-08-17 09:38:30.847075 2020-08-14 20:58:23.413276 | |
| 8.5 25 0 0 0 0 0 0 3070 active+clean 14h 3629'8629975 3629:12816091 [2,8,19,15]p2 [2,8,19,15]p2 2020-08-16 19:36:31.891044 2020-08-11 19:35:10.376406 | |
| 8.6 24 0 0 0 0 0 0 3026 active+clean 21h 3629'7978465 3629:11875308 [5,15,2,16]p5 [5,15,2,16]p5 2020-08-16 12:37:44.415756 2020-08-12 20:39:57.383255 | |
| 8.7 23 0 0 0 0 0 0 3081 active+clean 20h 3629'7045223 3629:10436503 [0,16,9,19]p0 [0,16,9,19]p0 2020-08-16 13:49:17.117870 2020-08-10 23:04:58.110588 | |
| 10.0 8727 0 0 0 3670473841 0 0 3059 active+clean 10h 3629'4319994 3629:27878140 [17,9,4,14]p17 [17,9,4,14]p17 2020-08-16 23:18:06.937879 2020-08-10 19:02:28.103491 | |
| 10.1 8733 0 0 0 3662648021 0 0 3077 active+clean 26h 3629'5147909 3629:31741562 [8,12,17,3]p8 [8,12,17,3]p8 2020-08-16 07:56:03.254946 2020-08-12 14:38:09.703348 | |
| 10.2 8710 0 0 0 3677817145 0 0 3067 active+clean 22h 3629'5527686 3629:33959754 [8,6,9,17]p8 [8,6,9,17]p8 2020-08-16 12:00:52.962034 2020-08-16 12:00:52.962034 | |
| 10.3 8688 0 0 0 3736528078 0 0 3053 active+clean 19h 3629'4468073 3629:27964068 [12,6,13,1]p12 [12,6,13,1]p12 2020-08-16 14:27:13.422638 2020-08-14 10:56:15.273485 | |
| 10.4 8773 0 0 0 3672511308 0 0 3054 active+clean 3h 3629'4131987 3629:27723907 [7,9,18,6]p7 [7,9,18,6]p7 2020-08-17 06:16:12.995135 2020-08-15 20:25:05.273445 | |
| 10.5 8549 0 0 0 3635015958 0 0 3047 active+clean 17h 3629'4734277 3629:27344475 [18,14,7,15]p18 [18,14,7,15]p18 2020-08-16 16:27:34.848142 2020-08-14 01:02:12.320648 | |
| 10.6 8723 0 0 0 3651236257 0 0 3042 active+clean 30h 3629'5111162 3629:33508238 [13,19,16,12]p13 [13,19,16,12]p13 2020-08-16 03:36:55.650689 2020-08-11 22:33:13.472993 | |
| 10.7 8623 0 0 0 3593613124 0 0 3053 active+clean 28h 3629'4433774 3629:25395614 [12,6,13,15]p12 [12,6,13,15]p12 2020-08-16 06:02:32.648055 2020-08-15 00:48:32.278486 | |
| 11.0 1 0 0 0 0 175636 484 264 active+clean 23h 3626'264 3629:5205 [5,13,0,10]p5 [5,13,0,10]p5 2020-08-16 10:55:15.816425 2020-08-11 19:47:25.967648 | |
| 11.1 5 0 0 0 0 842872 2620 3090 active+clean 24h 3629'7553373 3629:7563678 [13,15,16,4]p13 [13,15,16,4]p13 2020-08-16 09:24:39.030471 2020-08-15 05:17:56.146025 | |
| 11.2 4 0 0 0 0 977588 3472 3068 active+clean 26h 3629'3773773 3629:3781491 [9,11,10,12]p9 [9,11,10,12]p9 2020-08-16 07:39:36.931571 2020-08-13 19:56:13.839973 | |
| 11.3 0 0 0 0 0 0 0 3011 active+clean 15h 3605'7544511 3629:7551165 [4,2,5,19]p4 [4,2,5,19]p4 2020-08-16 18:46:04.571891 2020-08-16 18:46:04.571891 | |
| 11.4 2 0 0 0 0 56754276 190332 3007 active+clean 23m 3629'3871607 3629:3921914 [7,9,10,4]p7 [7,9,10,4]p7 2020-08-17 09:39:41.814879 2020-08-12 14:00:41.440724 | |
| 11.5 1 0 0 0 0 7596 32 3024 active+clean 34h 3620'3774307 3629:3779273 [11,1,16,14]p11 [11,1,16,14]p11 2020-08-15 23:27:34.148298 2020-08-15 23:27:34.148298 | |
| 11.6 2 0 0 0 0 16804608 59704 3067 active+clean 18h 3629'7579572 3629:7600332 [18,4,7,15]p18 [18,4,7,15]p18 2020-08-16 15:16:17.271006 2020-08-15 13:57:43.665912 | |
| 11.7 0 0 0 0 0 0 0 3041 active+clean 20h 3611'3775141 3629:3777308 [12,6,19,9]p12 [12,6,19,9]p12 2020-08-16 13:08:43.122319 2020-08-16 13:08:43.122319 | |
| 12.0 36724 0 0 0 2658760797 0 0 3021 active+clean 17h 3629'338200 3629:353651 [1,3,8,0]p1 [1,3,8,0]p1 2020-08-16 16:46:26.833264 2020-08-10 18:00:59.920643 | |
| 12.1 36560 0 0 0 2693060474 0 0 3044 active+clean 14h 3629'340743 3629:356297 [6,0,17,13]p6 [6,0,17,13]p6 2020-08-16 19:50:56.588301 2020-08-13 07:29:49.033271 | |
| 12.2 36744 0 0 0 2550191501 0 0 3076 active+clean 30h 3629'347211 3629:362431 [11,5,6,0]p11 [11,5,6,0]p11 2020-08-16 03:08:42.253178 2020-08-16 03:08:42.253178 | |
| 12.3 36976 0 0 0 2452558556 0 0 3083 active+clean 33h 3629'340114 3629:355304 [5,19,2,16]p5 [5,19,2,16]p5 2020-08-16 00:06:59.430645 2020-08-13 16:13:15.935250 | |
| 12.4 37167 0 0 0 2532685297 0 0 3065 active+clean 2h 3629'344231 3629:360201 [9,19,18,14]p9 [9,19,18,14]p9 2020-08-17 07:58:58.470487 2020-08-11 07:51:26.283423 | |
| 12.5 37005 0 0 0 2461994487 0 0 3071 active+clean 12m 3629'343955 3629:359523 [17,11,8,2]p17 [17,11,8,2]p17 2020-08-17 09:50:29.837769 2020-08-12 21:39:51.184699 | |
| 12.6 37091 0 0 0 2326678284 0 0 3029 active+clean 16h 3629'342832 3629:358174 [0,14,7,15]p0 [0,14,7,15]p0 2020-08-16 17:24:46.808204 2020-08-13 01:50:02.075255 | |
| 12.7 36975 0 0 0 2496638778 0 0 3100 active+clean 22h 3629'340118 3629:352610 [17,19,12,18]p17 [17,19,12,18]p17 2020-08-16 11:28:41.495468 2020-08-15 10:54:22.325510 | |
| 12.8 36745 0 0 0 2453736305 0 0 3047 active+clean 25h 3629'341425 3629:352764 [19,5,14,12]p19 [19,5,14,12]p19 2020-08-16 08:20:55.871363 2020-08-11 00:28:58.843200 | |
| 12.9 36761 0 0 0 2574355922 0 0 3092 active+clean 109m 3629'342468 3629:357534 [15,3,6,16]p15 [15,3,6,16]p15 2020-08-17 08:13:42.940428 2020-08-15 03:51:12.708397 | |
| 12.a 37207 0 0 0 2594481924 0 0 3096 active+clean 17h 3629'345495 3629:347041 [12,0,17,19]p12 [12,0,17,19]p12 2020-08-16 16:26:16.525778 2020-08-16 16:26:16.525778 | |
| 12.b 36624 0 0 0 2542103794 0 0 3058 active+clean 3h 3629'337926 3629:353373 [3,5,0,4]p3 [3,5,0,4]p3 2020-08-17 06:38:51.532865 2020-08-12 13:17:48.634712 | |
| 12.c 37146 0 0 0 2542888837 0 0 3046 active+clean 8h 3629'341297 3629:356994 [17,9,0,6]p17 [17,9,0,6]p17 2020-08-17 01:47:38.880347 2020-08-11 18:13:24.694080 | |
| 12.d 36882 0 0 0 2473040310 0 0 3063 active+clean 11h 3629'343050 3629:358433 [9,7,10,6]p9 [9,7,10,6]p9 2020-08-16 22:07:39.858356 2020-08-10 15:03:18.641927 | |
| 12.e 36977 0 0 0 2513728716 0 0 3099 active+clean 23h 3629'342535 3629:471615 [16,18,7,11]p16 [16,18,7,11]p16 2020-08-16 10:19:08.917672 2020-08-10 05:51:27.175188 | |
| 12.f 37170 0 0 0 2421662736 0 0 3091 active+clean 11h 3629'339856 3629:355477 [12,6,7,1]p12 [12,6,7,1]p12 2020-08-16 22:46:18.321426 2020-08-13 10:34:11.712596 | |
| 12.10 37063 0 0 0 2584770038 0 0 3050 active+clean 72m 3629'340702 3629:498660 [5,19,12,10]p5 [5,19,12,10]p5 2020-08-17 08:51:07.093497 2020-08-12 03:45:50.319811 | |
| 12.11 37166 0 0 0 2639362033 0 0 3013 active+clean 21h 3629'344753 3629:504304 [2,18,1,15]p2 [2,18,1,15]p2 2020-08-16 12:21:25.239991 2020-08-12 10:45:52.034024 | |
| 12.12 36937 0 0 0 2454977301 0 0 3061 active+clean 17h 3629'344690 3629:473964 [17,13,0,10]p17 [17,13,0,10]p17 2020-08-16 16:35:25.451262 2020-08-16 16:35:25.451262 | |
| 12.13 36989 0 0 0 2452289072 0 0 3001 active+clean 3h 3629'330247 3629:459049 [4,8,7,5]p4 [4,8,7,5]p4 2020-08-17 06:11:39.821627 2020-08-17 06:11:39.821627 | |
| 12.14 36936 0 0 0 2436184387 0 0 3077 active+clean 18h 3629'334511 3629:491996 [8,12,17,15]p8 [8,12,17,15]p8 2020-08-16 15:16:55.057592 2020-08-15 08:21:42.290408 | |
| 12.15 36841 0 0 0 2550933263 0 0 3042 active+clean 6h 3629'345296 3629:502422 [1,15,18,4]p1 [1,15,18,4]p1 2020-08-17 03:52:55.184909 2020-08-16 02:39:11.303006 | |
| 12.16 37146 0 0 0 2517689370 0 0 3094 active+clean 20h 3629'343214 3629:472912 [7,9,8,4]p7 [7,9,8,4]p7 2020-08-16 13:44:35.341577 2020-08-14 00:43:42.538689 | |
| 12.17 37235 0 0 0 2411367900 0 0 3086 active+clean 75m 3629'344474 3629:496483 [5,9,8,14]p5 [5,9,8,14]p5 2020-08-17 08:47:51.571280 2020-08-13 01:47:55.688179 | |
| 12.18 36729 0 0 0 2458800671 0 0 3098 active+clean 30h 3629'342808 3629:468672 [14,2,13,3]p14 [14,2,13,3]p14 2020-08-16 03:13:11.863020 2020-08-14 19:08:54.971752 | |
| 12.19 37311 0 0 0 2597837981 0 0 3088 active+clean 27h 3629'346758 3629:474436 [12,16,17,1]p12 [12,16,17,1]p12 2020-08-16 07:01:40.593159 2020-08-15 04:48:07.588319 | |
| 12.1a 37214 0 0 0 2632270936 0 0 3071 active+clean 2h 3629'351696 3629:494815 [5,15,14,6]p5 [5,15,14,6]p5 2020-08-17 07:44:07.798401 2020-08-16 04:51:26.455475 | |
| 12.1b 36940 0 0 0 2470626204 0 0 3053 active+clean 20h 3629'344732 3629:501605 [1,13,12,6]p1 [1,13,12,6]p1 2020-08-16 13:35:08.122944 2020-08-16 13:35:08.122944 | |
| 12.1c 36839 0 0 0 2492477640 0 0 3015 active+clean 2h 3629'343722 3629:445048 [3,1,6,0]p3 [3,1,6,0]p3 2020-08-17 07:42:13.521977 2020-08-15 22:22:48.911933 | |
| 12.1d 36549 0 0 0 2255784538 0 0 3067 active+clean 6h 3629'342857 3629:507769 [6,0,7,5]p6 [6,0,7,5]p6 2020-08-17 03:55:29.330992 2020-08-14 21:29:33.393053 | |
| 12.1e 36871 0 0 0 2452671913 0 0 3077 active+clean 28h 3629'342328 3629:605227 [2,14,5,1]p2 [2,14,5,1]p2 2020-08-16 05:41:59.725253 2020-08-15 01:28:05.207465 | |
| 12.1f 37234 0 0 0 2448717330 0 0 3077 active+clean 11h 3629'342758 3629:445978 [4,10,19,17]p4 [4,10,19,17]p4 2020-08-16 23:01:57.976326 2020-08-14 13:40:26.247849 | |
| 13.0 581 0 0 0 50331648 297297128 643500 3089 active+clean 11h 3629'332670 3629:358920 [17,19,2,12]p17 [17,19,2,12]p17 2020-08-16 22:22:54.073482 2020-08-14 05:44:32.910314 | |
| 13.1 578 0 0 0 41943040 332682568 720092 3056 active+clean 24h 3629'366771 3629:422292 [2,8,17,9]p2 [2,8,17,9]p2 2020-08-16 09:55:04.810883 2020-08-12 11:49:08.440021 | |
| 13.2 592 0 0 0 37402092 242490960 524872 3049 active+clean 21h 3629'360872 3629:410128 [11,7,12,6]p11 [11,7,12,6]p11 2020-08-16 12:34:53.339730 2020-08-15 03:14:11.641874 | |
| 13.3 606 0 0 0 34049588 207078008 448224 3067 active+clean 5h 3629'397683 3629:456105 [17,3,0,14]p17 [17,3,0,14]p17 2020-08-17 04:22:26.814605 2020-08-15 23:05:06.296695 | |
| 13.4 553 0 0 0 20972154 240751960 521108 3015 active+clean 25h 3629'852180 3629:888159 [9,11,18,16]p9 [9,11,18,16]p9 2020-08-16 08:22:47.112414 2020-08-16 08:22:47.112414 | |
| 13.5 580 0 0 0 20971680 292948688 634088 3009 active+clean 7h 3629'587527 3629:615665 [11,7,12,10]p11 [11,7,12,10]p11 2020-08-17 02:30:59.655047 2020-08-14 15:10:59.726903 | |
| 13.6 603 0 0 0 29360128 264796356 579740 3100 active+clean 12h 3629'1228251 3629:1268003 [10,6,15,9]p10 [10,6,15,9]p10 2020-08-16 22:00:12.848337 2020-08-14 07:04:49.213009 | |
| 13.7 559 0 0 0 25166390 290987928 629844 3072 active+clean 33h 3629'373240 3629:428639 [16,2,17,1]p16 [16,2,17,1]p16 2020-08-16 00:16:42.372384 2020-08-13 15:45:28.525122 | |
| 14.0 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [5,12]p5 [5,12]p5 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.1 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [18,7]p18 [18,7]p18 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.2 0 0 0 0 0 0 0 7 active+undersized+degraded 5d 3629'7 3629:21 [7,18]p7 [7,18]p7 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.3 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [8,5]p8 [8,5]p8 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.4 0 0 0 0 0 0 0 7 active+undersized+degraded 5d 3629'7 3629:21 [5,18]p5 [5,18]p5 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.5 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [16,1]p16 [16,1]p16 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.6 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [8,3]p8 [8,3]p8 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| 14.7 0 0 0 0 0 0 0 0 active+undersized 5d 0'0 3628:8 [18,3]p18 [18,3]p18 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 | |
| |
| * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. | |
| |
| </code> | |
| |
| |
| Thoughts: \\ | |
| * PG 14 is **empty** !!! | |
| * PG 14 is not scrub'in !!! | |
| * PG 14 does not backfill!!!!! | |
| \\ | |
| **WHY????????** | |
| |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph osd lspools | |
| 5 .rgw.root | |
| 6 default.rgw.control | |
| 7 default.rgw.meta | |
| 8 default.rgw.log | |
| 10 default.rgw.buckets.data | |
| 11 default.rgw.buckets.index | |
| 12 cephfs_data-ftp | |
| 13 cephfs_metadata-ftp | |
| 14 default.rgw.buckets.non-ec | |
| </code> | |
| Maybe ''PG 14'' is **EMPTY** cause pool 14 is unused!!!!!\\ | |
| \\ | |
| Pool is empty, but there are more empty pools and the ''warning'' comes only from this. | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph osd pool autoscale-status | |
| POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE | |
| default.rgw.buckets.non-ec 0 3.0 40940G 0.0000 1.0 8 warn | |
| cephfs_metadata-ftp 3049M 4.0 40940G 0.0003 1.0 8 on | |
| default.rgw.meta 16906k 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.buckets.index 73791k 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.control 0 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.buckets.data 113.7G 4.0 40940G 0.0111 1.0 8 on | |
| .rgw.root 1280k 4.0 40940G 0.0000 1.0 8 on | |
| cephfs_data-ftp 520.1G 4.0 40940G 0.0508 1.0 32 on | |
| default.rgw.log 0 4.0 40940G 0.0000 1.0 8 on | |
| </code> | |
| This pool **does not** have autoscale!!!!\\ | |
| Turning it on: | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph osd pool set default.rgw.buckets.non-ec pg_autoscale_mode on | |
| set pool 14 pg_autoscale_mode to on | |
| avmlp-osm-001 /home/ceph # ceph osd pool autoscale-status | |
| POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE | |
| default.rgw.buckets.non-ec 0 3.0 40940G 0.0000 1.0 8 on | |
| cephfs_metadata-ftp 3049M 4.0 40940G 0.0003 1.0 8 on | |
| default.rgw.meta 16906k 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.buckets.index 73791k 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.control 0 4.0 40940G 0.0000 1.0 8 on | |
| default.rgw.buckets.data 113.7G 4.0 40940G 0.0111 1.0 8 on | |
| .rgw.root 1280k 4.0 40940G 0.0000 1.0 8 on | |
| cephfs_data-ftp 520.1G 4.0 40940G 0.0508 1.0 32 on | |
| default.rgw.log 0 4.0 40940G 0.0000 1.0 8 on | |
| </code> | |
| check: | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph -s | |
| cluster: | |
| id: a3a799ce-f1d3-4230-a915-06e988fee767 | |
| health: HEALTH_WARN | |
| Degraded data redundancy: 8 pgs undersized | |
| | |
| services: | |
| mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d) | |
| mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminal.net | |
| mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby | |
| osd: 20 osds: 20 up (since 7w), 20 in (since 12M) | |
| rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net) | |
| | |
| data: | |
| pools: 9 pools, 96 pgs | |
| objects: 1.26M objects, 102 GiB | |
| usage: 659 GiB used, 39 TiB / 40 TiB avail | |
| pgs: 88 active+clean | |
| 8 active+undersized | |
| | |
| io: | |
| client: 1.1 MiB/s rd, 170 B/s wr, 16 op/s rd, 0 op/s wr | |
| | |
| avmlp-osm-001 /home/ceph # ceph health detail | |
| HEALTH_WARN Degraded data redundancy: 8 pgs undersized | |
| PG_DEGRADED Degraded data redundancy: 8 pgs undersized | |
| pg 14.0 is stuck undersized for 513135.338453, current state active+undersized, last acting [5,12] | |
| pg 14.1 is stuck undersized for 513135.375686, current state active+undersized, last acting [18,7] | |
| pg 14.2 is stuck undersized for 513135.291865, current state active+undersized, last acting [7,18] | |
| pg 14.3 is stuck undersized for 513135.370383, current state active+undersized, last acting [8,5] | |
| pg 14.4 is stuck undersized for 513135.338453, current state active+undersized, last acting [5,18] | |
| pg 14.5 is stuck undersized for 513135.317750, current state active+undersized, last acting [16,1] | |
| pg 14.6 is stuck undersized for 513135.370383, current state active+undersized, last acting [8,3] | |
| pg 14.7 is stuck undersized for 513135.375623, current state active+undersized, last acting [18,3] | |
| </code> | |
| ''ceph'' is moving!!! But still has ''warning'', looking for placement rule: | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph osd pool ls detail | grep "non-ec" | |
| pool 14 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3630 flags hashpspool stripe_width 0 application rgw | |
| </code> | |
| ''pool 14'' has the default placement rule, switching to ''CiberterminalRule'': | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph osd pool set default.rgw.buckets.non-ec crush_rule CiberterminalRule | |
| set pool 14 crush_rule to CiberterminalRule | |
| |
| avmlp-osm-001 /home/ceph # ceph osd pool ls detail | grep "non-ec" | |
| pool 14 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3631 flags hashpspool stripe_width 0 application rgw | |
| </code> | |
| Check: | |
| <code> | |
| avmlp-osm-001 /home/ceph # ceph health detail | |
| HEALTH_OK | |
| avmlp-osm-001 /home/ceph # ceph -s | |
| cluster: | |
| id: a3a799ce-f1d3-4230-a915-06e988fee767 | |
| health: HEALTH_OK | |
| | |
| services: | |
| mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d) | |
| mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminal.net | |
| mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby | |
| osd: 20 osds: 20 up (since 7w), 20 in (since 12M) | |
| rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net) | |
| | |
| data: | |
| pools: 9 pools, 96 pgs | |
| objects: 1.26M objects, 102 GiB | |
| usage: 659 GiB used, 39 TiB / 40 TiB avail | |
| pgs: 96 active+clean | |
| | |
| io: | |
| client: 15 KiB/s rd, 2.2 KiB/s wr, 1 op/s rd, 0 op/s wr | |
| </code> | |
| |
| **OUUUUU YEAHHHHHHHHH** | |
| |
| |
