linux:ceph:troubleshooting:too_many_pgs_per_osd
Table of Contents
[TROUBLESHOOT] Ceph too many pgs per osd
Documentation | |
---|---|
Name: | [TROUBLESHOOT] Ceph too many pgs per osd |
Description: | how to solve this “issue” |
Modification date : | 11/04/2019 |
Owner: | dodger |
Notify changes to: | Owner |
Tags: | ceph, object storage |
Scalate to: | Thefuckingbofh |
WARNING
This documents cover the TOO MANY PGs per OSD, not too few (documented here)
Understanding the problem
The solution is perfectly explained here:
https://stackoverflow.com/questions/39589696/ceph-too-many-pgs-per-osd-all-you-need-to-know
To keep the solution safe, I'll write here down it.
Rawpaste of pool usage
ceph pg dump | awk ' BEGIN { IGNORECASE = 1 } /^PG_STAT/ { col=1; while($col!="UP") {col++}; col++ } /^[0-9a-f]+\.[0-9a-f]+/ { match($0,/^[0-9a-f]+/); pool=substr($0, RSTART, RLENGTH); poollist[pool]=0; up=$col; i=0; RSTART=0; RLENGTH=0; delete osds; while(match(up,/[0-9]+/)>0) { osds[++i]=substr(up,RSTART,RLENGTH); up = substr(up, RSTART+RLENGTH) } for(i in osds) {array[osds[i],pool]++; osdlist[osds[i]];} } END { printf("\n"); printf("pool :\t"); for (i in poollist) printf("%s\t",i); printf("| SUM \n"); for (i in poollist) printf("--------"); printf("----------------\n"); for (i in osdlist) { printf("osd.%i\t", i); sum=0; for (j in poollist) { printf("%i\t", array[i,j]); sum+=array[i,j]; sumpool[j]+=array[i,j] }; printf("| %i\n",sum) } for (i in poollist) printf("--------"); printf("----------------\n"); printf("SUM :\t"); for (i in poollist) printf("%s\t",sumpool[i]); printf("|\n"); }'
That's all
Procedure for dropping pool: [HOWTO] Completely remove a POOL from cluster
Don't drop the pool, continue reading.
The "solution"
If you're sure that you need more PG's per os, change the configuration.
You must add:
mon pg warn max per osd = ${NEW_PG_PER_OSD_NUMBER}
on your ceph.conf
under the [global]
section.
And update the config worldwide.
You can also change the setting without having to restart:
# check ceph tell 'osd.*' config get mon_max_pg_per_osd # change ceph tell 'osd.*' config set mon_max_pg_per_osd ${NEW_PG_PER_OSD_NUMBER} # check again ceph tell 'osd.*' config get mon_max_pg_per_osd
linux/ceph/troubleshooting/too_many_pgs_per_osd.txt · Last modified: 2022/02/11 11:36 by 127.0.0.1