====== NFS gateway for CEPH object storage ======
^ Documentation ^|
^Name:| NFS gateway for CEPH object storage |
^Description:| Deploying NFS gateway on top of CEPH |
^Modification date :|14/01/2019|
^Owner:|dodger|
^Notify changes to:|Owner|
^Tags:| ceph, object storage|
^Scalate to:|The_fucking_bofh|
====== Variables used in this documentation ======
^ Name ^ Description ^ Sample ^
| ''${THESERVER}'' | Name of the gw node | export THESERVER="?????-osgw-00*" |
| ''${THEUSERNAME}'' | The username to be created | export THEUSERNAME="newuser" |
====== RAW ======
bvmld-osgw-101 /var/log/ceph # yum install nfs-ganesha-proxy.x86_64 nfs-ganesha.x86_64
Loaded plugins: fastestmirror, priorities, versionlock
Loading mirror speeds from cached hostfile
epel/x86_64/metalink | 33 kB 00:00:00
* base: mirror.gadix.com
* epel: mirror.ibcp.fr
* extras: centos.mirror.minorisa.net
* updates: mirror.gadix.com
base | 3.6 kB 00:00:00
ceph
....
Dependencies Resolved
============================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================
Installing:
nfs-ganesha x86_64 2.3.2-1.el7 epel 574 k
nfs-ganesha-proxy x86_64 2.3.2-1.el7 epel 41 k
Installing for dependencies:
gssproxy x86_64 0.7.0-21.el7 base 109 k
jemalloc x86_64 3.6.0-1.el7 epel 105 k
keyutils x86_64 1.5.8-3.el7 base 54 k
libntirpc x86_64 1.3.1-1.el7 epel 121 k
libtirpc x86_64 0.2.4-0.15.el7 base 89 k
libverto-tevent x86_64 0.2.5-4.el7 base 9.0 k
nfs-utils x86_64 1:1.3.0-0.61.el7 base 410 k
quota x86_64 1:4.01-17.el7 base 179 k
quota-nls noarch 1:4.01-17.el7 base 90 k
rpcbind x86_64 0.2.0-47.el7 base 60 k
tcp_wrappers x86_64 7.6-77.el7 base 78 k
Updating for dependencies:
libini_config x86_64 1.3.1-32.el7 base 64 k
Transaction Summary
============================================================================================================================================================
Install 2 Packages (+11 Dependent packages)
Upgrade ( 1 Dependent package)
====== External documentation ======
* https://access.redhat.com/solutions/3253661
* http://docs.ceph.com/docs/master/radosgw/nfs/
* https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_red_hat_enterprise_linux/#exporting-the-namespace-to-nfs-ganesha-rgw
===== Document 3253661 =====
Configuring NFS-Ganesha for Rados Gateway in Red Hat Ceph Storage 2.x, fails to export the RGW buckets.
The Rados Gateway logs show the following messages:
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL RGW
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:49): 1 validation errors in block FSAL
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:49): Errors processing block (FSAL)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:45): Unknown parameter (NFS_Protocols)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:46): Unknown parameter (Transport_Protocols)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:38): 2 errors while processing parameters for EXPORT
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:38): Errors processing block (EXPORT)
The service was restarted with:
# systemctl status nfs-ganesha.service
Resolución
Configured NFS-Ganesha in /etc/ganesha/ganesha.conf as following:
EXPORT {
Export_ID = 100;
Path = "/";
Pseudo = "/";
Access_Type = RW;
SecType = "sys";
NFS_Protocols = 4;
Transport_Protocols = TCP;
Squash = No_Root_Squash;
FSAL {
Name = RGW;
User_Id = s3user1;
Access_Key_Id ="";
Secret_Access_Key = "
Started rpcbind via systemd and ganesha using the binary and configuration file.
# systemctl start rpcbind
# /usr/bin/ganesha.nfsd -f /etc/ganesha/ganesha.conf
Verified if the process is running fine.
# ps -aux | grep -i ganesha
root 18347 0.0 0.0 107904 652 pts/2 S+ 01:04 0:00 tailf /var/log/ganesha.log
root 18927 25.0 0.5 4194528 22684 ? Ssl 01:24 0:00 /usr/bin/ganesha.nfsd -f /etc/ganesha/ganesha.conf
root 19282 0.0 0.0 112660 988 pts/0 S+ 01:24 0:00 grep --color=auto -i ganesh
Confirm if the export is showing up properly
# showmount -e
Export list for :
/ (everyone)
The logs at /var/log/messages showed the following, which ensured the process initiating properly.
Oct 5 01:08:27 localhost nfs-ganesha[18501]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
Oct 5 01:24:22 localhost nfs-ganesha[18926]: [main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 2.4.1
Oct 5 01:24:22 localhost nfs-ganesha[18927]: [main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
Oct 5 01:24:22 localhost nfs-ganesha[18927]: [main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:49): Unknown parameter (NFS_Protocols)
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:50): Unknown parameter (Transport_Protocols)
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
Oct 5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Oct 5 01:25:53 localhost nfs-ganesha[18927]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
Mount the share on a Client node, which has the nfs-utils package installed.
# yum install nfs-utils -y
# mount :/ /mnt -t nfs
# ls /mnt/
my-new-bucket
Create files in the mount point.
# for i in `seq 1 100`; do touch /mnt/my-new-bucket/file$i; echo "Creating /mnt/my-new-bucket/file$i"; done
Creating /mnt/my-new-bucket/file1
Creating /mnt/my-new-bucket/file2
Creating /mnt/my-new-bucket/file3
Creating /mnt/my-new-bucket/file4
..
....