User Tools

Site Tools


linux:ceph:nfs_gateway

NFS gateway for CEPH object storage

Documentation
Name: NFS gateway for CEPH object storage
Description: Deploying NFS gateway on top of CEPH
Modification date :14/01/2019
Owner:dodger
Notify changes to:Owner
Tags: ceph, object storage
Scalate to:The_fucking_bofh

Variables used in this documentation

Name Description Sample
${THESERVER} Name of the gw node
export THESERVER="?????-osgw-00*"
${THEUSERNAME} The username to be created
export THEUSERNAME="newuser"

RAW

bvmld-osgw-101 /var/log/ceph # yum install nfs-ganesha-proxy.x86_64 nfs-ganesha.x86_64 
Loaded plugins: fastestmirror, priorities, versionlock
Loading mirror speeds from cached hostfile
epel/x86_64/metalink                                                                                                                             |  33 kB  00:00:00     
 * base: mirror.gadix.com
 * epel: mirror.ibcp.fr
 * extras: centos.mirror.minorisa.net
 * updates: mirror.gadix.com
base                                                                                                                                             | 3.6 kB  00:00:00     
ceph                                                      
....
 
 
 
Dependencies Resolved
 
============================================================================================================================================================
 Package                                       Arch                                     Version                               Repository               Size
============================================================================================================================================================
Installing:
 nfs-ganesha                                   x86_64                                   2.3.2-1.el7                           epel                    574 k
 nfs-ganesha-proxy                             x86_64                                   2.3.2-1.el7                           epel                     41 k
Installing for dependencies:
 gssproxy                                      x86_64                                   0.7.0-21.el7                          base                    109 k
 jemalloc                                      x86_64                                   3.6.0-1.el7                           epel                    105 k
 keyutils                                      x86_64                                   1.5.8-3.el7                           base                     54 k
 libntirpc                                     x86_64                                   1.3.1-1.el7                           epel                    121 k
 libtirpc                                      x86_64                                   0.2.4-0.15.el7                        base                     89 k
 libverto-tevent                               x86_64                                   0.2.5-4.el7                           base                    9.0 k
 nfs-utils                                     x86_64                                   1:1.3.0-0.61.el7                      base                    410 k
 quota                                         x86_64                                   1:4.01-17.el7                         base                    179 k
 quota-nls                                     noarch                                   1:4.01-17.el7                         base                     90 k
 rpcbind                                       x86_64                                   0.2.0-47.el7                          base                     60 k
 tcp_wrappers                                  x86_64                                   7.6-77.el7                            base                     78 k
Updating for dependencies:
 libini_config                                 x86_64                                   1.3.1-32.el7                          base                     64 k
 
Transaction Summary
============================================================================================================================================================
Install  2 Packages (+11 Dependent packages)
Upgrade             (  1 Dependent package)

External documentation

Document 3253661

Configuring NFS-Ganesha for Rados Gateway in Red Hat Ceph Storage 2.x, fails to export the RGW buckets.

The Rados Gateway logs show the following messages:

01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] create_export :FSAL :CRIT :RGW module: librgw init failed (-5)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL RGW
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:49): 1 validation errors in block FSAL
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:49): Errors processing block (FSAL)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:45): Unknown parameter (NFS_Protocols)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:46): Unknown parameter (Transport_Protocols)
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:38): 2 errors while processing parameters for EXPORT
01/10/2017 22:42:25 : epoch 59d0fed1 : rgw_node1 : ganesha.nfsd-6112[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:38): Errors processing block (EXPORT)

The service was restarted with:

# systemctl status nfs-ganesha.service

Resolución Configured NFS-Ganesha in /etc/ganesha/ganesha.conf as following:

EXPORT {
        Export_ID = 100;
        Path = "/";
        Pseudo = "/";
        Access_Type = RW;
        SecType = "sys";
        NFS_Protocols = 4;
        Transport_Protocols = TCP;
        Squash = No_Root_Squash;

        FSAL {
                Name = RGW;
                User_Id = s3user1;
                Access_Key_Id ="<ACCESS_KEY>";
                Secret_Access_Key = "<SECRET_KEY";
        }
}

RGW {
    name = "client.rgw.vm250-217";
    ceph_conf = "/etc/ceph/ceph.conf";
    init_args = "--{arg}={arg-value}";
}

Started rpcbind via systemd and ganesha using the binary and configuration file.

# systemctl start rpcbind

# /usr/bin/ganesha.nfsd -f /etc/ganesha/ganesha.conf

Verified if the process is running fine.

# ps -aux | grep -i ganesha
root     18347  0.0  0.0 107904   652 pts/2    S+   01:04   0:00 tailf /var/log/ganesha.log
root     18927 25.0  0.5 4194528 22684 ?       Ssl  01:24   0:00 /usr/bin/ganesha.nfsd -f /etc/ganesha/ganesha.conf
root     19282  0.0  0.0 112660   988 pts/0    S+   01:24   0:00 grep --color=auto -i ganesh

Confirm if the export is showing up properly

# showmount -e

Export list for <NFS-server>:
/ (everyone)

The logs at /var/log/messages showed the following, which ensured the process initiating properly.

Oct  5 01:08:27 localhost nfs-ganesha[18501]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
Oct  5 01:24:22 localhost nfs-ganesha[18926]: [main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 2.4.1
Oct  5 01:24:22 localhost nfs-ganesha[18927]: [main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
Oct  5 01:24:22 localhost nfs-ganesha[18927]: [main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:49): Unknown parameter (NFS_Protocols)
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:50): Unknown parameter (Transport_Protocols)
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nsm_connect :NLM :CRIT :failed to connect to statd
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nsm_unmonitor_all :NLM :CRIT :Can not unmonitor all clnt_create returned NULL
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
Oct  5 01:24:23 localhost nfs-ganesha[18927]: [main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Oct  5 01:25:53 localhost nfs-ganesha[18927]: [reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE

Mount the share on a Client node, which has the nfs-utils package installed.

# yum install nfs-utils -y

# mount <server-name>:/ /mnt -t nfs

# ls /mnt/
my-new-bucket

Create files in the mount point.

# for i in `seq 1 100`; do touch /mnt/my-new-bucket/file$i; echo "Creating /mnt/my-new-bucket/file$i"; done
Creating /mnt/my-new-bucket/file1
Creating /mnt/my-new-bucket/file2
Creating /mnt/my-new-bucket/file3
Creating /mnt/my-new-bucket/file4
..
....
linux/ceph/nfs_gateway.txt · Last modified: 2022/02/11 11:36 by 127.0.0.1