How to Create a GPFS Filesystem on CentOS Linux

Hi everyone! This is my documentation about how to install and configure GPFS file system on top of CentOS Linux.

Just for your information, GPFS (General Parallel File System, brand name IBM Spectrum Scale) is a high-performance clustered file system software developed by IBM. It can be deployed in shared-disk or shared-nothing distributed parallel modes, or a combination of these. It is used by many of the world's largest commercial companies.

Here is the steps based on my experience on testing environment.

Install the dependencies first (https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=gpfs-software-requirements)

sudo yum install -y https://repo.ius.io/ius-release-el7.rpm

sudo yum update

sudo yum install nfs-utils ethtool rpcbind psmisc iputils ifupdown boost-regex kernel-devel cpp gcc gcc-c++ binutils ksh rpm-build m4

sudo yum install -y python36u python36u-libs python36u-devel python36u-pip python3

1. Download package from https://www.ibm.com/products/spectrum-scale/pricing

2. Extract downloaded zip > grant execution privilege > run the executable > press 1+enter on the License agreement, wait till finished.

3. Navigate to /usr/lpp/mmfs/5.1.1.3/gpfs_rpms/ then install the rpm package

rpm -ivh gpfs.base*.rpm gpfs.gpl*rpm gpfs.license*rpm gpfs.gskit*rpm gpfs.msg*rpm gpfs.compression*rpm gpfs.adv*rpm gpfs.crypto*rpm

4. Create portability layer

/usr/lpp/mmfs/bin/mmbuildgpl

5. Cluster creation. Follow the instruction from below link start from step 4

https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=nodes-installing-spectrum-scale-packages-linux-systems

OR

Method 1. Add machine one by one using  

mmcrcluster -N /folder/nodelist or mmcrcluster –N node1:quorum/quorum-manager,node2

Method 2. Create Nodeslist (text file) then fill with node/cluster member

Message when cluster created successfully

Verify cluster status

6. Accept GPFS server license, the arguments can contain server or client and can have more than one node

7. Start GPFS on node1, mmstartup command should included on system startup

The start process could take some times to change from arbitrating to active

Do the same on the other node

8. Create NSD Configuration

The disk used by GPFS are called Network Shared Disk (NSD).

The file diskdef.txt below contains the NSD configuration used by GPFS.

One NSD disk has been defined, the name is mynsd1 and the device files of these disks are /dev/sdb (8GB). The disk will be used to store data and metadata.

%nsd:

device=/dev/sdb   <==== the disk used for GPFS

nsd=mynsd1   <==== the name

usage=data and metadata

 

If more than one disk needed, just repeat the step above and edit device=/dev/xxx.

Execute the process using command /usr/lpp/mmfs/bin/mmcrnsd -F diskdef.txt

Check the configuration

9. Create GPFS File System

/usr/lpp/mmfs/bin/mmcrfs fs_gpfs01 -F diskdef.txt -A yes -T /fs_gpfs01

The command above create a GPFS filesystem called fs_gpfs01 using the NSD defined in diskdef.txt file and will be mounted on /fs_gpfs01 mount point

10. Mount & Unmount

Mount the GPFS volume on all nodes :  /usr/lpp/mmfs/bin/mmmount /fs_gpfs01 -a

 

Useful Links

• Install and configure GPFS 4.1 filesystem on Linux Centos 6.6 | Sysadmin continuous improvement (wordpress.com)
• https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=nodes-extracting-spectrum-scale-software-linux
• https://www.ibm.com/docs/en/spectrum-scale/5.1.1?topic=bgplln-using-mmbuildgpl-command-build-gpfs-portability-layer-linux-nodes
• https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=reference-mmcrcluster-command
• https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=reference-mmaddnode-command#mmaddnode
• https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=reference-mmdelnode-command#mmdelnode
• https://linuxize.com/post/how-to-install-python-3-on-centos-7/