GPFS

Installation

GPFS packages are provided by the gpfs.* packages.

Components

IBM Spectrum Scale RAID is the front end GUI that is used to manage GPFS filesets, snapshots, quotas, and collect performance metrics.

Tivoli Storage Manager is the front end GUI and is used to configure backups and integration with the tape library.

Usage

All GPFS management utilities are located under the /usr/lpp/mmfs/bin/ directory and commands typically start with mm.

Command	Description
mmccr flist	lists files stored in cluster configuration repository (CCR)
mmccr lsnodes	Shows all quorum nodes
mmshutdown -a	Run on ems1 to shut down IBM cluster
mmstartup -a	Run on ems1 to start up the IBM cluster
mmhealth node show	Shows the health of all nodes
mmlsfs all	Shows all file system attributes on the GPFS cluster.
mmlssnapshot gpfs0/1	will show all snapshots for a particular file system.
mmlsmount all_local	shows whether the file system on this node is being mounted by other nodes in the cluster.
/usr/lpp/mmfs/bin/mmlspdisk all --not-ok	shows all failed disks.
mmedquota -j gpfs1:fileset	used to change a quota of a fileset on the GPFS1 volume.
mmcrfileset gpfs1 morph --inode-space new --inode-limit 4194304:2097152	change inode quota

The IBM Spectrum Scale RAID GUI utilities are at /usr/lpp/mmfs/gui/cli/. Certain actions taken on the web interface will trigger one of these executables.

Snapshots

Snapshot management is done using the following utilities:

Command	Description
mmlssnapshot gpfs0	Lists all snapshots in the gpfs0 device
mmdelsnapshot gpfs0 snapshot_name -j fileset	Deletes a snapshot in the gpfs0 device on a particular fileset by name.
mmcrsnapshot gpfs0 snapshot_name -j fileset	Create a snapshot in the gpfs0 device for a particular fileset with the provided snapshot name.

Automated Snapshots

In CHGI, these commands are wrapped up in a custom perl script and configured with a configuration file. This script is then invoked daily with a root-user cronjob:

5 23 * * * /usr/local/bin/admSnapshots.plx /usr/local/etc/admSnapshots.conf

Where the configuration file defines the fileset and number of snapshots to keep:

ebg_gmb            3
ebg_web            3
qlong              3
snyder_irida       3

Tasks

Create Filesystem

mmcrfs gpfs1 -F vdisk.cfg -j scatter -T /gpfs/gpfs1 -B 16M --metadata-block-size 1M

Create a new GPFS filesystem on GPFS1, where vdisk.cfg contains the filesystem specifications.

%vdisk: vdiskName=gssio1_Data_16M_2p_1 rg=gssio1 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio1_MetaData_16M_2p_1 rg=gssio1 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio1_Data_16M_2p_1:::dataOnly:30:gssio1_Data_16M_2p_1:data
gssio1_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio1_MetaData_16M_2p_1:::metadataOnly:30:gssio1_MetaData_16M_2p_1:system
gssio1_MetaData_16M_2p_1:::metadataOnly:30::system
%vdisk: vdiskName=gssio2_Data_16M_2p_1 rg=gssio2 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio2_MetaData_16M_2p_1 rg=gssio2 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio2_Data_16M_2p_1:::dataOnly:30:gssio2_Data_16M_2p_1:data
gssio2_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio2_MetaData_16M_2p_1:::metadataOnly:30:gssio2_MetaData_16M_2p_1:system
gssio2_MetaData_16M_2p_1:::metadataOnly:30::system

Node Management

Use mmlsnode to show all nodes that are part of a nodeset.

[root@tsm /]# mmlsnode
GPFS nodeset    Node list
-------------   -------------------------------------------------------
   chgi-psc     essio1-ib essio2-ib ems1-ib crick-ib powerlc001-ib powerlc002-ib transmart-ib tsm-ib node001 node002 node003 node004 node005 node006 node007 node008 node009 node010 node011 node012 node013 node014 node015 node016 node017 node018 node019 node020 node021 node022 node023 node024 node025 node026 node027 node028 node029 node030 node031 node032 node035 node033 node036 node034 theia galaxy-dev-ib ebg01-ib snyder-ib

Additional information for each node can be found when listing cluster information with mmlscluster:

[root@tsm /]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         chgi-psc.gpfs.net
  GPFS cluster id:           2878855853427990921
  GPFS UID domain:           chgi-psc.gpfs.net
  Remote shell command:      sudo wrapper in use
  Remote file copy command:  sudo wrapper in use
  Repository type:           CCR

 Node  Daemon node name        IP address      Admin node name         Designation
-----------------------------------------------------------------------------------
   1   essio1-ib.gpfs.net      172.26.3.1      essio1-ib.gpfs.net      quorum-manager-perfmon
   2   essio2-ib.gpfs.net      172.26.3.2      essio2-ib.gpfs.net      quorum-manager-perfmon
   3   ems1-ib.gpfs.net        172.26.3.251    ems1-ib.gpfs.net        quorum-perfmon
   4   crick-ib.gpfs.net       172.26.4.252    crick-ib.gpfs.net       
   5   powerlc001-ib.gpfs.net  172.26.4.1      powerlc001-ib.gpfs.net  
   6   powerlc002-ib.gpfs.net  172.26.4.2      powerlc002-ib.gpfs.net  
   7   transmart-ib.gpfs.net   172.26.4.3      transmart-ib.gpfs.net   
   8   tsm-ib.gpfs.net         172.26.3.250    tsm-ib.gpfs.net         
   9   node001                 172.26.10.1     node001                 
  ...
  50   node034                 172.26.10.34    node034                 
  51   theia                   172.26.25.2     theia                   
  56   galaxy-dev-ib           172.26.10.116   galaxy-dev-ib           
  58   ebg01-ib                172.26.10.153   ebg01-ib                
  60   snyder-ib               172.26.10.105   snyder-ib

To add a new node:

# mmaddnode -N damona-ib

To remove a node:

# mmdelnode -N damona-ib
Verifying GPFS is stopped on all affected nodes ...
mmdelnode: [W] Could not cleanup the following unreached nodes:
damona-ib
mmdelnode: Command successfully completed
mmdelnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

Rebuild Shadow Database

The shadow database is a file used by mmbackup to quickly determine what files to back up. As per the documentation:

These databases shadow the inventory of objects in IBM Spectrum Protect so that only new changes will be backed up in the next incremental mmbackup. Failing to do so will needlessly back up some files additional times. The shadow database can also become out of date if mmbackup fails due to certain IBM Spectrum Protect server problems that prevent mmbackup from properly updating its shadow database after a backup. In these cases it is also required to issue the next mmbackup command with either the -q option or the --rebuild options.

—https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adm_backupusingmmbackup.htm

To rebuild the shadow database, I had to use a separate temp directory because the system temp was too small. For some reason however, this process failed but still generated a 10GB shadow file.

# mmbackup gpfs0 --rebuild -s /tmp2
--------------------------------------------------------
mmbackup: Shadow database rebuild of /gpfs begins at Wed Feb 19 09:35:36 MST 2020.
--------------------------------------------------------
Thu Feb 20 00:56:27 2020 mmbackup:Built query data file from TSM server: CHGI_TSM01 rc = 0
Thu Feb 20 00:56:28 2020 mmbackup:Scanning file system gpfs0
Thu Feb 20 04:55:32 2020 mmbackup:Reconstructing previous shadow file /gpfs/.mmbackupShadow.1.CHGI_TSM01.filesys from query data for CHGI_TSM01
mmbackup: tsbuhelper: parsePolicyShow: could not find mtime date
mmbackup: tsbuhelper: rebuildShadow: writing merged record fail rc = -1
mmbackup: tsbuhelper:rebuildShadow: Failed with rc=-1
Thu Feb 20 05:26:48 2020 mmbackup:/usr/lpp/mmfs/bin/tsbuhelper rebuildshadow /gpfs/.mmbackupQueryShadow.CHGI_TSM01.filesys /gpfs/.mmbackupCfg/prepFiles/list.mmbackup.1.CHGI_TSM01 /gpfs/.mmbackupShadow.1.CHGI_TSM01.filesys /tmp2 2>&1 returned rc=255.  Cannot rebuild shadow file.
Thu Feb 20 05:26:48 2020 mmbackup:Failed to reconstruct old shadow file.  Skipping CHGI_TSM01.
Thu Feb 20 05:26:48 2020 mmbackup:Done with shadow file database rebuilds
Thu Feb 20 05:26:48 2020 mmbackup:Incremental shadow database rebuild completely failed.
        TSM had 0 severe errors and returned 0. See the TSM log file for more information.
        0 files had errors,
 TSM exit status:  exit 12

----------------------------------------------------------
mmbackup: Shadow database rebuild of /gpfs completed with errors at Thu Feb 20 05:26:48 MST 2020.
----------------------------------------------------------
mmbackup: Command failed. Examine previous error messages to determine cause.

It was at 5:26 that the TSM server lost its membership to the GPFS filesystem for some reason which probably caused the failure...

Clear Stale Events on IBM Spectrum Scale RAID

To clear out old events, you will need to delete entries from the PostgreSQL server. On the server running the GUI, run:

# sudo -u postgres psql
psql (9.2.7)
Type "help" for help.

postgres=# select * FROM pg_stat_activity;

SELECT * FROM fscc.gss_state WHERE event_time <= '2019-12-12' ;

Fix Data Collection

If data collection is not happening or no data is being reported on IBM Spectrum Scale RAID, ensure that the pmsensors service is running on all GPFS nodes.

essio# systemctl restart pmsensors

If pmsensors isn't starting, or if the /opt/IBM/zimon/ZIMonSensors.cfg file is missing, ensure that the configuration file has the following contents. The colCandidates value should be set to the hostname of the GPFS management node for data collection.

colCandidates = "synergy-ib"
colRedundancy = 1
collectors = {
        host = "synergy-ib"
        port = "4739"
}
config = "/opt/IBM/zimon/ZIMonSensors.cfg"
ctdbstat = ""
daemonize = T
hostname = ""
ipfixinterface = "0.0.0.0"
logfile = "/var/log/zimon/ZIMonSensors.log"
loglevel = "info"
mmcmd = "/opt/IBM/zimon/MMCmdProxy"
mmdfcmd = "/opt/IBM/zimon/MMDFProxy"
mmpmon = "/opt/IBM/zimon/MmpmonSockProxy"
piddir = "/var/run"
release = "4.2.0-2"
sensors = {
        name = "CPU"
        period = 1
},
{
        name = "Load"
        period = 1
},
{
        name = "Memory"
        period = 1
},
{
        name = "Network"
        period = 1
},
{
        name = "Netstat"
        period = 0
},
{
        name = "Diskstat"
        period = 0
},
{
        name = "DiskFree"
        period = 600
},
{
        name = "GPFSDisk"
        period = 0
},
{
        name = "GPFSFilesystem"
        period = 1
},
{
        name = "GPFSNSDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSPoolIO"
        period = 0
},
{
        name = "GPFSVFS"
        period = 1
},
{
        name = "GPFSIOC"
        period = 0
},
{
        name = "GPFSVIO"
        period = 0
},
{
        name = "GPFSPDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSvFLUSH"
        period = 0
},
{
        name = "GPFSNode"
        period = 1
},
{
        name = "GPFSNodeAPI"
        period = 1
},
{
        name = "GPFSFilesystemAPI"
        period = 1
},
{
        name = "GPFSLROC"
        period = 0
},
{
        name = "GPFSCHMS"
        period = 0
},
{
        name = "GPFSAFM"
        period = 0
},
{
        name = "GPFSAFMFS"
        period = 0
},
{
        name = "GPFSAFMFSET"
        period = 0
},
{
        name = "GPFSRPCS"
        period = 0
},
{
        name = "GPFSFilesetQuota"
        period = 3600
},
{
        name = "GPFSDiskCap"
        period = 0
        restrict = "ems1-ib"
},
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Troubleshooting

To troubleshoot issues with GPFS, you may want to take a look at the logs from mmfsd at:

• /var/adm/ras/mmfs.log.*

See Also

• https://github.com/aleksyv/gpfs-monitoring
• https://github.com/aleksyv/gpfs-monitoring/blob/master/gpfs_cluster.py