GPFS

Installation

GPFS packages are provided by the gpfs.* packages.

Components

IBM Spectrum Scale RAID is the front end GUI that is used to manage GPFS filesets, snapshots, quotas, and collect performance metrics.

Tivoli Storage Manager is the front end GUI and is used to configure backups and integration with the tape library.

Usage

All GPFS management utilities are located under the /usr/lpp/mmfs/bin/ directory and commands typically start with mm.

Command	Description
`mmccr flist`	lists files stored in cluster configuration repository (CCR)
`mmccr lsnodes`	Shows all quorum nodes
`mmshutdown -a`	Run on ems1 to shut down IBM cluster
`mmstartup -a`	Run on ems1 to start up the IBM cluster
`mmhealth node show`	Shows the health of all nodes
`mmlsfs all`	Shows all file system attributes on the GPFS cluster.
`mmlssnapshot gpfs0/1`	will show all snapshots for a particular file system.
`mmlsmount all_local`	shows whether the file system on this node is being mounted by other nodes in the cluster.
`/usr/lpp/mmfs/bin/mmlspdisk all --not-ok`	shows all failed disks.
`mmedquota -j gpfs1:fileset`	used to change a quota of a fileset on the GPFS1 volume.
`mmcrfileset gpfs1 morph --inode-space new --inode-limit 4194304:2097152`	change inode quota

The IBM Spectrum Scale RAID GUI utilities are at /usr/lpp/mmfs/gui/cli/. Certain actions taken on the web interface will trigger one of these executables.

Snapshots

Snapshot management is done using the following utilities:

Command	Description
`mmlssnapshot gpfs0`	Lists all snapshots in the gpfs0 device
`mmdelsnapshot gpfs0 snapshot_name -j fileset`	Deletes a snapshot in the gpfs0 device on a particular fileset by name.
`mmcrsnapshot gpfs0 snapshot_name -j fileset`	Create a snapshot in the gpfs0 device for a particular fileset with the provided snapshot name.

Automated Snapshots

In CHGI, these commands are wrapped up in a custom perl script and configured with a configuration file. This script is then invoked daily with a root-user cronjob:

5 23 * * * /usr/local/bin/admSnapshots.plx /usr/local/etc/admSnapshots.conf

Where the configuration file defines the fileset and number of snapshots to keep:

ebg_gmb            3
ebg_web            3
qlong              3
snyder_irida       3

Tasks

Create Filesystem

mmcrfs gpfs1 -F vdisk.cfg -j scatter -T /gpfs/gpfs1 -B 16M --metadata-block-size 1M

Create a new GPFS filesystem on GPFS1, where vdisk.cfg contains the filesystem specifications.

%vdisk: vdiskName=gssio1_Data_16M_2p_1 rg=gssio1 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio1_MetaData_16M_2p_1 rg=gssio1 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio1_Data_16M_2p_1:::dataOnly:30:gssio1_Data_16M_2p_1:data
gssio1_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio1_MetaData_16M_2p_1:::metadataOnly:30:gssio1_MetaData_16M_2p_1:system
gssio1_MetaData_16M_2p_1:::metadataOnly:30::system
%vdisk: vdiskName=gssio2_Data_16M_2p_1 rg=gssio2 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio2_MetaData_16M_2p_1 rg=gssio2 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio2_Data_16M_2p_1:::dataOnly:30:gssio2_Data_16M_2p_1:data
gssio2_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio2_MetaData_16M_2p_1:::metadataOnly:30:gssio2_MetaData_16M_2p_1:system
gssio2_MetaData_16M_2p_1:::metadataOnly:30::system

Node Management

Use mmlsnode to show all nodes that are part of a nodeset.

[root@tsm /]# mmlsnode
GPFS nodeset    Node list
-------------   -------------------------------------------------------
   chgi-psc     essio1-ib essio2-ib ems1-ib crick-ib powerlc001-ib powerlc002-ib transmart-ib tsm-ib node001 node002 node003 node004 node005 node006 node007 node008 node009 node010 node011 node012 node013 node014 node015 node016 node017 node018 node019 node020 node021 node022 node023 node024 node025 node026 node027 node028 node029 node030 node031 node032 node035 node033 node036 node034 theia galaxy-dev-ib ebg01-ib snyder-ib

Additional information for each node can be found when listing cluster information with mmlscluster:

[root@tsm /]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         chgi-psc.gpfs.net
  GPFS cluster id:           2878855853427990921
  GPFS UID domain:           chgi-psc.gpfs.net
  Remote shell command:      sudo wrapper in use
  Remote file copy command:  sudo wrapper in use
  Repository type:           CCR

 Node  Daemon node name        IP address      Admin node name         Designation
-----------------------------------------------------------------------------------
   1   essio1-ib.gpfs.net      172.26.3.1      essio1-ib.gpfs.net      quorum-manager-perfmon
   2   essio2-ib.gpfs.net      172.26.3.2      essio2-ib.gpfs.net      quorum-manager-perfmon
   3   ems1-ib.gpfs.net        172.26.3.251    ems1-ib.gpfs.net        quorum-perfmon
   4   crick-ib.gpfs.net       172.26.4.252    crick-ib.gpfs.net       
   5   powerlc001-ib.gpfs.net  172.26.4.1      powerlc001-ib.gpfs.net  
   6   powerlc002-ib.gpfs.net  172.26.4.2      powerlc002-ib.gpfs.net  
   7   transmart-ib.gpfs.net   172.26.4.3      transmart-ib.gpfs.net   
   8   tsm-ib.gpfs.net         172.26.3.250    tsm-ib.gpfs.net         
   9   node001                 172.26.10.1     node001                 
  ...
  50   node034                 172.26.10.34    node034                 
  51   theia                   172.26.25.2     theia                   
  56   galaxy-dev-ib           172.26.10.116   galaxy-dev-ib           
  58   ebg01-ib                172.26.10.153   ebg01-ib                
  60   snyder-ib               172.26.10.105   snyder-ib

To add a new node:

# mmaddnode -N damona-ib

To remove a node:

# mmdelnode -N damona-ib
Verifying GPFS is stopped on all affected nodes ...
mmdelnode: [W] Could not cleanup the following unreached nodes:
damona-ib
mmdelnode: Command successfully completed
mmdelnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.

Clear Stale Events on IBM Spectrum Scale RAID

To clear out old events, you will need to delete entries from the PostgreSQL server. On the server running the GUI, run:

# sudo -u postgres psql
psql (9.2.7)
Type "help" for help.

postgres=# select * FROM pg_stat_activity;

SELECT * FROM fscc.gss_state WHERE event_time <= '2019-12-12' ;

Fix Data Collection

If data collection is not happening or no data is being reported on IBM Spectrum Scale RAID, ensure that the pmsensors service is running on all GPFS nodes.

essio# systemctl restart pmsensors

If pmsensors isn't starting, or if the /opt/IBM/zimon/ZIMonSensors.cfg file is missing, ensure that the configuration file has the following contents. The colCandidates value should be set to the hostname of the GPFS management node for data collection.

colCandidates = "synergy-ib"
colRedundancy = 1
collectors = {
        host = "synergy-ib"
        port = "4739"
}
config = "/opt/IBM/zimon/ZIMonSensors.cfg"
ctdbstat = ""
daemonize = T
hostname = ""
ipfixinterface = "0.0.0.0"
logfile = "/var/log/zimon/ZIMonSensors.log"
loglevel = "info"
mmcmd = "/opt/IBM/zimon/MMCmdProxy"
mmdfcmd = "/opt/IBM/zimon/MMDFProxy"
mmpmon = "/opt/IBM/zimon/MmpmonSockProxy"
piddir = "/var/run"
release = "4.2.0-2"
sensors = {
        name = "CPU"
        period = 1
},
{
        name = "Load"
        period = 1
},
{
        name = "Memory"
        period = 1
},
{
        name = "Network"
        period = 1
},
{
        name = "Netstat"
        period = 0
},
{
        name = "Diskstat"
        period = 0
},
{
        name = "DiskFree"
        period = 600
},
{
        name = "GPFSDisk"
        period = 0
},
{
        name = "GPFSFilesystem"
        period = 1
},
{
        name = "GPFSNSDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSPoolIO"
        period = 0
},
{
        name = "GPFSVFS"
        period = 1
},
{
        name = "GPFSIOC"
        period = 0
},
{
        name = "GPFSVIO"
        period = 0
},
{
        name = "GPFSPDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSvFLUSH"
        period = 0
},
{
        name = "GPFSNode"
        period = 1
},
{
        name = "GPFSNodeAPI"
        period = 1
},
{
        name = "GPFSFilesystemAPI"
        period = 1
},
{
        name = "GPFSLROC"
        period = 0
},
{
        name = "GPFSCHMS"
        period = 0
},
{
        name = "GPFSAFM"
        period = 0
},
{
        name = "GPFSAFMFS"
        period = 0
},
{
        name = "GPFSAFMFSET"
        period = 0
},
{
        name = "GPFSRPCS"
        period = 0
},
{
        name = "GPFSFilesetQuota"
        period = 3600
},
{
        name = "GPFSDiskCap"
        period = 0
        restrict = "ems1-ib"
},
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Troubleshooting

To troubleshoot issues with GPFS, you may want to take a look at the logs from mmfsd at:

/var/adm/ras/mmfs.log.*

GPFS

Installation

Components

Usage

Snapshots

Automated Snapshots

Tasks

Create Filesystem

Node Management

Clear Stale Events on IBM Spectrum Scale RAID

Fix Data Collection

Troubleshooting

See Also

Contents

Recent Changes

Recent Changes under Linux

Recent Changes under Guides

Ad-Free + Tracker-Free

Contents