GPFS: Difference between revisions

From Leo's Notes
This page was last edited on 25 February 2020, at 18:21.
← Older editNewer edit →
VisualWikitext
Line 417: Line 417:
}}
}}


I believe the {{code|gss_state_log_view}} pulls data from the {{code|gss_state_history}} table which is causing the significant slowdowns.
The {{code|gss_state_log_view}} view is defined as:
{{highlight|lang=terminal|code=
SELECT (-1) * s.state_id AS id, s.cluster_id, ev.colour, ev.message_code,
    pg_catalog.concat_ws(','::text, en.entity_name, ('"'::text || replace(s.current_info::text, '"'::text, '\\"'::text)) || '"'::text, en.parent_name) AS arguments,
    s.event_time, s.entity_type, s.entity_id, en.entity_name,
    s.expired AS acknowledged, sens.sensor, sens.sensor_category
  FROM (        SELECT st.state_id, st.cluster_id, st.entity_type,
                    st.entity_id, st.sensor, st.previous_state,
                    st.previous_info, st.current_state, st.current_info,
                    st.event_time, 'N'::text AS expired
                  FROM fscc.gss_state st
        UNION ALL
                SELECT sh.state_id, sh.cluster_id, sh.entity_type,
                    sh.entity_id, sh.sensor, sh.previous_state,
                    sh.previous_info, sh.current_state, sh.current_info,
                    sh.event_time, 'Y'::text AS expired
                  FROM fscc.gss_state_history sh) s
  JOIN fscc.gss_state_events ev ON s.entity_type::text = ev.entity_type::text AND s.sensor::text = ev.sensor::text AND s.current_state::text = ev.state::text
  JOIN fscc.gss_entity_view en ON s.entity_type::text = en.entity_type AND s.entity_id = en.entity_id
  JOIN fscc.gss_sensor sens ON s.entity_type::text = sens.entity_type::text AND s.sensor::text = sens.sensor::text;
}}


Soo... I just deleted them:
I cleared out the {{code|gss_state_history}} table.
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2019-01-1' AND sensor = 'GPFS_STATE';
postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2019-01-1' AND sensor = 'GPFS_STATE';

Revision as of 18:21, 25 February 2020

Installation

GPFS packages are provided by the gpfs.* packages.

Components

IBM Spectrum Scale RAID is the front end GUI that is used to manage GPFS filesets, snapshots, quotas, and collect performance metrics.

Tivoli Storage Manager is the front end GUI and is used to configure backups and integration with the tape library.

Usage

All GPFS management utilities are located under the /usr/lpp/mmfs/bin/ directory and commands typically start with mm.

Command Description
mmccr flist lists files stored in cluster configuration repository (CCR)
mmccr lsnodes Shows all quorum nodes
mmshutdown -a Run on ems1 to shut down IBM cluster
mmstartup -a Run on ems1 to start up the IBM cluster
mmhealth node show Shows the health of all nodes
mmlsfs all Shows all file system attributes on the GPFS cluster.
mmlssnapshot gpfs0/1 will show all snapshots for a particular file system.
mmlsmount all_local shows whether the file system on this node is being mounted by other nodes in the cluster.
/usr/lpp/mmfs/bin/mmlspdisk all --not-ok shows all failed disks.
mmedquota -j gpfs1:fileset used to change a quota of a fileset on the GPFS1 volume.
mmcrfileset gpfs1 morph --inode-space new --inode-limit 4194304:2097152 change inode quota

The IBM Spectrum Scale RAID GUI utilities are at /usr/lpp/mmfs/gui/cli/. Certain actions taken on the web interface will trigger one of these executables.

Snapshots

Snapshot management is done using the following utilities:

Command Description
mmlssnapshot gpfs0 Lists all snapshots in the gpfs0 device
mmdelsnapshot gpfs0 snapshot_name -j fileset Deletes a snapshot in the gpfs0 device on a particular fileset by name.
mmcrsnapshot gpfs0 snapshot_name -j fileset Create a snapshot in the gpfs0 device for a particular fileset with the provided snapshot name.

Automated Snapshots

In CHGI, these commands are wrapped up in a custom perl script and configured with a configuration file. This script is then invoked daily with a root-user cronjob: 

5 23 * * * /usr/local/bin/admSnapshots.plx /usr/local/etc/admSnapshots.conf

Where the configuration file defines the fileset and number of snapshots to keep: 

ebg_gmb            3
ebg_web            3
qlong              3
snyder_irida       3

Tasks

Create Filesystem

mmcrfs gpfs1 -F vdisk.cfg -j scatter -T /gpfs/gpfs1 -B 16M --metadata-block-size 1M

Create a new GPFS filesystem on GPFS1, where vdisk.cfg contains the filesystem specifications. 

%vdisk: vdiskName=gssio1_Data_16M_2p_1 rg=gssio1 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio1_MetaData_16M_2p_1 rg=gssio1 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio1_Data_16M_2p_1:::dataOnly:30:gssio1_Data_16M_2p_1:data
gssio1_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio1_MetaData_16M_2p_1:::metadataOnly:30:gssio1_MetaData_16M_2p_1:system
gssio1_MetaData_16M_2p_1:::metadataOnly:30::system
%vdisk: vdiskName=gssio2_Data_16M_2p_1 rg=gssio2 da=DA1 blocksize=16M size=700T raidCode=8+2p diskUsage=dataOnly failureGroup=30  pool=data
%vdisk: vdiskName=gssio2_MetaData_16M_2p_1 rg=gssio2 da=DA1 blocksize=1M size=7T raidCode=3WayReplication diskUsage=metadataOnly failureGroup=30  pool=system
# %nsd:gssio2_Data_16M_2p_1:::dataOnly:30:gssio2_Data_16M_2p_1:data
gssio2_Data_16M_2p_1:::dataOnly:30::data
# %nsd:gssio2_MetaData_16M_2p_1:::metadataOnly:30:gssio2_MetaData_16M_2p_1:system
gssio2_MetaData_16M_2p_1:::metadataOnly:30::system

Node Management

Use mmlsnode to show all nodes that are part of a nodeset. 

[root@tsm /]# mmlsnode
GPFS nodeset    Node list
-------------   -------------------------------------------------------
   chgi-psc     essio1-ib essio2-ib ems1-ib crick-ib powerlc001-ib powerlc002-ib transmart-ib tsm-ib node001 node002 node003 node004 node005 node006 node007 node008 node009 node010 node011 node012 node013 node014 node015 node016 node017 node018 node019 node020 node021 node022 node023 node024 node025 node026 node027 node028 node029 node030 node031 node032 node035 node033 node036 node034 theia galaxy-dev-ib ebg01-ib snyder-ib

Additional information for each node can be found when listing cluster information with mmlscluster: 

[root@tsm /]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         chgi-psc.gpfs.net
  GPFS cluster id:           2878855853427990921
  GPFS UID domain:           chgi-psc.gpfs.net
  Remote shell command:      sudo wrapper in use
  Remote file copy command:  sudo wrapper in use
  Repository type:           CCR

 Node  Daemon node name        IP address      Admin node name         Designation
-----------------------------------------------------------------------------------
   1   essio1-ib.gpfs.net      172.26.3.1      essio1-ib.gpfs.net      quorum-manager-perfmon
   2   essio2-ib.gpfs.net      172.26.3.2      essio2-ib.gpfs.net      quorum-manager-perfmon
   3   ems1-ib.gpfs.net        172.26.3.251    ems1-ib.gpfs.net        quorum-perfmon
   4   crick-ib.gpfs.net       172.26.4.252    crick-ib.gpfs.net       
   5   powerlc001-ib.gpfs.net  172.26.4.1      powerlc001-ib.gpfs.net  
   6   powerlc002-ib.gpfs.net  172.26.4.2      powerlc002-ib.gpfs.net  
   7   transmart-ib.gpfs.net   172.26.4.3      transmart-ib.gpfs.net   
   8   tsm-ib.gpfs.net         172.26.3.250    tsm-ib.gpfs.net         
   9   node001                 172.26.10.1     node001                 
  ...
  50   node034                 172.26.10.34    node034                 
  51   theia                   172.26.25.2     theia                   
  56   galaxy-dev-ib           172.26.10.116   galaxy-dev-ib           
  58   ebg01-ib                172.26.10.153   ebg01-ib                
  60   snyder-ib               172.26.10.105   snyder-ib

To add a new node: 

# mmaddnode -N damona-ib

To remove a node: 

# mmdelnode -N damona-ib
Verifying GPFS is stopped on all affected nodes ...
mmdelnode: [W] Could not cleanup the following unreached nodes:
damona-ib
mmdelnode: Command successfully completed
mmdelnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.


Rebuild Shadow Database

The shadow database is a file used by mmbackup to quickly determine what files to back up. As per the documentation:

These databases shadow the inventory of objects in IBM Spectrum Protect so that only new changes will be backed up in the next incremental mmbackup. Failing to do so will needlessly back up some files additional times. The shadow database can also become out of date if mmbackup fails due to certain IBM Spectrum Protect server problems that prevent mmbackup from properly updating its shadow database after a backup. In these cases it is also required to issue the next mmbackup command with either the -q option or the --rebuild options.
—https://www.ibm.com/support/knowledgecenter/en/STXKQY_5.0.0/com.ibm.spectrum.scale.v5r00.doc/bl1adm_backupusingmmbackup.htm

To rebuild the shadow database, I had to use a separate temp directory because the system temp was too small. For some reason however, this process failed but still generated a 10GB shadow file. 

# mmbackup gpfs0 --rebuild -s /tmp2
--------------------------------------------------------
mmbackup: Shadow database rebuild of /gpfs begins at Wed Feb 19 09:35:36 MST 2020.
--------------------------------------------------------
Thu Feb 20 00:56:27 2020 mmbackup:Built query data file from TSM server: CHGI_TSM01 rc = 0
Thu Feb 20 00:56:28 2020 mmbackup:Scanning file system gpfs0
Thu Feb 20 04:55:32 2020 mmbackup:Reconstructing previous shadow file /gpfs/.mmbackupShadow.1.CHGI_TSM01.filesys from query data for CHGI_TSM01
mmbackup: tsbuhelper: parsePolicyShow: could not find mtime date
mmbackup: tsbuhelper: rebuildShadow: writing merged record fail rc = -1
mmbackup: tsbuhelper:rebuildShadow: Failed with rc=-1
Thu Feb 20 05:26:48 2020 mmbackup:/usr/lpp/mmfs/bin/tsbuhelper rebuildshadow /gpfs/.mmbackupQueryShadow.CHGI_TSM01.filesys /gpfs/.mmbackupCfg/prepFiles/list.mmbackup.1.CHGI_TSM01 /gpfs/.mmbackupShadow.1.CHGI_TSM01.filesys /tmp2 2>&1 returned rc=255.  Cannot rebuild shadow file.
Thu Feb 20 05:26:48 2020 mmbackup:Failed to reconstruct old shadow file.  Skipping CHGI_TSM01.
Thu Feb 20 05:26:48 2020 mmbackup:Done with shadow file database rebuilds
Thu Feb 20 05:26:48 2020 mmbackup:Incremental shadow database rebuild completely failed.
        TSM had 0 severe errors and returned 0. See the TSM log file for more information.
        0 files had errors,
 TSM exit status:  exit 12

----------------------------------------------------------
mmbackup: Shadow database rebuild of /gpfs completed with errors at Thu Feb 20 05:26:48 MST 2020.
----------------------------------------------------------
mmbackup: Command failed. Examine previous error messages to determine cause.

It was at 5:26 that the TSM server lost its membership to the GPFS filesystem for some reason which probably caused the failure...

Clear Stale Events on IBM Spectrum Scale RAID

To clear out old events, you will need to delete entries from the PostgreSQL server. On the server running the GUI, run: 

# sudo -u postgres psql
psql (9.2.7)
Type "help" for help.

postgres=# select * FROM pg_stat_activity;

SELECT * FROM fscc.gss_state WHERE event_time <= '2019-12-12' ;

Fix Data Collection

If data collection is not happening or no data is being reported on IBM Spectrum Scale RAID, ensure that the pmsensors service is running on all GPFS nodes. 

essio# systemctl restart pmsensors

If pmsensors isn't starting, or if the /opt/IBM/zimon/ZIMonSensors.cfg file is missing, ensure that the configuration file has the following contents. The colCandidates value should be set to the hostname of the GPFS management node for data collection. 

colCandidates = "synergy-ib"
colRedundancy = 1
collectors = {
        host = "synergy-ib"
        port = "4739"
}
config = "/opt/IBM/zimon/ZIMonSensors.cfg"
ctdbstat = ""
daemonize = T
hostname = ""
ipfixinterface = "0.0.0.0"
logfile = "/var/log/zimon/ZIMonSensors.log"
loglevel = "info"
mmcmd = "/opt/IBM/zimon/MMCmdProxy"
mmdfcmd = "/opt/IBM/zimon/MMDFProxy"
mmpmon = "/opt/IBM/zimon/MmpmonSockProxy"
piddir = "/var/run"
release = "4.2.0-2"
sensors = {
        name = "CPU"
        period = 1
},
{
        name = "Load"
        period = 1
},
{
        name = "Memory"
        period = 1
},
{
        name = "Network"
        period = 1
},
{
        name = "Netstat"
        period = 0
},
{
        name = "Diskstat"
        period = 0
},
{
        name = "DiskFree"
        period = 600
},
{
        name = "GPFSDisk"
        period = 0
},
{
        name = "GPFSFilesystem"
        period = 1
},
{
        name = "GPFSNSDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSPoolIO"
        period = 0
},
{
        name = "GPFSVFS"
        period = 1
},
{
        name = "GPFSIOC"
        period = 0
},
{
        name = "GPFSVIO"
        period = 0
},
{
        name = "GPFSPDDisk"
        period = 1
        restrict = "nsdNodes"
},
{
        name = "GPFSvFLUSH"
        period = 0
},
{
        name = "GPFSNode"
        period = 1
},
{
        name = "GPFSNodeAPI"
        period = 1
},
{
        name = "GPFSFilesystemAPI"
        period = 1
},
{
        name = "GPFSLROC"
        period = 0
},
{
        name = "GPFSCHMS"
        period = 0
},
{
        name = "GPFSAFM"
        period = 0
},
{
        name = "GPFSAFMFS"
        period = 0
},
{
        name = "GPFSAFMFSET"
        period = 0
},
{
        name = "GPFSRPCS"
        period = 0
},
{
        name = "GPFSFilesetQuota"
        period = 3600
},
{
        name = "GPFSDiskCap"
        period = 0
        restrict = "ems1-ib"
},
{
        name = "NFSIO"
        period = 0
        proxyCmd = "/opt/IBM/zimon/GaneshaProxy"
        restrict = "cesNodes"
        type = "Generic"
}
smbstat = ""

Troubleshooting

To troubleshoot issues with GPFS, you may want to take a look at the logs from mmfsd at:

  • /var/adm/ras/mmfs.log.*
  • /var/adm/ras/trace

Spectrum Scale logs can be found at:

  • /var/log/cnlog/mgtsrv/*log
  • /var/lib/mmfs/gui/preferences.xml


Cluster Manager connection broke

The TSM server kept on getting disconnected every morning. Logs show the following: 

Fri Feb 21 05:08:04.735 2020: [E] The TCP connection to IP address 172.26.3.2 essio2-ib.gpfs.net <c0n1> (socket 76) state is unexpected: ca_state=4 unacked=0 rto=210000
Fri Feb 21 05:08:04.749 2020: [I] tscCheckTcpConn: Sending debug data collection request to node 172.26.3.2 essio2-ib.gpfs.net
Fri Feb 21 05:08:04.750 2020: Sending request to collect TCP debug data to essio2-ib.gpfs.net localNode
Fri Feb 21 05:08:04.809 2020: [I] Calling user exit script gpfsSendRequestToNodes: event sendRequestToNodes, Async command /usr/lpp/mmfs/bin/mmcommon.
Fri Feb 21 05:17:55.685 2020: [E] Timed out in 300 seconds waiting for a commMsgCheckMessages reply from node 172.26.3.2 essio2-ib.gpfs.net. Sending expel message.
Fri Feb 21 05:17:55.716 2020: Sending request to collect expel debug data to essio2-ib.gpfs.net localNode
Fri Feb 21 05:17:55.721 2020: [I] Calling user exit script gpfsSendRequestToNodes: event sendRequestToNodes, Async command /usr/lpp/mmfs/bin/mmcommon.
Fri Feb 21 05:18:05.782 2020: [N] Request sent to 172.26.3.1 (essio1-ib.gpfs.net) to expel 172.26.3.2 (essio2-ib.gpfs.net) from cluster chgi-psc.gpfs.net
Fri Feb 21 05:18:05.783 2020: [N] This node will be expelled from cluster chgi-psc.gpfs.net due to expel msg from 172.26.3.250 (tsm-ib.gpfs.net)
Fri Feb 21 05:18:21.075 2020: [N] VERBS RDMA closed connection to 172.26.3.1 (essio1-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 1
Fri Feb 21 05:18:21.077 2020: [N] VERBS RDMA closed connection to 172.26.3.1 (essio1-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 3
Fri Feb 21 05:18:21.078 2020: [N] VERBS RDMA closed connection to 172.26.3.1 (essio1-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 2
Fri Feb 21 05:18:21.079 2020: [N] VERBS RDMA closed connection to 172.26.3.1 (essio1-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 11
Fri Feb 21 05:18:21.080 2020: [I] Cluster Manager connection broke. Probing cluster chgi-psc.gpfs.net
Fri Feb 21 05:18:21.112 2020: [N] VERBS RDMA closed connection to 172.26.3.2 (essio2-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 4
Fri Feb 21 05:18:21.114 2020: [N] VERBS RDMA closed connection to 172.26.3.2 (essio2-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 7
Fri Feb 21 05:18:21.115 2020: [N] VERBS RDMA closed connection to 172.26.3.2 (essio2-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 15
Fri Feb 21 05:18:21.116 2020: [N] VERBS RDMA closed connection to 172.26.3.2 (essio2-ib.gpfs.net) on mlx5_0 port 1 fabnum 0 index 14
Fri Feb 21 05:18:21.582 2020: [E] Unable to contact any quorum nodes during cluster probe.
Fri Feb 21 05:18:21.583 2020: [E] Lost membership in cluster chgi-psc.gpfs.net. Unmounting file systems.
Fri Feb 21 05:18:22.307 2020: [I] Calling user exit script mmUnmountFs: event unmount, Async command /usr/lpp/mmfs/lib/mmsysmon/sendRasEventToMonitor.
Fri Feb 21 05:18:22.719 2020: [I] The quota client for file system gpfs0 has terminated.
Fri Feb 21 05:18:22 MST 2020: [I] sendRasEventToMonitor: Successfully send a filesystem event to monitor
Fri Feb 21 05:18:23.270 2020: [N] Connecting to 172.26.3.1 essio1-ib.gpfs.net <c0p0>
Fri Feb 21 05:18:33.317 2020: [N] Connecting to 172.26.3.2 essio2-ib.gpfs.net <c0p1>
Fri Feb 21 05:18:43.327 2020: [N] Connecting to 172.26.3.251 ems1-ib.gpfs.net <c0p2>

You can check the timeout times using mmfsadm dump cfgmgr: 

# mmfsadm dump cfgmgr | grep failureDetectionTime
lease config: dynamic yes failureDetectionTime 35.0 usePR no recoveryWait 60 dmsTimeout 40
lease config: dynamic yes failureDetectionTime 0.0 usePR ? recoveryWait 60 dmsTimeout 40


Spectrum Scale Web Interface Slow

For some reason, the Spectrum Scale web UI is extremely slow. The server seems to be pegged by Postgres running nonstop: 

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                        
16624 postgres  20   0  184896  46208  33600 R  99.7  0.2   1:22.49 postgres                                                                                                                                       
16680 postgres  20   0  188352  50112  33088 R  99.7  0.2   0:52.04 postgres

Looking further, it's running a stupid looking SQL query: 

postgres=# select * FROM pg_stat_activity;
...

2020-02-24 18:22:08.600034-06 | f       | active              | SELECT l.*,l.acknowledged || translate(right(l.colour,1),'DWNY','1234') AS severity, m.text AS description,m.explanation, m.system_action, m.repair
_action, replace(replace(replace(m.text, '{0}', split_part(arguments, ',', 1)), '{1}', split_part(arguments, ',', 2)), '{2}', split_part(arguments, ',', 3)) as formatted_description FROM ( SELECT id,cluster_id,c
olour,message_code,arguments,event_time,entity_type,entity_id,entity_name,acknowledged,sensor,sensor_category FROM fscc.gss_state_log_view UNION ALL SELECT * FROM ( SELECT id,cluster_id,colour,message_code,argum
ents,event_time,ilog.entity_type,entity_id,entity_name,acknowledged,'',isens.sensor_category FROM fscc.gss_log AS ilog LEFT JOIN fscc.gss_sensor AS isens ON ilog.entity_type=isens.entity_type AND isens.sensor='D
EFAULT' ) AS i ) AS l JOIN fscc.messages AS m ON l.message_code=m.message_code AND m.locale=$1 WHERE (ACKNOWLEDGED = 'N') and ((COLOUR = 'RED') or (COLOUR = 'YELLOW'))  ORDER BY SEVERITY DESC OFFSET $2 LIMIT $3

Looking at a database dump of fscc, the largest table by far is gss_state_history as it has at least 900000 entries. 

10120:COPY gss_state_history (state_id, cluster_id, entity_type, entity_id, sensor, previous_state, previous_info, current_state, current_info, event_time) FROM stdin;
912165:COPY gss_vdisk (vdisk_id, cluster_id, vd_name, da_id, rg_id, raidcode, capacity, remarks, track_size, checksum_g) FROM stdin;

The gss_state_log_view view is defined as: 

SELECT (-1) * s.state_id AS id, s.cluster_id, ev.colour, ev.message_code, 
    pg_catalog.concat_ws(','::text, en.entity_name, ('"'::text

I cleared out the gss_state_history table. 

postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2019-01-1' AND sensor = 'GPFS_STATE';
 count  
--------
 391441
(1 row)

postgres=# DELETE FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2019-01-1' AND sensor = 'GPFS_STATE';
DELETE 391441
postgres=# SELECT COUNT(*) FROM fscc.gss_state_history;
 count  
--------
 510596
(1 row)

postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2019-01-1';
 count 
-------
  3420
(1 row)

postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2020-01-1';
 count  
--------
 434680
(1 row)

postgres=# SELECT COUNT(*) FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2020-01-1' AND sensor = 'GPFS_STATE';
 count  
--------
 422628
(1 row)

postgres=# DELETE FROM fscc.gss_state_history WHERE event_time >= '2018-01-01' AND event_time <= '2020-01-1' AND sensor = 'GPFS_STATE';
DELETE 422628

See Also