IBM Spectrum Protect: Difference between revisions

Revision as of 22:29, 5 February 2020

IBM Spectrum Protect, formerly known as Tivoli Storage Manager or TSM is used for managing backups.

Usage

Admin CLI is used to configure the TSM system. The TSM client CLI is used to list and restore files on the client system.

TSM Command Line Usage

The TSM command line utility can be invoked by running dsmc. The client interface will only show a subset of parameters that you will see in the admin console. It looks like it shows only values applicable to the host that it is running on (ie. filespaces, incl/excl options, etc.).

A nice guide: https://help.it.ox.ac.uk/hfs/help/dsmc

Here is what the TSM client interface looks like.

[root@tsm ~]# dsmc
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 7, Release 1, Level 6.5 
  Client date/time: 02/04/2020 16:01:52
(c) Copyright by IBM Corporation and other(s) 1990, 2017. All Rights Reserved. 

Node Name: TSM
Session established with server CHGI_TSM01: Linux/ppc64
  Server Version 7, Release 1, Level 7.200
  Server date/time: 02/04/2020 16:01:52  Last access: 02/04/2020 05:29:10

tsm>

List Include / Exclude Options

You can see all include/exclude with query inclexcl, which is similar to what is shown in the Admin Console when running query cloptset but only limited to this particular host.

tsm> q inclexcl
*** FILE INCLUDE/EXCLUDE ***
Mode Function  Pattern (match from top down)  Source File
---- --------- ------------------------------ -----------------
No exclude filespace statements defined.
Excl Directory /.../.TsmCacheDir              TSM
Exclude All       /gpfs/cbousman/.../*           Server
Exclude All       /.../.snapshots/.../*          Server
Exclude All       /tiered/vetmed_data/.../*      Server
Exclude All       /tiered/snyder_data/.../*      Server
...
Exclude All       /.../*.dbv                     Server
Include All       /gpfs/forever/.../*            Server
No DFS include/exclude statements defined.

List Files

To list all files that are available for restore:

tsm> query backup /path/to/file
           Size        Backup Date                Mgmt Class           A/I File
           ----        -----------                ----------           --- ----
         4,096  B  11/26/2019 06:43:38            INDEFINITE            A  /gpfs/home/lleung/.cache
         4,096  B  11/26/2019 06:43:54            INDEFINITE            A  /gpfs/home/lleung/.config
         4,096  B  11/26/2019 06:47:09            INDEFINITE            A  /gpfs/home/lleung/.dbus
         4,096  B  11/26/2019 12:06:04            INDEFINITE            A  /gpfs/home/lleung/.gnupg
...

Additional options:

-inactive to show previous versions of the files.
-subdir=yes to show sub directories

Restoring Files

To restore a file, use the restore source [destination] function. If destination is omitted, TSM will restore over the original location. TSM will restore the most recent version (active) of the file.

tsm> restore /gpfs/home/lleung/.zsh /gpfs/home/lleung/restore/
Restore function invoked.

Restoring           4,096 /gpfs/home/lleung/.zsh --> /gpfs/home/lleung/restore/.zsh [Done]

Restore processing finished.
                                  
Total number of objects restored:             1
Total number of objects failed:               0
Total number of bytes transferred:            0  B
Data transfer time:                        0.00 sec
Network data transfer rate:                0.00 KB/sec
Aggregate data transfer rate:              0.00 KB/sec
Elapsed processing time:               00:00:03

Other options:

-su=yes is required when restoring the entire directory.
-inactive -pick to restore an inactive file with option to pick which revision to restore.

Administrative Command Line Usage

The administrative TSM command line can be entered either from the web GUI via the 'Command Builder' or by running dsmadmc utility. In both cases, you must provide the TSM credentials.

You will be greeted with a prompt similar to the one below.

Session established with server CHGI_TSM01: Linux/ppc64
  Server Version 7, Release 1, Level 7.200
  Server date/time: 11/19/2019 14:18:43  Last access: 11/12/2019 17:15:04
  
tsm: CHGI_TSM01>

Commands and parameters can be shortened so long as it is not ambiguous. query volume can therefore be shortened to q vol, for example.

Additionally, SQL queries can be executed in this prompt which executes against the underlying DB2 database.

Documentation: https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.3/srv.reference/r_cmdline_adclient_options.html

SQL

To show all file spaces and their respective storage pool location when backed up:

tsm: CHGI_TSM01> select node_name, filespace_name, physical_mb, stgpool_name FROM occupancy WHERE stgpool_name = 'BACKUPTAPE'

     NODE_NAME: TSM
FILESPACE_NAME: /tiered
   PHYSICAL_MB: 686244989.70
  STGPOOL_NAME: BACKUPTAPE

     NODE_NAME: TSM
FILESPACE_NAME: /gpfs
   PHYSICAL_MB: 616886529.62
  STGPOOL_NAME: BACKUPTAPE

All entries listed here defines which storage pool backups go to. In the example above, /tiered and /gpfs will be stored on the BACKUPTAPE storage pool.

Query

The query command is used to obtain TSM objects.

Documentation available at https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/srv.reference/r_cmd_query.html

Tapes

To show all tapes in use, their capacity and usage, run query volume.

tsm: CHGI_TSM01> query volume
Volume Name	Storage Pool Name	Device Class Name	Estimated Capacity	Pct Util	Volume Status
A00091L7	BACKUPTAPE	LTO	127.339 G	98.291	Full
A00105L7	BACKUPTAPE	LTO	13.642 T	1.331	Filling
A00106L7	BACKUPTAPE	LTO	167.739 G	83.773	Full
A00107L7	BACKUPTAPE	LTO	203.538 G	97.105	Full

To show contents of a particular tape, run query content tapeid.

tsm: CHGI_TSM01> query content A00091L7

Node Name           Type     Filespace      FSID     Client's Name for File                
                             Name                    
---------------     ----     ----------     ----     --------------------------------------
TSM                 Bkup     /gpfs            10     /home/atyndall/.cache/mozilla/firefox/
                                                      td76tbw4.default/cache2/entries/929BC
                                                      F811537CE5A1B05BC367E7D5FCD9D1512C2  
TSM                 Bkup     /gpfs            10     /home/atyndall/.mozilla/firefox/td76tb
                                                      w4.default/.parentlock               
TSM                 Bkup     /gpfs            10     /home/atyndall/.mozilla/firefox/td76tb
                                                      w4.default/SiteSecurityServiceState.t
                                                      xt

Schedules

To show all configured backup schedules, run query schedules [format=detailed].

tsm: CHGI_TSM01> query schedules

Domain           *     Schedule Name        Action     Start Date/Time          Duration     Period     Day
------------     -     ----------------     ------     --------------------     --------     ------     ---
SERVERS                DAILY_INCR           Inc Bk     07/04/2017 18:00:00          1 H        1 D      Any
SERVERS                MMBACKUP             CMD        07/04/2017 02:10:00          1 H        1 D      Any
STANDARD               DAILY_INCR           Inc Bk     07/04/2017 18:00:00          1 H        1 D      Any

Side note: The detailed view is the same output as the TSM Client's output when running query schedule.

Backup Filespace

To show all partitions that are made available to be backed up, run query filespace. These are partitions that have the potential of being backed up based on schedules and policies but aren't necessarily backed up.

tsm: CHGI_TSM01> query filespace

Node Name           Filespace N     FSID     Platform     Filespace     Is Filespa        Capacity     Pct U
                    ame                                    Type         ce Unicode                       til
                                                                            ?                          
---------------     -----------     ----     --------     ---------     ----------     -----------     -----
CRICK.CHGI.UCAL     /                  1     LinuxPPC     EXT4              No              879 GB      24.9
 GARY.CA                                      64LE                                                          
TSM                 /tsminst1          1     LinuxPPC     EXT4              No           30,109 MB      52.9
                                              64                                                            
TSM                 /                  2     LinuxPPC     EXT4              No           50,268 MB      74.6
                                              64                                                            
TSM                 /boot              3     LinuxPPC     EXT4              No              468 MB      68.6
                                              64                                                            
TSM                 /tsmdb0            4     LinuxPPC     EXT4              No              196 GB      45.1
                                              64                                                            
TSM                 /tsmdb1            5     LinuxPPC     EXT4              No              196 GB      45.1
                                              64                                                            
TSM                 /tsmdb2            6     LinuxPPC     EXT4              No              196 GB      45.1
                                              64                                                            
TSM                 /tsmdb3            7     LinuxPPC     EXT4              No              196 GB      45.1
                                              64                                                            
TSM                 /tsmlog            8     LinuxPPC     EXT4              No              245 GB      57.0
                                              64                                                            
TSM                 /tsmarchlog        9     LinuxPPC     EXT4              No              492 GB       5.0
                                              64                                                            
TSM                 /gpfs             10     LinuxPPC     GPFS              No            1,292 TB      77.9
                                              64                                                            
TSM                 /tiered           14     LinuxPPC     GPFS              No            1,400 TB      92.8
                                              64

Scripts

Custom scripts are a set of SQL queries that can be executed using the run command. An example script to show tape information is given below.

tsm: CHGI_TSM01>query script tapes

Name                Description                                            Managing profile    
---------------     --------------------------------------------------     --------------------
TAPES               list information about tape usage                                          

tsm: CHGI_TSM01>query script tapes format=raw
select stgpool_name, count(*) as "Tapes" from volumes -
where devclass_name = 'LTO' group by stgpool_name
select count(*) as "Total Storage Pool Tapes" from volumes -
where devclass_name = 'LTO'
select count(*) as "Total Stg. Pool Tapes in Lib." from volumes -
where devclass_name='LTO' and volume_name in (select volume_name from libvolumes where library_name = 'CHGI_TSM')
select status, count(*) as "Tapes" from libvolumes -
where library_name = 'CHGI_TSM' group by status order by status desc
select count (*) as "Total LTO tapes in library" from libvolumes -
where library_name = 'CHGI_TSM'
issue message i "The following tapes are outside the library"
q media * stg=* wherestate=mountablenotinlib

Invoke the script with run scriptname:

tsm: CHGI_TSM01>run tapes  

STGPOOL_NAME                                Tapes
--------------------------------     ------------
BACKUPTAPE                                    273

 Total Storage Pool Tapes
-------------------------
                      273

 Total Stg. Pool Tapes in Lib.
------------------------------
                           272

STATUS                       Tapes
-----------------     ------------
Scratch                         82
Private                        283

 Total LTO tapes in library
---------------------------
                        365
ANR1496I The following tapes are outside the library

Volume N     State                          Location                Automated LibNa
ame                                                                 me             
--------     --------------------------     -------------------     ---------------
A00121L7     Mountable not in library       ?                                      
ANR1462I RUN: Command script TAPES completed successfully.

Option Set and Client Options

Backups can include or exclude specific files based on Option Set and its associated Client Options.

Use the TSM Administrative CLI to define a new Option Set (define cloptset) and new Client Options (define clientopt optionset option value [seq = N]). Use the INCLEXCL option to define what files to include or exclude. The ordering of these rules can be changed by specifying a sequence number.

An example Option Set and its list of options.

tsm: CHGI_TSM01>query cloptset

Optionset                     Description                   Last Update by      Managing profile         Replica Option 
                                                            (administrator)                              Set            
-------------------------     -------------------------     ---------------     --------------------     ---------------
CRICK                                                       JOERG                                        No             

Option                        Sequence     Use Option     Option Value                                                
                                number      Set Value     
                                              (FORCE)     
-------------------------     --------     ----------     ------------------------------------------------------------
DOMAIN                               0             No     all-local -/gpfs                                            
INCLEXCL                             0             No     Exclude /unix/                                              
INCLEXCL                             1             No     Exclude /.../core                                           
INCLEXCL                             2             No     Exclude.dir /unix/                                          
INCLEXCL                             3             No     Include /gpfs/forever/.../* INDEFINITE                      
RESOURCEUTILIZATION                  0             No     5                                                           
SCHEDMODE                            0             No     prompted                                                    
TXNBYTELIMIT                         0             No     2G

The INCLEXCL directive defines what files to include or exclude. To exclude all files under any directory named 'temp' on Windows, we could use the following client option. Note the *:\ to match any drive, ...\ to match any number of directories, and * to match any file.

tsm: CHGI_TSM01> define clientopt WINDOWS inclexcl "Exclude *:\...\temp\...\*"
ANR2050I DEFINE CLIENTOPT: Option INCLEXCL defined in optionset WINDOWS.

To exclude a specific directory on Linux, we define a similar rule as above but without a drive letter and with forward slashes instead. Similarily, .../ to match any number of directories and * to match any files.

tsm: CHGI_TSM01> define clientopt TSM_IX inclexcl "Exclude /gpfs/home/sbagheri/Human_genome_mapping/.../*"
ANR2050I DEFINE CLIENTOPT: Option INCLEXCL defined in optionset TSM_IX.

Automated Backups

There are two scheduled tasks defined:

Execution of /usr/local/bin/mmbackup.ksh at 2:10AM, daily
An incremental backup at 6:00PM, daily.

It looks like /usr/local/bin/mmbackup.ksh runs mmbackup which renders the client options into a policy file at /var/mmfs/mmbackup/.mmbackupRules.gpfs0 and then executes mmapplypolicy that uses this policy file to do the actual backups. A pstree of the entire command execution looks like:

mmbackup.ksh(77050)───mmbackup(77054)───tsbackup33(77192)───sh(77314)───mmapplypolicy(77315)───tsapolicy(77387)─┬─tsapolicy(77394)─┬─{tsapolicy}(77396)
...
                                                                                                                │                  └─{tsapolicy}(77605)
                                                                                                                ├─{tsapolicy}(77395)
...

tsapolicy generates a list of files that requires to be backed up and will then generate a file list that will then get passed to dsmc selective -filelist=/gpfs/.mmbackupCfg/mmbackupChanged..... -servername=CHGI_TSM01 -verbose -subdir=no that does the actual backups. It looks like file statuses are saved on the DB2 server, so near the end of the process where file states need to be updated, the database server will be extremely busy.

I'm unsure if the incremental backup at 6PM actually does anything then, if backups are done daily using the script...

An example client option set and backup policy is provided below.

tsm> query cloptset
ANS1138E The 'QUERY' command must be followed by a subcommand
tsm> query inclexcl
Session established with server CHGI_TSM01: Linux/ppc64
  Server Version 7, Release 1, Level 7.200
  Server date/time: 02/05/2020 11:19:28  Last access: 02/05/2020 05:32:30

*** FILE INCLUDE/EXCLUDE ***
Mode Function  Pattern (match from top down)  Source File
---- --------- ------------------------------ -----------------
No exclude filespace statements defined.
Excl Directory /.../.TsmCacheDir              TSM
Exclude All       /gpfs/cbousman/.../*           Server
Exclude All       /.../.snapshots/.../*          Server
Exclude All       /tiered/vetmed_data/.../*      Server
Exclude All       /tiered/snyder_data/.../*      Server
Exclude All       /tiered/mtgraovac/.../*        Server
Exclude All       /tiered/kkurek/.../*           Server
Exclude All       /tiered/danderson/.../*        Server
Exclude All       /tiered/achri_data/.../*       Server
Exclude All       /tiered/ewang/.../*            Server
Exclude All       /tiered/morph/.../*            Server
Exclude All       /tiered/ewang_scratch/.../*    Server
Exclude All       /gpfs/gallo/.../*              Server
Exclude All       /gpfs/charb_data/.../*         Server
Exclude All       /gpfs/vetmed_stage/.../*       Server
Exclude All       /gpfs/vetmed_data/.../*        Server
Exclude All       /gpfs/snyder_work/.../*        Server
Exclude All       /gpfs/ebg_work/.../*           Server
Exclude All       /gpfs/qlong/.../*              Server
Exclude All       /.../tmp/.../*                 Server
Exclude All       /.../core                      Server
Exclude All       /tsmarchlog/.../*              Server
Exclude All       /tsmlog/.../*                  Server
Exclude All       /tsmdb*/.../*                  Server
Exclude All       /.../*.dsm                     Server
Exclude All       /.../*.dbv                     Server
Include All       /gpfs/forever/.../*            Server
No DFS include/exclude statements defined.

/* Auto-generated GPFS policy rules file

 * Generated on Wed Feb  5 02:10:14 2020
 */

/*   Server rules for backup server 1
 ***   CHGI_TSM01   ***
 */
RULE EXTERNAL LIST 'mmbackup.1.CHGI_TSM01' EXEC '/gpfs/.mmbackupCfg/BAexecScript.gpfs0' OPTS '"/gpfs/.mmbackupShadow.1.CHGI_TSM01.filesys.update" "-servername=CHGI_TSM01"  "-auditlogname=/gpfs/mmbackup.audit.gpfs0.CHGI_TSM01" "NONE"' 
RULE 'BackupRule' LIST 'mmbackup.1.CHGI_TSM01' DIRECTORIES_PLUS 
     SHOW(VARCHAR(MODIFICATION_TIME) || ' ' || VARCHAR(CHANGE_TIME) || ' ' ||
          VARCHAR(FILE_SIZE)         || ' ' || VARCHAR(FILESET_NAME) ||
          ' ' || 'resdnt' )
     WHERE  
      ( 
        NOT 
        ( (PATH_NAME LIKE '/%/.mmbackup%') OR 
          (PATH_NAME LIKE '/%/.mmLockDir' AND MODE LIKE 'd%') OR 
          (PATH_NAME LIKE '/%/.mmLockDir/%') OR 
          (MODE LIKE 's%') 
        ) 
      ) 
      AND  
        (MISC_ATTRIBUTES LIKE '%u%')
      AND 
      (
        NOT 
        ( (PATH_NAME LIKE '/%/.SpaceMan' AND MODE LIKE 'd%') OR 
          (PATH_NAME LIKE '/%/.SpaceMan/%')
        )
      )
     AND (NOT (PATH_NAME LIKE '/%/.TsmCacheDir' AND MODE LIKE 'd%') AND NOT (PATH_NAME LIKE '/%/.TsmCacheDir/%')) 
     AND  (NOT (PATH_NAME LIKE '/gpfs/cbousman/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/%/.snapshots/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/vetmed_data/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/snyder_data/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/mtgraovac/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/kkurek/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/danderson/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/achri_data/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/ewang/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/morph/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tiered/ewang_scratch/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/gallo/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/charb_data/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/vetmed_stage/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/vetmed_data/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/snyder_work/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/ebg_work/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/gpfs/qlong/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/%/tmp/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/%/core' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tsmarchlog/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tsmlog/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/tsmdb%/%' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/%.dsm' AND NOT MODE LIKE 'd%'))
     AND  (NOT (PATH_NAME LIKE '/%.dbv' AND NOT MODE LIKE 'd%'))
     OR ((PATH_NAME LIKE '/gpfs/forever/%') AND  (MISC_ATTRIBUTES LIKE '%u%'))

Database Information

TSM requires DB2. In CHGI, the database files are stored in /tsmdb0, /tsmdb1, /tsmdb2, /tsmdb3, /tsmlog, each about 200GB in size. The database space can be listed in the TSM command line by running:

tsm: CHGI_TSM01>q db                                                                                                          

Database Name       Total Pages     Usable Pages       Used Pages       Free Pages
--------------     ------------     ------------     ------------     ------------
TSMDB1               14,806,035       14,802,931       14,023,207          779,724

tsm: CHGI_TSM01> query dbspace
Location                           Total Space of      Used Space on F      Free Space(MB)
                                   File System (MB     ile System (MB)     
                                                 )                         
------------------------------     ---------------     ---------------     ---------------
/tsmdb0                                 201,459.91           90,837.06          110,622.85
/tsmdb1                                 201,459.91           90,836.96          110,622.95
/tsmdb2                                 201,459.91           90,837.04          110,622.87
/tsmdb3                                 201,459.91           90,837.00          110,622.91

tsm: CHGI_TSM01>q log

Total Space(MB)      Used Space(MB)      Free Space(MB)
---------------     ---------------     ---------------
        131,072               1,119             129,953

@@ Line 327: / Line 327: @@
 }}
-tsapolicy generates a list of files that requires to be backed up and will then generate a file list that will then get passed to {{code|dsmc selective -filelist=/gpfs/.mmbackupCfg/mmbackupChanged..... -servername=CHGI_TSM01 -verbose -subdir=no}} that does the actual backups. It looks like file statuses are saved on the DB2 server, so near the end of the process where file states need to be updated, the database server will be extremely busy.
+tsapolicy generates a list of files that requires to be backed up and will then generate a file list that will then get passed to {{code|1=dsmc selective -filelist=/gpfs/.mmbackupCfg/mmbackupChanged..... -servername=CHGI_TSM01 -verbose -subdir=no}} that does the actual backups. It looks like file statuses are saved on the DB2 server, so near the end of the process where file states need to be updated, the database server will be extremely busy.
 I'm unsure if the incremental backup at 6PM actually does anything then, if backups are done daily using the script...