IBM Spectrum Protect: Difference between revisions
| Line 540: | Line 540: | ||
}} | }} | ||
I set the access mode to readonly, then initiated another reclamation with {{code|reclaim stgpool backuptape}}. Tape library complained of a few errors from the tape drive and the access state went back to "Unavailable" on TSM. | |||
I moved the tape to the IO port with: | |||
{{highlight|lang=terminal|code= | |||
tsm: CHGI_TSM01>move media c00013l7 stgpool=backuptape remove=yes | |||
ANR0609I MOVE MEDIA started as process 360. | |||
}} | |||
=== Reclamation Issues === | === Reclamation Issues === | ||
Revision as of 23:59, 11 February 2020
IBM Spectrum Protect, formerly known as Tivoli Storage Manager or TSM is used for managing backups.
Usage
Admin CLI is used to configure the TSM system. The TSM client CLI is used to list and restore files on the client system.
TSM Command Line Usage
The TSM command line utility can be invoked by running dsmc. The client interface will only show a subset of parameters that you will see in the admin console. It looks like it shows only values applicable to the host that it is running on (ie. filespaces, incl/excl options, etc.).
A nice guide: https://help.it.ox.ac.uk/hfs/help/dsmc
Here is what the TSM client interface looks like.
[root@tsm ~]# dsmc
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
Client Version 7, Release 1, Level 6.5
Client date/time: 02/04/2020 16:01:52
(c) Copyright by IBM Corporation and other(s) 1990, 2017. All Rights Reserved.
Node Name: TSM
Session established with server CHGI_TSM01: Linux/ppc64
Server Version 7, Release 1, Level 7.200
Server date/time: 02/04/2020 16:01:52 Last access: 02/04/2020 05:29:10
tsm>
List Include / Exclude Options
You can see all include/exclude with query inclexcl, which is similar to what is shown in the Admin Console when running query cloptset but only limited to this particular host.
tsm> q inclexcl
*** FILE INCLUDE/EXCLUDE ***
Mode Function Pattern (match from top down) Source File
---- --------- ------------------------------ -----------------
No exclude filespace statements defined.
Excl Directory /.../.TsmCacheDir TSM
Exclude All /gpfs/cbousman/.../* Server
Exclude All /.../.snapshots/.../* Server
Exclude All /tiered/vetmed_data/.../* Server
Exclude All /tiered/snyder_data/.../* Server
...
Exclude All /.../*.dbv Server
Include All /gpfs/forever/.../* Server
No DFS include/exclude statements defined.
List Files
To list all files that are available for restore:
tsm> query backup /path/to/file
Size Backup Date Mgmt Class A/I File
---- ----------- ---------- --- ----
4,096 B 11/26/2019 06:43:38 INDEFINITE A /gpfs/home/lleung/.cache
4,096 B 11/26/2019 06:43:54 INDEFINITE A /gpfs/home/lleung/.config
4,096 B 11/26/2019 06:47:09 INDEFINITE A /gpfs/home/lleung/.dbus
4,096 B 11/26/2019 12:06:04 INDEFINITE A /gpfs/home/lleung/.gnupg
...
Additional options:
-inactiveto show previous versions of the files.-subdir=yesto show sub directories
Restoring Files
To restore a file, use the restore source [destination] function. If destination is omitted, TSM will restore over the original location. TSM will restore the most recent version (active) of the file.
tsm> restore /gpfs/home/lleung/.zsh /gpfs/home/lleung/restore/
Restore function invoked.
Restoring 4,096 /gpfs/home/lleung/.zsh --> /gpfs/home/lleung/restore/.zsh [Done]
Restore processing finished.
Total number of objects restored: 1
Total number of objects failed: 0
Total number of bytes transferred: 0 B
Data transfer time: 0.00 sec
Network data transfer rate: 0.00 KB/sec
Aggregate data transfer rate: 0.00 KB/sec
Elapsed processing time: 00:00:03
Other options:
-su=yesis required when restoring the entire directory.-inactive -pickto restore an inactive file with option to pick which revision to restore.
Administrative Command Line Usage
The administrative TSM command line can be entered either from the web GUI via the 'Command Builder' or by running dsmadmc utility. In both cases, you must provide the TSM credentials.
You will be greeted with a prompt similar to the one below.
Session established with server CHGI_TSM01: Linux/ppc64
Server Version 7, Release 1, Level 7.200
Server date/time: 11/19/2019 14:18:43 Last access: 11/12/2019 17:15:04
tsm: CHGI_TSM01>
Commands and parameters can be shortened so long as it is not ambiguous. query volume can therefore be shortened to q vol, for example.
Additionally, SQL queries can be executed in this prompt which executes against the underlying DB2 database.
Documentation: https://www.ibm.com/support/knowledgecenter/SSGSG7_7.1.3/srv.reference/r_cmdline_adclient_options.html
SQL
To show all file spaces and their respective storage pool location when backed up:
tsm: CHGI_TSM01> select node_name, filespace_name, physical_mb, stgpool_name FROM occupancy WHERE stgpool_name = 'BACKUPTAPE'
NODE_NAME: TSM
FILESPACE_NAME: /tiered
PHYSICAL_MB: 686244989.70
STGPOOL_NAME: BACKUPTAPE
NODE_NAME: TSM
FILESPACE_NAME: /gpfs
PHYSICAL_MB: 616886529.62
STGPOOL_NAME: BACKUPTAPE
All entries listed here defines which storage pool backups go to. In the example above, /tiered and /gpfs will be stored on the BACKUPTAPE storage pool.
Query
The query command is used to obtain TSM objects.
Documentation available at https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/srv.reference/r_cmd_query.html
Tapes
To show all tapes in use, their capacity and usage, run query volume.
tsm: CHGI_TSM01> query volume
Volume Name Storage Pool Name Device Class Name Estimated Capacity Pct Util Volume Status
A00091L7 BACKUPTAPE LTO 127.339 G 98.291 Full
A00105L7 BACKUPTAPE LTO 13.642 T 1.331 Filling
A00106L7 BACKUPTAPE LTO 167.739 G 83.773 Full
A00107L7 BACKUPTAPE LTO 203.538 G 97.105 Full
To show contents of a particular tape, run query content tapeid.
tsm: CHGI_TSM01> query content A00091L7
Node Name Type Filespace FSID Client's Name for File
Name
--------------- ---- ---------- ---- --------------------------------------
TSM Bkup /gpfs 10 /home/atyndall/.cache/mozilla/firefox/
td76tbw4.default/cache2/entries/929BC
F811537CE5A1B05BC367E7D5FCD9D1512C2
TSM Bkup /gpfs 10 /home/atyndall/.mozilla/firefox/td76tb
w4.default/.parentlock
TSM Bkup /gpfs 10 /home/atyndall/.mozilla/firefox/td76tb
w4.default/SiteSecurityServiceState.t
xt
Schedules
To show all configured backup schedules, run query schedules [format=detailed].
tsm: CHGI_TSM01> query schedules
Domain * Schedule Name Action Start Date/Time Duration Period Day
------------ - ---------------- ------ -------------------- -------- ------ ---
SERVERS DAILY_INCR Inc Bk 07/04/2017 18:00:00 1 H 1 D Any
SERVERS MMBACKUP CMD 07/04/2017 02:10:00 1 H 1 D Any
STANDARD DAILY_INCR Inc Bk 07/04/2017 18:00:00 1 H 1 D Any
Side note: The detailed view is the same output as the TSM Client's output when running query schedule.
Backup Filespace
To show all partitions that are made available to be backed up, run query filespace. These are partitions that have the potential of being backed up based on schedules and policies but aren't necessarily backed up.
tsm: CHGI_TSM01> query filespace
Node Name Filespace N FSID Platform Filespace Is Filespa Capacity Pct U
ame Type ce Unicode til
?
--------------- ----------- ---- -------- --------- ---------- ----------- -----
CRICK.CHGI.UCAL / 1 LinuxPPC EXT4 No 879 GB 24.9
GARY.CA 64LE
TSM /tsminst1 1 LinuxPPC EXT4 No 30,109 MB 52.9
64
TSM / 2 LinuxPPC EXT4 No 50,268 MB 74.6
64
TSM /boot 3 LinuxPPC EXT4 No 468 MB 68.6
64
TSM /tsmdb0 4 LinuxPPC EXT4 No 196 GB 45.1
64
TSM /tsmdb1 5 LinuxPPC EXT4 No 196 GB 45.1
64
TSM /tsmdb2 6 LinuxPPC EXT4 No 196 GB 45.1
64
TSM /tsmdb3 7 LinuxPPC EXT4 No 196 GB 45.1
64
TSM /tsmlog 8 LinuxPPC EXT4 No 245 GB 57.0
64
TSM /tsmarchlog 9 LinuxPPC EXT4 No 492 GB 5.0
64
TSM /gpfs 10 LinuxPPC GPFS No 1,292 TB 77.9
64
TSM /tiered 14 LinuxPPC GPFS No 1,400 TB 92.8
64
Scripts
Custom scripts are a set of SQL queries that can be executed using the run command. A list of all scripts can be found with query script. To show the contents of a particular script, run query script name format=raw.
List a particular script:
tsm: CHGI_TSM01>query script tapes
Name Description Managing profile
--------------- -------------------------------------------------- --------------------
TAPES list information about tape usage
Commands of a particular script:
tsm: CHGI_TSM01>query script tapes format=raw
select stgpool_name, count(*) as "Tapes" from volumes -
where devclass_name = 'LTO' group by stgpool_name
select count(*) as "Total Storage Pool Tapes" from volumes -
where devclass_name = 'LTO'
select count(*) as "Total Stg. Pool Tapes in Lib." from volumes -
where devclass_name='LTO' and volume_name in (select volume_name from libvolumes where library_name = 'CHGI_TSM')
select status, count(*) as "Tapes" from libvolumes -
where library_name = 'CHGI_TSM' group by status order by status desc
select count (*) as "Total LTO tapes in library" from libvolumes -
where library_name = 'CHGI_TSM'
issue message i "The following tapes are outside the library"
q media * stg=* wherestate=mountablenotinlib
Invoke the script with run scriptname:
tsm: CHGI_TSM01>run tapes
STGPOOL_NAME Tapes
-------------------------------- ------------
BACKUPTAPE 273
Total Storage Pool Tapes
-------------------------
273
Total Stg. Pool Tapes in Lib.
------------------------------
272
STATUS Tapes
----------------- ------------
Scratch 82
Private 283
Total LTO tapes in library
---------------------------
365
ANR1496I The following tapes are outside the library
Volume N State Location Automated LibNa
ame me
-------- -------------------------- ------------------- ---------------
A00121L7 Mountable not in library ?
ANR1462I RUN: Command script TAPES completed successfully.
Event
Query event to show events that have happened recently. Useful to see if backups are running successfully.
tsm: CHGI_TSM01>query event * * begindate=-1 enddate=today format=detailed
Policy Domain Name Schedule Name Node Name Scheduled Start Actual Start Completed Status Result Reason
------------------------------ ------------------------------ -------------------- -------------------- -------------------- -------------------- --------------- --------------- ---------------------
SERVERS MMBACKUP TSM 02/04/2020 02:10:00 02/04/2020 02:10:06 02/04/2020 05:29:10 Completed 0 All operations comple
ted successfully.
SERVERS DAILY_INCR TSM 02/04/2020 18:00:00 02/04/2020 18:00:07 02/04/2020 18:12:29 Completed 4 The operation complet
ed successfully, but
some files were not
processed.
SERVERS MMBACKUP TSM 02/05/2020 02:10:00 02/05/2020 02:10:08 02/05/2020 05:32:30 Completed 0 All operations comple
ted successfully.
SERVERS DAILY_INCR TSM 02/05/2020 18:00:00 Future
Option Set and Client Options
Backups can include or exclude specific files based on Option Set and its associated Client Options.
Use the TSM Administrative CLI to define a new Option Set (define cloptset) and new Client Options (define clientopt optionset option value [seq = N]). Use the INCLEXCL option to define what files to include or exclude. The ordering of these rules can be changed by specifying a sequence number.
An example Option Set and its list of options.
tsm: CHGI_TSM01>query cloptset
Optionset Description Last Update by Managing profile Replica Option
(administrator) Set
------------------------- ------------------------- --------------- -------------------- ---------------
CRICK JOERG No
Option Sequence Use Option Option Value
number Set Value
(FORCE)
------------------------- -------- ---------- ------------------------------------------------------------
DOMAIN 0 No all-local -/gpfs
INCLEXCL 0 No Exclude /unix/
INCLEXCL 1 No Exclude /.../core
INCLEXCL 2 No Exclude.dir /unix/
INCLEXCL 3 No Include /gpfs/forever/.../* INDEFINITE
RESOURCEUTILIZATION 0 No 5
SCHEDMODE 0 No prompted
TXNBYTELIMIT 0 No 2G
The INCLEXCL directive defines what files to include or exclude. To exclude all files under any directory named 'temp' on Windows, we could use the following client option. Note the *:\ to match any drive, ...\ to match any number of directories, and * to match any file.
tsm: CHGI_TSM01> define clientopt WINDOWS inclexcl "Exclude *:\...\temp\...\*"
ANR2050I DEFINE CLIENTOPT: Option INCLEXCL defined in optionset WINDOWS.
To exclude a specific directory on Linux, we define a similar rule as above but without a drive letter and with forward slashes instead. Similarily, .../ to match any number of directories and * to match any files.
tsm: CHGI_TSM01> define clientopt TSM_IX inclexcl "Exclude /gpfs/home/sbagheri/Human_genome_mapping/.../*"
ANR2050I DEFINE CLIENTOPT: Option INCLEXCL defined in optionset TSM_IX.
See Also:
- https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/srv.reference/r_cmd_cloptset_define.html
- https://www.ibm.com/support/knowledgecenter/SSEQVQ_8.1.9/srv.reference/r_cmd_clientopt_define.html
- Inclexcl matching help - https://kb.wisc.edu/helpdesk/page.php?id=25684
Automated Backups
There are two scheduled tasks defined:
- Execution of /usr/local/bin/mmbackup.ksh at 2:10AM, daily
- An incremental backup at 6:00PM, daily.
It looks like /usr/local/bin/mmbackup.ksh runs mmbackup which renders the client options into a policy file at /var/mmfs/mmbackup/.mmbackupRules.gpfs0 and then executes mmapplypolicy that uses this policy file to do the actual backups. A pstree of the entire command execution looks like:
mmbackup.ksh(77050)───mmbackup(77054)───tsbackup33(77192)───sh(77314)───mmapplypolicy(77315)───tsapolicy(77387)─┬─tsapolicy(77394)─┬─{tsapolicy}(77396)
...
│ └─{tsapolicy}(77605)
├─{tsapolicy}(77395)
...
tsapolicy generates a list of files that requires to be backed up and will then generate a file list that will then get passed to dsmc selective -filelist=/gpfs/.mmbackupCfg/mmbackupChanged..... -servername=CHGI_TSM01 -verbose -subdir=no that does the actual backups. It looks like file statuses are saved on the DB2 server, so near the end of the process where file states need to be updated, the database server will be extremely busy.
I'm unsure if the incremental backup at 6PM actually does anything then, if backups are done daily using the script...
An example client option set and backup policy is provided below.
tsm> query cloptset
ANS1138E The 'QUERY' command must be followed by a subcommand
tsm> query inclexcl
Session established with server CHGI_TSM01: Linux/ppc64
Server Version 7, Release 1, Level 7.200
Server date/time: 02/05/2020 11:19:28 Last access: 02/05/2020 05:32:30
*** FILE INCLUDE/EXCLUDE ***
Mode Function Pattern (match from top down) Source File
---- --------- ------------------------------ -----------------
No exclude filespace statements defined.
Excl Directory /.../.TsmCacheDir TSM
Exclude All /gpfs/cbousman/.../* Server
Exclude All /.../.snapshots/.../* Server
Exclude All /tiered/vetmed_data/.../* Server
Exclude All /tiered/snyder_data/.../* Server
Exclude All /tiered/mtgraovac/.../* Server
Exclude All /tiered/kkurek/.../* Server
Exclude All /tiered/danderson/.../* Server
Exclude All /tiered/achri_data/.../* Server
Exclude All /tiered/ewang/.../* Server
Exclude All /tiered/morph/.../* Server
Exclude All /tiered/ewang_scratch/.../* Server
Exclude All /gpfs/gallo/.../* Server
Exclude All /gpfs/charb_data/.../* Server
Exclude All /gpfs/vetmed_stage/.../* Server
Exclude All /gpfs/vetmed_data/.../* Server
Exclude All /gpfs/snyder_work/.../* Server
Exclude All /gpfs/ebg_work/.../* Server
Exclude All /gpfs/qlong/.../* Server
Exclude All /.../tmp/.../* Server
Exclude All /.../core Server
Exclude All /tsmarchlog/.../* Server
Exclude All /tsmlog/.../* Server
Exclude All /tsmdb*/.../* Server
Exclude All /.../*.dsm Server
Exclude All /.../*.dbv Server
Include All /gpfs/forever/.../* Server
No DFS include/exclude statements defined.
|
Database Information
TSM requires DB2. In CHGI, the database files are stored in /tsmdb0, /tsmdb1, /tsmdb2, /tsmdb3, /tsmlog, each about 200GB in size. The database space can be listed in the TSM command line by running:
tsm: CHGI_TSM01>q db
Database Name Total Pages Usable Pages Used Pages Free Pages
-------------- ------------ ------------ ------------ ------------
TSMDB1 14,806,035 14,802,931 14,023,207 779,724
tsm: CHGI_TSM01> query dbspace
Location Total Space of Used Space on F Free Space(MB)
File System (MB ile System (MB)
)
------------------------------ --------------- --------------- ---------------
/tsmdb0 201,459.91 90,837.06 110,622.85
/tsmdb1 201,459.91 90,836.96 110,622.95
/tsmdb2 201,459.91 90,837.04 110,622.87
/tsmdb3 201,459.91 90,837.00 110,622.91
tsm: CHGI_TSM01>q log
Total Space(MB) Used Space(MB) Free Space(MB)
--------------- --------------- ---------------
131,072 1,119 129,953
Troubleshooting
Tape Issue
From the TSM web GUI, looking at events under the TSM server, I saw the following:
Feb 11, 2020, 12:28:28 AM ANR8944E Hardware or media error on drive DRIVE4 (/dev/IBMtape4) with volume C00013L7(OP=READ, Error Number= 5, CC=0, KEY=03, ASC=11, ASCQ=00, SENSE=F0.00.03.00.04.00.00.58.00.00.00.00.11.00.36.00.70.60.1F.47.04.01.43.30.30.30.31.33.4C.0A.00.00.83.40.CE.00.00.01.05.CE.80.08.60.4C.37.00.3D.BB.A6.00.01.90.CC.70.60.1F.47.70.60.1F.47.70.60.1F.47.70.60.00.01.15.60.03.E0.00.50.8C.00.B8.04.D1.00.00.3F.27.54.01.4D.48.31.52.58.31.55.57.44.36, Description=An undetermined error has occurred). Refer to the IBM Tivoli Storage Manager documentation on I/O error code descriptions. (PROCESS: 351)
Feb 11, 2020, 12:28:28 AM ANR8359E Media fault detected on LTO volume C00013L7 in drive DRIVE4 (/dev/IBMtape4) of library CHGI_TSM. (PROCESS: 351)
The tape library also logged an error with this cartridge:
Type: Cartridge
Location: Frame 1, Column 4, Row 3
State:
Time: Mon, Feb 10, 2020 11:11:27 PM MST
User: Service
Description: Cartridge C00013L7 could not be read from because of faulty media or a faulty drive.
Error Code: 0005
The volume was automatically set to 'unavailable' by TSM:
tsm: CHGI_TSM01>query vol C00013L7 format=detailed
Volume Name: C00013L7
Storage Pool Name: BACKUPTAPE
Device Class Name: LTO
Estimated Capacity: 7.2 T
Scaled Capacity Applied:
Pct Util: 0.3
Volume Status: Full
Access: Unavailable
Pct. Reclaimable Space: 99.7
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 66
Write Pass Number: 1
Approx. Date Last Written: 08/04/2019 02:53:45
Approx. Date Last Read: 02/11/2020 00:28:28
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 3
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 08/02/2019 12:36:02
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: IBM Tivoli Storage Manager
Logical Block Protected: No
I set the access mode to readonly, then initiated another reclamation with reclaim stgpool backuptape. Tape library complained of a few errors from the tape drive and the access state went back to "Unavailable" on TSM.
I moved the tape to the IO port with:
tsm: CHGI_TSM01>move media c00013l7 stgpool=backuptape remove=yes
ANR0609I MOVE MEDIA started as process 360.
Reclamation Issues
When triggering a reclamation, I got the following messages:
tsm: CHGI_TSM01>reclaim stgpool backuptape
ANR3638W Space reclamation skipped volume C00266L7 because the spanned volume C00285L7 in storage pool BACKUPTAPE is inaccessible.
ANR3638W Space reclamation skipped volume C00234L7 because the spanned volume C00231L7 in storage pool BACKUPTAPE is inaccessible.
ANR3638W Space reclamation skipped volume A00144L7 because the spanned volume A00140L7 in storage pool BACKUPTAPE is inaccessible.
ANR2110I RECLAIM STGPOOL started as process 359.
ANR4930I Reclamation process 359 started for primary storage pool BACKUPTAPE manually, threshold=60, duration=None.
ANS8003I Process number 359 started.
The tape's status is set to unavailable for some reason:
tsm: CHGI_TSM01>query vol c00285l7 format=detailed
Volume Name: C00285L7
Storage Pool Name: BACKUPTAPE
Device Class Name: LTO
Estimated Capacity: 13.6 T
Scaled Capacity Applied:
Pct Util: 2.3
Volume Status: Filling
Access: Unavailable
Pct. Reclaimable Space: 2.4
Scratch Volume?: Yes
In Error State?: Yes
Number of Writable Sides: 1
Number of Times Mounted: 1
Write Pass Number: 1
Approx. Date Last Written: 10/02/2019 06:29:24
Approx. Date Last Read: 10/01/2019 19:53:09
Date Became Pending:
Number of Write Errors: 1
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator):
Last Update Date/Time: 10/01/2019 19:49:01
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: IBM Tivoli Storage Manager
Logical Block Protected: No
I think these tapes may have been bad or the drive had issues reading it. Perhaps try the reclamation process again after setting the access to readonly using UPDATE VOL tapeid access=readonly.
See Also
- Overview of rention policies and human readable definition of terms in TSM - https://oit.duke.edu/about/policies/tsm-backup-retention-policies
- TSM overview - https://wiki.fysik.dtu.dk/it/TSM-server-configuration