IBM Spectrum Archive
Usage
The ltfs* binaries are typically installed at /opt/ibm/ltfsee/bin/ and are provided by the ltfs-*.rpm RPM packages.
You will most likely interface with LTFS using the ltfsee command.
LTFS Library Status
[root@ltfs ~]# ltfsee status
Ctrl Node MD MMM Library
172.26.3.249 Active Active T_ARCH
Tapes
To show all tape drives:
[root@ltfs ~]# ltfsee info drives
Drive S/N Status Type Role Library Address Node ID Tape Node Group
0007807A4B In use LTO8 mrg T_ARCH 260 4 T00031L8 G0
0007807A0B Mounted LTO8 mrg T_ARCH 261 4 - G0
000780765B In use LTO8 mrg T_ARCH 262 4 T00054L8 G0
To show all tapes in the library, use ltfsee info tapes. Tape status description available at https://www.ibm.com/support/knowledgecenter/ST9MBR_1.2.6/ltfs_ee_ltfsee_info_tapes.html.
[root@ltfs ~]# ltfsee info tapes
Tape ID Status Type Capacity(GiB) Used(GiB) Free(GiB) Reclaimable(GiB) Pool Library Address Drive Appendable
T00000L8 Valid L8 10907 0 10907 0 TIERED1 T_ARCH 1161 - yes
T00001L8 Valid L8 10907 0 10907 0 TIERED1 T_ARCH 1160 - yes
T00003L8 Valid L8 10907 0 10907 0 TIERED1 T_ARCH 1158 - yes
T00004L8 Valid L8 10907 0 10907 0 TIERED1 T_ARCH 1157 - yes
Tape Types
Tapes that end with L8 are LTO-8.
If drives are in use, you can list all jobs to see what is actually being done.
[root@ltfs ~]# ltfsee info jobs
Job Type Status Idle(sec) Scan ID Tape Pool Library Node File Name or inode
Reclaim(Source) In-progress 573811 2944737025 T00031L8 TIERED1 T_ARCH 4 -
Reclaim(Target) In-progress 573683 2944737281 T00054L8 TIERED1 T_ARCH 4 -
Validate Unscheduled 4469 1602429441 T00054L8 TIERED1 T_ARCH - -
To add tapes to a particular pool, use ltfsee pool add -p pool -t tape ...:
ltfsee pool add -p TIERED1 -t $tape -t $tape2 ...
To remove tapes from a particular pool, use ltfsee pool remove -p pool -t tape ...:
ltfsee pool remove -p TIERED1 -t $tape -r
TODO: When removing, how does that affect replicas?
If a tape is in a drive, you can move it back to its homeslot using ltfsee tape move homeslot -t tape -p pool:
# ltfsee tape move homeslot -t T00037L8 -p TIERED1
GLESL373I(00890): Moving tape T00037L8.
Tape T00037L8 is unmounted because it is inserted into the drive.
Tape T00037L8 is moved successfully.
Alternatively, use ieslot to move it to the IO port of the tape library for extraction.
LTFS Information
ltfsee info pools shows all pools.
Pool Name Total(TiB) Used(TiB) Free(TiB) Reclaimable(TiB) Tapes Type Library Node Group
TIERED1 644.3 84.6 559.7 0.0 61 LTO T_ARCH G0
TIERED2 638.9 80.8 558.1 0.0 61 LTO T_ARCH G0
ltfsee info libraries shows all libraries with LTFS.
[root@ltfs ~]# ltfsee info libraries
Library Name Status Model Serial Number Ctrl Node
T_ARCH Active 03584L32 0000078BA6130402 172.26.3.249
Show Tiered Files
When files are tiered to tape, the full path of the file is stored in a database as 'migrated'. You can find a file's tiered tape cartridge using ltfsee info files -f filepath. Eg:
Files that are 'migrated' or tiered off to tape. Files not migrated are 'resident' and resides on spinning disk.
Example output below shows tape ID @ library for files migrated
# ltfsee info files -f /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR302*
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021220_2.fastq
Tape id:- Status: resident
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021242_1.fastq
Tape id:T00042L8@T_ARCH:T00102L8@T_ARCH Status: migrated
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021288_1.fastq
Tape id:T00055L8@T_ARCH:T00107L8@T_ARCH Status: migrated
Side Notes on Migrated Files
Migrated files will have a stub file with zero size. To count sizes withdu, you need to use --apparent-size Eg.
[root@node001 xuexu]# du -sh dbgap_UK_OTTO
23T dbgap_UK_OTTO
[root@node001 xuexu]# du -sh --apparent-size dbgap_UK_OTTO
35T dbgap_UK_OTTO
[root@node001 dbgap_UK_OTTO]# du SRR3021288_2.fastq
0 SRR3021288_2.fastq
[root@node001 dbgap_UK_OTTO]# du -sh --apparent-size SRR3021288_2.fastq
15G SRR3021288_2.fastq
Troubleshooting
Logging
LTFS logs are located at /var/log/ltfsee.log.
Additional logs worth investigating:
/opt/tivoli/tsm/client/hsm/bin/dsmerror.log
The reclamation process failed
Two tape drives have tapes in it but doesn't appear to be doing anything. Looking at /var/log/ltfsee.log, we can see that the reclamation process failed. The T00037L8 tape was in a drive that stopped functioning, showing a '5' in the single character display with the cartridge ejected.
Possibly bad tape drive that caused the job to fail, which subsequently locked everything up?
2020-01-20T10:23:05.514522-07:00 ltfs reclaim_target[10652]: GLESA112E(00590): The following command failed with (rc:256:1) : /bin/cp /ltfs/T00037L8/.LTFSEE_DATA/10991691794470275686-15679318773138264748-78339434-18551699-0 /ltfs/T00022L8/.LTFSEE_DATA 2>&1.
2020-01-20T10:23:14.925509-07:00 ltfs reclaim_target[10652]: GLESA112E(00590): The following command failed with (rc:256:1) : /bin/cp /ltfs/T00037L8/.LTFSEE_DATA/10991691794470275686-15679318773138264748-78339434-18551699-0 /ltfs/T00022L8/.LTFSEE_DATA 2>&1.
2020-01-20T10:23:14.925818-07:00 ltfs reclaim_target[10652]: GLESR035E(01182): The copy process from source to destination tape failed for the file 10991691794470275686-15679318773138264748-78339434-18551699-0.
2020-01-20T10:23:14.926141-07:00 ltfs reclaim_target[10652]: GLESR004E(01930): Processing file /tiered/ewang/xuexu/dbgap_tcga_germline/SRR3341182_SRR3341183_varscan.pileup failed: exiting the reclamation driver.
2020-01-20T10:23:14.926413-07:00 ltfs reclaim_target[10652]: GLESR026E(00158): The reclamation process failed (1932).#012 Have a look for previous messages.
2020-01-20T10:23:14.927489-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_TGTT00022L8 failed.
2020-01-20T10:23:15.501824-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 1447695617 have been successful: 1 failed.
2020-01-20T10:23:15.502533-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_SRCT00037L8 failed.
2020-01-20T10:23:16.130885-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 1442649089 have been successful: 1 failed.
2020-01-20T10:23:16.132052-07:00 ltfs ltfsee[14335]: GLESL082E(01590): Reclamation failed while reclaiming tape T00037L8 to target tape T00022L8.
2020-01-26T05:00:02.516999-07:00 ltfs ltfsee[23826]: GLESL668E(00979): Unable to get the state of tape T00030L8. Skip to reclaim. Consult the log files. (rc=1040)
2020-01-26T05:00:02.823888-07:00 ltfs ltfsee[23826]: GLESL682E(01028): Tape with ID: T00030L8 is an invalid state. Source tapes must be in state either "Valid LTFS" or "Warning".
2020-01-26T05:04:36.512781-07:00 ltfs reclaim_target[31377]: GLESR030E(00178): The reclamation process failed. (1655)#012 Have a look for previous messages.
2020-01-26T05:04:36.513142-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_TGTT00029L8 failed.
2020-01-26T05:04:36.620168-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 2135234305 have been successful: 1 failed.
The two drives that are still in use are idling for the past week. Interestingly, ltfsee log did not show any information for 31L8 and 54L8 tapes.
[root@ltfs ~]# ltfsee info jobs
Job Type Status Idle(sec) Scan ID Tape Pool Library Node File Name or inode
Reclaim(Source) In-progress 576566 2944737025 T00031L8 TIERED1 T_ARCH 4 -
Reclaim(Target) In-progress 576438 2944737281 T00054L8 TIERED1 T_ARCH 4 -
Validate Unscheduled 7224 1602429441 T00054L8 TIERED1 T_ARCH - -
The job cannot be stopped. Force a LTFS stop using ltfsee stop -f and wait. The tapes drives should eventually eject the tapes. I got the following error when stopping, but the tapes got ejected and everything seemed to have stopped.
[root@ltfs ~]# ltfsee stop -f
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running stop command - sending request and waiting for the completion.
GLESL030E(00909): Unable to connect to the MMM service. Check whether the IBM Spectrum Archive EE has been started.
GLESL358E(00494): Error on processing tape T00054L8 (1).
GLESL661E(00104): IPC got failure result (result=1).
GLESL646E(00164): Unable to stop the IBM Spectrum Archive EE monitor daemon for library T_ARCH.
Cannot Start LTFS
[root@ltfs ~]# /opt/ibm/ltfsee/bin/ltfsee start
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running start command - sending request : T_ARCH.
Running start command - waiting for completion : T_ARCH.
...
GLESL657E(00191): Fail to start the IBM Spectrum Archive EE service (MMM) for library T_ARCH.
Use the 'ltfsee info nodes' command to see the error modules.
The monitor daemon will start the recovery sequence.
[root@ltfs ~]# ltfsee info nodes
Spectrum Archive EE service (MMM) for library T_ARCH fails to start or is not running on ltfs-ib.gpfs.net Node ID:4
Problem Detected:
Node ID Error Modules
4 MMM;
Looking at /var/log/ltfsee.log, we see:
2020-02-03T13:15:35.266926-07:00 ltfs mmm[31142]: GLESM709E(00369): Assign tape (T00099L8) command error: 172.26.3.249:7600 (4): Request Error (070E): [Cartridge.cc:61]: Cartridge add is failed: 7c05 LTFSI1079E The operation is not allowed.
Seemed to have started up by itself?
[root@ltfs ~]# /opt/ibm/ltfsee/bin/ltfsee start
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
GLESL519I(00344): The IBM Spectrum Archive EE service (ltfsee_md) for library T_ARCH is already running.
[root@ltfs ~]# ltfsee info nodes
Node ID Status Node IP Drives Ctrl Node Library Node Group Host Name
4 Available 172.26.3.249 3 yes(active) T_ARCH G0 ltfs-ib.gpfs.net
Not sure what happened there...
Invalid Tapes
[root@ltfs ~]# ltfsee info tapes
Tape ID Status Type Capacity(GiB) Used(GiB) Free(GiB) Reclaimable(GiB) Pool Library Address Drive Appendable
T00030L8 Invalid L8 10907 10895 0 22 TIERED1 T_ARCH 1131 - no
T00099L8 Invalid L8 10907 10883 0 0 TIERED2 T_ARCH 1031 - no
According to IBM's documentation:
The Invalid status indicates that the cartridge is inconsistent with the LTFS format.To check and repair this tape before you add it to a tape storage pool, use the ltfsee pool add command with the check option.
In order to add the tape back into the pool, it has to be removed first.
[root@ltfs ~]# ltfsee pool add -p TIERED2 -t T00099L8 -c
GLESL042I(00894): Adding tape T00099L8 to storage pool TIERED2.
GLESL346E(00949): No tapes are found in the unassigned list. T00099L8 is already assigned to another storage pool, is in an IE slot, or does not exist in the library.
[root@ltfs ~]# ltfsee pool remove -p TIERED2 -t T00099L8
GLESL043I(01134): Removing tape T00099L8 from storage pool TIERED2.
Removed tape T00099L8 from pool TIERED2 successfully.
[root@ltfs ~]# ltfsee pool add -p TIERED2 -t T00099L8 -c
GLESL042I(00894): Adding tape T00099L8 to storage pool TIERED2.
Tape T00099L8 successfully checked.
Added tape T00099L8 to pool TIERED2 successfully.
Drive status is 'locating' and ltfsee info jobs show the recovery job idling...
[root@ltfs ~]# ltfsee info jobs
Job Type Status Idle(sec) Scan ID Tape Pool Library Node File Name or inode
Recovery Scheduled 694 2181628673 T00099L8 TIERED2 T_ARCH 4 -
but this is normal.
Tasks
Cronjobs
The following cronjob is installed.
# Cron Job Used to Move Data Via policy to LTFS Tape at 1:00 AM
00 1 * * * /mmpolicies/movetogpfs1.sh
# Cron Job Used to Move Data Via policy to LTFS Tape at 3:00 AM
00 3 * * * /mmpolicies/movetogpfs0.sh
# Cron Job Used to Reconcile disk data with LTFS Tape data Pool TIERED1
00 20 * * * /opt/ibm/ltfsee/bin/ltfsee reconcile -p TIERED1 -l T_ARCH -g /tiered
# Cron Job Used to Reconcile disk data with LTFS Tape data Pool TIERED2
00 22 * * * /opt/ibm/ltfsee/bin/ltfsee reconcile -p TIERED2 -l T_ARCH -g /tiered
# Cron Job Used to Reclaim LTFS Tape data in Pool TIERED1
00 5 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED1 -l T_ARCH
# Cron Job Used to Reclaim LTFS Tape Data in Pool TIERED2
00 7 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED2 -l T_ARCH