IBM Spectrum Archive: Difference between revisions

From Leo's Notes
This page was last edited on 3 February 2020, at 19:28.
← Older editNewer edit →
VisualWikitext
Created page with " == Usage == The {{code|ltfs*}} binaries are typically installed at {{code|/opt/ibm/ltfsee/bin/}} and are provided by the {{code|ltfs-*.rpm}} RPM packages. You will most lik..."
Line 1: Line 1:
IBM Spectrum Archive Enterprise Edition (EE) can be used to tier data from a GPFS storage pool to a Tape Library.  Files stored on tape can be accessed like any other files stored on spinning disk via LTFS.
 


== Usage ==
== Usage ==
Use the {{code|ltfsee}} command to interact with Spectrum Archive.  Other binaries related to LTFS can be found at {{code|/opt/ibm/ltfsee/bin/}}.
The {{code|ltfs*}} binaries are typically installed at {{code|/opt/ibm/ltfsee/bin/}} and are provided by the {{code|ltfs-*.rpm}} RPM packages.
 
You will most likely interface with LTFS using the {{code|ltfsee}} command.


=== Spectrum Archive Status ===
{{code|ltfsee info libraries}} shows all libraries.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info libraries
Library Name  Status  Model    Serial Number    Ctrl Node 
T_ARCH        Active  03584L32  0000078BA6130402  172.26.3.249
}}


Shows whether MD and MMM are active.
=== LTFS Library Status ===
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee status
[root@ltfs ~]# ltfsee status
Line 19: Line 14:
}}
}}


{{code|ltfsee info pools}} shows all pools.
=== Tapes ===
{{highlight|lang=terminal|code=
All tape related usage.
Pool Name  Total(TiB)  Used(TiB)  Free(TiB)  Reclaimable(TiB)  Tapes Type  Library  Node Group
TIERED1        644.3      84.6      559.7              0.0    61  LTO  T_ARCH  G0       
TIERED2        638.9      80.8      558.1              0.0    61  LTO  T_ARCH  G0       
}}


=== Tapes ===
To show all drives:
==== List Tape Drives ====
To show all tape drives:
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info drives
[root@ltfs ~]# ltfsee info drives
Line 37: Line 26:
}}
}}


If drives are in use, you can list all in-progress jobs to see what task is using the drive.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info jobs
Job Type        Status      Idle(sec)  Scan ID    Tape      Pool    Library  Node  File Name or inode
Reclaim(Source)  In-progress    573811  2944737025  T00031L8  TIERED1  T_ARCH      4  -               
Reclaim(Target)  In-progress    573683  2944737281  T00054L8  TIERED1  T_ARCH      4  -               
Validate        Unscheduled      4469  1602429441  T00054L8  TIERED1  T_ARCH      -  -               
}}
A detailed list of drive status can be found at https://www.ibm.com/support/knowledgecenter/ST9MBR_1.2.3/ltfs_ee_ltfsee_info_drives.html#ltfs_ee_ltfsee_info_drives
==== List Tape Cartridges ====
To show all tapes in the library, run {{code|ltfsee info tapes}}.  Common tape status codes are:
{| class="wikitable"
! Status Code
! Description
|-
| Critical
| The Critical status indicates that an attempt to write to this cartridge failed.
To avoid data loss, recover the data on this cartridge by using the ltfsee recover command, then discard this cartridge.
|-
| Error
| The Error status indicates that the cartridge has an error and cannot be recovered.
This cartridge cannot be used with the IBM Spectrum Archive Enterprise Edition.
|-
| Invalid
| The Invalid status indicates that the cartridge is inconsistent with the LTFS format.
To check and repair this tape before you add it to a tape storage pool, use the ltfsee pool add command with the check option.
|-
| Unavailable
| The Unavailable status indicates that the cartridge is not available in the IBM Spectrum Archive Enterprise Edition system.
Tapes that are newly inserted into the tape library have an Unavailable status. To recall files from a tape with this status, first import the tape by using the ltfsee import command. To add a tape with this status to the IBM Spectrum Archive Enterprise Edition system:
First, move the tape to a home slot by using the ltfsee tape move command.
Then, add the tape to a tape storage pool by using the ltfsee pool command.
|-
| Valid
| The Valid status indicates that the cartridge is valid.
|}


Other tape statuses and descriptions can be found at:
{{code|ltfsee info tapes}} shows all tapes all libraries.
* https://www.ibm.com/support/knowledgecenter/ST9MBR_1.2.6/ltfs_ee_ltfsee_info_tapes.html


{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
Line 87: Line 37:
T00004L8  Valid        L8            10907          0      10907                0  TIERED1  T_ARCH  1157    -          yes       
T00004L8  Valid        L8            10907          0      10907                0  TIERED1  T_ARCH  1157    -          yes       
}}
}}
<!--
{{Info|1=Tape Types|2=Tapes that end with L8 are LTO-8.}}
{{Info|1=Tape Types|2=Tapes that end with L8 are LTO-8.}}
-->


==== Managing Tape Cartridges ====
If drives are in use, you can list all jobs to see what is actually being done.
To add tapes to a particular pool, use {{code|ltfsee pool add -p pool -t tape ...}}:
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee pool add -p TIERED2 -t T00099L8
[root@ltfs ~]# ltfsee info jobs
GLESL042I(00894): Adding tape T00099L8 to storage pool TIERED2.
Job Type        Status      Idle(sec)  Scan ID    Tape      Pool    Library  Node  File Name or inode
Added tape T00099L8 to pool TIERED2 successfully.
Reclaim(Source)  In-progress    573811  2944737025  T00031L8  TIERED1  T_ARCH      4  -                
Reclaim(Target) In-progress    573683  2944737281  T00054L8  TIERED1  T_ARCH      4  -               
Validate        Unscheduled      4469  1602429441  T00054L8  TIERED1  T_ARCH      -  -               
}}
}}


To remove tapes from a particular pool, use {{code|ltfsee pool remove -p pool -t tape ...}}.  All active files stored on the tape must first be recalled.  This can only be done if the data on the tape is recovered using the {{code|ltfsee recover}} command.  Files on the tape to be removed has to be in the "resident" state before LTFS allows you to remove the tape.
To add tapes to a particular pool, use {{code|ltfsee pool add -p pool -t tape ...}}:
{{code|ltfsee pool add -p TIERED1 -t $tape -t $tape2 ...}}


If Spectrum Archive is confused about the state of the file (eg. a file is premigrated, and is neither resident or migrated) and a {{code|ltfsee repair}} isn't capable of fixing the status or if you don't care about the data that's on the tape, you can force remove with the {{code|-r}} force remove flag.
To remove tapes from a particular pool, use {{code|ltfsee pool remove -p pool -t tape ...}}:
{{Warning
{{code|ltfsee pool remove -p TIERED1 -t $tape -r}}  
|1=Removing a tape already in tape drive
 
|2=If you force remove a tape that is spooled in a drive, Spectrum Archive will 'forget' that it's in a drive once the tape is removed. The tape will be unloaded but won't be moved to any slots. You must manually move it using the Tape Library interface.
 
}}
TODO: When removing, how does that affect replicas?


=== LTFS Information ===
{{code|ltfsee info pools}} shows all pools.
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
# ltfsee pool remove -p TIERED1 -t T00037L8
Pool Name  Total(TiB)  Used(TiB) Free(TiB) Reclaimable(TiB) Tapes  Type  Library  Node Group
GLESL043I(01134): Removing tape T00037L8 from storage pool TIERED1.
TIERED1         644.3      84.6      559.7              0.0    61  LTO  T_ARCH  G0       
GLESL357E(01223): Tape T00037L8 has migrated files or saved files. It has not been removed from the pool.
TIERED2        638.9      80.8      558.1              0.0    61  LTO  T_ARCH  G0       
# ltfsee pool remove -p TIERED1 -t T00037L8 -r
}}
GLESL043I(01134): Removing tape T00037L8 from storage pool TIERED1.
GLESL361W(01199): Tape T00037L8 was removed from pool TIERED1 forcefully. Cannot recall files on tape T00037L8. Add the tape to the same pool again if you need to recall files from the tape without formatting.


# ltfsee pool remove -p TIERED1 -t T00030L8
GLESL043I(01134): Removing tape T00030L8 from storage pool TIERED1.
Removed tape T00030L8 from pool TIERED1 successfully.
}}


If a tape is in a drive, you can move it back to its homeslot using {{code|ltfsee tape move homeslot -t tape -p pool}}. Alternatively, use {{code|ieslot}} to move it to the IO slot of the tape library for extraction.
{{code|ltfsee info libraries}} shows all libraries with LTFS.  
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
# ltfsee tape move homeslot -t T00037L8 -p TIERED1
[root@ltfs ~]# ltfsee info libraries
GLESL373I(00890): Moving tape T00037L8.
Library Name  Status  Model    Serial Number    Ctrl Node 
Tape T00037L8 is unmounted because it is inserted into the drive.
T_ARCH        Active  03584L32  0000078BA6130402  172.26.3.249
Tape T00037L8 is moved successfully.
}}
}}


=== Show Tiered Files ===
=== Show Tiered Files ===
A file is '''migrated''' when the file contents have been copied to one or more tape and its original GPFS location is turned into a file stub. All files that are only stored on GPFS are '''resident'''. Files that are in the process of being migrated, where the data has been copied to the LTFSEE storage and is waiting to be copied to tape are '''pre-migrated'''.
When files are tiered to tape, the full path of the file is stored in a database as 'migrated'. You can find a file's tiered tape cartridge using {{code|ltfsee info files -f filepath}}. Eg:
 
Files that are 'migrated' or tiered off to tape. Files not migrated are 'resident' and resides on spinning disk.


File statuses can be viewed with {{code|ltfsee info files -f filepath}}. The location (or locations if multiple copies were made) is given as the tape ID and library name under each file entry.
Example output below shows tape ID @ library for files migrated
{{highlight|lang=terminal|code=
{{highlight|lang=terminal|code=
# ltfsee info files -f /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR302*
# ltfsee info files -f /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR302*
Line 161: Line 109:
}}
}}


To find all files that are stored on a particular tape, the easiest way is to look at the {{code|.schema}} metadata files for the library stored in {{code|/tiered/.ltfsee/meta/$library-id/volume_cache/*.schema}}.


For example, files stored on tape {{code|T00037L8}} can be found this way:
{{highlight|lang=terminal|code=
# cd /tiered/.ltfsee/meta/0000078BA6130402/volume_cache
# grep gpfs.path -A 1 T00037L8.schema {{!}} grep value
<value>/tiered/ewang/xuexu/TCGA_bam_germline_12_4/f02e8ee1-349f-42ff-9ed9-ac37435901fa/cdb2f568f3cbdff5a9cd2edd692613bb_gdc_realn.bam</value>
<value>/tiered/ewang/xuexu/TCGA_bam_germline_12_4/TCGA-A2-A0D2_1.fastq</value>
...
}}


=== Movement Policies ===
File movements are defined and applied by {{code|mmapplypolicy}}. A scheduled job invoking {{code|mmapplypolicy}} daily can be used to tier files on a regular basis.
The primary policy is on the LTFS server located at {{code|/mmpolicies/mmpolicyLATEST.txt}} and are applied with a daily cronjob that invokes:
{{highlight|lang=terminal|code=
# mmapplypolicy /dev/gpfs1 -P /mmpolicies/mmpolicyLATEST.txt >/dev/null 2>&1
# mmapplypolicy /dev/gpfs0 -P /mmpolicies/mmpolicyLATEST.txt >/dev/null 2>&1
}}
The policy file contains the rules that govern which files are moved off to tape. LTFS/GPFS related files, specific filesystem logs, Space Manager(?) should not be migrated.
To prevent tiering of small amounts of data which could cause wear loading/unloading tape excessively, tiering can also be configured to happen only if filesystem usage exceeds 90% and will attempt to lower filesystem usage down to 80% as defined by the {{code|THRESHOLD}} values. Small files can also be ignored with the {{code|FILE_SIZE}} condition.
{{highlight|lang=text|code=
define(user_exclude_list,(PATH_NAME LIKE '/ibm/gpfs/.ltfsee/%' OR PATH_NAME LIKE'/ibm/gpfs/.SpaceMan/%' OR NAME LIKE 'dsmerror.log'))
define(user_include_list,(PATH_NAME LIKE '/tiered/%'))
define(is_premigrated,(MISC_ATTRIBUTES LIKE '%M%' AND MISC_ATTRIBUTES NOT LIKE'%V%'))
define(is_migrated,(MISC_ATTRIBUTES LIKE '%V%'))
define(is_resident,(NOT MISC_ATTRIBUTES LIKE '%M%'))
RULE 'DATA_POOL_PLACEMENT_RULE' SET POOL 'data'
RULE EXTERNAL POOL 'LTFSEE_FILES'
EXEC '/opt/ibm/ltfsee/bin/ltfsee'
OPTS '-p TIERED1 TIERED2'
RULE 'LTFSEE_FILES_RULE' MIGRATE FROM POOL 'data'
THRESHOLD(90,80)
TO POOL 'LTFSEE_FILES'
WHERE FILE_SIZE > 1048576
AND (CURRENT_TIMESTAMP - ACCESS_TIME > INTERVAL '525600' MINUTES )
AND (CURRENT_TIMESTAMP - MODIFICATION_TIME > INTERVAL '525600' MINUTES )
AND is_resident OR is_premigrated
AND NOT user_exclude_list
AND user_include_list
}}


== Troubleshooting ==
== Troubleshooting ==
Line 215: Line 119:
Additional logs worth investigating:
Additional logs worth investigating:
* {{code|/opt/tivoli/tsm/client/hsm/bin/dsmerror.log}}
* {{code|/opt/tivoli/tsm/client/hsm/bin/dsmerror.log}}
=== {{code|The reclamation process failed}} ===
Two tape drives have tapes in it but doesn't appear to be doing anything. Looking at {{code|/var/log/ltfsee.log}}, we can see that the reclamation process failed. The {{code|T00037L8}} tape was in a drive that stopped functioning, showing a '5' in the single character display with the cartridge ejected.
{{Info
|1=Addendum: Bad Tape?
|2=After a week, the same error occurred again with the same tape cartridge. It's most likely that the {{code|T00037L8}} tape cartridge is faulty.
}}
{{highlight|lang=text|code=
2020-01-20T10:23:05.514522-07:00 ltfs reclaim_target[10652]: GLESA112E(00590): The following command failed with (rc:256:1) : /bin/cp /ltfs/T00037L8/.LTFSEE_DATA/10991691794470275686-15679318773138264748-78339434-18551699-0 /ltfs/T00022L8/.LTFSEE_DATA 2>&1.
2020-01-20T10:23:14.925509-07:00 ltfs reclaim_target[10652]: GLESA112E(00590): The following command failed with (rc:256:1) : /bin/cp /ltfs/T00037L8/.LTFSEE_DATA/10991691794470275686-15679318773138264748-78339434-18551699-0 /ltfs/T00022L8/.LTFSEE_DATA 2>&1.
2020-01-20T10:23:14.925818-07:00 ltfs reclaim_target[10652]: GLESR035E(01182): The copy process from source to destination tape failed for the file 10991691794470275686-15679318773138264748-78339434-18551699-0.
2020-01-20T10:23:14.926141-07:00 ltfs reclaim_target[10652]: GLESR004E(01930): Processing file /tiered/ewang/xuexu/dbgap_tcga_germline/SRR3341182_SRR3341183_varscan.pileup failed: exiting the reclamation driver.
2020-01-20T10:23:14.926413-07:00 ltfs reclaim_target[10652]: GLESR026E(00158): The reclamation process failed (1932).#012          Have a look for previous messages.
2020-01-20T10:23:14.927489-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_TGTT00022L8 failed.
2020-01-20T10:23:15.501824-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 1447695617 have been successful: 1 failed.
2020-01-20T10:23:15.502533-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_SRCT00037L8 failed.
2020-01-20T10:23:16.130885-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 1442649089 have been successful: 1 failed.
2020-01-20T10:23:16.132052-07:00 ltfs ltfsee[14335]: GLESL082E(01590): Reclamation failed while reclaiming tape T00037L8 to target tape T00022L8.
2020-01-26T05:00:02.516999-07:00 ltfs ltfsee[23826]: GLESL668E(00979): Unable to get the state of tape T00030L8. Skip to reclaim. Consult the log files. (rc=1040)
2020-01-26T05:00:02.823888-07:00 ltfs ltfsee[23826]: GLESL682E(01028): Tape with ID: T00030L8 is an invalid state. Source tapes must be in state either "Valid LTFS" or "Warning".
2020-01-26T05:04:36.512781-07:00 ltfs reclaim_target[31377]: GLESR030E(00178): The reclamation process failed. (1655)#012          Have a look for previous messages.
2020-01-26T05:04:36.513142-07:00 ltfs mmm[2692]: GLESM221E(02136): Generic job with identifier REC_TGTT00029L8 failed.
2020-01-26T05:04:36.620168-07:00 ltfs mmm[2692]: GLESM223E(02024): Not all generic requests for session 2135234305 have been successful: 1 failed.
}}
The two drives that are still in use are idling for the past week. Interestingly, ltfsee log did not show any information for 31L8 and 54L8 tapes.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info jobs
Job Type        Status      Idle(sec)  Scan ID    Tape      Pool    Library  Node  File Name or inode
Reclaim(Source)  In-progress    576566  2944737025  T00031L8  TIERED1  T_ARCH      4  -               
Reclaim(Target)  In-progress    576438  2944737281  T00054L8  TIERED1  T_ARCH      4  -               
Validate        Unscheduled      7224  1602429441  T00054L8  TIERED1  T_ARCH      -  -               
}}
The job cannot be stopped. Force a LTFS stop using {{code|ltfsee stop -f}} and wait. The tapes drives should eventually eject the tapes. I got the following error when stopping, but the tapes got ejected and everything seemed to have stopped.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee stop -f
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running stop command - sending request and waiting for the completion.
GLESL030E(00909): Unable to connect to the MMM service. Check whether the IBM Spectrum Archive EE has been started.
GLESL358E(00494): Error on processing tape T00054L8 (1).
GLESL661E(00104): IPC got failure result (result=1).
GLESL646E(00164): Unable to stop the IBM Spectrum Archive EE monitor daemon for library T_ARCH.
}}
=== Cannot Start LTFS ===
{{highlight|lang=terminal|code=
[root@ltfs ~]# /opt/ibm/ltfsee/bin/ltfsee start
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running start command - sending request : T_ARCH.
Running start command - waiting for completion : T_ARCH.
...
GLESL657E(00191): Fail to start the IBM Spectrum Archive EE service (MMM) for library T_ARCH.
                  Use the 'ltfsee info nodes' command to see the error modules.
                  The monitor daemon will start the recovery sequence.
[root@ltfs ~]# ltfsee info nodes
Spectrum Archive EE service (MMM) for library T_ARCH fails to start or is not running on ltfs-ib.gpfs.net Node ID:4
Problem Detected:
Node ID  Error Modules
      4  MMM;
}}
Looking at {{code|/var/log/ltfsee.log}}, we see:
{{highlight|lang=text|code=
2020-02-03T13:15:35.266926-07:00 ltfs mmm[31142]: GLESM709E(00369): Assign tape (T00099L8) command error: 172.26.3.249:7600 (4): Request Error (070E): [Cartridge.cc:61]: Cartridge add is failed: 7c05 LTFSI1079E The operation is not allowed.
}}
Seemed to have started up by itself?
{{highlight|lang=terminal|code=
[root@ltfs ~]# /opt/ibm/ltfsee/bin/ltfsee start
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
GLESL519I(00344): The IBM Spectrum Archive EE service (ltfsee_md) for library T_ARCH is already running.
[root@ltfs ~]# ltfsee info nodes
Node ID  Status    Node IP      Drives  Ctrl Node    Library  Node Group  Host Name     
4        Available  172.26.3.249      3  yes(active)  T_ARCH  G0          ltfs-ib.gpfs.net
}}
Not sure what happened there...
=== Invalid Tapes ===
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info tapes
Tape ID  Status      Type  Capacity(GiB)  Used(GiB)  Free(GiB)  Reclaimable(GiB)  Pool    Library  Address  Drive      Appendable
T00030L8  Invalid      L8            10907      10895          0                22  TIERED1  T_ARCH  1131    -          no           
T00099L8  Invalid      L8            10907      10883          0                0  TIERED2  T_ARCH  1031    -          no       
}}
According to IBM's documentation:
{{quote|The Invalid status indicates that the cartridge is inconsistent with the LTFS format.
To check and repair this tape before you add it to a tape storage pool, use the ltfsee pool add command with the check option.
|cite=IBM https://www.ibm.com/support/knowledgecenter/ST9MBR_1.2.6/ltfs_ee_ltfsee_info_tapes.html}}
In order to add the tape back into the pool, it has to be removed first.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee pool add -p TIERED2 -t T00099L8 -c
GLESL042I(00894): Adding tape T00099L8 to storage pool TIERED2.
GLESL346E(00949): No tapes are found in the unassigned list. T00099L8 is already assigned to another storage pool, is in an IE slot, or does not exist in the library.
[root@ltfs ~]# ltfsee pool remove -p TIERED2 -t T00099L8
GLESL043I(01134): Removing tape T00099L8 from storage pool TIERED2.
Removed tape T00099L8 from pool TIERED2 successfully.
[root@ltfs ~]# ltfsee pool add -p TIERED2 -t T00099L8 -c
GLESL042I(00894): Adding tape T00099L8 to storage pool TIERED2.
Tape T00099L8 successfully checked.
Added tape T00099L8 to pool TIERED2 successfully.
}}
The checking step takes a very long time, upwards to an hour for this LTO8 cartridge. The tape drive status will be shown as 'locating' and {{code|ltfsee info jobs}} show the recovery job idling. This however appears to be normal since it eventually finished.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info jobs
Job Type  Status    Idle(sec)  Scan ID    Tape      Pool    Library  Node  File Name or inode
Recovery  Scheduled        694  2181628673  T00099L8  TIERED2  T_ARCH      4  -               
}}
=== Error Tape Drives ===
One of the tape drives is in 'Error' state.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info drives
Drive S/N  Status  Type  Role  Library  Address  Node ID  Tape      Node Group
0007807A4B  In use  LTO8  mrg  T_ARCH  260      4        T00000L8  G0       
0007807A0B  Error  LTO8  mrg  T_ARCH  261      4        -        G0       
000780765B  In use  LTO8  mrg  T_ARCH  262      4        T00037L8  G0       
}}
The drive shows up as online to the tape library. No visible errors displayed on the tape drive itself.
I can't remove/readd it:
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee drive remove -d 0007807A0B
GLESL132E(00247): Could not remove a drive 0007807A0B. Drive is not in mount or not mounted state. The tape drive status:2.
}}
Stopping and starting LTFSEE seemed to have cleared this error.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee stop
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running stop command - sending request and waiting for the completion.
...
Stopped the IBM Spectrum Archive EE services for library T_ARCH.
[root@ltfs ~]# ltfsee start
Library name: T_ARCH, library serial: 0000078BA6130402, control node (ltfsee_md) IP address: 172.26.3.249.
Running start command - sending request : T_ARCH.
Running start command - waiting for completion : T_ARCH.
.....................................
Started the IBM Spectrum Archive EE services for library T_ARCH with good status.
[root@ltfs ~]# ltfsee info drives
Drive S/N  Status      Type  Role  Library  Address  Node ID  Tape  Node Group
0007807A4B  Not mounted  LTO8  mrg  T_ARCH  260      4        -    G0       
0007807A0B  Not mounted  LTO8  mrg  T_ARCH  261      4        -    G0       
000780765B  Not mounted  LTO8  mrg  T_ARCH  262      4        -    G0       
}}
Interestingly, all tapes that were in drives were stowed away. I believe this is the behavior when shutting down LTFS.
=== Bad Tape Drive ===
The TS4500 tape library reported a tape drive issue with the error "Drive internal power-on self-tests failed." code 001F.  The LTFS operations that were running at the time on this tape drive stopped and the tape cartridge itself is marked as 'Critical'.
{{code|/var/log/ltfsee.log}} showed:
{{highlight|lang=text|code=
2020-03-24T19:43:39.873263-06:00 ltfs ltfseecp[21628]: GLESG081E(00468): Migrating data of GPFS file /tiered/kkurek/sbarclay/speedseq_align/73.L005.realign.bam: write failed to tape T00086L8 and file /ltfs/T00086L8/.LTFSEE_DATA/10991691794470275686-15679318773138264748-228651564-67167434-0 (data length: 524288, rc: -1, errno: 5).
2020-03-24T19:43:39.873579-06:00 ltfs ltfseecp[21628]: GLESG506E(00803): Migration file (/tiered/kkurek/sbarclay/speedseq_align/73.L005.realign.bam) to tape T00086L8 failed (1091).
2020-03-24T19:43:39.873870-06:00 ltfs ltfseecp[21628]: GLESC003E(01158): Redundant copy for file /tiered/kkurek/sbarclay/speedseq_align/73.L005.realign.bam to tape T00086L8 failed.
2020-03-24T19:43:39.979980-06:00 ltfs mmm[28988]: GLESM110W(00210): Tape T00086L8 got critical.
2020-03-24T19:44:09.243830-06:00 ltfs mmm[28988]: GLESM709E(00442): Unmount T00086L8 command error: 172.26.3.249:7600 (4): Request Error (077E): [Cartridge.cc:146]: Cartridge unmount is failed: 578a LTFSI1086E This operation is not allowed on a cartridge with a critical error.
2020-03-24T19:44:09.244112-06:00 ltfs mmm[28988]: GLESM118E(00060): Unmount of tape T00086L8 failed (drive 000780765B). Check the state of tapes and drives.
2020-03-24T19:44:39.166261-06:00 ltfs mmm[28988]: GLESM709E(00442): Unmount T00086L8 command error: 172.26.3.249:7600 (4): Request Error (077E): [Cartridge.cc:146]: Cartridge unmount is failed: 5b10 LTFSI1086E This operation is not allowed on a cartridge with a critical error.
2020-03-24T19:44:39.166568-06:00 ltfs mmm[28988]: GLESM118E(00060): Unmount of tape T00086L8 failed (drive 000780765B). Check the state of tapes and drives.
2020-03-24T19:45:09.283571-06:00 ltfs mmm[28988]: GLESM709E(00442): Unmount T00086L8 command error: 172.26.3.249:7600 (4): Request Error (077E): [Cartridge.cc:146]: Cartridge unmount is failed: 5d71 LTFSI1086E This operation is not allowed on a cartridge with a critical error.
2020-03-24T19:45:09.283890-06:00 ltfs mmm[28988]: GLESM118E(00060): Unmount of tape T00086L8 failed (drive 000780765B). Check the state of tapes and drives.
...
2020-03-25T01:19:14.689053-06:00 ltfs ltfsee[14508]: GLESL062E(00577): Tape with ID: T00086L8 is in an invalid state. Tapes must be either in state "Valid LTFS" or "Unknown".
2020-03-25T01:19:14.695195-06:00 ltfs ltfsee[14501]: GLESL062E(00577): Tape with ID: T00086L8 is in an invalid state. Tapes must be either in state "Valid LTFS" or "Unknown".
2020-03-25T01:19:14.700692-06:00 ltfs ltfsee[14503]: GLESL062E(00577): Tape with ID: T00086L8 is in an invalid state. Tapes must be either in state "Valid LTFS" or "Unknown".
2020-03-25T01:19:14.706384-06:00 ltfs ltfsee[14504]: GLESL062E(00577): Tape with ID: T00086L8 is in an invalid state. Tapes must be either in state "Valid LTFS" or "Unknown".
}}
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info tapes
Tape ID  Status      Type  Capacity(GiB)  Used(GiB)  Free(GiB)  Reclaimable(GiB)  Pool    Library  Address  Drive      Appendable               
T00086L8  Critical    L8            10907          0          0                0  TIERED2  T_ARCH  262      000780765B  no                     
...
T00084L8  Critical    L8            10907      10859          0                0  TIERED2  T_ARCH  262      000780765B  no              (this happened the next day)
}}
You must recover all files from the tape prior to moving it: {{highlight|lang=terminal|code=
# ltfsee tape move homeslot -t T00084L8 -p TIERED2
GLESL373I(00890): Moving tape T00084L8.
GLESL630E(00463): Cannot move tape T00084L8 because the status is "Critical".
}}
The recovery process from IBM's documentation can be read at https://www.ibm.com/support/knowledgecenter/en/ST9MBR_1.2.4/ltfs_ee_recovering_critical_tapes.html
In summary, the recovery process is to:
# Recover the files from tape back to Spectrum Scale. {{highlight|lang=terminal|code=
## Show all files to recover
# ltfsee recover -s -p TIERED2 -l T_ARCH -t T00084L8
## Recover these files
# ltfsee recover -c -p TIERED2 -l T_ARCH -t T00084L8
}}
# Remove the tape from the LTFS library. {{highlight|lang=terminal|code=
# ltfsee recover -r -p TIERED2 -l T_ARCH -t T00084L8
}}
# Move the tape to the IO port. {{highlight|lang=terminal|code=
# ltfsee tape move homeslot -t T00084L8 -p TIERED2
}}
If files are stuck in a premigrated state, the recover process won't succeed:
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee recover -r -p TIERED1 -l T_ARCH -t T00037L8
Scanning GPFS file systems to find migrated/saved objects in tape T00037L8.
Tape T00037L8 has 14 files to be recovered. The list is saved to /tmp/ltfs.29601.tiered.recoverlist.
GLESL613E(00822): Cannot remove tape T00037L8 because there are files to be recovered.
}}
You can try to use the {{code|ltfsee recover}} command to change the status of the files from premigrated to resident.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee repair "`cat /tmp/a`"
ANS9294I No files matching '/tiered/morph/....czi' were found.
GLESL257I(00083): Non-empty regular file /tiered/morph/....czi was in premigrated state.  The file is repaired to resident state.
}}
But for some reason, that doesn't actually do what it says it did:
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee info files -f "`cat /tmp/a`"
Name: /tiered/morph/....czi
Tape id:T00037L8@T_ARCH:T00069L8@T_ARCH Status: premigrated
}}
Since the files causing the issues are stored on another tape, stored on Spectrum Archive storage for premigration, and also stored on the GPFS storage, it should be safe to force remove the tape cartridge.
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee pool remove -p TIERED1 -t T00037L8 -r
GLESL043I(01134): Removing tape T00037L8 from storage pool TIERED1.
GLESL361W(01199): Tape T00037L8 was removed from pool TIERED1 forcefully. Cannot recall files on tape T00037L8. Add the tape to the same pool again if you need to recall files from the tape without formatting.
}}
The tape was removed from the pool but was left in the tape drive. Because the tape isn't part of LTFSEE anymore, you have to move it to its homeslot using the tape library interface. The bad drive was then removed:
{{highlight|lang=terminal|code=
[root@ltfs tiered]# ltfsee drive remove -d 000780765B -n 4 -l T_ARCH                                                                              │[REQUIRED PARAMETERS]
GLESL121I(00279): Drive serial 000780765B is removed from the tape drive list.                                                                    │  -p <pool_name>
}}
The 'bad' tape can then be re-added.


== Tasks ==
== Tasks ==
=== Shutdown and Startup ===
If the tape library needs to go offline, turn off LTFS to ensure drives are not accessed.
{{highlight|lang=terminal|code=
# ltfsee stop
}}
When completed, start up LTFS:
{{highlight|lang=terminal|code=
# ltfsee start
}}
=== Remove or Reclaim Tape ===
If a tape needs to be removed from the pool or tape library, you need to first reclaim its data so that migrated or saved files on the tape are retained.
To start a reclaimation for a specific tape, use {{code|ltfsee reclaim -p pool -t tapeid}}:
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee reclaim -p TIERED1 -t T00037L8
Start reclaiming 1 tapes in the following list of tapes:
T00037L8 .
Files in tape T00037L8 are copied to tape T00000L8.
}}
Once completed, you should be able to remove the tape from the pool and then move it to the IO slot.
If you don't reclaim the contents first, you will get these error messages:
{{highlight|lang=terminal|code=
[root@ltfs ~]# ltfsee pool remove -p TIERED1 -t T00037L8
GLESL043I(01134): Removing tape T00037L8 from storage pool TIERED1.
GLESL357E(01223): Tape T00037L8 has migrated files or saved files. It has not been removed from the pool.
[root@ltfs ~]# ltfsee tape move ieslot -t T00037L8 -p TIERED1
GLESL170E(00472): Failed to move tape T00037L8 because tape is assigned to a pool and not offline.
}}
=== Cronjobs ===
=== Cronjobs ===
The following cronjob is installed.
The following cronjob is installed.
Line 523: Line 138:
00 7 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED2 -l T_ARCH
00 7 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED2 -l T_ARCH
}}
}}


== See Also ==
== See Also ==
* https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Quickly%20determining%20the%20file%20state%20or%20status%20of%20all%20files%20within%20Spectrum%20Archive%20EE
* https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Quickly%20determining%20the%20file%20state%20or%20status%20of%20all%20files%20within%20Spectrum%20Archive%20EE

Revision as of 19:28, 3 February 2020


Usage

The ltfs* binaries are typically installed at /opt/ibm/ltfsee/bin/ and are provided by the ltfs-*.rpm RPM packages.

You will most likely interface with LTFS using the ltfsee command.


LTFS Library Status

[root@ltfs ~]# ltfsee status
Ctrl Node     MD      MMM     Library
172.26.3.249  Active  Active  T_ARCH

Tapes

All tape related usage.

To show all drives: 

[root@ltfs ~]# ltfsee info drives
Drive S/N   Status   Type  Role  Library  Address  Node ID  Tape      Node Group
0007807A4B  In use   LTO8   mrg  T_ARCH   260      4        T00031L8  G0        
0007807A0B  Mounted  LTO8   mrg  T_ARCH   261      4        -         G0        
000780765B  In use   LTO8   mrg  T_ARCH   262      4        T00054L8  G0


ltfsee info tapes shows all tapes all libraries. 

[root@ltfs ~]# ltfsee info tapes
Tape ID   Status       Type  Capacity(GiB)  Used(GiB)  Free(GiB)  Reclaimable(GiB)  Pool     Library  Address  Drive       Appendable
T00000L8  Valid        L8            10907          0      10907                 0  TIERED1  T_ARCH   1161     -           yes       
T00001L8  Valid        L8            10907          0      10907                 0  TIERED1  T_ARCH   1160     -           yes       
T00003L8  Valid        L8            10907          0      10907                 0  TIERED1  T_ARCH   1158     -           yes       
T00004L8  Valid        L8            10907          0      10907                 0  TIERED1  T_ARCH   1157     -           yes
Tape Types
Tapes that end with L8 are LTO-8.


If drives are in use, you can list all jobs to see what is actually being done. 

[root@ltfs ~]# ltfsee info jobs
Job Type         Status       Idle(sec)  Scan ID     Tape      Pool     Library  Node  File Name or inode
Reclaim(Source)  In-progress     573811  2944737025  T00031L8  TIERED1  T_ARCH      4  -                 
Reclaim(Target)  In-progress     573683  2944737281  T00054L8  TIERED1  T_ARCH      4  -                 
Validate         Unscheduled       4469  1602429441  T00054L8  TIERED1  T_ARCH      -  -

To add tapes to a particular pool, use ltfsee pool add -p pool -t tape ...: ltfsee pool add -p TIERED1 -t $tape -t $tape2 ...

To remove tapes from a particular pool, use ltfsee pool remove -p pool -t tape ...: ltfsee pool remove -p TIERED1 -t $tape -r


TODO: When removing, how does that affect replicas?

LTFS Information

ltfsee info pools shows all pools. 

Pool Name  Total(TiB)  Used(TiB)  Free(TiB)  Reclaimable(TiB)  Tapes  Type  Library  Node Group
TIERED1         644.3       84.6      559.7               0.0     61  LTO   T_ARCH   G0        
TIERED2         638.9       80.8      558.1               0.0     61  LTO   T_ARCH   G0


ltfsee info libraries shows all libraries with LTFS. 

[root@ltfs ~]# ltfsee info libraries 
Library Name  Status  Model     Serial Number     Ctrl Node   
T_ARCH        Active  03584L32  0000078BA6130402  172.26.3.249


Show Tiered Files

When files are tiered to tape, the full path of the file is stored in a database as 'migrated'. You can find a file's tiered tape cartridge using ltfsee info files -f filepath. Eg:

Files that are 'migrated' or tiered off to tape. Files not migrated are 'resident' and resides on spinning disk.

Example output below shows tape ID @ library for files migrated 

# ltfsee info files -f /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR302*
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021220_2.fastq
Tape id:-          Status: resident
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021242_1.fastq
Tape id:T00042L8@T_ARCH:T00102L8@T_ARCH Status: migrated
Name: /tiered/ewang_scratch/xuexu/dbgap_UK_OTTO/SRR3021288_1.fastq
Tape id:T00055L8@T_ARCH:T00107L8@T_ARCH Status: migrated
Side Notes on Migrated Files
Migrated files will have a stub file with zero size. To count sizes with du, you need to use --apparent-size Eg. 
[root@node001 xuexu]# du -sh dbgap_UK_OTTO
23T	dbgap_UK_OTTO

[root@node001 xuexu]# du -sh --apparent-size dbgap_UK_OTTO
35T	dbgap_UK_OTTO 

[root@node001 dbgap_UK_OTTO]# du SRR3021288_2.fastq
0	SRR3021288_2.fastq

[root@node001 dbgap_UK_OTTO]# du -sh --apparent-size SRR3021288_2.fastq
15G	SRR3021288_2.fastq



Troubleshooting

Logging

LTFS logs are located at /var/log/ltfsee.log.

Additional logs worth investigating:

  • /opt/tivoli/tsm/client/hsm/bin/dsmerror.log

Tasks

Cronjobs

The following cronjob is installed. 

# Cron Job Used to Move Data Via policy to LTFS Tape at 1:00 AM
00 1 * * * /mmpolicies/movetogpfs1.sh 
# Cron Job Used to Move Data Via policy to LTFS Tape at 3:00 AM
00 3 * * * /mmpolicies/movetogpfs0.sh 
# Cron Job Used to Reconcile disk data with LTFS Tape data Pool TIERED1
00 20 * * * /opt/ibm/ltfsee/bin/ltfsee reconcile -p TIERED1 -l T_ARCH -g /tiered
# Cron Job Used to Reconcile disk data with LTFS Tape data Pool TIERED2
00 22 * * * /opt/ibm/ltfsee/bin/ltfsee reconcile -p TIERED2 -l T_ARCH -g /tiered
# Cron Job Used to Reclaim LTFS Tape data in Pool TIERED1
00 5 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED1 -l T_ARCH
# Cron Job Used to Reclaim LTFS Tape Data in Pool TIERED2
00 7 * * sun /opt/ibm/ltfsee/bin/ltfsee reclaim -p TIERED2 -l T_ARCH


See Also