Dying Hard Disk

From Leo's Notes
Last edited on 3 April 2021, at 03:01.

April 1, 2021

A Hitachi/HGST Ultrastar 7K4000 drive in my NAS failed.

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-147.8.1.el8_1.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Hitachi/HGST Ultrastar 7K4000
Device Model:     Hitachi HUS724030ALE641
Serial Number:    P8GH7GVR
LU WWN Device Id: 5 000cca 22cc6eda4
Firmware Version: MJ8OA5F0
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Apr  2 20:48:36 2021 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121) The previous self-test completed having
                                        the read element of the test failed.
Total time to complete Offline
data collection:                (   24) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (   1) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   092   092   016    Pre-fail  Always       -       40
  2 Throughput_Performance  0x0005   101   101   054    Pre-fail  Offline      -       199
  3 Spin_Up_Time            0x0007   129   129   024    Pre-fail  Always       -       492 (Average 470)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       77
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       9
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   119   119   020    Pre-fail  Offline      -       35
  9 Power_On_Hours          0x0012   093   093   000    Old_age   Always       -       53285
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       76
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       1177
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       1178
194 Temperature_Celsius     0x0002   187   187   000    Old_age   Always       -       32 (Min/Max 19/43)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       11
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       16
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       13
...
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     53257         1000
# 2  Extended offline    Completed without error       00%     28435         -
# 3  Vendor (0xb0)       Completed without error       00%     28182         -
# 4  Vendor (0x71)       Completed without error       00%     28182         -

I tried to dd into the drive, but it bailed out almost immediately.

# dd if=/dev/zero of=/dev/sdb bs=512 status=progress
dd: error writing '/dev/sdb': Input/output error
1001+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 3.23109 s, 158 kB/s

# dd if=/dev/sdb of=/dev/null bs=512 status=progress   
509440 bytes (509 kB, 498 KiB) copied, 3 s, 158 kB/s
dd: error reading '/dev/sdb': Input/output error
1000+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 6.33653 s, 80.8 kB/s

Kernel logs showed:

[Fri Apr  2 20:47:15 2021] ata2.00: exception Emask 0x0 SAct 0x100 SErr 0x100000 action 0x0
[Fri Apr  2 20:47:15 2021] ata2.00: irq_stat 0x40000008
[Fri Apr  2 20:47:15 2021] ata2: SError: { Dispar }
[Fri Apr  2 20:47:15 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr  2 20:47:15 2021] ata2.00: cmd 60/08:40:e8:03:00/00:00:00:00:00/40 tag 8 ncq dma 4096 in
                                    res 51/40:08:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr  2 20:47:15 2021] ata2.00: status: { DRDY ERR }
[Fri Apr  2 20:47:15 2021] ata2.00: error: { UNC }
[Fri Apr  2 20:47:15 2021] ata2.00: configured for UDMA/133
[Fri Apr  2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr  2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 Sense Key : Medium Error [current]
[Fri Apr  2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr  2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 CDB: Read(16) 88 00 00 00 00 00 00 00 03 e8 00 00 00 08 00 00
[Fri Apr  2 20:47:15 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr  2 20:47:15 2021] ata2: EH complete
[Fri Apr  2 20:47:18 2021] ata2.00: exception Emask 0x0 SAct 0x800 SErr 0x0 action 0x0
[Fri Apr  2 20:47:18 2021] ata2.00: irq_stat 0x40000008
[Fri Apr  2 20:47:18 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr  2 20:47:18 2021] ata2.00: cmd 60/f8:58:08:02:00/01:00:00:00:00/40 tag 11 ncq dma 258048 in
                                    res 51/40:18:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr  2 20:47:18 2021] ata2.00: status: { DRDY ERR }
[Fri Apr  2 20:47:18 2021] ata2.00: error: { UNC }
[Fri Apr  2 20:47:18 2021] ata2.00: configured for UDMA/133
[Fri Apr  2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr  2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 Sense Key : Medium Error [current]
[Fri Apr  2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr  2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 CDB: Read(16) 88 00 00 00 00 00 00 00 02 08 00 00 01 f8 00 00
[Fri Apr  2 20:47:18 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr  2 20:47:18 2021] ata2: EH complete
[Fri Apr  2 20:47:21 2021] ata2.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x0
[Fri Apr  2 20:47:21 2021] ata2.00: irq_stat 0x40000008
[Fri Apr  2 20:47:21 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr  2 20:47:21 2021] ata2.00: cmd 60/08:90:e8:03:00/00:00:00:00:00/40 tag 18 ncq dma 4096 in
                                    res 51/40:08:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr  2 20:47:21 2021] ata2.00: status: { DRDY ERR }
[Fri Apr  2 20:47:21 2021] ata2.00: error: { UNC }
[Fri Apr  2 20:47:21 2021] ata2.00: configured for UDMA/133
[Fri Apr  2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr  2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 Sense Key : Medium Error [current]
[Fri Apr  2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr  2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 CDB: Read(16) 88 00 00 00 00 00 00 00 03 e8 00 00 00 08 00 00
[Fri Apr  2 20:47:21 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr  2 20:47:21 2021] Buffer I/O error on dev sdb, logical block 125, async page read
[Fri Apr  2 20:47:21 2021] ata2: EH complete

July 30, 2019

The Seagate ST3000DM001 drive I replaced last month failed. I replaced it with another Hitachi drive I had spare.

This failed Seagate now has a failure at a specific LBA.

leo # smartctl -a /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.6.14-300.fc32.x86_64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-9YN166
Serial Number:    W1F0SJ27
LU WWN Device Id: 5 000c50 0529df4f3
Firmware Version: CC4B
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Jul 31 09:38:57 2020 MDT

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 116)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline 
data collection: 		(  575) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 344) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x3085)	SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   092   006    Pre-fail  Always       -       7678832
  3 Spin_Up_Time            0x0003   096   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       311
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       712
  7 Seek_Error_Rate         0x000f   075   060   030    Pre-fail  Always       -       33217073
  9 Power_On_Hours          0x0032   086   086   000    Old_age   Always       -       12793
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       64
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       284
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1 1 1
189 High_Fly_Writes         0x003a   090   090   000    Old_age   Always       -       10
190 Airflow_Temperature_Cel 0x0022   059   047   045    Old_age   Always       -       41 (Min/Max 27/50)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       224
193 Load_Cycle_Count        0x0032   090   090   000    Old_age   Always       -       21487
194 Temperature_Celsius     0x0022   041   053   000    Old_age   Always       -       41 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   097   000    Old_age   Always       -       32
198 Offline_Uncorrectable   0x0010   100   097   000    Old_age   Offline      -       32
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       354
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       2222h+26m+07.853s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       108286374264413
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       13374937181101

SMART Error Log Version: 1
ATA Error Count: 76 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 76 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 70 ff ff ff 4f 00   2d+09:59:18.586  READ FPDMA QUEUED
  60 00 80 ff ff ff 4f 00   2d+09:59:18.581  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00   2d+09:59:18.574  READ FPDMA QUEUED
  60 00 c0 ff ff ff 4f 00   2d+09:59:18.574  READ FPDMA QUEUED
  60 00 10 ff ff ff 4f 00   2d+09:59:18.574  READ FPDMA QUEUED

Error 75 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 c0 ff ff ff 4f 00   2d+09:59:15.327  READ FPDMA QUEUED
  61 00 80 ff ff ff 4f 00   2d+09:59:15.302  WRITE FPDMA QUEUED
  60 00 80 ff ff ff 4f 00   2d+09:59:15.301  READ FPDMA QUEUED
  61 00 98 ff ff ff 4f 00   2d+09:59:15.289  WRITE FPDMA QUEUED
  61 00 00 ff ff ff 4f 00   2d+09:59:15.287  WRITE FPDMA QUEUED

Error 74 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 80 ff ff ff 4f 00   2d+09:59:12.251  WRITE FPDMA QUEUED
  61 00 40 ff ff ff 4f 00   2d+09:59:12.251  WRITE FPDMA QUEUED
  61 00 40 ff ff ff 4f 00   2d+09:59:12.250  WRITE FPDMA QUEUED
  61 00 40 ff ff ff 4f 00   2d+09:59:12.243  WRITE FPDMA QUEUED
  61 00 40 ff ff ff 4f 00   2d+09:59:12.242  WRITE FPDMA QUEUED

Error 73 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 10 ff ff ff 4f 00   2d+09:59:09.247  READ FPDMA QUEUED
  60 00 10 ff ff ff 4f 00   2d+09:59:09.247  READ FPDMA QUEUED
  60 00 10 10 0a 00 40 00   2d+09:59:09.247  READ FPDMA QUEUED
  60 00 c0 ff ff ff 4f 00   2d+09:59:09.247  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00   2d+09:59:09.247  READ FPDMA QUEUED

Error 72 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 80 ff ff ff 4f 00   2d+09:59:04.864  WRITE FPDMA QUEUED
  61 00 28 ff ff ff 4f 00   2d+09:59:04.863  WRITE FPDMA QUEUED
  61 00 00 ff ff ff 4f 00   2d+09:59:04.863  WRITE FPDMA QUEUED
  60 00 80 ff ff ff 4f 00   2d+09:59:04.863  READ FPDMA QUEUED
  61 00 40 ff ff ff 4f 00   2d+09:59:04.861  WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       40%     12786         4005685760
# 2  Extended offline    Completed without error       00%      9311         -
# 3  Extended offline    Completed without error       00%      5247         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Attempting to write to this bad LBA with dd results in an I/O error:

leo # dd if=/dev/sdb of=/dev/null bs=512 skip=4005685760 count=1 
dd: error reading '/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 5.7316 s, 0.0 kB/s

## A few blocks before are fine
leo # dd if=/dev/sdb of=/dev/null bs=512 skip=4005685750 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.00874588 s, 58.5 kB/s

June 7, 2019

I unwisely rebuilt the ZFS pool with these ST3000DM001 drives in late 2019 because they still _seemed_ good and perhaps the issues I've had were related to flaky SATA cables.

The same drive that faulted in September failed again. This time, it no longer can be recognized by SMART. I replaced this failed drive in the array with the same model with serial number W1F0SJ27.

September 20, 2019

One of the 3TB disks in my array went offline this morning.

# zpool status storage
...
        NAME                                      STATE     READ WRITE CKSUM
        storage                                   DEGRADED     0     0     0
          raidz1-0                                DEGRADED     0     0     0
            ata-Hitachi_HUS724030ALE641_P8GH7GVR  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F4HHZ7       FAULTED      0     0     0  too many errors
            ata-Hitachi_HUS724030ALE641_P8GBAW7P  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F4HGS6       ONLINE       0     0     0
            ata-Hitachi_HUS724030ALE641_P8G9NZSR  ONLINE       0     0     0

# smartctl -a /dev/sdd
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.0.9-100.fc28.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST3000DM001-1CH166
Serial Number:    Z1F4HHZ7
LU WWN Device Id: 5 000c50 065c6f541
Firmware Version: CC29
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Sep 20 09:20:05 2019 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  584) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 319) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   097   097   006    Pre-fail  Always       -       42489445
  3 Spin_Up_Time            0x0003   094   094   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       632
  5 Reallocated_Sector_Ct   0x0033   099   099   010    Pre-fail  Always       -       1160
  7 Seek_Error_Rate         0x000f   086   060   030    Pre-fail  Always       -       406992069
  9 Power_On_Hours          0x0032   064   064   000    Old_age   Always       -       32374
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       68
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   086   086   000    Old_age   Always       -       14
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
189 High_Fly_Writes         0x003a   096   096   000    Old_age   Always       -       4
190 Airflow_Temperature_Cel 0x0022   071   055   045    Old_age   Always       -       29 (Min/Max 25/33)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       524
193 Load_Cycle_Count        0x0032   079   079   000    Old_age   Always       -       42761
194 Temperature_Celsius     0x0022   029   045   000    Old_age   Always       -       29 (0 16 0 0 0)
197 Current_Pending_Sector  0x0012   001   001   000    Old_age   Always       -       39672
198 Offline_Uncorrectable   0x0010   001   001   000    Old_age   Offline      -       39672
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       18628h+23m+18.929s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       104853181293
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       166965854582

SMART Error Log Version: 1
ATA Error Count: 14 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 14 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00  28d+08:28:46.753  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:46.752  READ FPDMA QUEUED
  ea 00 00 00 00 00 a0 00  28d+08:28:46.658  FLUSH CACHE EXT
  61 00 08 ff ff ff 4f 00  28d+08:28:46.624  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00  28d+08:28:46.624  WRITE FPDMA QUEUED

Error 13 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00  28d+08:28:40.736  READ FPDMA QUEUED
  ea 00 00 00 00 00 a0 00  28d+08:28:40.248  FLUSH CACHE EXT
  61 00 08 ff ff ff 4f 00  28d+08:28:40.247  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00  28d+08:28:40.247  WRITE FPDMA QUEUED
  61 00 08 00 0b 00 40 00  28d+08:28:40.246  WRITE FPDMA QUEUED

Error 12 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00  28d+08:28:37.790  WRITE FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:36.075  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:36.000  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:35.990  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:35.939  READ FPDMA QUEUED

Error 11 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: WP at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  61 00 08 ff ff ff 4f 00  28d+08:28:32.734  WRITE FPDMA QUEUED
  61 00 08 ff ff ff 4f 00  28d+08:28:32.734  WRITE FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:29.681  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:29.620  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:29.544  READ FPDMA QUEUED

Error 10 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 ff ff ff 4f 00  28d+08:28:23.596  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:23.567  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:23.542  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:23.532  READ FPDMA QUEUED
  60 00 08 ff ff ff 4f 00  28d+08:28:23.496  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     20997         -
# 2  Extended offline    Completed without error       00%     20860         -
# 3  Extended offline    Completed without error       00%      9897         -
# 4  Extended offline    Interrupted (host reset)      00%      6482         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

May 18, 2017

Here's a dmesg log of another dying disk.

May 17 15:43:18 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 17 15:43:18 nas kernel: ata14.00: failed command: READ DMA EXT
May 17 15:43:18 nas kernel: ata14.00: cmd 25/00:08:88:08:f6/00:00:3c:00:00/e0 tag 0 dma 4096 in#012         res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 17 15:43:18 nas kernel: ata14.00: status: { DRDY }
May 17 15:43:18 nas kernel: ata14: hard resetting link
May 17 15:43:18 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:23 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:28 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:28 nas kernel: ata14: hard resetting link
May 17 15:43:28 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:33 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:38 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:38 nas kernel: ata14: hard resetting link
May 17 15:43:38 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:43 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:44:13 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:44:13 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 17 15:44:13 nas kernel: ata14: hard resetting link
May 17 15:44:13 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:13 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:13 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:19 nas kernel: ata14: hard resetting link
May 17 15:44:19 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:19 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:19 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:24 nas kernel: ata14: hard resetting link
May 17 15:44:24 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:25 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:25 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:25 nas kernel: ata14.00: disabled
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Sense Key : Not Ready [current]
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Add. Sense: Logical unit not ready, hard reset required
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: Read(10) 28 00 3c f6 08 88 00 00 08 00
May 17 15:44:25 nas kernel: blk_update_request: I/O error, dev sdi, sector 1022757000
May 17 15:44:25 nas kernel: ata14: EH complete
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 19 00:51:51 nas kernel: ata14: SATA max UDMA/133 cmd 0x960 ctl 0xb60 bmdma 0xc808 irq 23
May 19 00:51:56 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:51:56 nas kernel: ata14.00: ATA-8: ST2000DM001-9YN164, CC4C, max UDMA/133
May 19 00:51:56 nas kernel: ata14.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 0/32)
May 19 00:51:56 nas kernel: ata14.00: configured for UDMA/133
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 4096-byte physical blocks
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write Protect is off
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 00:51:56 nas kernel: sdi: sdi1
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Attached SCSI disk
May 19 00:53:41 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:53:41 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:53:41 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012         res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:53:41 nas kernel: ata14.00: status: { DRDY }
May 19 00:53:41 nas kernel: ata14: hard resetting link
May 19 00:53:41 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:53:47 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:53:49 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:53:49 nas kernel: ata14.00: configured for UDMA/133
May 19 00:53:49 nas kernel: ata14: EH complete
May 19 00:54:45 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:54:45 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:54:45 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012         res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:54:45 nas kernel: ata14.00: status: { DRDY }
May 19 00:54:45 nas kernel: ata14: hard resetting link
May 19 00:54:45 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:54:51 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:54:55 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:54:55 nas kernel: ata14: hard resetting link
May 19 00:54:55 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:01 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:05 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:05 nas kernel: ata14: hard resetting link
May 19 00:55:05 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:11 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:40 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:40 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 19 00:55:40 nas kernel: ata14: hard resetting link
May 19 00:55:40 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:45 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:45 nas kernel: ata14: reset failed, giving up
May 19 00:55:45 nas kernel: ata14.00: disabled
May 19 00:55:45 nas kernel: ata14: EH complete
May 19 01:06:47 nas kernel: ata14: SATA max UDMA/100 mmio m1024@0xfd7ff000 tf 0xfd7ff2c0 irq 17
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 4096-byte physical blocks
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write Protect is off
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Attached SCSI disk
May 19 01:06:49 nas kernel: ata14: SATA link down (SStatus 0 SControl 310)
May 19 01:13:54 nas kernel: sdi:
May 19 01:15:52 nas kernel: sdi: sdi1

The disk went offline pretty quick, so ZFS took it off the pool equally as quick.



Disks

# zpool status storage
  pool: storage
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0 in 7h8m with 0 errors on Fri Apr 14 10:23:51 2017
config:

        NAME                                                STATE     READ WRITE CKSUM
        storage                                             DEGRADED     0     0     0
          raidz1-0                                          DEGRADED     0     0     0
            ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0877352-part1  ONLINE       0     0     0
            ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0840640-part1  ONLINE       0     0     0
            ata-ST2000DM001-9YN164_W2F0B1GV-part1           UNAVAIL      0     0     0
            ata-ST2000DM001-1CH164_Z1E37XQR-part1           ONLINE       0     0     0
            ata-ST2000DL003-9VT166_5YD3VR3A-part1           ONLINE       0     0     0

errors: No known data errors

An older Entry

I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing it because it's only one drive and it's acting like a raid 0 (right?).

Unfortunately, my backups failed for a few files and then this turned up:

root@nas:/mnt/nas4/images/christine/2015-01-25  Annual Dinner- we host# zpool status -xv
  pool: data
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 5h45m with 0 errors on Tue Jun 17 00:13:18 2014
config:

        NAME                                 STATE     READ WRITE CKSUM
        data                                 DEGRADED     0     0 6.31K
          raidz1-0                           DEGRADED     0     0 13.2K
            ata-ST3000DM001-1CH166_Z1F4HHZ7  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F4HGS6  ONLINE       0     0     0
            ata-ST3000DM001-9YN166_W1F0SJ27  FAULTED      1     2     0  too many errors

errors: Permanent errors have been detected in the following files:

        data:<0x0>
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3255.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3256.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3257.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3258.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3259.JPG

It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it:

smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /8:0:0:0
Product:
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

The logs show the drive failing:

2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2270 Jan 25 22:13:54 nas kernel: ata7.00: failed command: READ DMA EXT
2271 Jan 25 22:13:54 nas kernel: ata7.00: cmd 25/00:00:6f:7c:55/00:02:37:00:00/e0 tag 0 dma 262144 in
2272 Jan 25 22:13:54 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2273 Jan 25 22:13:54 nas kernel: ata7.00: status: { Busy }
2274 Jan 25 22:13:54 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2275 Jan 25 22:13:54 nas kernel: ata7: hard resetting link
2276 Jan 25 22:13:59 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2277 Jan 25 22:13:59 nas kernel: ata7.00: configured for UDMA/33
2278 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2279 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2280 Jan 25 22:13:59 nas kernel: Descriptor sense data with sense descriptors (in hex):
2281 Jan 25 22:13:59 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2282 Jan 25 22:13:59 nas kernel:        d8 d8 d8 d8
2283 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2284 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 37 55 7c 6f 00 02 00 00
2285 Jan 25 22:13:59 nas kernel: ata7: EH complete
2286 Jan 25 22:15:09 nas kernel: NFS: v4 server 10.1.1.6 does not accept raw uid/gids. Reenabling the idmapper.
2287 Jan 25 22:15:41 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2288 Jan 25 22:15:41 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2289 Jan 25 22:15:41 nas kernel: ata7.00: failed command: READ DMA EXT
2290 Jan 25 22:15:41 nas kernel: ata7.00: cmd 25/00:80:c0:2d:c0/00:01:65:00:00/e0 tag 0 dma 196608 in
2291 Jan 25 22:15:41 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2292 Jan 25 22:15:41 nas kernel: ata7.00: status: { Busy }
2293 Jan 25 22:15:41 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2294 Jan 25 22:15:41 nas kernel: ata7: hard resetting link
2295 Jan 25 22:15:46 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2296 Jan 25 22:15:46 nas kernel: ata7.00: configured for UDMA/33
2297 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2298 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2299 Jan 25 22:15:46 nas kernel: Descriptor sense data with sense descriptors (in hex):
2300 Jan 25 22:15:46 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2301 Jan 25 22:15:46 nas kernel:        d8 d8 d8 d8
2302 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2303 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2d c0 00 01 80 00
2304 Jan 25 22:15:46 nas kernel: ata7: EH complete
2305 Jan 25 22:15:46 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2306 Jan 25 22:15:46 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2307 Jan 25 22:15:46 nas kernel: ata7.00: failed command: READ DMA EXT
2308 Jan 25 22:15:46 nas kernel: ata7.00: cmd 25/00:00:40:2f:c0/00:03:65:00:00/e0 tag 0 dma 393216 in
2309 Jan 25 22:15:46 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2310 Jan 25 22:15:46 nas kernel: ata7.00: status: { Busy }
2311 Jan 25 22:15:46 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2312 Jan 25 22:15:46 nas kernel: ata7: hard resetting link

2313 Jan 25 22:15:51 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2314 Jan 25 22:15:51 nas kernel: ata7.00: configured for UDMA/33
2315 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2316 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2317 Jan 25 22:15:51 nas kernel: Descriptor sense data with sense descriptors (in hex):
2318 Jan 25 22:15:51 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2319 Jan 25 22:15:51 nas kernel:        d8 d8 d8 d8
2320 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2321 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2f 40 00 03 00 00
2322 Jan 25 22:15:51 nas kernel: ata7: EH complete
2323 Jan 25 22:15:51 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2324 Jan 25 22:15:51 nas kernel: ata7: SError: { PHYRdyChg }
2325 Jan 25 22:15:51 nas kernel: ata7.00: failed command: READ DMA EXT
2326 Jan 25 22:15:51 nas kernel: ata7.00: cmd 25/00:14:70:0f:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2327 Jan 25 22:15:51 nas kernel:         res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2328 Jan 25 22:15:51 nas kernel: ata7.00: status: { Busy }
2329 Jan 25 22:15:51 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2330 Jan 25 22:15:51 nas kernel: ata7: hard resetting link
2331 Jan 25 22:15:56 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2332 Jan 25 22:15:56 nas kernel: ata7.00: configured for UDMA/33
2333 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2334 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2335 Jan 25 22:15:56 nas kernel: Descriptor sense data with sense descriptors (in hex):
2336 Jan 25 22:15:56 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2337 Jan 25 22:15:56 nas kernel:        d0 d0 d0 d0
2338 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2339 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 0f 70 00 00 14 00
2340 Jan 25 22:15:56 nas kernel: ata7: EH complete
2341 Jan 25 22:15:57 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2342 Jan 25 22:15:57 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2343 Jan 25 22:15:57 nas kernel: ata7.00: failed command: READ DMA EXT
2344 Jan 25 22:15:57 nas kernel: ata7.00: cmd 25/00:14:77:10:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2345 Jan 25 22:15:57 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2346 Jan 25 22:15:57 nas kernel: ata7.00: status: { Busy }
2347 Jan 25 22:15:57 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2348 Jan 25 22:15:57 nas kernel: ata7: hard resetting link
2349 Jan 25 22:16:02 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2350 Jan 25 22:16:02 nas kernel: ata7.00: configured for UDMA/33
2351 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2352 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2353 Jan 25 22:16:02 nas kernel: Descriptor sense data with sense descriptors (in hex):
2354 Jan 25 22:16:02 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2355 Jan 25 22:16:02 nas kernel:        d8 d8 d8 d8
2356 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2357 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 10 77 00 00 14 00
2358 Jan 25 22:16:02 nas kernel: ata7: EH complete
2359 Jan 25 22:16:02 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2360 Jan 25 22:16:02 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2361 Jan 25 22:16:02 nas kernel: ata7.00: failed command: READ DMA EXT
2362 Jan 25 22:16:02 nas kernel: ata7.00: cmd 25/00:80:c0:3a:c0/00:02:65:00:00/e0 tag 0 dma 327680 in

2363 Jan 25 22:16:02 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2364 Jan 25 22:16:02 nas kernel: ata7.00: status: { Busy }
2365 Jan 25 22:16:02 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2366 Jan 25 22:16:02 nas kernel: ata7: hard resetting link
2367 Jan 25 22:16:07 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2368 Jan 25 22:16:07 nas kernel: ata7.00: configured for UDMA/33
2369 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2370 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2371 Jan 25 22:16:07 nas kernel: Descriptor sense data with sense descriptors (in hex):
2372 Jan 25 22:16:07 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2373 Jan 25 22:16:07 nas kernel:        d8 d8 d8 d8
2374 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2375 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 3a c0 00 02 80 00
2376 Jan 25 22:16:07 nas kernel: ata7: EH complete
2377 Jan 25 22:16:07 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2378 Jan 25 22:16:07 nas kernel: ata7: SError: { PHYRdyChg }
2379 Jan 25 22:16:07 nas kernel: ata7.00: failed command: WRITE DMA EXT
2380 Jan 25 22:16:07 nas kernel: ata7.00: cmd 35/00:03:d6:04:48/00:00:69:00:00/e0 tag 0 dma 1536 out
2381 Jan 25 22:16:07 nas kernel:         res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2382 Jan 25 22:16:07 nas kernel: ata7.00: status: { Busy }
2383 Jan 25 22:16:07 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2384 Jan 25 22:16:07 nas kernel: ata7: hard resetting link
2385 Jan 25 22:16:12 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2386 Jan 25 22:16:12 nas kernel: ata7.00: configured for UDMA/33
2387 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2388 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2389 Jan 25 22:16:12 nas kernel: Descriptor sense data with sense descriptors (in hex):
2390 Jan 25 22:16:12 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2391 Jan 25 22:16:12 nas kernel:        d0 d0 d0 d0
2392 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2393 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] CDB: Write(10): 2a 00 69 48 04 d6 00 00 03 00
2394 Jan 25 22:16:12 nas kernel: ata7: EH complete
2395 Jan 25 22:16:13 nas kernel: ata7: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2396 Jan 25 22:16:13 nas kernel: ata7: SError: { PHYRdyChg }
2397 Jan 25 22:16:13 nas kernel: ata7: hard resetting link
2398 Jan 25 22:16:18 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2399 Jan 25 22:16:18 nas kernel: ata7.00: configured for UDMA/33
2400 Jan 25 22:16:18 nas kernel: ata7: EH complete
2401 Jan 25 22:16:24 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2402 Jan 25 22:16:24 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2403 Jan 25 22:16:24 nas kernel: ata7.00: failed command: READ DMA EXT
2404 Jan 25 22:16:24 nas kernel: ata7.00: cmd 25/00:00:c0:0b:c3/00:03:65:00:00/e0 tag 0 dma 393216 in
2405 Jan 25 22:16:24 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2406 Jan 25 22:16:24 nas kernel: ata7.00: status: { Busy }
2407 Jan 25 22:16:24 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2408 Jan 25 22:16:24 nas kernel: ata7: hard resetting link
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33

... this goes on for another 12 minutes.


So, lesson learned. Test backups and replace dead drives ASAP.