Dying Hard Disk
From Leo's Notes
Last edited on 3 April 2021, at 03:01.
April 1, 2021
A Hitachi/HGST Ultrastar 7K4000 drive in my NAS failed.
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.18.0-147.8.1.el8_1.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724030ALE641
Serial Number: P8GH7GVR
LU WWN Device Id: 5 000cca 22cc6eda4
Firmware Version: MJ8OA5F0
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Apr 2 20:48:36 2021 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: ( 24) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 1) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 092 092 016 Pre-fail Always - 40
2 Throughput_Performance 0x0005 101 101 054 Pre-fail Offline - 199
3 Spin_Up_Time 0x0007 129 129 024 Pre-fail Always - 492 (Average 470)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 77
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 9
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 119 119 020 Pre-fail Offline - 35
9 Power_On_Hours 0x0012 093 093 000 Old_age Always - 53285
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 76
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1177
193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 1178
194 Temperature_Celsius 0x0002 187 187 000 Old_age Always - 32 (Min/Max 19/43)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 11
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 16
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 13
...
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 53257 1000
# 2 Extended offline Completed without error 00% 28435 -
# 3 Vendor (0xb0) Completed without error 00% 28182 -
# 4 Vendor (0x71) Completed without error 00% 28182 -
I tried to dd into the drive, but it bailed out almost immediately.
# dd if=/dev/zero of=/dev/sdb bs=512 status=progress
dd: error writing '/dev/sdb': Input/output error
1001+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 3.23109 s, 158 kB/s
# dd if=/dev/sdb of=/dev/null bs=512 status=progress
509440 bytes (509 kB, 498 KiB) copied, 3 s, 158 kB/s
dd: error reading '/dev/sdb': Input/output error
1000+0 records in
1000+0 records out
512000 bytes (512 kB, 500 KiB) copied, 6.33653 s, 80.8 kB/s
Kernel logs showed:
[Fri Apr 2 20:47:15 2021] ata2.00: exception Emask 0x0 SAct 0x100 SErr 0x100000 action 0x0
[Fri Apr 2 20:47:15 2021] ata2.00: irq_stat 0x40000008
[Fri Apr 2 20:47:15 2021] ata2: SError: { Dispar }
[Fri Apr 2 20:47:15 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr 2 20:47:15 2021] ata2.00: cmd 60/08:40:e8:03:00/00:00:00:00:00/40 tag 8 ncq dma 4096 in
res 51/40:08:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr 2 20:47:15 2021] ata2.00: status: { DRDY ERR }
[Fri Apr 2 20:47:15 2021] ata2.00: error: { UNC }
[Fri Apr 2 20:47:15 2021] ata2.00: configured for UDMA/133
[Fri Apr 2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr 2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 Sense Key : Medium Error [current]
[Fri Apr 2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr 2 20:47:15 2021] sd 2:0:0:0: [sdb] tag#8 CDB: Read(16) 88 00 00 00 00 00 00 00 03 e8 00 00 00 08 00 00
[Fri Apr 2 20:47:15 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr 2 20:47:15 2021] ata2: EH complete
[Fri Apr 2 20:47:18 2021] ata2.00: exception Emask 0x0 SAct 0x800 SErr 0x0 action 0x0
[Fri Apr 2 20:47:18 2021] ata2.00: irq_stat 0x40000008
[Fri Apr 2 20:47:18 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr 2 20:47:18 2021] ata2.00: cmd 60/f8:58:08:02:00/01:00:00:00:00/40 tag 11 ncq dma 258048 in
res 51/40:18:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr 2 20:47:18 2021] ata2.00: status: { DRDY ERR }
[Fri Apr 2 20:47:18 2021] ata2.00: error: { UNC }
[Fri Apr 2 20:47:18 2021] ata2.00: configured for UDMA/133
[Fri Apr 2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr 2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 Sense Key : Medium Error [current]
[Fri Apr 2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr 2 20:47:18 2021] sd 2:0:0:0: [sdb] tag#11 CDB: Read(16) 88 00 00 00 00 00 00 00 02 08 00 00 01 f8 00 00
[Fri Apr 2 20:47:18 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr 2 20:47:18 2021] ata2: EH complete
[Fri Apr 2 20:47:21 2021] ata2.00: exception Emask 0x0 SAct 0x40000 SErr 0x0 action 0x0
[Fri Apr 2 20:47:21 2021] ata2.00: irq_stat 0x40000008
[Fri Apr 2 20:47:21 2021] ata2.00: failed command: READ FPDMA QUEUED
[Fri Apr 2 20:47:21 2021] ata2.00: cmd 60/08:90:e8:03:00/00:00:00:00:00/40 tag 18 ncq dma 4096 in
res 51/40:08:e8:03:00/00:00:00:00:00/40 Emask 0x409 (media error) <F>
[Fri Apr 2 20:47:21 2021] ata2.00: status: { DRDY ERR }
[Fri Apr 2 20:47:21 2021] ata2.00: error: { UNC }
[Fri Apr 2 20:47:21 2021] ata2.00: configured for UDMA/133
[Fri Apr 2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Fri Apr 2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 Sense Key : Medium Error [current]
[Fri Apr 2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 Add. Sense: Unrecovered read error - auto reallocate failed
[Fri Apr 2 20:47:21 2021] sd 2:0:0:0: [sdb] tag#18 CDB: Read(16) 88 00 00 00 00 00 00 00 03 e8 00 00 00 08 00 00
[Fri Apr 2 20:47:21 2021] print_req_error: I/O error, dev sdb, sector 1000 flags 0
[Fri Apr 2 20:47:21 2021] Buffer I/O error on dev sdb, logical block 125, async page read
[Fri Apr 2 20:47:21 2021] ata2: EH complete
July 30, 2019
The Seagate ST3000DM001 drive I replaced last month failed. I replaced it with another Hitachi drive I had spare.
This failed Seagate now has a failure at a specific LBA.
leo # smartctl -a /dev/sdb
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.6.14-300.fc32.x86_64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-9YN166
Serial Number: W1F0SJ27
LU WWN Device Id: 5 000c50 0529df4f3
Firmware Version: CC4B
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Jul 31 09:38:57 2020 MDT
==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/223651en
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 116) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: ( 575) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 344) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 105 092 006 Pre-fail Always - 7678832
3 Spin_Up_Time 0x0003 096 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 311
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 712
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 33217073
9 Power_On_Hours 0x0032 086 086 000 Old_age Always - 12793
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 64
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 284
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 1 1
189 High_Fly_Writes 0x003a 090 090 000 Old_age Always - 10
190 Airflow_Temperature_Cel 0x0022 059 047 045 Old_age Always - 41 (Min/Max 27/50)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 224
193 Load_Cycle_Count 0x0032 090 090 000 Old_age Always - 21487
194 Temperature_Celsius 0x0022 041 053 000 Old_age Always - 41 (0 17 0 0 0)
197 Current_Pending_Sector 0x0012 100 097 000 Old_age Always - 32
198 Offline_Uncorrectable 0x0010 100 097 000 Old_age Offline - 32
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 354
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 2222h+26m+07.853s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 108286374264413
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 13374937181101
SMART Error Log Version: 1
ATA Error Count: 76 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 76 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 70 ff ff ff 4f 00 2d+09:59:18.586 READ FPDMA QUEUED
60 00 80 ff ff ff 4f 00 2d+09:59:18.581 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 2d+09:59:18.574 READ FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 2d+09:59:18.574 READ FPDMA QUEUED
60 00 10 ff ff ff 4f 00 2d+09:59:18.574 READ FPDMA QUEUED
Error 75 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 c0 ff ff ff 4f 00 2d+09:59:15.327 READ FPDMA QUEUED
61 00 80 ff ff ff 4f 00 2d+09:59:15.302 WRITE FPDMA QUEUED
60 00 80 ff ff ff 4f 00 2d+09:59:15.301 READ FPDMA QUEUED
61 00 98 ff ff ff 4f 00 2d+09:59:15.289 WRITE FPDMA QUEUED
61 00 00 ff ff ff 4f 00 2d+09:59:15.287 WRITE FPDMA QUEUED
Error 74 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 80 ff ff ff 4f 00 2d+09:59:12.251 WRITE FPDMA QUEUED
61 00 40 ff ff ff 4f 00 2d+09:59:12.251 WRITE FPDMA QUEUED
61 00 40 ff ff ff 4f 00 2d+09:59:12.250 WRITE FPDMA QUEUED
61 00 40 ff ff ff 4f 00 2d+09:59:12.243 WRITE FPDMA QUEUED
61 00 40 ff ff ff 4f 00 2d+09:59:12.242 WRITE FPDMA QUEUED
Error 73 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 10 ff ff ff 4f 00 2d+09:59:09.247 READ FPDMA QUEUED
60 00 10 ff ff ff 4f 00 2d+09:59:09.247 READ FPDMA QUEUED
60 00 10 10 0a 00 40 00 2d+09:59:09.247 READ FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 2d+09:59:09.247 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 2d+09:59:09.247 READ FPDMA QUEUED
Error 72 occurred at disk power-on lifetime: 11403 hours (475 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 80 ff ff ff 4f 00 2d+09:59:04.864 WRITE FPDMA QUEUED
61 00 28 ff ff ff 4f 00 2d+09:59:04.863 WRITE FPDMA QUEUED
61 00 00 ff ff ff 4f 00 2d+09:59:04.863 WRITE FPDMA QUEUED
60 00 80 ff ff ff 4f 00 2d+09:59:04.863 READ FPDMA QUEUED
61 00 40 ff ff ff 4f 00 2d+09:59:04.861 WRITE FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 40% 12786 4005685760
# 2 Extended offline Completed without error 00% 9311 -
# 3 Extended offline Completed without error 00% 5247 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Attempting to write to this bad LBA with dd results in an I/O error:
leo # dd if=/dev/sdb of=/dev/null bs=512 skip=4005685760 count=1
dd: error reading '/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 5.7316 s, 0.0 kB/s
## A few blocks before are fine
leo # dd if=/dev/sdb of=/dev/null bs=512 skip=4005685750 count=1
1+0 records in
1+0 records out
512 bytes copied, 0.00874588 s, 58.5 kB/s
June 7, 2019
I unwisely rebuilt the ZFS pool with these ST3000DM001 drives in late 2019 because they still _seemed_ good and perhaps the issues I've had were related to flaky SATA cables.
The same drive that faulted in September failed again. This time, it no longer can be recognized by SMART. I replaced this failed drive in the array with the same model with serial number W1F0SJ27.
September 20, 2019
One of the 3TB disks in my array went offline this morning.
# zpool status storage
...
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-Hitachi_HUS724030ALE641_P8GH7GVR ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HHZ7 FAULTED 0 0 0 too many errors
ata-Hitachi_HUS724030ALE641_P8GBAW7P ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HGS6 ONLINE 0 0 0
ata-Hitachi_HUS724030ALE641_P8G9NZSR ONLINE 0 0 0
# smartctl -a /dev/sdd
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.0.9-100.fc28.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1CH166
Serial Number: Z1F4HHZ7
LU WWN Device Id: 5 000c50 065c6f541
Firmware Version: CC29
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Sep 20 09:20:05 2019 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 584) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 319) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 097 097 006 Pre-fail Always - 42489445
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 632
5 Reallocated_Sector_Ct 0x0033 099 099 010 Pre-fail Always - 1160
7 Seek_Error_Rate 0x000f 086 060 030 Pre-fail Always - 406992069
9 Power_On_Hours 0x0032 064 064 000 Old_age Always - 32374
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 68
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 086 086 000 Old_age Always - 14
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4
190 Airflow_Temperature_Cel 0x0022 071 055 045 Old_age Always - 29 (Min/Max 25/33)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 524
193 Load_Cycle_Count 0x0032 079 079 000 Old_age Always - 42761
194 Temperature_Celsius 0x0022 029 045 000 Old_age Always - 29 (0 16 0 0 0)
197 Current_Pending_Sector 0x0012 001 001 000 Old_age Always - 39672
198 Offline_Uncorrectable 0x0010 001 001 000 Old_age Offline - 39672
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 18628h+23m+18.929s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 104853181293
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 166965854582
SMART Error Log Version: 1
ATA Error Count: 14 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 14 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:46.753 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:46.752 READ FPDMA QUEUED
ea 00 00 00 00 00 a0 00 28d+08:28:46.658 FLUSH CACHE EXT
61 00 08 ff ff ff 4f 00 28d+08:28:46.624 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:46.624 WRITE FPDMA QUEUED
Error 13 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:40.736 READ FPDMA QUEUED
ea 00 00 00 00 00 a0 00 28d+08:28:40.248 FLUSH CACHE EXT
61 00 08 ff ff ff 4f 00 28d+08:28:40.247 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:40.247 WRITE FPDMA QUEUED
61 00 08 00 0b 00 40 00 28d+08:28:40.246 WRITE FPDMA QUEUED
Error 12 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 28d+08:28:37.790 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:36.075 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:36.000 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:35.990 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:35.939 READ FPDMA QUEUED
Error 11 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 28d+08:28:32.734 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:32.734 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.681 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.620 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.544 READ FPDMA QUEUED
Error 10 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:23.596 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.567 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.542 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.532 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.496 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 20997 -
# 2 Extended offline Completed without error 00% 20860 -
# 3 Extended offline Completed without error 00% 9897 -
# 4 Extended offline Interrupted (host reset) 00% 6482 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
May 18, 2017
Here's a dmesg log of another dying disk.
May 17 15:43:18 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 17 15:43:18 nas kernel: ata14.00: failed command: READ DMA EXT
May 17 15:43:18 nas kernel: ata14.00: cmd 25/00:08:88:08:f6/00:00:3c:00:00/e0 tag 0 dma 4096 in#012 res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 17 15:43:18 nas kernel: ata14.00: status: { DRDY }
May 17 15:43:18 nas kernel: ata14: hard resetting link
May 17 15:43:18 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:23 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:28 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:28 nas kernel: ata14: hard resetting link
May 17 15:43:28 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:33 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:38 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:38 nas kernel: ata14: hard resetting link
May 17 15:43:38 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:43 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:44:13 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:44:13 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 17 15:44:13 nas kernel: ata14: hard resetting link
May 17 15:44:13 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:13 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:13 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:19 nas kernel: ata14: hard resetting link
May 17 15:44:19 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:19 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:19 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:24 nas kernel: ata14: hard resetting link
May 17 15:44:24 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:25 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:25 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:25 nas kernel: ata14.00: disabled
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Sense Key : Not Ready [current]
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Add. Sense: Logical unit not ready, hard reset required
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: Read(10) 28 00 3c f6 08 88 00 00 08 00
May 17 15:44:25 nas kernel: blk_update_request: I/O error, dev sdi, sector 1022757000
May 17 15:44:25 nas kernel: ata14: EH complete
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 19 00:51:51 nas kernel: ata14: SATA max UDMA/133 cmd 0x960 ctl 0xb60 bmdma 0xc808 irq 23
May 19 00:51:56 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:51:56 nas kernel: ata14.00: ATA-8: ST2000DM001-9YN164, CC4C, max UDMA/133
May 19 00:51:56 nas kernel: ata14.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 0/32)
May 19 00:51:56 nas kernel: ata14.00: configured for UDMA/133
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 4096-byte physical blocks
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write Protect is off
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 00:51:56 nas kernel: sdi: sdi1
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Attached SCSI disk
May 19 00:53:41 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:53:41 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:53:41 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012 res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:53:41 nas kernel: ata14.00: status: { DRDY }
May 19 00:53:41 nas kernel: ata14: hard resetting link
May 19 00:53:41 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:53:47 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:53:49 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:53:49 nas kernel: ata14.00: configured for UDMA/133
May 19 00:53:49 nas kernel: ata14: EH complete
May 19 00:54:45 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:54:45 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:54:45 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012 res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:54:45 nas kernel: ata14.00: status: { DRDY }
May 19 00:54:45 nas kernel: ata14: hard resetting link
May 19 00:54:45 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:54:51 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:54:55 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:54:55 nas kernel: ata14: hard resetting link
May 19 00:54:55 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:01 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:05 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:05 nas kernel: ata14: hard resetting link
May 19 00:55:05 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:11 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:40 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:40 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 19 00:55:40 nas kernel: ata14: hard resetting link
May 19 00:55:40 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:45 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:45 nas kernel: ata14: reset failed, giving up
May 19 00:55:45 nas kernel: ata14.00: disabled
May 19 00:55:45 nas kernel: ata14: EH complete
May 19 01:06:47 nas kernel: ata14: SATA max UDMA/100 mmio m1024@0xfd7ff000 tf 0xfd7ff2c0 irq 17
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 4096-byte physical blocks
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write Protect is off
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Attached SCSI disk
May 19 01:06:49 nas kernel: ata14: SATA link down (SStatus 0 SControl 310)
May 19 01:13:54 nas kernel: sdi:
May 19 01:15:52 nas kernel: sdi: sdi1
The disk went offline pretty quick, so ZFS took it off the pool equally as quick.
Disks
# zpool status storage
pool: storage
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 7h8m with 0 errors on Fri Apr 14 10:23:51 2017
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0877352-part1 ONLINE 0 0 0
ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0840640-part1 ONLINE 0 0 0
ata-ST2000DM001-9YN164_W2F0B1GV-part1 UNAVAIL 0 0 0
ata-ST2000DM001-1CH164_Z1E37XQR-part1 ONLINE 0 0 0
ata-ST2000DL003-9VT166_5YD3VR3A-part1 ONLINE 0 0 0
errors: No known data errors
An older Entry
I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing it because it's only one drive and it's acting like a raid 0 (right?).
Unfortunately, my backups failed for a few files and then this turned up:
root@nas:/mnt/nas4/images/christine/2015-01-25 Annual Dinner- we host# zpool status -xv
pool: data
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 5h45m with 0 errors on Tue Jun 17 00:13:18 2014
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 6.31K
raidz1-0 DEGRADED 0 0 13.2K
ata-ST3000DM001-1CH166_Z1F4HHZ7 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HGS6 ONLINE 0 0 0
ata-ST3000DM001-9YN166_W1F0SJ27 FAULTED 1 2 0 too many errors
errors: Permanent errors have been detected in the following files:
data:<0x0>
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3255.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3256.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3257.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3258.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3259.JPG
It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it:
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
Vendor: /8:0:0:0
Product:
User Capacity: 600,332,565,813,390,450 bytes [600 PB]
Logical block size: 774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
The logs show the drive failing:
2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2270 Jan 25 22:13:54 nas kernel: ata7.00: failed command: READ DMA EXT
2271 Jan 25 22:13:54 nas kernel: ata7.00: cmd 25/00:00:6f:7c:55/00:02:37:00:00/e0 tag 0 dma 262144 in
2272 Jan 25 22:13:54 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2273 Jan 25 22:13:54 nas kernel: ata7.00: status: { Busy }
2274 Jan 25 22:13:54 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2275 Jan 25 22:13:54 nas kernel: ata7: hard resetting link
2276 Jan 25 22:13:59 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2277 Jan 25 22:13:59 nas kernel: ata7.00: configured for UDMA/33
2278 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2279 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2280 Jan 25 22:13:59 nas kernel: Descriptor sense data with sense descriptors (in hex):
2281 Jan 25 22:13:59 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2282 Jan 25 22:13:59 nas kernel: d8 d8 d8 d8
2283 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2284 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 37 55 7c 6f 00 02 00 00
2285 Jan 25 22:13:59 nas kernel: ata7: EH complete
2286 Jan 25 22:15:09 nas kernel: NFS: v4 server 10.1.1.6 does not accept raw uid/gids. Reenabling the idmapper.
2287 Jan 25 22:15:41 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2288 Jan 25 22:15:41 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2289 Jan 25 22:15:41 nas kernel: ata7.00: failed command: READ DMA EXT
2290 Jan 25 22:15:41 nas kernel: ata7.00: cmd 25/00:80:c0:2d:c0/00:01:65:00:00/e0 tag 0 dma 196608 in
2291 Jan 25 22:15:41 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2292 Jan 25 22:15:41 nas kernel: ata7.00: status: { Busy }
2293 Jan 25 22:15:41 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2294 Jan 25 22:15:41 nas kernel: ata7: hard resetting link
2295 Jan 25 22:15:46 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2296 Jan 25 22:15:46 nas kernel: ata7.00: configured for UDMA/33
2297 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2298 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2299 Jan 25 22:15:46 nas kernel: Descriptor sense data with sense descriptors (in hex):
2300 Jan 25 22:15:46 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2301 Jan 25 22:15:46 nas kernel: d8 d8 d8 d8
2302 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2303 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2d c0 00 01 80 00
2304 Jan 25 22:15:46 nas kernel: ata7: EH complete
2305 Jan 25 22:15:46 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2306 Jan 25 22:15:46 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2307 Jan 25 22:15:46 nas kernel: ata7.00: failed command: READ DMA EXT
2308 Jan 25 22:15:46 nas kernel: ata7.00: cmd 25/00:00:40:2f:c0/00:03:65:00:00/e0 tag 0 dma 393216 in
2309 Jan 25 22:15:46 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2310 Jan 25 22:15:46 nas kernel: ata7.00: status: { Busy }
2311 Jan 25 22:15:46 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2312 Jan 25 22:15:46 nas kernel: ata7: hard resetting link
2313 Jan 25 22:15:51 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2314 Jan 25 22:15:51 nas kernel: ata7.00: configured for UDMA/33
2315 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2316 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2317 Jan 25 22:15:51 nas kernel: Descriptor sense data with sense descriptors (in hex):
2318 Jan 25 22:15:51 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2319 Jan 25 22:15:51 nas kernel: d8 d8 d8 d8
2320 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2321 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2f 40 00 03 00 00
2322 Jan 25 22:15:51 nas kernel: ata7: EH complete
2323 Jan 25 22:15:51 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2324 Jan 25 22:15:51 nas kernel: ata7: SError: { PHYRdyChg }
2325 Jan 25 22:15:51 nas kernel: ata7.00: failed command: READ DMA EXT
2326 Jan 25 22:15:51 nas kernel: ata7.00: cmd 25/00:14:70:0f:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2327 Jan 25 22:15:51 nas kernel: res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2328 Jan 25 22:15:51 nas kernel: ata7.00: status: { Busy }
2329 Jan 25 22:15:51 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2330 Jan 25 22:15:51 nas kernel: ata7: hard resetting link
2331 Jan 25 22:15:56 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2332 Jan 25 22:15:56 nas kernel: ata7.00: configured for UDMA/33
2333 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2334 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2335 Jan 25 22:15:56 nas kernel: Descriptor sense data with sense descriptors (in hex):
2336 Jan 25 22:15:56 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2337 Jan 25 22:15:56 nas kernel: d0 d0 d0 d0
2338 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2339 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 0f 70 00 00 14 00
2340 Jan 25 22:15:56 nas kernel: ata7: EH complete
2341 Jan 25 22:15:57 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2342 Jan 25 22:15:57 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2343 Jan 25 22:15:57 nas kernel: ata7.00: failed command: READ DMA EXT
2344 Jan 25 22:15:57 nas kernel: ata7.00: cmd 25/00:14:77:10:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2345 Jan 25 22:15:57 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2346 Jan 25 22:15:57 nas kernel: ata7.00: status: { Busy }
2347 Jan 25 22:15:57 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2348 Jan 25 22:15:57 nas kernel: ata7: hard resetting link
2349 Jan 25 22:16:02 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2350 Jan 25 22:16:02 nas kernel: ata7.00: configured for UDMA/33
2351 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2352 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2353 Jan 25 22:16:02 nas kernel: Descriptor sense data with sense descriptors (in hex):
2354 Jan 25 22:16:02 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2355 Jan 25 22:16:02 nas kernel: d8 d8 d8 d8
2356 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2357 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 10 77 00 00 14 00
2358 Jan 25 22:16:02 nas kernel: ata7: EH complete
2359 Jan 25 22:16:02 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2360 Jan 25 22:16:02 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2361 Jan 25 22:16:02 nas kernel: ata7.00: failed command: READ DMA EXT
2362 Jan 25 22:16:02 nas kernel: ata7.00: cmd 25/00:80:c0:3a:c0/00:02:65:00:00/e0 tag 0 dma 327680 in
2363 Jan 25 22:16:02 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2364 Jan 25 22:16:02 nas kernel: ata7.00: status: { Busy }
2365 Jan 25 22:16:02 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2366 Jan 25 22:16:02 nas kernel: ata7: hard resetting link
2367 Jan 25 22:16:07 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2368 Jan 25 22:16:07 nas kernel: ata7.00: configured for UDMA/33
2369 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2370 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2371 Jan 25 22:16:07 nas kernel: Descriptor sense data with sense descriptors (in hex):
2372 Jan 25 22:16:07 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2373 Jan 25 22:16:07 nas kernel: d8 d8 d8 d8
2374 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2375 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 3a c0 00 02 80 00
2376 Jan 25 22:16:07 nas kernel: ata7: EH complete
2377 Jan 25 22:16:07 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2378 Jan 25 22:16:07 nas kernel: ata7: SError: { PHYRdyChg }
2379 Jan 25 22:16:07 nas kernel: ata7.00: failed command: WRITE DMA EXT
2380 Jan 25 22:16:07 nas kernel: ata7.00: cmd 35/00:03:d6:04:48/00:00:69:00:00/e0 tag 0 dma 1536 out
2381 Jan 25 22:16:07 nas kernel: res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2382 Jan 25 22:16:07 nas kernel: ata7.00: status: { Busy }
2383 Jan 25 22:16:07 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2384 Jan 25 22:16:07 nas kernel: ata7: hard resetting link
2385 Jan 25 22:16:12 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2386 Jan 25 22:16:12 nas kernel: ata7.00: configured for UDMA/33
2387 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2388 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2389 Jan 25 22:16:12 nas kernel: Descriptor sense data with sense descriptors (in hex):
2390 Jan 25 22:16:12 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2391 Jan 25 22:16:12 nas kernel: d0 d0 d0 d0
2392 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2393 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] CDB: Write(10): 2a 00 69 48 04 d6 00 00 03 00
2394 Jan 25 22:16:12 nas kernel: ata7: EH complete
2395 Jan 25 22:16:13 nas kernel: ata7: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2396 Jan 25 22:16:13 nas kernel: ata7: SError: { PHYRdyChg }
2397 Jan 25 22:16:13 nas kernel: ata7: hard resetting link
2398 Jan 25 22:16:18 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2399 Jan 25 22:16:18 nas kernel: ata7.00: configured for UDMA/33
2400 Jan 25 22:16:18 nas kernel: ata7: EH complete
2401 Jan 25 22:16:24 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2402 Jan 25 22:16:24 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2403 Jan 25 22:16:24 nas kernel: ata7.00: failed command: READ DMA EXT
2404 Jan 25 22:16:24 nas kernel: ata7.00: cmd 25/00:00:c0:0b:c3/00:03:65:00:00/e0 tag 0 dma 393216 in
2405 Jan 25 22:16:24 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2406 Jan 25 22:16:24 nas kernel: ata7.00: status: { Busy }
2407 Jan 25 22:16:24 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2408 Jan 25 22:16:24 nas kernel: ata7: hard resetting link
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33
... this goes on for another 12 minutes.
So, lesson learned. Test backups and replace dead drives ASAP.