Dying Hard Disk
From Leo's Notes
This page was last edited on 20 September 2019, at 15:26.
September 20, 2019
One of the 3TB disks in my array went offline this morning.
# zpool status storage
...
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-Hitachi_HUS724030ALE641_P8GH7GVR ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HHZ7 FAULTED 0 0 0 too many errors
ata-Hitachi_HUS724030ALE641_P8GBAW7P ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HGS6 ONLINE 0 0 0
ata-Hitachi_HUS724030ALE641_P8G9NZSR ONLINE 0 0 0
# smartctl -a /dev/sdd
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-5.0.9-100.fc28.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1CH166
Serial Number: Z1F4HHZ7
LU WWN Device Id: 5 000c50 065c6f541
Firmware Version: CC29
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Sep 20 09:20:05 2019 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 584) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 319) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 097 097 006 Pre-fail Always - 42489445
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 632
5 Reallocated_Sector_Ct 0x0033 099 099 010 Pre-fail Always - 1160
7 Seek_Error_Rate 0x000f 086 060 030 Pre-fail Always - 406992069
9 Power_On_Hours 0x0032 064 064 000 Old_age Always - 32374
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 68
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 086 086 000 Old_age Always - 14
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4
190 Airflow_Temperature_Cel 0x0022 071 055 045 Old_age Always - 29 (Min/Max 25/33)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 524
193 Load_Cycle_Count 0x0032 079 079 000 Old_age Always - 42761
194 Temperature_Celsius 0x0022 029 045 000 Old_age Always - 29 (0 16 0 0 0)
197 Current_Pending_Sector 0x0012 001 001 000 Old_age Always - 39672
198 Offline_Uncorrectable 0x0010 001 001 000 Old_age Offline - 39672
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 18628h+23m+18.929s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 104853181293
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 166965854582
SMART Error Log Version: 1
ATA Error Count: 14 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 14 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:46.753 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:46.752 READ FPDMA QUEUED
ea 00 00 00 00 00 a0 00 28d+08:28:46.658 FLUSH CACHE EXT
61 00 08 ff ff ff 4f 00 28d+08:28:46.624 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:46.624 WRITE FPDMA QUEUED
Error 13 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:40.736 READ FPDMA QUEUED
ea 00 00 00 00 00 a0 00 28d+08:28:40.248 FLUSH CACHE EXT
61 00 08 ff ff ff 4f 00 28d+08:28:40.247 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:40.247 WRITE FPDMA QUEUED
61 00 08 00 0b 00 40 00 28d+08:28:40.246 WRITE FPDMA QUEUED
Error 12 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 28d+08:28:37.790 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:36.075 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:36.000 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:35.990 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:35.939 READ FPDMA QUEUED
Error 11 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 28d+08:28:32.734 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 28d+08:28:32.734 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.681 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.620 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:29.544 READ FPDMA QUEUED
Error 10 occurred at disk power-on lifetime: 32368 hours (1348 days + 16 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 28d+08:28:23.596 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.567 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.542 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.532 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 28d+08:28:23.496 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 20997 -
# 2 Extended offline Completed without error 00% 20860 -
# 3 Extended offline Completed without error 00% 9897 -
# 4 Extended offline Interrupted (host reset) 00% 6482 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
May 18, 2017
Here's a dmesg log of another dying disk.
May 17 15:43:18 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 17 15:43:18 nas kernel: ata14.00: failed command: READ DMA EXT
May 17 15:43:18 nas kernel: ata14.00: cmd 25/00:08:88:08:f6/00:00:3c:00:00/e0 tag 0 dma 4096 in#012 res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 17 15:43:18 nas kernel: ata14.00: status: { DRDY }
May 17 15:43:18 nas kernel: ata14: hard resetting link
May 17 15:43:18 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:23 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:28 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:28 nas kernel: ata14: hard resetting link
May 17 15:43:28 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:33 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:43:38 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:43:38 nas kernel: ata14: hard resetting link
May 17 15:43:38 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:43:43 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 17 15:44:13 nas kernel: ata14: SRST failed (errno=-16)
May 17 15:44:13 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 17 15:44:13 nas kernel: ata14: hard resetting link
May 17 15:44:13 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:13 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:13 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:19 nas kernel: ata14: hard resetting link
May 17 15:44:19 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:19 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:19 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:24 nas kernel: ata14: hard resetting link
May 17 15:44:24 nas kernel: ata14: nv: skipping hardreset on occupied port
May 17 15:44:25 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 17 15:44:25 nas kernel: ata14.00: revalidation failed (errno=-2)
May 17 15:44:25 nas kernel: ata14.00: disabled
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Sense Key : Not Ready [current]
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 Add. Sense: Logical unit not ready, hard reset required
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: Read(10) 28 00 3c f6 08 88 00 00 08 00
May 17 15:44:25 nas kernel: blk_update_request: I/O error, dev sdi, sector 1022757000
May 17 15:44:25 nas kernel: ata14: EH complete
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is on
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Write Protect is off
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:44:25 nas kernel: sd 13:0:0:0: [sdi] Sense not available.
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 16:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 18:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 22:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 17 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 00:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 01:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 02:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 03:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 04:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 05:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 06:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 07:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 08:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 09:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 10:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 11:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 12:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 13:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:17:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 14:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 15:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 16:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 17:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 18:47:20 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 19:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 20:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 21:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 22:47:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:17:22 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 18 23:47:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 00 00 00 00 00 00 00 00 00 00 e5 00
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
May 19 00:17:21 nas kernel: sd 13:0:0:0: [sdi] tag#0 CDB: ATA command pass through(16) 85 06 2c 00 da 00 00 00 00 00 4f 00 c2 00 b0 00
May 19 00:51:51 nas kernel: ata14: SATA max UDMA/133 cmd 0x960 ctl 0xb60 bmdma 0xc808 irq 23
May 19 00:51:56 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:51:56 nas kernel: ata14.00: ATA-8: ST2000DM001-9YN164, CC4C, max UDMA/133
May 19 00:51:56 nas kernel: ata14.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 0/32)
May 19 00:51:56 nas kernel: ata14.00: configured for UDMA/133
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] 4096-byte physical blocks
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write Protect is off
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 00:51:56 nas kernel: sdi: sdi1
May 19 00:51:56 nas kernel: sd 12:0:0:0: [sdi] Attached SCSI disk
May 19 00:53:41 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:53:41 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:53:41 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012 res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:53:41 nas kernel: ata14.00: status: { DRDY }
May 19 00:53:41 nas kernel: ata14: hard resetting link
May 19 00:53:41 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:53:47 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:53:49 nas kernel: ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
May 19 00:53:49 nas kernel: ata14.00: configured for UDMA/133
May 19 00:53:49 nas kernel: ata14: EH complete
May 19 00:54:45 nas kernel: ata14.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
May 19 00:54:45 nas kernel: ata14.00: failed command: IDENTIFY DEVICE
May 19 00:54:45 nas kernel: ata14.00: cmd ec/00:01:00:00:00/00:00:00:00:00/00 tag 0 pio 512 in#012 res 40/00:01:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
May 19 00:54:45 nas kernel: ata14.00: status: { DRDY }
May 19 00:54:45 nas kernel: ata14: hard resetting link
May 19 00:54:45 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:54:51 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:54:55 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:54:55 nas kernel: ata14: hard resetting link
May 19 00:54:55 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:01 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:05 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:05 nas kernel: ata14: hard resetting link
May 19 00:55:05 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:11 nas kernel: ata14: link is slow to respond, please be patient (ready=0)
May 19 00:55:40 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:40 nas kernel: ata14: limiting SATA link speed to 1.5 Gbps
May 19 00:55:40 nas kernel: ata14: hard resetting link
May 19 00:55:40 nas kernel: ata14: nv: skipping hardreset on occupied port
May 19 00:55:45 nas kernel: ata14: SRST failed (errno=-16)
May 19 00:55:45 nas kernel: ata14: reset failed, giving up
May 19 00:55:45 nas kernel: ata14.00: disabled
May 19 00:55:45 nas kernel: ata14: EH complete
May 19 01:06:47 nas kernel: ata14: SATA max UDMA/100 mmio m1024@0xfd7ff000 tf 0xfd7ff2c0 irq 17
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 3907029168 512-byte logical blocks: (2.00 TB/1.82 TiB)
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] 4096-byte physical blocks
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write Protect is off
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 19 01:06:48 nas kernel: sd 9:0:0:0: [sdi] Attached SCSI disk
May 19 01:06:49 nas kernel: ata14: SATA link down (SStatus 0 SControl 310)
May 19 01:13:54 nas kernel: sdi:
May 19 01:15:52 nas kernel: sdi: sdi1
The disk went offline pretty quick, so ZFS took it off the pool equally as quick.
Disks
# zpool status storage
pool: storage
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 7h8m with 0 errors on Fri Apr 14 10:23:51 2017
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0877352-part1 ONLINE 0 0 0
ata-WDC_WD20EARS-00J99B0_WD-WCAWZ0840640-part1 ONLINE 0 0 0
ata-ST2000DM001-9YN164_W2F0B1GV-part1 UNAVAIL 0 0 0
ata-ST2000DM001-1CH164_Z1E37XQR-part1 ONLINE 0 0 0
ata-ST2000DL003-9VT166_5YD3VR3A-part1 ONLINE 0 0 0
errors: No known data errors
An older Entry
I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing it because it's only one drive and it's acting like a raid 0 (right?).
Unfortunately, my backups failed for a few files and then this turned up:
root@nas:/mnt/nas4/images/christine/2015-01-25 Annual Dinner- we host# zpool status -xv
pool: data
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 5h45m with 0 errors on Tue Jun 17 00:13:18 2014
config:
NAME STATE READ WRITE CKSUM
data DEGRADED 0 0 6.31K
raidz1-0 DEGRADED 0 0 13.2K
ata-ST3000DM001-1CH166_Z1F4HHZ7 ONLINE 0 0 0
ata-ST3000DM001-1CH166_Z1F4HGS6 ONLINE 0 0 0
ata-ST3000DM001-9YN166_W1F0SJ27 FAULTED 1 2 0 too many errors
errors: Permanent errors have been detected in the following files:
data:<0x0>
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3255.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3256.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3257.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3258.JPG
/data/images/christine/2015-01-25 Annual Dinner- we host/IMG_3259.JPG
It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it:
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
Vendor: /8:0:0:0
Product:
User Capacity: 600,332,565,813,390,450 bytes [600 PB]
Logical block size: 774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
The logs show the drive failing:
2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2270 Jan 25 22:13:54 nas kernel: ata7.00: failed command: READ DMA EXT
2271 Jan 25 22:13:54 nas kernel: ata7.00: cmd 25/00:00:6f:7c:55/00:02:37:00:00/e0 tag 0 dma 262144 in
2272 Jan 25 22:13:54 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2273 Jan 25 22:13:54 nas kernel: ata7.00: status: { Busy }
2274 Jan 25 22:13:54 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2275 Jan 25 22:13:54 nas kernel: ata7: hard resetting link
2276 Jan 25 22:13:59 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2277 Jan 25 22:13:59 nas kernel: ata7.00: configured for UDMA/33
2278 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2279 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2280 Jan 25 22:13:59 nas kernel: Descriptor sense data with sense descriptors (in hex):
2281 Jan 25 22:13:59 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2282 Jan 25 22:13:59 nas kernel: d8 d8 d8 d8
2283 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2284 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 37 55 7c 6f 00 02 00 00
2285 Jan 25 22:13:59 nas kernel: ata7: EH complete
2286 Jan 25 22:15:09 nas kernel: NFS: v4 server 10.1.1.6 does not accept raw uid/gids. Reenabling the idmapper.
2287 Jan 25 22:15:41 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2288 Jan 25 22:15:41 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2289 Jan 25 22:15:41 nas kernel: ata7.00: failed command: READ DMA EXT
2290 Jan 25 22:15:41 nas kernel: ata7.00: cmd 25/00:80:c0:2d:c0/00:01:65:00:00/e0 tag 0 dma 196608 in
2291 Jan 25 22:15:41 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2292 Jan 25 22:15:41 nas kernel: ata7.00: status: { Busy }
2293 Jan 25 22:15:41 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2294 Jan 25 22:15:41 nas kernel: ata7: hard resetting link
2295 Jan 25 22:15:46 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2296 Jan 25 22:15:46 nas kernel: ata7.00: configured for UDMA/33
2297 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2298 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2299 Jan 25 22:15:46 nas kernel: Descriptor sense data with sense descriptors (in hex):
2300 Jan 25 22:15:46 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2301 Jan 25 22:15:46 nas kernel: d8 d8 d8 d8
2302 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2303 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2d c0 00 01 80 00
2304 Jan 25 22:15:46 nas kernel: ata7: EH complete
2305 Jan 25 22:15:46 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2306 Jan 25 22:15:46 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2307 Jan 25 22:15:46 nas kernel: ata7.00: failed command: READ DMA EXT
2308 Jan 25 22:15:46 nas kernel: ata7.00: cmd 25/00:00:40:2f:c0/00:03:65:00:00/e0 tag 0 dma 393216 in
2309 Jan 25 22:15:46 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2310 Jan 25 22:15:46 nas kernel: ata7.00: status: { Busy }
2311 Jan 25 22:15:46 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2312 Jan 25 22:15:46 nas kernel: ata7: hard resetting link
2313 Jan 25 22:15:51 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2314 Jan 25 22:15:51 nas kernel: ata7.00: configured for UDMA/33
2315 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2316 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2317 Jan 25 22:15:51 nas kernel: Descriptor sense data with sense descriptors (in hex):
2318 Jan 25 22:15:51 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2319 Jan 25 22:15:51 nas kernel: d8 d8 d8 d8
2320 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2321 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2f 40 00 03 00 00
2322 Jan 25 22:15:51 nas kernel: ata7: EH complete
2323 Jan 25 22:15:51 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2324 Jan 25 22:15:51 nas kernel: ata7: SError: { PHYRdyChg }
2325 Jan 25 22:15:51 nas kernel: ata7.00: failed command: READ DMA EXT
2326 Jan 25 22:15:51 nas kernel: ata7.00: cmd 25/00:14:70:0f:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2327 Jan 25 22:15:51 nas kernel: res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2328 Jan 25 22:15:51 nas kernel: ata7.00: status: { Busy }
2329 Jan 25 22:15:51 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2330 Jan 25 22:15:51 nas kernel: ata7: hard resetting link
2331 Jan 25 22:15:56 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2332 Jan 25 22:15:56 nas kernel: ata7.00: configured for UDMA/33
2333 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2334 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2335 Jan 25 22:15:56 nas kernel: Descriptor sense data with sense descriptors (in hex):
2336 Jan 25 22:15:56 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2337 Jan 25 22:15:56 nas kernel: d0 d0 d0 d0
2338 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2339 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 0f 70 00 00 14 00
2340 Jan 25 22:15:56 nas kernel: ata7: EH complete
2341 Jan 25 22:15:57 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2342 Jan 25 22:15:57 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2343 Jan 25 22:15:57 nas kernel: ata7.00: failed command: READ DMA EXT
2344 Jan 25 22:15:57 nas kernel: ata7.00: cmd 25/00:14:77:10:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2345 Jan 25 22:15:57 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2346 Jan 25 22:15:57 nas kernel: ata7.00: status: { Busy }
2347 Jan 25 22:15:57 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2348 Jan 25 22:15:57 nas kernel: ata7: hard resetting link
2349 Jan 25 22:16:02 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2350 Jan 25 22:16:02 nas kernel: ata7.00: configured for UDMA/33
2351 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2352 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2353 Jan 25 22:16:02 nas kernel: Descriptor sense data with sense descriptors (in hex):
2354 Jan 25 22:16:02 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2355 Jan 25 22:16:02 nas kernel: d8 d8 d8 d8
2356 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2357 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 10 77 00 00 14 00
2358 Jan 25 22:16:02 nas kernel: ata7: EH complete
2359 Jan 25 22:16:02 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2360 Jan 25 22:16:02 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2361 Jan 25 22:16:02 nas kernel: ata7.00: failed command: READ DMA EXT
2362 Jan 25 22:16:02 nas kernel: ata7.00: cmd 25/00:80:c0:3a:c0/00:02:65:00:00/e0 tag 0 dma 327680 in
2363 Jan 25 22:16:02 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2364 Jan 25 22:16:02 nas kernel: ata7.00: status: { Busy }
2365 Jan 25 22:16:02 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2366 Jan 25 22:16:02 nas kernel: ata7: hard resetting link
2367 Jan 25 22:16:07 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2368 Jan 25 22:16:07 nas kernel: ata7.00: configured for UDMA/33
2369 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2370 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2371 Jan 25 22:16:07 nas kernel: Descriptor sense data with sense descriptors (in hex):
2372 Jan 25 22:16:07 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2373 Jan 25 22:16:07 nas kernel: d8 d8 d8 d8
2374 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2375 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 3a c0 00 02 80 00
2376 Jan 25 22:16:07 nas kernel: ata7: EH complete
2377 Jan 25 22:16:07 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2378 Jan 25 22:16:07 nas kernel: ata7: SError: { PHYRdyChg }
2379 Jan 25 22:16:07 nas kernel: ata7.00: failed command: WRITE DMA EXT
2380 Jan 25 22:16:07 nas kernel: ata7.00: cmd 35/00:03:d6:04:48/00:00:69:00:00/e0 tag 0 dma 1536 out
2381 Jan 25 22:16:07 nas kernel: res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2382 Jan 25 22:16:07 nas kernel: ata7.00: status: { Busy }
2383 Jan 25 22:16:07 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2384 Jan 25 22:16:07 nas kernel: ata7: hard resetting link
2385 Jan 25 22:16:12 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2386 Jan 25 22:16:12 nas kernel: ata7.00: configured for UDMA/33
2387 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2388 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2389 Jan 25 22:16:12 nas kernel: Descriptor sense data with sense descriptors (in hex):
2390 Jan 25 22:16:12 nas kernel: 72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2391 Jan 25 22:16:12 nas kernel: d0 d0 d0 d0
2392 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2393 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] CDB: Write(10): 2a 00 69 48 04 d6 00 00 03 00
2394 Jan 25 22:16:12 nas kernel: ata7: EH complete
2395 Jan 25 22:16:13 nas kernel: ata7: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2396 Jan 25 22:16:13 nas kernel: ata7: SError: { PHYRdyChg }
2397 Jan 25 22:16:13 nas kernel: ata7: hard resetting link
2398 Jan 25 22:16:18 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2399 Jan 25 22:16:18 nas kernel: ata7.00: configured for UDMA/33
2400 Jan 25 22:16:18 nas kernel: ata7: EH complete
2401 Jan 25 22:16:24 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2402 Jan 25 22:16:24 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2403 Jan 25 22:16:24 nas kernel: ata7.00: failed command: READ DMA EXT
2404 Jan 25 22:16:24 nas kernel: ata7.00: cmd 25/00:00:c0:0b:c3/00:03:65:00:00/e0 tag 0 dma 393216 in
2405 Jan 25 22:16:24 nas kernel: res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2406 Jan 25 22:16:24 nas kernel: ata7.00: status: { Busy }
2407 Jan 25 22:16:24 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2408 Jan 25 22:16:24 nas kernel: ata7: hard resetting link
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33
... this goes on for another 12 minutes.
So, lesson learned. Test backups and replace dead drives ASAP.