Dying Hard Disk: Difference between revisions

From Leo's Notes
This page was last edited on 30 March 2017, at 20:15.
← Older editNewer edit →
VisualWikitext
No edit summary
m formatting
Line 1: Line 1:


I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing because it's only one drive and it's acting like a raid 0 (right?).
I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing it because it's only one drive and it's acting like a raid 0 (right?).


Unfortunately, my backups failed for a few files and then this turned up:
Unfortunately, my backups failed for a few files and then this turned up:


<syntaxhighlight lang="text">
{{highlight|lang=terminal|code=
root@nas:/mnt/nas4/images/christine/2015-01-25  Annual Dinner- we host# zpool status -xv
root@nas:/mnt/nas4/images/christine/2015-01-25  Annual Dinner- we host# zpool status -xv
   pool: data
   pool: data
Line 31: Line 31:
         /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3258.JPG
         /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3258.JPG
         /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3259.JPG
         /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3259.JPG
</syntaxhighlight>
}}


It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it:
It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it:


<syntaxhighlight lang="text">
{{highlight|lang=terminal|code=
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
Line 45: Line 45:
>> Terminate command early due to bad response to IEC mode page
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
 
}}
</syntaxhighlight>


The logs show the drive failing:
The logs show the drive failing:


<syntaxhighlight lang="text">
{{highlight|lang=terminal|code=
2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
Line 196: Line 195:
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33
}}
... this goes on for another 12 minutes.


</syntaxhighlight>


.. this goes on for another 12 minutes.
So, lesson learned. Test backups and replace dead drives ASAP.

Revision as of 20:15, 30 March 2017

I noticed that one of the drives in my ZFS array went down last month, but I couldn't bother myself with fixing it because it's only one drive and it's acting like a raid 0 (right?).

Unfortunately, my backups failed for a few files and then this turned up: 

root@nas:/mnt/nas4/images/christine/2015-01-25  Annual Dinner- we host# zpool status -xv
  pool: data
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 5h45m with 0 errors on Tue Jun 17 00:13:18 2014
config:

        NAME                                 STATE     READ WRITE CKSUM
        data                                 DEGRADED     0     0 6.31K
          raidz1-0                           DEGRADED     0     0 13.2K
            ata-ST3000DM001-1CH166_Z1F4HHZ7  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F4HGS6  ONLINE       0     0     0
            ata-ST3000DM001-9YN166_W1F0SJ27  FAULTED      1     2     0  too many errors

errors: Permanent errors have been detected in the following files:

        data:<0x0>
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3255.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3256.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3257.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3258.JPG
        /data/images/christine/2015-01-25  Annual Dinner- we host/IMG_3259.JPG

It seems like I lost 5 images that were recently saved on the storage pool. The drive is so far gone that I couldn't even run smartctl on it: 

smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-431.17.1.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

Vendor:               /8:0:0:0
Product:
User Capacity:        600,332,565,813,390,450 bytes [600 PB]
Logical block size:   774843950 bytes
>> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

The logs show the drive failing: 

2268 Jan 25 22:13:54 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2269 Jan 25 22:13:54 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2270 Jan 25 22:13:54 nas kernel: ata7.00: failed command: READ DMA EXT
2271 Jan 25 22:13:54 nas kernel: ata7.00: cmd 25/00:00:6f:7c:55/00:02:37:00:00/e0 tag 0 dma 262144 in
2272 Jan 25 22:13:54 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2273 Jan 25 22:13:54 nas kernel: ata7.00: status: { Busy }
2274 Jan 25 22:13:54 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2275 Jan 25 22:13:54 nas kernel: ata7: hard resetting link
2276 Jan 25 22:13:59 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2277 Jan 25 22:13:59 nas kernel: ata7.00: configured for UDMA/33
2278 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2279 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2280 Jan 25 22:13:59 nas kernel: Descriptor sense data with sense descriptors (in hex):
2281 Jan 25 22:13:59 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2282 Jan 25 22:13:59 nas kernel:        d8 d8 d8 d8
2283 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2284 Jan 25 22:13:59 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 37 55 7c 6f 00 02 00 00
2285 Jan 25 22:13:59 nas kernel: ata7: EH complete
2286 Jan 25 22:15:09 nas kernel: NFS: v4 server 10.1.1.6 does not accept raw uid/gids. Reenabling the idmapper.
2287 Jan 25 22:15:41 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2288 Jan 25 22:15:41 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2289 Jan 25 22:15:41 nas kernel: ata7.00: failed command: READ DMA EXT
2290 Jan 25 22:15:41 nas kernel: ata7.00: cmd 25/00:80:c0:2d:c0/00:01:65:00:00/e0 tag 0 dma 196608 in
2291 Jan 25 22:15:41 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2292 Jan 25 22:15:41 nas kernel: ata7.00: status: { Busy }
2293 Jan 25 22:15:41 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2294 Jan 25 22:15:41 nas kernel: ata7: hard resetting link
2295 Jan 25 22:15:46 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2296 Jan 25 22:15:46 nas kernel: ata7.00: configured for UDMA/33
2297 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2298 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2299 Jan 25 22:15:46 nas kernel: Descriptor sense data with sense descriptors (in hex):
2300 Jan 25 22:15:46 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2301 Jan 25 22:15:46 nas kernel:        d8 d8 d8 d8
2302 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2303 Jan 25 22:15:46 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2d c0 00 01 80 00
2304 Jan 25 22:15:46 nas kernel: ata7: EH complete
2305 Jan 25 22:15:46 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2306 Jan 25 22:15:46 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2307 Jan 25 22:15:46 nas kernel: ata7.00: failed command: READ DMA EXT
2308 Jan 25 22:15:46 nas kernel: ata7.00: cmd 25/00:00:40:2f:c0/00:03:65:00:00/e0 tag 0 dma 393216 in
2309 Jan 25 22:15:46 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2310 Jan 25 22:15:46 nas kernel: ata7.00: status: { Busy }
2311 Jan 25 22:15:46 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2312 Jan 25 22:15:46 nas kernel: ata7: hard resetting link

2313 Jan 25 22:15:51 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2314 Jan 25 22:15:51 nas kernel: ata7.00: configured for UDMA/33
2315 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2316 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2317 Jan 25 22:15:51 nas kernel: Descriptor sense data with sense descriptors (in hex):
2318 Jan 25 22:15:51 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2319 Jan 25 22:15:51 nas kernel:        d8 d8 d8 d8
2320 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2321 Jan 25 22:15:51 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 2f 40 00 03 00 00
2322 Jan 25 22:15:51 nas kernel: ata7: EH complete
2323 Jan 25 22:15:51 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2324 Jan 25 22:15:51 nas kernel: ata7: SError: { PHYRdyChg }
2325 Jan 25 22:15:51 nas kernel: ata7.00: failed command: READ DMA EXT
2326 Jan 25 22:15:51 nas kernel: ata7.00: cmd 25/00:14:70:0f:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2327 Jan 25 22:15:51 nas kernel:         res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2328 Jan 25 22:15:51 nas kernel: ata7.00: status: { Busy }
2329 Jan 25 22:15:51 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2330 Jan 25 22:15:51 nas kernel: ata7: hard resetting link
2331 Jan 25 22:15:56 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2332 Jan 25 22:15:56 nas kernel: ata7.00: configured for UDMA/33
2333 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2334 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2335 Jan 25 22:15:56 nas kernel: Descriptor sense data with sense descriptors (in hex):
2336 Jan 25 22:15:56 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2337 Jan 25 22:15:56 nas kernel:        d0 d0 d0 d0
2338 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2339 Jan 25 22:15:56 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 0f 70 00 00 14 00
2340 Jan 25 22:15:56 nas kernel: ata7: EH complete
2341 Jan 25 22:15:57 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2342 Jan 25 22:15:57 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2343 Jan 25 22:15:57 nas kernel: ata7.00: failed command: READ DMA EXT
2344 Jan 25 22:15:57 nas kernel: ata7.00: cmd 25/00:14:77:10:5e/00:00:68:00:00/e0 tag 0 dma 10240 in
2345 Jan 25 22:15:57 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2346 Jan 25 22:15:57 nas kernel: ata7.00: status: { Busy }
2347 Jan 25 22:15:57 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2348 Jan 25 22:15:57 nas kernel: ata7: hard resetting link
2349 Jan 25 22:16:02 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2350 Jan 25 22:16:02 nas kernel: ata7.00: configured for UDMA/33
2351 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2352 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2353 Jan 25 22:16:02 nas kernel: Descriptor sense data with sense descriptors (in hex):
2354 Jan 25 22:16:02 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2355 Jan 25 22:16:02 nas kernel:        d8 d8 d8 d8
2356 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2357 Jan 25 22:16:02 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 68 5e 10 77 00 00 14 00
2358 Jan 25 22:16:02 nas kernel: ata7: EH complete
2359 Jan 25 22:16:02 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0xd0000 action 0xe frozen
2360 Jan 25 22:16:02 nas kernel: ata7: SError: { PHYRdyChg CommWake 10B8B }
2361 Jan 25 22:16:02 nas kernel: ata7.00: failed command: READ DMA EXT
2362 Jan 25 22:16:02 nas kernel: ata7.00: cmd 25/00:80:c0:3a:c0/00:02:65:00:00/e0 tag 0 dma 327680 in

2363 Jan 25 22:16:02 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2364 Jan 25 22:16:02 nas kernel: ata7.00: status: { Busy }
2365 Jan 25 22:16:02 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2366 Jan 25 22:16:02 nas kernel: ata7: hard resetting link
2367 Jan 25 22:16:07 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2368 Jan 25 22:16:07 nas kernel: ata7.00: configured for UDMA/33
2369 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2370 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2371 Jan 25 22:16:07 nas kernel: Descriptor sense data with sense descriptors (in hex):
2372 Jan 25 22:16:07 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d8 d8
2373 Jan 25 22:16:07 nas kernel:        d8 d8 d8 d8
2374 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2375 Jan 25 22:16:07 nas kernel: sd 6:0:0:0: [sdf] CDB: Read(10): 28 00 65 c0 3a c0 00 02 80 00
2376 Jan 25 22:16:07 nas kernel: ata7: EH complete
2377 Jan 25 22:16:07 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2378 Jan 25 22:16:07 nas kernel: ata7: SError: { PHYRdyChg }
2379 Jan 25 22:16:07 nas kernel: ata7.00: failed command: WRITE DMA EXT
2380 Jan 25 22:16:07 nas kernel: ata7.00: cmd 35/00:03:d6:04:48/00:00:69:00:00/e0 tag 0 dma 1536 out
2381 Jan 25 22:16:07 nas kernel:         res d0/d0:d0:d0:d0:d0/d0:d0:d0:d0:d0/d0 Emask 0x12 (ATA bus error)
2382 Jan 25 22:16:07 nas kernel: ata7.00: status: { Busy }
2383 Jan 25 22:16:07 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2384 Jan 25 22:16:07 nas kernel: ata7: hard resetting link
2385 Jan 25 22:16:12 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2386 Jan 25 22:16:12 nas kernel: ata7.00: configured for UDMA/33
2387 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
2388 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Sense Key : Aborted Command [current] [descriptor]
2389 Jan 25 22:16:12 nas kernel: Descriptor sense data with sense descriptors (in hex):
2390 Jan 25 22:16:12 nas kernel:        72 0b 47 00 00 00 00 0c 00 0a 80 00 00 00 d0 d0
2391 Jan 25 22:16:12 nas kernel:        d0 d0 d0 d0
2392 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] Add. Sense: Scsi parity error
2393 Jan 25 22:16:12 nas kernel: sd 6:0:0:0: [sdf] CDB: Write(10): 2a 00 69 48 04 d6 00 00 03 00
2394 Jan 25 22:16:12 nas kernel: ata7: EH complete
2395 Jan 25 22:16:13 nas kernel: ata7: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
2396 Jan 25 22:16:13 nas kernel: ata7: SError: { PHYRdyChg }
2397 Jan 25 22:16:13 nas kernel: ata7: hard resetting link
2398 Jan 25 22:16:18 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2399 Jan 25 22:16:18 nas kernel: ata7.00: configured for UDMA/33
2400 Jan 25 22:16:18 nas kernel: ata7: EH complete
2401 Jan 25 22:16:24 nas kernel: ata7.00: exception Emask 0x10 SAct 0x0 SErr 0x90000 action 0xe frozen
2402 Jan 25 22:16:24 nas kernel: ata7: SError: { PHYRdyChg 10B8B }
2403 Jan 25 22:16:24 nas kernel: ata7.00: failed command: READ DMA EXT
2404 Jan 25 22:16:24 nas kernel: ata7.00: cmd 25/00:00:c0:0b:c3/00:03:65:00:00/e0 tag 0 dma 393216 in
2405 Jan 25 22:16:24 nas kernel:         res d8/d8:d8:d8:d8:d8/d8:d8:d8:d8:d8/d8 Emask 0x12 (ATA bus error)
2406 Jan 25 22:16:24 nas kernel: ata7.00: status: { Busy }
2407 Jan 25 22:16:24 nas kernel: ata7.00: error: { ICRC UNC IDNF }
2408 Jan 25 22:16:24 nas kernel: ata7: hard resetting link
2409 Jan 25 22:16:29 nas kernel: ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
2410 Jan 25 22:16:29 nas kernel: ata7.00: configured for UDMA/33

... this goes on for another 12 minutes.


So, lesson learned. Test backups and replace dead drives ASAP.

Recent Changes

Recent Changes under Linux

Recent Changes under Guides

Ad-Free + Tracker-Free

I respect your privacy. This site does not serve ads, use trackers/analytics, or load resources from 3rd party websites.

Cookies may be created by MediaWiki but will contain no user identifiable information.