[OmniOS-discuss] 4kn or 512e with ashift=12
Fred Liu
Fred_Liu at issi.com
Mon Mar 28 08:56:57 UTC 2016
> -----Original Message-----
> From: Fred Liu
> Sent: 星期四, 三月 24, 2016 18:26
> To: 'Chris Siebenmann'; Richard Jahnel
> Cc: omnios-discuss at lists.omniti.com
> Subject: RE: [OmniOS-discuss] 4kn or 512e with ashift=12
>
>
>
> > -----Original Message-----
> > From: Chris Siebenmann [mailto:cks at cs.toronto.edu]
> > Sent: 星期三, 三月 23, 2016 23:33
> > To: Richard Jahnel
> > Cc: Chris Siebenmann; Fred Liu; omnios-discuss at lists.omniti.com
> > Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> >
> > > It should be noted that using a 512e disk as a 512n disk subjects
> > > you to a significant risk of silent corruption in the event of power loss.
> > > Because 512e disks does a read>modify>write operation to modify
> > > 512byte chunk of a 4k sector, zfs won't know about the other
> > > 7 corrupted 512e sectors in the event of a power loss during a write
> > > operation. So when discards the incomplete txg on reboot, it won't
> > > do anything about the other 7 512e sectors it doesn't know were affected.
> >
> > This is true; under normal circumstances you do not want to use a
> > 512e drive in an ashift=9 vdev. However, if you have a dead 512n drive
> > and you have no remaining 512n spares, your choices are to run without
> > redundancy, to wedge in a 512e drive and accept the potential problems
> > on power failure (problems that can likely be fixed by scrubbing the
> > pool afterwards), or obtain enough additional drives (and perhaps
> > server(s)) to entirely rebuild the pool on 512e drives with ashift=12.
> >
> > In this situation, running with a 512e drive and accepting the
> > performance issues and potential exposure to power failures is
> > basically the lesser evil. (I wish ZFS was willing to accept this, but
> > it isn't.)
> >
> [Fred Liu]: I have a similar test here:
>
> [root at 00-25-90-74-f5-04 ~]# zpool status
> pool: tank
> state: ONLINE
> scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
> config:
>
> NAME STATE READ WRITE CKSUM
> tank ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> c2t45d0 ONLINE 0 0 0
> c2t46d0 ONLINE 0 0 0
> c2t47d0 ONLINE 0 0 0
> c2t48d0 ONLINE 0 0 0
> c2t49d0 ONLINE 0 0 0
> c2t52d0 ONLINE 0 0 0
> c2t53d0 ONLINE 0 0 0
> c2t44d0 ONLINE 0 0 0
> spares
> c0t5000CCA6A0C791CBd0 AVAIL
>
> errors: No known data errors
>
> pool: zones
> state: ONLINE
> scan: scrub repaired 0 in 2h45m with 0 errors on Tue Aug 12 20:24:30 2014
> config:
>
> NAME STATE READ WRITE CKSUM
> zones ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> c0t5000C500584AC07Bd0 ONLINE 0 0 0
> c0t5000C500584AC557d0 ONLINE 0 0 0
> c0t5000C500584ACB1Fd0 ONLINE 0 0 0
> c0t5000C500584AD7B3d0 ONLINE 0 0 0
> c0t5000C500584C30DBd0 ONLINE 0 0 0
> c0t5000C500586E54A3d0 ONLINE 0 0 0
> c0t5000C500586EF0CBd0 ONLINE 0 0 0
> c0t5000C50058426A0Fd0 ONLINE 0 0 0
> logs
> c4t0d0 ONLINE 0 0 0
> c4t1d0 ONLINE 0 0 0
> cache
> c0t55CD2E404BE9CB7Ed0 ONLINE 0 0 0
>
> errors: No known data errors
>
> [root at 00-25-90-74-f5-04 ~]# format
> Searching for disks...done
>
>
> AVAILABLE DISK SELECTIONS:
> 0. c0t55CD2E404BE9CB7Ed0 <ATA-INTEL
> SSDSC2BW18-DC32-167.68GB>
> /scsi_vhci/disk at g55cd2e404be9cb7e
> 1. c0t5000C500584AC07Bd0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500584ac07b
> 2. c0t5000C500584AC557d0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500584ac557
> 3. c0t5000C500584ACB1Fd0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500584acb1f
> 4. c0t5000C500584AD7B3d0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500584ad7b3
> 5. c0t5000C500584C30DBd0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500584c30db
> 6. c0t5000C500586E54A3d0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500586e54a3
> 7. c0t5000C500586EF0CBd0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c500586ef0cb
> 8. c0t5000C50058426A0Fd0
> <SEAGATE-ST91000640SS-0004-931.51GB>
> /scsi_vhci/disk at g5000c50058426a0f
> 9. c0t5000CCA6A0C791CBd0 <ATA-Hitachi HTS54101-A480-931.51GB>
> /scsi_vhci/disk at g5000cca6a0c791cb
> 10. c0t50000F0056425331d0 <ATA-SAMSUNG
> MMCRE28G-AS1Q-119.24GB>
> /scsi_vhci/disk at g50000f0056425331
> 11. c2t44d0 <ATA-Hitachi HTS54101-A480-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2c,0
> 12. c2t45d0 <ATA-Hitachi HTS54101-A480-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2d,0
> 13. c2t46d0 <ATA-ST1000LM024 HN-M-0002-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2e,0
> 14. c2t47d0 <ATA-ST1000LM024 HN-M-0002-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 2f,0
> 15. c2t48d0 <ATA-WDC WD10JPVT-08A-1A01-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 30,0
> 16. c2t49d0 <ATA-WDC WD10JPVT-75A-1A01-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 31,0
> 17. c2t52d0 <ATA-ST1000LM024 HN-M-0001-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 34,0
> 18. c2t53d0 <ATA-ST1000LM024 HN-M-0001-931.51GB>
> /pci at 0,0/pci8086,1c10 at 1c/pci1000,3140 at 0/sd at 35,0
> 19. c4t0d0 <ATA-ANS9010_2NNN2NNN-_200-1.78GB>
> /pci at 0,0/pci15d9,624 at 1f,2/disk at 0,0
> 20. c4t1d0 <ATA-ANS9010_2NNN2NNN-_200-1.78GB>
> /pci at 0,0/pci15d9,624 at 1f,2/disk at 1,0
>
>
>
> [root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0
> c0t5000CCA6A0C791CBd0 cannot replace c2t44d0 with
> c0t5000CCA6A0C791CBd0: devices have different sector alignment
>
> But in fact "c2t44d0" and "c0t5000CCA6A0C791CBd0" are in the same model --
> ATA-Hitachi HTS54101-A480-931.51GB.
> That is HTS541010A9E680
> (https://www.hgst.com/sites/default/files/resources/TS5K1000_ds.pdf) which
> is a 512e HDD.
> The *only* difference is that "c2t44d0" is attached to a LSI 1068 HBA and
> "c0t5000CCA6A0C791CBd0" is attached to a LSI 2308 HBA.
>
> [root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c2t44d0s0 | grep ashift
> ashift: 9
> ashift: 9
> ashift: 9
> ashift: 9
>
> [root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c0t5000CCA6A0C791CBd0s0 |
> grep ashift
> ashift: 12
> ashift: 12
> ashift: 12
> ashift: 12
> format> inq
> Vendor: ATA
> Product: Hitachi HTS54101
> Revision: A480
> format> q
>
> adding " "ATA Hitachi HTS54101", "physical-block-size:512"," into sd.conf
>
> [root at 00-25-90-74-f5-04 ~]# update_drv -vf sd Cannot unload module: sd Will
> be unloaded upon reboot.
> Forcing update of sd.conf.
> sd.conf updated in the kernel.
>
> Reboot the server for "cfgadm -c unconfigure" can't work here.
>
> [root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c0t5000CCA6A0C791CBd0s0 |
> grep ashift
> [root at 00-25-90-74-f5-04 ~]#
>
> No ashift in output now.
>
> [root at 00-25-90-74-f5-04 ~]# zdb -l /dev/dsk/c2t44d0s0 | grep ashift
> ashift: 9
> ashift: 9
> ashift: 9
> ashift: 9
>
> same like before
>
> [root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0
> c0t5000CCA6A0C791CBd0 cannot replace c2t44d0 with
> c0t5000CCA6A0C791CBd0: devices have different sector alignment
>
> Remove the spare:
> [root at 00-25-90-74-f5-04 ~]# zpool remove tank c0t5000CCA6A0C791CBd0
> [root at 00-25-90-74-f5-04 ~]#
>
> Add it back:
> [root at 00-25-90-74-f5-04 ~]# zpool add tank spare c0t5000CCA6A0C791CBd0
> [root at 00-25-90-74-f5-04 ~]#
>
> [root at 00-25-90-74-f5-04 ~]# zpool status tank
> pool: tank
> state: ONLINE
> scan: resilvered 187G in 21h9m with 0 errors on Thu Jan 15 08:05:16 2015
> config:
>
> NAME STATE READ WRITE CKSUM
> tank ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> c2t45d0 ONLINE 0 0 0
> c2t46d0 ONLINE 0 0 0
> c2t47d0 ONLINE 0 0 0
> c2t48d0 ONLINE 0 0 0
> c2t49d0 ONLINE 0 0 0
> c2t52d0 ONLINE 0 0 0
> c2t53d0 ONLINE 0 0 0
> c2t44d0 ONLINE 0 0 0
> spares
> c0t5000CCA6A0C791CBd0 AVAIL
>
> errors: No known data errors
>
> Still not working;
>
> [root at 00-25-90-74-f5-04 ~]# zpool replace tank c2t44d0
> c0t5000CCA6A0C791CBd0 cannot replace c2t44d0 with
> c0t5000CCA6A0C791CBd0: devices have different sector alignment
>
>
> maybe the sd.conf update is not correct.
>
>
It looks ZoL is a really helpful tool to override ashift when sd.conf doesn't work even with following errors:
PANIC: blkptr at ffff8807e4b34000 has invalid CHECKSUM 10
Showing stack for process 11419
Pid: 11419, comm: z_zvol Tainted: P -- ------------ 2.6.32-573.3.1.el6.x86_64 #1
Call Trace:
[<ffffffffa0472e9d>] ? spl_dumpstack+0x3d/0x40 [spl]
[<ffffffffa0472f2d>] ? vcmn_err+0x8d/0xf0 [spl]
[<ffffffff815391da>] ? schedule_timeout+0x19a/0x2e0
[<ffffffff81089c10>] ? process_timeout+0x0/0x10
[<ffffffff810a1697>] ? finish_wait+0x67/0x80
[<ffffffffa046e4bf>] ? spl_kmem_cache_alloc+0x38f/0x8c0 [spl]
[<ffffffffa0526e62>] ? zfs_panic_recover+0x52/0x60 [zfs]
[<ffffffffa04c7220>] ? arc_read_done+0x0/0x320 [zfs]
[<ffffffffa0577283>] ? zfs_blkptr_verify+0x83/0x420 [zfs]
[<ffffffff810a14b0>] ? autoremove_wake_function+0x0/0x40
[<ffffffffa0578292>] ? zio_read+0x42/0x100 [zfs]
[<ffffffff81178cbd>] ? __kmalloc_node+0x4d/0x60
[<ffffffffa04c7220>] ? arc_read_done+0x0/0x320 [zfs]
[<ffffffffa04c9721>] ? arc_read+0x341/0xa70 [zfs]
[<ffffffffa04d1b34>] ? dbuf_prefetch+0x1f4/0x2e0 [zfs]
[<ffffffffa04d892a>] ? dmu_prefetch+0x1da/0x210 [zfs]
[<ffffffff8127e51d>] ? alloc_disk_node+0xad/0x110
[<ffffffffa0584ce7>] ? zvol_create_minor_impl+0x607/0x630 [zfs]
[<ffffffffa0585298>] ? zvol_create_minors_cb+0x88/0xf0 [zfs]
[<ffffffffa04dac36>] ? dmu_objset_find_impl+0x106/0x420 [zfs]
[<ffffffffa0585210>] ? zvol_create_minors_cb+0x0/0xf0 [zfs]
[<ffffffffa04dacfa>] ? dmu_objset_find_impl+0x1ca/0x420 [zfs]
[<ffffffffa0585210>] ? zvol_create_minors_cb+0x0/0xf0 [zfs]
[<ffffffffa04dacfa>] ? dmu_objset_find_impl+0x1ca/0x420 [zfs]
[<ffffffffa0585210>] ? zvol_create_minors_cb+0x0/0xf0 [zfs]
[<ffffffffa0585210>] ? zvol_create_minors_cb+0x0/0xf0 [zfs]
[<ffffffffa04dafa2>] ? dmu_objset_find+0x52/0x80 [zfs]
[<ffffffffa046dd26>] ? spl_kmem_alloc+0x96/0x1a0 [spl]
[<ffffffffa05850a2>] ? zvol_task_cb+0x392/0x3b0 [zfs]
[<ffffffffa0470ebf>] ? taskq_thread+0x25f/0x540 [spl]
[<ffffffff810672b0>] ? default_wake_function+0x0/0x20
[<ffffffffa0470c60>] ? taskq_thread+0x0/0x540 [spl]
[<ffffffff810a101e>] ? kthread+0x9e/0xc0
[<ffffffff8100c28a>] ? child_rip+0xa/0x20
[<ffffffff810a0f80>] ? kthread+0x0/0xc0
[<ffffffff8100c280>] ? child_rip+0x0/0x20
INFO: task z_zvol:11419 blocked for more than 120 seconds.
Tainted: P -- ------------ 2.6.32-573.3.1.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
z_zvol D 0000000000000003 0 11419 2 0x00000000
ffff8807e5daf610 0000000000000046 0000000000000000 0000000000000000
0000000000000000 ffff8807e5daf5e8 000000b57f764afb 0000000000020000
ffff8807e5daf5b0 0000000100074cde ffff8807e68bbad8 ffff8
The pool can still be imported by ZoL and processed by zpool replace -o ashift=9:
[root at livecd ~]# zpool status
pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Mar 28 08:36:30 2016
1.95G scanned out of 1.47T at 1.76M/s, 242h30m to go
246M resilvered, 0.13% done
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ata-Hitachi_HTS541010A9E680_J8400076GJU97C ONLINE 0 0 0
ata-ST1000LM024_HN-M101MBB_S2R8J9BC502817 ONLINE 0 0 0
ata-ST1000LM024_HN-M101MBB_S2R8J9KC505621 ONLINE 0 0 0
ata-WDC_WD10JPVT-08A1YT2_WD-WXD1A4355927 ONLINE 0 0 0
ata-WDC_WD10JPVT-75A1YT0_WXP1EA2KFK12 ONLINE 0 0 0
ata-ST1000LM024_HN-M101MBB_S318J9AF191087 ONLINE 0 0 0
ata-ST1000LM024_HN-M101MBB_S318J9AF191090 ONLINE 0 0 0
spare-7 ONLINE 0 0 0
ata-Hitachi_HTS541010A9E680_J8400076GJ0KZD ONLINE 0 0 0
sdw ONLINE 0 0 0 (resilvering)
spares
sdw INUSE currently in use
It can be good remedy for the old storage systems with 512n HDDs and without 512n spares.
Thanks
Fred
More information about the OmniOS-discuss
mailing list