<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Are you using the same record size of 4K on your zfs pool as you used with your linux test system?<div class=""><br class=""></div><div class="">If the record size for the zpool and slog is set at the default value of 128K, it will greatly reduce the measured IOPS relative to that measured with a recordsize of 4K.</div><div class=""><br class=""></div><div class="">Min Kim</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Oct 22, 2015, at 12:26 PM, Matej Zerovnik <<a href="mailto:matej@zunaj.si" class="">matej@zunaj.si</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Interesting…<div class=""><br class=""></div><div class="">Although, I’m not sure if this is really the problem.</div><div class=""><br class=""></div><div class="">For test, I booted up linux and put both ZeusRAM to raid1 software raid and repeated the test. I got full 48kIOPS in the test, meaning there was 96kIOPS sent to JBOD (48k IOPS for each drive).</div><div class=""><br class=""></div><div class="">On the OmniOS test bed, there are 28k IOPS sent to ZIL and X amount to spindles when flushing write cache, but no more then 1000 IOPS (100 iops/drive * 10). Comparing that to the case above, IOPS shouldn’t be a limit.</div><div class=""><br class=""></div><div class="">Maybe I could try building my pools with hard drives that aren’t near ZIL drive, which is in bay 0. I could take hard drives from bays 4-15, which probably use different SAS lanes.</div><div class=""><br class=""></div><div class="">lp, Matej</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On 22 Oct 2015, at 21:10, Min Kim <<a href="mailto:minkim1@gmail.com" class="">minkim1@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I believe this is an known issue with SAS expanders.<div class=""><br class=""></div><div class="">Please see here:</div><div class=""><br class=""></div><div class=""><a href="http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas" class="">http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas</a></div><div class=""><br class=""></div><div class="">When you are stress-testing the Zeusram by itself, all the IOPs and bandwidth of the expander are allocated to that device alone. Once you add all the other drives, you lose some of that as you have to share it with the other disks.</div><div class=""><br class=""></div><div class="">Min Kim</div><div class=""><br class=""><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Oct 22, 2015, at 12:02 PM, Matej Zerovnik <<a href="mailto:matej@zunaj.si" class="">matej@zunaj.si</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">Hello,<br class=""><br class="">I'm building a new system and I'm having a bit of a performance problem. Well, its either that or I'm not getting the whole ZIL idea:)<br class=""><br class="">My system is following:<br class="">- IBM xServer 3550 M4 server (dual CPU with 160GB memory)<br class="">- LSI 9207 HBA (P19 firmware)<br class="">- Supermicro JBOD with SAS expander<br class="">- 4TB SAS3 drives<br class="">- ZeusRAM for ZIL<br class="">- LTS Omnios (all patches applied)<br class=""><br class="">If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k IOPS out of it, no problem there.<br class=""><br class="">If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL and set sync=always, I can only squeeze 14k IOPS out of the system.<br class="">Is that normal or should I be getting 48k IOPS on the 2nd pool as well, since this is the performance ZeusRAM can deliver?<br class=""><br class="">I'm testing with fio:<br class="">fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 --numjobs=16 --runtime=60 --group_reporting --name=4ktest<br class=""><br class="">thanks, Matej_______________________________________________<br class="">OmniOS-discuss mailing list<br class=""><a href="mailto:OmniOS-discuss@lists.omniti.com" class="">OmniOS-discuss@lists.omniti.com</a><br class=""><a href="http://lists.omniti.com/mailman/listinfo/omnios-discuss" class="">http://lists.omniti.com/mailman/listinfo/omnios-discuss</a><br class=""></div></blockquote></div><br class=""></div></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></body></html>