<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi everyone!<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I just setup a small network with 2 nodes:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>* 1 proxmox host on Debian Wheezy hosting KVM VMs<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>* 1 napp-it host on OmniOS stable<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The systems are currently connected through a 1 GBit link for general WAN and LAN communitcation and a 20 GBit link (two 10 GBit links aggregated) for the iSCSI communication.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Both connection's bandwidth was confirmed using iperf.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The napp-it system currently has one pool (tank) consisting of 2 mirror vdevs. The 4 disks are SAS3 disks connected to a SAS2 backplane and directly attached (no expander) to the LSI SAS3008 (9300-8i) HBA.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Comstar is running on that Machine with 1 target (vm-storage) in 1 target group (vm-storage-group).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Proxmox has this iSCSI target configured as a "ZFS over iSCSI" storage using a block size of 8k and the "Write cache" option enabled.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>This is where the problem starts:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>dd if=/dev/zero of=/tank/test bs=1G count=20 conv=fdatasync<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>This dd test yields around 300 MB/s directly on the napp-it system.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>dd if=/dev/zero of=/home/test bs=1G count=20 conv=fdatasync<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>This dd test yields around 100 MB/s on a VM with it's disk on the napp-it system connected via iSCSI.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>The problem here is not the absolute numbers as these tests do not provide accurate numbers, the problem is the difference between the two values. I expected at least something around 80% of the local bandwidth, but this is usually around 30% or less.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>What I noticed during the tests: When running the test locally on the napp-it system, all disks will be fully utilized (read using iostat -x 1). When running the test inside a VM, the disk utilization barely reaches 30% (which seems to reflect the results of the bandwidth displayed by dd).<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>These 30% are only reached, if the locical unit of the VM disk has the writeback cache enabled. Disabling it results in 20-30 MB/s with the dd test mentioned above. Enabling it also increases the disk utilization.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>These values are also seen during the disk migration. Migrating one disk results in slow speed and low disk utilization. Migrating several disks in parallel will evetually cause 100% disk utilization.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I also tested a NFS share as VM storage in proxmox. Running the same test inside a VM on the NFS share yields results around 200-220 MB/s. This is better (and shows that the traffic is going over the fast link between the servers), but not really yet as I still lose a third.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>I am fairly new to the Solaris and ZFS world, so any help is greatly appreciated.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal>Thanks in advance!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Steffen<o:p></o:p></p></div></body></html>