<div dir="ltr">Not sure if it's the same issue we reported 3 years ago. We applied our patch and haven't seen this issue ever since.<div><br></div><div><a href="https://illumos.topicbox.com/groups/developer/Te5808458a5a5a14f-M74735db9aeccaa5d8c3a70a4">https://illumos.topicbox.com/groups/developer/Te5808458a5a5a14f-M74735db9aeccaa5d8c3a70a4</a></div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 3, 2018 at 1:55 PM, Schweiss, Chip <span dir="ltr"><<a href="mailto:chip@innovates.com" target="_blank">chip@innovates.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hopefully the patch Marcel is talking about fixes this. I've at least figured out enough to predict when the problem is imminent. <div><br></div><div>We have been migrating to using automounter instead of hard mounts which could to be related to this problem growing over time.<br><div><br></div><div>Just an FYI: I've kept the server running in this state, but moved its storage pool to a sister server. The port binding problem remains with NO NFS clients connected, but neither pfiles or lsof shows rpcbind as the culprit:</div><div><br></div><div><div><font face="monospace, monospace" size="1"># netstat -an|grep BOUND|wc -l</font></div><div><font face="monospace, monospace" size="1">32739</font></div></div><div><font face="monospace, monospace" size="1"><br></font></div><div><div><font face="monospace, monospace" size="1"># /opt/ozmt/bin/SunOS/lsof -i:41155</font></div><div><font face="monospace, monospace" size="1"><br></font></div></div><div><font face="monospace, monospace" size="1">{nothing returned}</font></div><div><font face="monospace, monospace" size="1"><br></font></div><div><div><font face="monospace, monospace" size="1"># pfiles `pgrep rpcbind`</font></div><div><font face="monospace, monospace" size="1">449: /usr/sbin/rpcbind</font></div><div><font face="monospace, monospace" size="1"> Current rlimit: 65536 file descriptors</font></div><div><font face="monospace, monospace" size="1"> 0: S_IFCHR mode:0666 dev:527,0 ino:70778888 uid:0 gid:3 rdev:135,2</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/mm@0:null</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 1: S_IFCHR mode:0666 dev:527,0 ino:70778888 uid:0 gid:3 rdev:135,2</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/mm@0:null</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 2: S_IFCHR mode:0666 dev:527,0 ino:70778888 uid:0 gid:3 rdev:135,2</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/mm@0:null</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 3: S_IFCHR mode:0000 dev:527,0 ino:61271 uid:0 gid:0 rdev:231,64</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET6 :: port: 111</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp6@0:udp6</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 4: S_IFCHR mode:0000 dev:527,0 ino:50998 uid:0 gid:0 rdev:231,59</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET6 :: port: 0</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp6@0:udp6</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 5: S_IFCHR mode:0000 dev:527,0 ino:61264 uid:0 gid:0 rdev:231,58</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET6 :: port: 60955</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp6@0:udp6</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 6: S_IFCHR mode:0000 dev:527,0 ino:64334 uid:0 gid:0 rdev:224,57</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET6 :: port: 111</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tcp6@0:tcp6</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 7: S_IFCHR mode:0000 dev:527,0 ino:64333 uid:0 gid:0 rdev:224,56</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET6 :: port: 0</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tcp6@0:tcp6</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 8: S_IFCHR mode:0000 dev:527,0 ino:64332 uid:0 gid:0 rdev:230,55</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET 0.0.0.0 port: 111</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp@0:udp</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 9: S_IFCHR mode:0000 dev:527,0 ino:64330 uid:0 gid:0 rdev:230,54</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET 0.0.0.0 port: 0</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp@0:udp</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 10: S_IFCHR mode:0000 dev:527,0 ino:64331 uid:0 gid:0 rdev:230,53</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET 0.0.0.0 port: 60994</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/udp@0:udp</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 11: S_IFCHR mode:0000 dev:527,0 ino:64327 uid:0 gid:0 rdev:223,52</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET 0.0.0.0 port: 111</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tcp@0:tcp</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 12: S_IFCHR mode:0000 dev:527,0 ino:64326 uid:0 gid:0 rdev:223,51</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> sockname: AF_INET 0.0.0.0 port: 0</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tcp@0:tcp</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 13: S_IFCHR mode:0000 dev:527,0 ino:64324 uid:0 gid:0 rdev:226,32</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticlts</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 14: S_IFCHR mode:0000 dev:527,0 ino:64328 uid:0 gid:0 rdev:226,33</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticlts</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 15: S_IFCHR mode:0000 dev:527,0 ino:64324 uid:0 gid:0 rdev:226,35</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticlts</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 16: S_IFCHR mode:0000 dev:527,0 ino:64322 uid:0 gid:0 rdev:226,36</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticotsord</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 17: S_IFCHR mode:0000 dev:527,0 ino:64321 uid:0 gid:0 rdev:226,37</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticotsord</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 18: S_IFCHR mode:0000 dev:527,0 ino:64030 uid:0 gid:0 rdev:226,39</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticots</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 19: S_IFCHR mode:0000 dev:527,0 ino:64029 uid:0 gid:0 rdev:226,40</font></div><div><font face="monospace, monospace" size="1"> O_RDWR</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/tl@0:ticots</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div><div><font face="monospace, monospace" size="1"> 20: S_IFIFO mode:0000 dev:525,0 ino:206 uid:1 gid:12 rdev:0,0</font></div><div><font face="monospace, monospace" size="1"> O_RDWR|O_NONBLOCK</font></div><div><font face="monospace, monospace" size="1"> 21: S_IFIFO mode:0000 dev:525,0 ino:206 uid:1 gid:12 rdev:0,0</font></div><div><font face="monospace, monospace" size="1"> O_RDWR|O_NONBLOCK</font></div><div><font face="monospace, monospace" size="1"> 23: S_IFCHR mode:0000 dev:527,0 ino:33089 uid:0 gid:0 rdev:129,21273</font></div><div><font face="monospace, monospace" size="1"> O_WRONLY FD_CLOEXEC</font></div><div><font face="monospace, monospace" size="1"> /devices/pseudo/log@0:conslog</font></div><div><font face="monospace, monospace" size="1"> offset:0</font></div></div><div><br></div><div>Restarting rpcbind doesn't affect it either:</div><div><br></div><div><font face="monospace, monospace" size="1"># svcadm restart svc:/network/rpc/bind:default<br></font></div><div><font face="monospace, monospace" size="1"><br></font></div><div><div><font face="monospace, monospace" size="1"># netstat -an|grep BOUND|wc -l</font></div><div><font face="monospace, monospace" size="1">32739</font></div></div><div><font face="monospace, monospace" size="1"><br></font></div><div>In the interim of this patch getting integrated I'll monitor the number of bound ports to know when I should fail my pool over again.</div><div><br></div></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 3, 2018 at 10:32 AM, Marcel Telka <span dir="ltr"><<a href="mailto:marcel@telka.sk" target="_blank">marcel@telka.sk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Wed, Jan 03, 2018 at 10:02:43AM -0600, Schweiss, Chip wrote:<br> > The problem occurred again starting last night. I have another clue, but I<br> > still don't know how it is occurring or how to fix it.<br> ><br> > It looks like all the TCP ports are in "bound" state, but not being<br> > released.<br> ><br> > How can I isolate the cause of this?<br><br></span>This is a bug in rpcmod, very likely related to<br><a href="https://www.illumos.org/issues/1616" rel="noreferrer" target="_blank">https://www.illumos.org/issues<wbr>/1616</a><br><br> I discussed this few weeks back with some guy who faced the same issue. It<br> looks like he found the cause and have a fix for it. I thought he will post a<br> review request, but that didn't happened for some reason yet.<br><br> I'll try to push this forward...<br><br><br> Thanks.<br><span class="gmail-m_-2726035397427601039HOEnZb"><font color="#888888"><br> --<br> +-----------------------------<wbr>--------------+<br> | Marcel Telka e-mail: <a href="mailto:marcel@telka.sk" target="_blank">marcel@telka.sk</a> |<br> | homepage: <a href="http://telka.sk/" rel="noreferrer" target="_blank">http://telka.sk/</a> |<br> | jabber: <a href="mailto:marcel@jabber.sk" target="_blank">marcel@jabber.sk</a> |<br> +-----------------------------<wbr>--------------+<br><br> ------------------------------<wbr>------------<br> illumos-zfs<br> Archives: <a href="https://illumos.topicbox.com/groups/zfs/discussions/T8f10bde64dc0d5c5-Mb17ca753ce6f6fbed5124147" rel="noreferrer" target="_blank">https://illumos.topicbox.com/g<wbr>roups/zfs/discussions/T8f10bde<wbr>64dc0d5c5-Mb17ca753ce6f6fbed51<wbr>24147</a><br></font></span><div class="gmail-m_-2726035397427601039HOEnZb"><div class="gmail-m_-2726035397427601039h5">Powered by Topicbox: <a href="https://topicbox.com" rel="noreferrer" target="_blank">https://topicbox.com</a><br></div></div></blockquote></div><br></div></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div id="gmail-m_-2726035397427601039topicbox-footer" style="margin:10px 0px 0px;border-top:1px solid rgba(0,0,0,0.15);border-right-color:rgba(0,0,0,0.15);border-bottom-color:rgba(0,0,0,0.15);border-left-color:rgba(0,0,0,0.15);padding:7px 0px"> <strong>illumos-zfs</strong> | <a href="https://illumos.topicbox.com/groups/zfs/discussions/T8f10bde64dc0d5c5-M4fe740db1a7d9f22a30655f7" target="_blank">Archives</a> | Powered by <a href="https://topicbox.com" target="_blank">Topicbox</a> </div> </div></div></blockquote></div><br></div></div></div>