[OmniOS-discuss] slow ssh login, maybee to many locales?

[PÁSZTOR György]

Hi,

"Zach Malone" <wzmalone at gmail.com> írta 2015-04-20 10:29-kor:
> Alternatively, you can define reverse entries for your SSH clients (if
> you control the DNS server), or just disable reverse DNS lookup in
> your sshd config altogether.  Are SSH logins to localhost slow?
> 127.0.0.1 should be defined, so if ssh-ing from your host to itself is
> quick, that would indicate a DNS issue.

Maybe I did not wrote some detail, which I thought obvious for me.
Yes, ssh to localhost is sloow too.
It is my home network, and also I have proper dns records forward and in
reverse direction.
As I already said: when I log in from the same workstation to my
openindiana box or into an openindiana zoen, the ssh is subsecond fast!
The issue comes only, when I log in to my omnios zone.

> On Mon, Apr 20, 2015 at 10:17 AM, Schweiss, Chip <chip at innovates.com> wrote:
> >> I faced with that, the login onto my new omnios zone is slow.
> >> I tried to debug.
> >> Many of the symptoms seemed pretty the same as this:
> >>
> >> http://broken.net/uncategorized/resolving-slow-ssh-login-performance-problems-on-openindiana/

Please read this link first!
As I already wrote: I checked the "basic" things first.
Also I ruled out that case, what Jason writes about:
It is not the tpm module.

However the simptoms are similar: My ssh client is also wait at the same
phase: after the kexinit was sent.
(However, it not minutes in my case, only ~10 secs)

> >> There are tons's of file openings in the /usr/lib/locale dir at that
> >> point:
> >>
> >> 24560:   8.8232 stat("/usr/lib/locale/is_IS.UTF-8", 0x08047D48) = 0
> >> 24560:   8.8234 open("/usr/lib/locale//is_IS.UTF-8/LC_CTYPE/LCL_DATA",

I don't know what happens at this point on server side, or what it tries to
read / check in locale settings, or why...
BUT
On one of my openindiana box:
pasztor at sb:~$ ls -l /usr/lib/locale/
total 14
drwxr-xr-x 8 root bin  8 Mar  5  2014 C
lrwxrwxrwx 1 root root 3 Mar  5  2014 POSIX -> ./C

And compare that to omnios:
pasztor at omni:~$ ls -l /usr/lib/locale/
total 2984
drwxr-xr-x 8 root bin  8 ápr.  18 14:00 af_ZA.UTF-8
...
drwxr-xr-x 8 root bin  8 ápr.  18 14:00 zh_SG.UTF-8
drwxr-xr-x 8 root bin  8 ápr.  18 14:00 zh_TW.UTF-8
pasztor at omni:~$ ls -l /usr/lib/locale/ |wc -l
223

And another difference:
pasztor at sb:~$ locale
LANG=
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

vs.
pasztor at omni:~$ locale
LANG=hu_HU.UTF-8
LC_CTYPE="hu_HU.UTF-8"
LC_NUMERIC="hu_HU.UTF-8"
LC_TIME="hu_HU.UTF-8"
LC_COLLATE="hu_HU.UTF-8"
LC_MONETARY="hu_HU.UTF-8"
LC_MESSAGES="hu_HU.UTF-8"
LC_ALL=


So, Now I tried this:
root at omni:/usr/lib/locale# mkdir _
root at omni:/usr/lib/locale# mv * _ ; ln -s _/{C,POSIX} .
mv: cannot move ?_? to a subdirectory of itself, ?_/_?
root at omni:/usr/lib/locale# ls -l
total 15
lrwxrwxrwx   1 root root   3 Apr 21 23:18 C -> _/C
lrwxrwxrwx   1 root root   7 Apr 21 23:18 POSIX -> _/POSIX
drwxr-xr-x 223 root root 224 Apr 21 23:18 _

pasztor at intrepid:/tmp$ time ssh omni date
Tue Apr 21 23:19:02 UTC 2015

real	0m0.248s
user	0m0.004s
sys	0m0.008s


So, the problem is definitely about the locales.
Sshd check into every locale dir on the kexinit phase.
I think it's a bug. Why on earth does this needed to an ssh login?

Proof 2:
Now I revert back locale dir into it's original state:
root at omni:/usr/lib/locale# rm C POSIX ; mv _/* .
root at omni:/usr/lib/locale# rmdir _

Aaaand:
pasztor at intrepid:/tmp$ time ssh omni date
2015. április 21. 23:21:18 UTC

real	0m9.575s
user	0m0.028s
sys	0m0.004s

So, do you understand it now?
It is definitely not dns.
I quoted the output of truss on purpose, not just some L'art pour Lart
thing.
I also choose the subject carefully, not just because I was bored ;-)
The situation changed now, it's not "maybe" now. We have definitely a
proof about that.

Kind regards,
György Pásztor