Kellogg's Korner

Yesterday I ran into something on one of our servers that caught me by surprise. I was unable to log into a server that I was working on without an issue the day before. When I tried to log in I got :

login: <user>
password:
Could not chdir to home directory /home/<user>: Permission denied
/bin/bash: Permission denied

At first I thought is was just a simple permissions issue, so I asked Rick if he could login. When he tried he got the same results. The same thing happened to Joe and Scott. At this point  we thought it was more that a simple permissions, We went over to login locally as root only to find out that we could not.

We started looking at what had changed on the server. We had recently installed Symantec HBSS and their HIPS. These are intrusion detection and prevention systems. We found that the HIPS software was degrading system performance on the server, it was removed the day before. With that, and the fact that HIPS and HBSS have caused issues befor, we went with the thought Hips caused the issue.

We spent the next few hours working that angle. we worked on getting the system online with a rescue CD to get root access. Getting the HBSS back on the server did not fix the login issue, so we moved on to getting HIPS back one. During the install of HIPS a “Kernel Tainted” error showed up. When HIPS did not fix the problem one of the guys thought it might be a kernel issue, I did not think it was the kernel.

At this point i was thinking  the issue was a pam,d issue.  I remember one having a security script write over the pam.d of a x64 with a per hardened pam.d of a x32 Linux system. When rebuilding the kernel did not work we began to work with the pam.d. I copied a pam.d off a working system to copy over the problem system.This did not work either.

Now were were stumped. They started working on any random thought that popped up. I moved in to check out the forums to see if there were any similar issues. after about a half hour of searching and reading I found the fix. It turned out to be a permission issue after all, but one we never even thought of. After checking the forum i tried out the fix. I returned to the system where the others were working and checked the permissions on the / (root) directory. Upon checking this i found that they were set to 700. This setting closed eliminated any one but root from logging in. We changed the permissions 755, what they should be after instillation.  this finaly fixed the problem, we were once again able to log into the system.

We figured out that the root access was disabled by the HBSS and HIPS, this is something we will need to fix latter. Although a great security setting, it had slowed down our progress greatly. we needed to reoob several times to gain root access and then again to check settings.

Symptoms:

1. talent and ssh attempts to non-root accounts get an error “permission denied /bin/bash (or what ever shell the account uses)
2. local login attempts of non-root accounts log out immediately. This can bee seen in audit logs.
3. root account has not issues. (our issues with root were not related to the permissions problem)

Solution:

1. Check permissions on / (root) directory. should be set to 705 minimum.

Popularity: 21% [?]