There are several servers that seemed to be setup identically with a local user account called
user1 which may be found using
getent passwd user1. However, for one reason or another, the account had become inaccessible and returned an error. Although a reboot of the CentOS 6 server resolved the issue, I wanted to know more. It is Linux after all, and I shouldn't have to reboot the server, except for Kernel updates, for now...
The symptom was re-creatable with this command.
[root@centos6 ~]# sudo su user1 su: cannot set user id: Resource temporarily unavailable
After the reboot.
[root@centos6~]# sudo su user1 [user1@centos6 root]$
The issue is that the affected user has reached the maximum number of process specified in the file
The following commands can reveal the number of processes and open files for a given user.
ps -U username | wc -l lsof | grep username | wc -l
/etc/security/limits.conf for the user.Log in as this user and run:
Look at the number of user processes(threads)
ps -u testuser -L | wc -l
su: pam_keyinit(su-l:session): Unable to change UID to 24074 temporarily su: pam_keyinit(su-l:session): Unable to change UID to 24074 temporarily
To fix the issue increase the NPROC Soft limit according to the user and applications needs.
It turns out that there were entries for
user1 in the
/etc/security/limits.conf which by default is generally a commented file. So the resolution would be increase the numbers.
[root@centos6~]# grep -v "^#" /etc/security/limits.conf
Through the contents of the configuration file
/etc/security/limits.conf resource limits are placed on users' sessions. The user root (and other users with
uid=0) are not affected by this restriction.
Each line of the configuration file describes a limit for a user in the form:
The fields listed above should be filled as follows:
- a username
- a groupname, with @group syntax
- the wild-card *, for default entry
- the wild-card %, for maxlogins limit only, can also be used with %group syntax
can have the three values:
- hard for enforcing hard resource limits. These limits are set by the superuser and enforced by the Linux kernel. The user cannot raise his requirement of system resources above such values;
- soft for enforcing soft resource limits. These limits are ones that the user can adjust within the range permitted by any pre-exisiting hard limits. The values specified with this token can be thought of as default values, for normal system usage;
- - for enforcing both soft and hard limits together.
can be one of the following:
- core- limits the core file size (KB)
- data- max data size (KB)
- fsize- maximum filesize (KB)
- memlock- max locked-in-memory address space (KB)
- nofile- max number of open files
- rss- max resident set size (KB)
- stack- max stack size (KB)
- cpu- max CPU time (MIN)
- nproc- max number of processes
- as- address space limit
- maxlogins- max number of logins for this user
- maxsyslogins- max number of logins on system
- priority- the priority to run user process with (negative values boost process priority)
- locks- max locked files
If a type of "-'' is specified without supplying the item and value fields then no limits are enforced.
The first entry of the form which applies to the authenticating user will override all other entries in the limits configuration file.
In general, individual limits have priority over group limits, so if no limits is imposed for a given group, but one of the members in this group have a limits line, the user will have its limits set according to this line.
Please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.
The following is an example configuration file:
# EXAMPLE /etc/security/limits.conf file: # ======================================= # * soft nproc 2047 * hard nproc 16384 * soft nofile 2048 * hard nofile 65536
Having this file means, for instance, that every user can start a maximum of 2047 processes and can raise this limit up to 16384.