In your java application, if you had seen the below error in logs.

SOURCE:java.lang.OutOfMemoryError: unable to create new native thread

The above error means that when java process wants to create a new thread and requests for a new process to the kernel, but the kernel was not able to allocate new processes because of the number of processes created by that user reached current limits on max user processes.

You can find the current limits of the user by running the below command

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15244
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

If you check above max user processes, which is set to 1024, meaning the current user can create 1024 processes, if current processes count exceeds this limit, and if any application asks for a new process, the kernel will not allocate a new process.

Note there is also a max limit at the system level; this we can find from

$ sudo sysctl -a | grep kernel.pid_max
kernel.pid_max = 32768

The above means system-wide for all users, kernel can create 32768 processes.

It is up to the process to react when this happens. In the case of java we will see SOURCE:java.lang.OutOfMemoryError: unable to create new native thread

You can also find the average load of processes created.

$ more /proc/loadavg
7.35 7.83 7.56 6/7802 5393

In the above, you can see that, on average, 7802 processes are getting created.

How to know the current number of processes running for the current user?
How do we know if the above limits got applied?

How to know the current number of processes running for the current user?

$ ps -ef | grep ^app \
   | awk '{print $2}' | grep -v PID \
   | xargs -i sh -c 'ps -p {} -lfT | grep -v PID | wc -l' | awk '{sum+=$1} END {print sum}'

7335

The above command searches for all processes created by the app user, and for each process, finds the count of all sub-process/threads and sums up all the counts. 7335 processes are currently running for that user app in the above case.

You can use a sample program like the below to create threads and verify at what point the java process will get OutOfMemoryError

public class ThreadCreator {
	public static void main(String[] args) {
    int count = 0;
		while(count<775){
		    new Thread(new Runnable(){
			public void run() {
			    try {
				Thread.sleep(10000000);
			    } catch(InterruptedException e) { }
			}
		    }).start();
		    count++;
		}
	}
}

Note: Above creates 775 threads; you can increase this number To fix this, if your system has enough memory, then you can increase the user processes limit. ulimit -u 16384

To make it permanent, you include in ~/.bash_profile

$ more ~/.bash_profile
ulimit -u 16384

or in /etc/security/limits.conf add below lines

app           soft    nproc          16384
app           hard    nproc          16384

To reflect the above changes, you should restart the java process.

How do we know if the above limits got applied?

You can find the limits of all the java processes running in that system below the command.

$ ps -ef | grep "java" \
  | grep -v grep | awk '{print $2}' \
  | grep -v PID | xargs -i sh -c 'echo {}; cat /proc/{}/limits | grep "Max processes"'

Max processes             8192                 8192                 processes
59525
Max processes             8192                 8192                 processes
60232
Max processes             16384                16384                processes