With the current defaults, it is relatively easy to push the broker's heap memory utilization past the
-Xmx512mheap limit passed to the JVM in the start script. When that happens the broker begins to fail in various places with
java.lang.OutOfMemoryError: unable to create new native threador
java.lang.OutOfMemoryError: Java heap space.
What to do?
Well Google is your friend but there is also the ActiveMQ FAQ and particularly the entry that deals with the likely causes and relevant configuration that can alleviate ActiveMQ OutOfMemoryError Exceptions.
In short, the answer is nearly always configuration and the intent is that the OutOfMemory FAQ entry will provide a comprehensive reference for the relevant options. Let it be your first port of call.