I wonder if anybody has ever experienced similar problem as we do in our Tigase project.

The same Java application (not necessarily Tigase server) which works very well, using relatively little memory, like no more than 10MB on a dedicated machine on Mac OS X or Ubuntu goes wild and uses hundreds of MBs on VM, specifically OpenVZ.

Even if you assign 1GB for JVM on 2.5GB instance the JVM seems to be allocating the whole memory rendering machine (VM) unusable after a while.

Setting max memory limits with the option:

-Xmx1000M

Seems to work just fine on dedicated machine whereas it seems to be ignored on the OpenVZ. I have seen many articles on the internet discussing privvmpages, vmguarpages and oomguarpages settings for the VM but it still does not explain JVM behaviour. It should respect -Xmx settings anyway.