Utilities such as htop, top, and the command free -mh all report a high usage of memory. The thing is nothing was running. It turns out to be explainable.
Using the following commands, it was determined that VMware virtual server has enabled the vmw_balloon.
[root@server1~]# lsmod | grep balloon vmw_balloon 18190 0 [root@server1~]# vmware-toolbox-cmd stat balloon 17960 MB
VMware Tools Balloon Defined
The VMware Tools balloon driver is among the utilities installed in the guest operating system with VMware Tools. It cooperates with the hypervisor to allocate memory from the operating system and return it to the hypervisor.
ESXi is decoupled from the virtual machines running on it; the memory management techniques employed within a virtual machine are completely opaque to the hypervisor. The hypervisor therefore has no knowledge of how memory allocated to a virtual machine is being used, and when it swaps memory to disk or compresses memory, it could swap out memory that is still in use. The VMware Tools balloon driver (also called the “guest balloon”) helps the hypervisor reclaim memory more efficiently. The balloon driver runs as a process within the virtual machine, allocates and pins unused memory, and communicates the pages back to the hypervisor. The memory owned by the balloon driver can then be temporarily decoupled from the virtual machine and used elsewhere.
ESXi gives the balloon driver a target size, and the balloon driver attempts to fulfill the request. Under the control of the ESXi hypervisor, balloons in each host virtual machine expand or shrink depending on the shifting requirements of the virtual machines. ESXi calculates balloon targets based on virtual machine activity; a less active virtual machine gets a higher balloon target and the reclaimed memory moves to the more active virtual machines.
A balloon in the guest operating system forces the OS to use memory more conservatively. Operating systems use every bit of available memory to improve system performance. However, caching inactive data in one virtual machine while another virtual machine has scarce memory for immediate needs is poor use of memory in a virtualization environment. Expanding the balloon in a virtual machine constrains the OS from using memory for low priority storage and allows the hypervisor to distribute physical memory resources where they are most needed.
From what I can tell, no action is to be taken. This is a dynamic memory driver. So, if you should start up your applications again, they should reclaim this space. To test this theory, run the vmware-toolbox-cmd stat balloon command to check if anything changed. It should have.
[root@server1~]# vmware-toolbox-cmd stat balloon 0 MB
Also htop, free -hm, and top all reported comparable results.