Zenoss 2.5.2 performence problems

Last week I upgraded our Zenoss (network and server monitoring system) from 2.4.x to 2.5.2 version. As a results zenoss services became very unstable – lot of alerts about (zensnmp) heartbeat failure, web interface inaccessible for some time (minutes) and holes in graphs.

When I investigated the problem, I noticed almost all 4G physical memory is consumed (which might not be a problem – caches tends to fill whole memory in Linux for better performance), but the problem was that also 4G swap tend to be about 1/2 consumed.

When I checked load (uptime), I got a bit over 5, but CPU itself was utilized just for about 20%. Classical waits for heavy IO – swapping. We increased a physical memory (to 8G) and zenoss is stable again.