Sources of Linux memory usage not in /proc/meminfo

I’m running a Linux server that shows weird patterns in memory usage that I’m trying to investigate.

During normal operation, around 200MB of memory are used for applications according to munin, but every now and then (maybe every couple of days), there’s a sudden increase by around 1GB, as visible in this graph:

I’ve been trying to find the cause of this, but I’m not sure what it could be. The fact that munin attributes this memory to “apps” suggests to me that one of the running processes is simply using a lot of memory, but looking at top when this happens shows that none of the processes uses nearly as much memory; in fact, the top 15 memory consuming processes don’t change at all when this occurs and the top one only uses 2.2% of the memory. Also, the memory info from ps doesn’t nearly add up to this much memory, but rather something in the ballpark of 200MB.

The height of the increase is usually the same, and as visible in this example, it just disappears after a few hours.

I’ve found that a reliable way to trigger this behaviour is by downloading a large file (1.4G) using wget via an ssh tunnel over the network to the local disk (network activity alone doesn’t trigger it).

To see the memory usage in more detail, I’ve started to log /proc/meminfo every 10 seconds and made a tarball with graphs from this data. The download causing the behaviour in question started at 10:35:32 and was finished at 10:38:53. The original log data is also included in the tarball.

In /proc/meminfo, there’s no value that goes up enough to explain that huge decrease of free memory (this is essentially what munin attributes to “apps”), which leaves me clueless. The amount of used memory that munin reports is also reported by free and top, but there are no processes in the process list that would explain such high memory usage, or the sudden increase when downloading a file.

What I’m trying to understand is:

• Is it normal that the memory usage behaves in this way without any
  particular process using the increased memory?


• What could be other sources of memory that’s not accounted for in
  /proc/meminfo?

Or could this be a sign of the system being compromised?