[Aide] AIDE memory leak [Re: AIDE performance problems]

Discussion:

Paul Hessels

2015-04-10 17:40:49 UTC

Is this a memory leak? Or is there a legit reason to be using 12G of memory
to scan this number or size of files?

Is this the wrong email list for this?

Thank you Hannes, I will give it a try. It takes a long time to run my
tests of course.
With the old version I think I am encountering a memory leak, eventually
the aide process is using at 12G of ram and the machine dies. Hopefully
the new version doesn't have this problem.
On Wed, Apr 8, 2015 at 3:48 PM, Hannes von Haugwitz <

Hi,

# aide --version
Aide 0.15.1

Last night I uploaded the version of Debian jessie
(0.16~a2.git20130520-3) to wheezy-backports. Please give this version a
try. It might perform a bit better.
Best regards
Hannes
_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide

David V Duccini

2015-04-10 17:50:44 UTC

Permalink

We could try running aide against valgrind to see if there is a leak

-dvd

Okay, so I don't have enough memory to run an aide init. The machine has 12G of memory and its running against 2.5TB and about 20 million files.
Over the 12 hour period it comes very close to finishing but doesn't make it. Here is an process listing and memory stats right before it died.
root 13396 58.7 97.3 12047724 12003268 pts/1 D Apr09 536:49 /usr/bin/aide --config /var/lib/aide/aide.conf.autogenerated --init
root 13651 0.0 0.0 7832 868 pts/1 S+ 13:11 0:00 grep autog
total used free shared buffers cached
Mem: 12045 11928 116 0 3 61
-/+ buffers/cache: 11862 182
Swap: 0 0 0
Here is the memory graph from munin
https://www.dropbox.com/s/5i1jdtxa4922j18/Screenshot%202015-04-10%2013.38.09.png?dl=0
Is this a memory leak? Or is there a legit reason to be using 12G of memory to scan this number or size of files?
Is this the wrong email list for this?
Thank you Hannes, I will give it a try. It takes a long time to run my tests of course.
With the old version I think I am encountering a memory leak, eventually the aide process is using at 12G of ram and the machine dies. Hopefully the new version doesn't have this problem.
Hi,

# aide --version
Aide 0.15.1

Paul Hessels

2015-04-10 20:01:38 UTC

Permalink

I am in no way a valgrind expert, but I have it running on a subset of
files with memcheck enabled. We'll see how it does.

Is it possible that 12G of ram is how much memory is required and this
isn't a memory leak?

Post by David V Duccini
We could try running aide against valgrind to see if there is a leak
-dvd
Okay, so I don't have enough memory to run an aide init. The machine has
12G of memory and its running against 2.5TB and about 20 million files.
Over the 12 hour period it comes very close to finishing but doesn't make
it. Here is an process listing and memory stats right before it died.
root 13396 58.7 97.3 12047724 12003268 pts/1 D Apr09 536:49
/usr/bin/aide --config /var/lib/aide/aide.conf.autogenerated --init
root 13651 0.0 0.0 7832 868 pts/1 S+ 13:11 0:00 grep autog
total used free shared buffers cached
Mem: 12045 11928 116 0 3 61
-/+ buffers/cache: 11862 182
Swap: 0 0 0
Here is the memory graph from munin
https://www.dropbox.com/s/5i1jdtxa4922j18/Screenshot%202015-04-10%2013.38.09.png?dl=0
Is this a memory leak? Or is there a legit reason to be using 12G of
memory to scan this number or size of files?
Is this the wrong email list for this?

Thank you Hannes, I will give it a try. It takes a long time to run my tests of course.
With the old version I think I am encountering a memory leak, eventually
the aide process is using at 12G of ram and the machine dies. Hopefully
the new version doesn't have this problem.
On Wed, Apr 8, 2015 at 3:48 PM, Hannes von Haugwitz <

Hi,

# aide --version
Aide 0.15.1

_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide
_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide

Paul Hessels

2015-04-10 23:54:15 UTC

Permalink

Okay, valgrind is done. It didn't report any memory leaks.

Post by Paul Hessels
I am in no way a valgrind expert, but I have it running on a subset of
files with memcheck enabled. We'll see how it does.
Is it possible that 12G of ram is how much memory is required and this
isn't a memory leak?

Hi,

# aide --version
Aide 0.15.1

Richard van den Berg

2015-04-11 11:02:06 UTC

Permalink

Post by Paul Hessels
Okay, valgrind is done. It didn't report any memory leaks.

What is the size of the largest file you are running aide on? It really shouldn't be using 12GB of RAM.

Kind regards,

Richard

Paul Hessels

2015-04-12 00:54:54 UTC

Permalink

It looks like the largest file is 102G. That file doesn't show up until
about half way through the scan and it doesn't appear to have a huge impact
on the memory based on the graphs.

Post by Richard van den Berg

Post by Paul Hessels
Okay, valgrind is done. It didn't report any memory leaks.

What is the size of the largest file you are running aide on? It really
shouldn't be using 12GB of RAM.
Kind regards,
Richard
_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide

Rami Lehti

2015-04-12 08:25:14 UTC

Permalink

aide --init constructs a tree datastructure in memory that contains all information about all files. This can take quite a bit of memory. One solution would be to write the database lines out as they are ready and free memory as soon as the lines are written.

Kind regards,
Rami Lehti

Post by Paul Hessels
It looks like the largest file is 102G. That file doesn't show up until
about half way through the scan and it doesn't appear to have a huge impact
on the memory based on the graphs.
On Sat, Apr 11, 2015 at 7:02 AM, Richard van den Berg

Post by Richard van den Berg

Post by Paul Hessels
Okay, valgrind is done. It didn't report any memory leaks.

What is the size of the largest file you are running aide on? It

really

Post by Richard van den Berg
shouldn't be using 12GB of RAM.
Kind regards,
Richard
_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide

------------------------------------------------------------------------
_______________________________________________
Aide mailing list
https://mailman.cs.tut.fi/mailman/listinfo/aide

--
Vapaa Internet ry
Hallituksen puheenjohtaja