[Aide] Minimum checks for file integrity?

[Andy Lawrence]

I've just now discovered aide, many thanks to those involved. I am using
it solely for the purpose of detecting file integrity, corruption, mystery
bit flips etc. Using the NORMAL rule, FIPSR+sha512 a complete scan took a
little over 15 hours to complete. Looking to speed this up I'm wondering
if for only file integrity concerns if there would be a better rule that
would be safe but also faster?

Thanks
Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.tut.fi/pipermail/aide/attachments/20140120/6a5dfb4d/attachment.html>

[Richard van den Berg]

Looking to speed this up I'm wondering if for only file integrity concerns if there would be a better rule that would be safe but also faster?

Was the system CPU constrained or IO constrained during the scan?

Kind regards,

Richard

[Andy Lawrence]

Post by Richard van den Berg

Looking to speed this up I'm wondering if for only file integrity

concerns if there would be a better rule that would be safe but also faster?

Post by Richard van den Berg
Was the system CPU constrained or IO constrained during the scan?
Kind regards,
Richard

Hi Richard,

Most definitely CPU bound.

Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.tut.fi/pipermail/aide/attachments/20140120/8ac85ab6/attachment-0001.html>

[Richard van den Berg]

Post by Andy Lawrence
Most definitely CPU bound.

In that case the sha256 from FIPSR and the sha512 are the bottleneck. Maybe search for a hash
algorithm that performs better on your hardware. Select a subset of files and test the performance
before you settle on a final aide.conf to use on your whole system.

Kind regards,

Richard

[Andy Lawrence]

I'm currently running a complete test on two different machines, on each of
the following:

NORMAL
sha256
sha512
md5

Do you happen to know, are all of the above capable of detecting corrupt
files or single bit flips? I would imagine md5 would be the quickest, but
is it enough protection? I'm not trying to keep the NSA out, ha ha.

Post by Richard van den Berg

Post by Andy Lawrence
Most definitely CPU bound.

In that case the sha256 from FIPSR and the sha512 are the bottleneck.
Maybe search for a hash algorithm that performs better on your hardware.
Select a subset of files and test the performance before you settle on a
final aide.conf to use on your whole system.
Kind regards,
Richard
_______________________________________________
Aide mailing list
Aide at cs.tut.fi
https://mailman.cs.tut.fi/mailman/listinfo/aide

[Richard van den Berg]

Post by Andy Lawrence
NORMAL
sha256
sha512
md5
Do you happen to know, are all of the above capable of detecting corrupt files or single bit flips? I would imagine md5 would be the quickest, but is it enough protection? I'm not trying to keep the NSA out, ha ha.

What is your NORMAL macro defined as? It probably has several hashes as well. One hash is really enough for what you need. Even md5 will do. There are known collisions for md5 but they are not going to happen by accident.


Kind regards,

Richard

[Andy Lawrence]

NORMAL by default was FIPSR + sha512.

Well, my study was cut short when I saw how much faster md5 was! All from
the same box, 450k files, 6TB total, time in minutes:

NORMAL = 902
sha512 = 416
md5 = 280

md5 is plenty fast and appears to do what I need, Richard many thanks for
the help.

Andy

Post by Andy Lawrence

Post by Andy Lawrence
I'm currently running a complete test on two different machines, on each
NORMAL
sha256
sha512
md5
Do you happen to know, are all of the above capable of detecting corrupt

files or single bit flips? I would imagine md5 would be the quickest, but
is it enough protection? I'm not trying to keep the NSA out, ha ha.
What is your NORMAL macro defined as? It probably has several hashes as
well. One hash is really enough for what you need. Even md5 will do. There
are known collisions for md5 but they are not going to happen by accident.
Kind regards,
Richard
_______________________________________________
Aide mailing list
Aide at cs.tut.fi
https://mailman.cs.tut.fi/mailman/listinfo/aide

[Marc Haber]

Post by Andy Lawrence
I've just now discovered aide, many thanks to those involved. I am using
it solely for the purpose of detecting file integrity, corruption, mystery
bit flips etc. Using the NORMAL rule, FIPSR+sha512 a complete scan took a
little over 15 hours to complete. Looking to speed this up I'm wondering
if for only file integrity concerns if there would be a better rule that
would be safe but also faster?

Out of curiosity, how many files, how much data, and what kind of
machine? SSDs or rotating rust?

I have not seen a CPU-bound aide in years.

Greetings
Marc

[Andy Lawrence]

Hi Marc,

About 450k files, 6TB total. The drive array is fast, capable of 1.5GB/sec,
16 spinners. Note I didn't profile it while running just the md5 hash, it
might have been IO bound.

I'm anxious for Btrfs to become stable! However I'll likely use aide for
many years to come, the portable db feature is key.

Andy

Post by Andy Lawrence

Post by Andy Lawrence
I've just now discovered aide, many thanks to those involved. I am using
it solely for the purpose of detecting file integrity, corruption,

mystery

Post by Andy Lawrence
bit flips etc. Using the NORMAL rule, FIPSR+sha512 a complete scan took

a

Post by Andy Lawrence
little over 15 hours to complete. Looking to speed this up I'm wondering
if for only file integrity concerns if there would be a better rule that
would be safe but also faster?

Out of curiosity, how many files, how much data, and what kind of
machine? SSDs or rotating rust?
I have not seen a CPU-bound aide in years.
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Mannheim, Germany | lose things." Winona Ryder | Fon: *49 621 31958061
Nordisch by Nature | How to make an American Quilt | Fax: *49 621 31958062
_______________________________________________
Aide mailing list
Aide at cs.tut.fi
https://mailman.cs.tut.fi/mailman/listinfo/aide