slowly sending e-mail with txrep on

[rodoman]

some e-mails sending very long from e-mail client program. When i look in spamd.log i saw that this e-mails has scantime=30.3 and about it time . But other e-mail's has less then 1seconds sbantime. Why could it happends? Thanks for the answer .

[truxoft]

Hello Rodoman,

TxRep performs several database lookups, so possibly your database engine is not properly configured, or the server has insufficient resources. You can also run the scan in debug mode, where you should then see the processing times for individual steps of TxRep, and can so better identify the bottleneck.

Regards,

Ivo

From: rodoman [mailto:notifications@github.com]
Sent: Wednesday, October 8, 2014 3:32 PM
To: truxoft/TxRep
Subject: [TxRep] slowly sending e-mail with txrep on (#6)

some e-mails sending very long from e-mail client program. When i look in spamd.log i saw that this e-mails has scantime=30.3 and about it time . But other e-mail's has less then 1seconds sbantime. Why could it happends? Thanks for the answer .

—
Reply to this email directly or view it on GitHub #6 . https://github.com/notifications/beacon/6275830__eyJzY29wZSI6Ik5ld3NpZXM6QmVhY29uIiwiZXhwaXJlcyI6MTcyODM5NDI5OCwiZGF0YSI6eyJpZCI6NDUxNzQxMzl9fQ==--85968bea4887b7f7ffbba5c393a44dcc9ded8a61.gif

[rodoman]

how could i run txrep in debug mode ?
when there is this delays I online see top and load average not more that
0.1 .Thanks

[truxoft]

Just like at any other SA plugin – with the option flag D (see more in the documentation). For example:

spamassassin -D TxRep,auto-whitelist < /path/someSpamMessageFile > & debug.txt

or (for debugging the SA Learn process):

sa-learn --spam -D TxRep,auto-whitelist /path/someSpamMessageFile > & debug.txt

You should also read the TxRep documentation, especially the section Optimizing TxRep (http://truxoft.com/resources/txrep.htm#OPTIMIZING-TXREP). In case your database lookups are slow, you can change or optimize the database engine, and you can also significantly reduce the number of lookups.

From: rodoman [mailto:notifications@github.com]

how could i run txrep in debug mode ?
when there is this delays I online see top and load average not more that
0.1 .Thanks

[jpotter]

I found that, when running spamd, I had to move to SQL for the data storage. The tx-reputation DB file was getting locked with a .mutex file by one of the spamd workers, and presumably that lock file acquisition has a 30 second timeout.

So, essentially, what we saw looked like a bug in TxRep not releasing the mutex file, but could be stepped around by switching to SQL.

[jpotter]

Also: when the stale mutex happens, it means that one of the spamd children will be holding the mutex, so if the message being scanned gets hit by that process, then it flys through. Since spamd by default doesn't round-robin stuff, if you have a light enough load, i.e. usually only 1 message at a time, it'll work fast. But when you suddenly have 2 messages, the second one will fail with the 30 second delay.

[rodoman]

i could optimezed this delay when create a index in txrep mysql base . It
work not so fast like without txrep on - but i didnt have 30sec delay .
Thank you .

[jpotter]

It's not an issue of their being an index in MySQL -- it's that if you used the flat file storage, BerkelyDB, there's a .mutex lock file with the PID of one of the running spamd processes, and no other process can get that .mutex lock. Since spamd is multi-forked, not multi-threaded, that means that the other processes can't get that lock file.

If I had to guess, open_storages / line 1627 opens the locks for the process, but never releases them. It works fine if you don't run it via spamd, because the process isn't long-lived; and it works fine if you use MySQL, because there's no PID-based lock.