I never did figure out the following... but I did figure out the root
of my problem.

spam-milter.sock != spamd.sock

Bill


Bill Levering
idbill@...
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC




On Nov 8, 2009, at 9:11 PM, Bill Levering wrote:

> WHat does this mean?
>
> Nov  9 05:10:04 syreen milter-greylist: greylist: mi_stop=1
> Nov  9 05:10:04 syreen milter-greylist: smfi_main() returned 0
>
> Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
> (greylist): read returned -1: Connection reset by
> 159.139.11-93.rev.gaoland.net
> Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
> (greylist): to error state
> Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
> (greylist): init failed to open
> Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
> (greylist): to error state
> Nov  9 05:10:10 syreen sm-mta[31068]: nA959kRx031068: Milter
> (greylist): write(D) returned -1, expected 23: Broken pipe
> Nov  9 05:10:10 syreen sm-mta[31068]: nA959kRx031068: Milter
> (greylist): to error state
>
>
> Bill Levering
> idbill@...
> KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC
>
>
>
>

WHat does this mean?

Nov  9 05:10:04 syreen milter-greylist: greylist: mi_stop=1
Nov  9 05:10:04 syreen milter-greylist: smfi_main() returned 0

Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
(greylist): read returned -1: Connection reset by
159.139.11-93.rev.gaoland.net
Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
(greylist): to error state
Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
(greylist): init failed to open
Nov  9 05:10:04 syreen sm-mta[31086]: nA95A41n031086: Milter
(greylist): to error state
Nov  9 05:10:10 syreen sm-mta[31068]: nA959kRx031068: Milter
(greylist): write(D) returned -1, expected 23: Broken pipe
Nov  9 05:10:10 syreen sm-mta[31068]: nA959kRx031068: Milter
(greylist): to error state


Bill Levering
idbill@...
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC

New issue...

Nov  8 19:42:54 syreen milter-greylist: spamd connect failed:
Permission denied
Nov  8 19:42:54 syreen milter-greylist: ACL evaluation failure

At first, I thought this was a spamd issue, but I've tweaked
permissions on the bayes, whitelist, socket... etc.
That and spamd works fine  with the spam-milter... I turned on verbose
logging and spamd isn't complaining about anything unusual (missing
vuserinfo/valias/vpop).

So I'm back to trying to find out why milter-greylist is having a
problem.


I turned on milter-greylist verbose loggings... and still no new info.

Debian 5.0.3
Milter-greylist 4.2.3
SpamAssassin Server version 3.2.5
    running on Perl 5.10.0
    with zlib support (Compress::Zlib 2.012)

sendmail running as root :(
root     23248  0.0  0.6  66168  3148 ?        Ss   20:17   0:00
sendmail: MTA: accepting connections

spamd running as:
root     20421  0.1 10.2 116204 53760 ?        Ss   19:37   0:05 /usr/
sbin/spamd --create-prefs --max-children 5 --helper-home-dir -u spam -
d --pidfile=/var/run/spamd.pid -v

greylist runnings as:
greylist 21933  0.0  0.5 116940  2760 ?        Ssl  19:59   0:01 /usr/
local/bin/milter-greylist -w 8m -P /var/run/greylist.pid -u greylist -
p /var/run/milter-greylist/milter-greylist.sock

milter config entry for spamd:
spamdsock unix "/var/run/spamass/spamass.sock"
dacl blacklist spamd > 10 msg "Your message is considered spam"


If I set the perms on spamd.sock from:
root@syreen:/usr/local/src/milter-greylist-4.2.3 > ls -l /var/run/
spamass/
total 4
-rw-r--r-- 1 spamass-milter nogroup 5 Oct 28 17:58 spamass.pid
srw------- 1 root           root    0 Oct 28 17:58 spamass.sock


to:
root@syreen:/usr/local/src/milter-greylist-4.2.3 > ls -l /var/run/
spamass/
total 4
-rw-r--r-- 1 spamass-milter nogroup 5 Oct 28 17:58 spamass.pid
srw-rw-rw- 1 root           root    0 Oct 28 17:58 spamass.sock


I now get:
Nov  8 20:05:47 syreen milter-greylist: spamd read failed: Connection
reset by peer

(I have an almost identical setup on CentOS 4.8, which works.
(Sendmail/greylist run as smmsp & spamd runs as spamd) )

Bill

PS. Have I mentioned that Debian appears to be lacking in sendmail
support? (missing files, etc)

Bill Levering
idbill@...
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC

New issue...

Nov  8 19:42:54 syreen milter-greylist: spamd connect failed:
Permission denied
Nov  8 19:42:54 syreen milter-greylist: ACL evaluation failure

At first, I thought this was a spamd issue, but I've tweaked
permissions on the bayes, whitelist, socket... etc.
That and spamd works fine  with the spam-milter... I turned on verbose
logging and spamd isn't complaining about anything.

So I'm back to trying to find out why milter-greylist is having a
problem.


I turned on milter-greylist verbose loggings... and still no new info.


Bill

Bill Levering
idbill@...
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC

On 11/5/09 5:33 AM, manu@... wrote:
>
> If you want to use DKIM, you will have to whitelist the sender's domain
> at RCPT stage ACL:
>
> racl whitelist from /@foo\.com$/
> dacl whitelist from /@foo\.com$/ dkim pass
> dacl blacklist from /@foo\.com$/ msg "foo.com must use DKIM"

Yes, I thought of doing that for Gmail/googlemail. In my opinion, that's
a more elegant way of accepting mail for Gmail's multiple "floating"
relays. Greylisted mails tends to be resent by another (or several
other) IPs. (Of course, if you already use lazyaw its usually not a
problem).

But maintaining a list for others that use DKIM is too much manual
labour for being worth it, at least for my taste.

Re,
/P

I uploaded a patch to improve the dacl handling and fix the dkim bug. The patch
('mg-4.3.5_to_mg-4.3.5-dacl.patch') can be found here:

http://tech.groups.yahoo.com/group/milter-greylist/files/

(Sorry, Yahoo does not allow direct links). The file patches against the latest
CVS check-out.

The modification will base the precedence of racl and dacl lines on the order in
which they appear in the conf file.

As an example, a conf file like

dacl whitelist dkim pass
racl greylist default

will whitelist a sender if it passes dkim, and default to greylisting if not.

The modifications to the code turned out to be quite simple; all of the
necessary hooks were already in there. The logic as the message passes through
the milter stages is as follows:

1. The connecting host and sender are processed in the rcpt stage of the SMTP
connection and checked for matches against racl statements

2. If a racl matches, the milter checks to see if there are any dacl statements
before the matching racl statement. If so, the result of the racl match is
stored and the message proceeds to the data and body stage of the SMPT
connection.

If there are no dacl statements before the matching racl line, no data stage
processing is needed and the racl result is processed immediately in the rcpt
stage (saving CPU).

3. If there are dacl lines before the matching racl line, the milter will
continue to the data and body stages of the SMPT connection. It will process the
dacl commands up to the matching racl line. In case of a match, the dacl result
will be processed; no match means the racl results will be restored.

Although the processing logic may sound complicated, the result is very
intuitive and greatly increases performance by not passing messages through the
dacl stage when not needed.

Take this example:

racl whitelist spf pass
dacl whitelist dkim pass
racl greylist default

If a sender matches the spf clause, the message is accepted immediately and does
not pass through the data stage.

The line based dacl precedence will also work for black listing based on content
scanning:

racl whitelist spf
dacl whitelist dkim pass
dacl blacklist spamd
racl greylist default

In this case, all messages that match spf or dkim will be whitelisted, whereas
any other messages which are flagged by Spamassasin will be rejected. All other
messages will be greylisted.

Please go ahead and try the patch. I am very interested to hear your feedback.
By turning on 'verbose' in the conf file, the program will clearly log its
racl/dacl decisions. The logic should become quite obvious after a couple of
messages.

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the milter-greylist
group.

   File        : /mg-4.3.5_to_mg-4.3.5-dacl.patch
   Uploaded by : reschauzier <reschauzier@...>
   Description : Improve handling of data stage commands

You can access this file at the URL:
http://groups.yahoo.com/group/milter-greylist/files/mg-4.3.5_to_mg-4.3.5-dacl.pa\
tch

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/general.htmlfiles

Regards,

reschauzier <reschauzier@...>

In the course of testing a modification to improve the handling of dacl
statements, I noticed the strangest thing. It looks like there is a bug in
dkimcheck.c that will prevent dkim from working.

Note this piece of code from the function dkimcheck_error():

   106 	 break;
   107  }
   108
   109  if (priv->priv_dkim != DKIM_STAT_OK) {
   110 	 (void)dkim_free(priv->priv_dkim);
   111 	 priv->priv_dkim = NULL;
   112  }
   113
   114  return retval;
   115 }

Line 109 checks the pointer priv->priv_dkim against an integer value. Since
DKIM_STAT_OK equals 0, every time the pointer is not NULL, it will be reset.
This is exactly what happens as a message goes through the SMTP stages in
milter-greylist. As a result, no message is ever checked for dkim.

The correct code should be:

   109  if (priv->priv_dkimstat != DKIM_STAT_OK) {

With this change the dkim checking seems to work properly.

Does this really mean no-one was ever able to run a successful DKIM check? Or am
I overlooking something?

reschauzier <reschauzier@...> wrote:

> The milter does not work, on the other hand, for message authentication
> systems such as DomainKeys and DKIM. reason is that the dacl can
> override a racl whitelist, but not a racl greylist or blacklist.

Of course: if a RCPT stage ACL reject the message for all recipients,
there is no DATA stage ACL.

If you want to use DKIM, you will have to whitelist the sender's domain
at RCPT stage ACL:

racl whitelist from /@foo\.com$/
dacl whitelist from /@foo\.com$/ dkim pass
dacl blacklist from /@foo\.com$/ msg "foo.com must use DKIM"

Of course it you do that with multiple domains, a list will be usefull.
Or even better, a DNSRBL of domains using DKIM.

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Thanks for the info. It confirms what I found: the current milter-greylist will
work for spam and virus filtering (albeit with a significant performance hit, as
you mention).

The milter does not work, on the other hand, for message authentication systems
such as DomainKeys and DKIM.

The reason is that the dacl can override a racl whitelist, but not a racl
greylist or blacklist.

A simple conf file like

racl greylist default
dacl whitelist dkim pass

will not function as expected (allow all dkim authenticated messages, greylist
all others), as will an even simpler file like

dacl whitelist dkim pass

The racl will greylist by default, and a dacl whitelist will not override a racl
greylist (if it sounds confusing, that's because it is; it took me couple of
hours sorting through the source to figure this out)

Is there anyone that can prove me wrong and has a working DKIM/DK setup with
milter-greylist?

Thanks,
Rudy.


--- In milter-greylist@yahoogroups.com, chasd <chasd@...> wrote:
>
> > For my interest, what have you been using the dacls for?
> > Interfacing w/ spamd, DKIM check or something else?
>
> To see if applying virus or spam filtering would be worthwhile at
> that mail processing point.
> I hadn't realized that one of the reasons the delivery times
> increased so much was because _every_ message was being processed. It
> makes sense now. The performance issue was why I didn't move forward
> toward implementation, I just did testing. With finer-grained control
> of dacl processing, I'd revisit my tests.
>
>
> --
> Charles Dostale
> System Admin - Silver Oaks Communications
> http://www.silveroaks.com/
> 824 17th Street, Moline  IL  61265
>

> For my interest, what have you been using the dacls for?
> Interfacing w/ spamd, DKIM check or something else?

To see if applying virus or spam filtering would be worthwhile at
that mail processing point.
I hadn't realized that one of the reasons the delivery times
increased so much was because _every_ message was being processed. It
makes sense now. The performance issue was why I didn't move forward
toward implementation, I just did testing. With finer-grained control
of dacl processing, I'd revisit my tests.


--
Charles Dostale
System Admin - Silver Oaks Communications
http://www.silveroaks.com/
824 17th Street, Moline  IL  61265

On Tue, Nov 03, 2009 at 12:56:15PM -0000, reschauzier wrote:
> For my interest, what have you been using the dacls for? Interfacing w/ spamd,
DKIM check or something else?

I use it as a cheap content filter, using the base64 encoded win32
executable header.

--
Emmanuel Dreyfus
manu@...

For my interest, what have you been using the dacls for? Interfacing w/ spamd,
DKIM check or something else?

Thanks,
Rudy.

--- In milter-greylist@yahoogroups.com, chasd <chasd@...> wrote:
>
> I prefer 1, although I have only used dacl in testing, not in
> production.
>
>
> --
> Charles Dostale
> System Admin - Silver Oaks Communications
> http://www.silveroaks.com/
> 824 17th Street, Moline  IL  61265
>

I prefer 1, although I have only used dacl in testing, not in
production.


--
Charles Dostale
System Admin - Silver Oaks Communications
http://www.silveroaks.com/
824 17th Street, Moline  IL  61265

Bill Levering schreef:
> I'm moving a mailserver from freebsd to debian.
>
> The apt version of milter-greylist is 3.x... which is too old for my
> existing configuration.
>
> But I'm unable to get 4.2.3 to compile.
>
> Generally, I get:
> Required libmilter not found. Use --with-libmilter

This is what I configure it with:

./configure --with-user=smmsp --with-libspf2  \
--sysconfdir=/etc/milter-greylist \
--with-libcurl --with-libmilter=/usr --disable-rpath \
--with-conffile=/etc/milter-greylist/greylist.conf \
--with-dumpfile=/var/lib/milter-greylist/greylist.db \

If I do not add --disable-rpath on linux it just doesn't configure properly.

Regards,

Seth

Bill Levering <idbill@...> wrote:

> What am I missing?

Please send me config.log (in private mail, don't pollute the list with
that)

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

I'm moving a mailserver from freebsd to debian.

The apt version of milter-greylist is 3.x... which is too old for my
existing configuration.

But I'm unable to get 4.2.3 to compile.

Generally, I get:
Required libmilter not found. Use --with-libmilter


I've installed libmilter-dev

If I point directly to libmilter.so, I start see all sorts of warnings:
configure: WARNING: sys/param.h: present but cannot be compiled
configure: WARNING: sys/param.h:     check for missing prerequisite
headers?
configure: WARNING: sys/param.h: see the Autoconf documentation
configure: WARNING: sys/param.h:     section "Present But Cannot Be
Compiled"
configure: WARNING: sys/param.h: proceeding with the preprocessor's
result
configure: WARNING: sys/param.h: in the future, the compiler will take
precedence
configure: WARNING:     ## ------------------------------ ##
configure: WARNING:     ## Report this to manu@... ##
configure: WARNING:     ## ------------------------------ ##

What am I missing?

Bill

PS. So far I'm not impresses with Debian support of sendmail, and the
crond doesn't support timezone!

Bill Levering
idbill@...
KFP: 0C38 4D7E 5B50 94FE 992D  406D 6C81 DE33 5459 A1AC

Kouhei Sutou <kou@...> wrote:

> chmod(2) should be used instead of umask(2):
>   http://www.clear-code.com/~kou/patches/milter-greylist-socket-mode.diff

Got it!

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Hi,

The socket mode configuration(*) affects PID file mode
because milter-greylist uses umask(2) for the configuration.

(*) socket "/var/milter-greylist/milter-greylist.sock" 660

We will use the socket mode configuration to access
milter-greylist socket by other process that belongs to
group of milter-greylist process. In the case, we will use
660 for permitting group access. But it causes that PID file
also gets 660 permission. It's not expected behavior.

chmod(2) should be used instead of umask(2):
   http://www.clear-code.com/~kou/patches/milter-greylist-socket-mode.diff


Thanks,
--
kou

Thanks for your feedback. You are describing exactly what I was running into
with the dacl commands in the current greylist, and why I think it needs a major
overhaul.

Basing precedence on the config order is intuitive to most users (except maybe
for SMTP and milter die-hards) and it elegantly extends the milter-greylist
philosophy. No additional keywords needed.

I'd be very interested to hear more feedback from users that are using dacl
statements in their current setup, or have tried using them in the past but have
given up.

Thanks,
Rudy.


--- In milter-greylist@yahoogroups.com, Fredrik Pettai <pettai@...> wrote:
> rest", the dacl with the rules for checking for example dkim, will
> become more or less useless. So, in order to do whitelisting of

> Mixing racl and dacl will be more CPU intensive than doing them in the
> current order, but I think it adds more flexibility if it's possible
> to be able to do whitelisting (or greylisting and perhaps even
> blacklisting) on both RCPT and DATA before falling back on the default.
>
> As an example, I would like to be able to do something similar to this.
> (where the acls are parsed in linear order)
>
> racl whitelist list "my network"
> racl whitelist list "awl admin recipients"
> racl whitelist list "broken mta"
> dacl whitelist dkim pass
> dacl whitelist body /^-----BEGIN PGP.*MESSAGE-----$/
> racl greylist sm_macro "maybe_forged" delay 1h autowhite 3d
> racl greylist dnsrbl "SPAMHAUS" delay 24h autowhite 3d
> dacl greylist dkim fail delay 1h autowhite 3d
> *acl greylist default delay 30m autowhite 30d
>
> *I think "racl greylist default" would be misleading in this
> configuration case. Correct me if I'm wrong here, but I also believe
> that the way "racl greylist default delay ... autowhite ..." works,
> wasn't possible to configure with dacl (see it as another suggestion
> for enhancement :-)).
>
> Regards,
> /P
>

Fredrik Pettai <pettai@...> wrote:

> an way that gets that functionality to mix racls   & dacls

An idea about this:

We have something called properties, that can be fetched from a LDAP
directory or a web service. I use it for have LDAP-stored, per-recipient
filtering settings. It does somthing like this:

ldapcheck "senderpref" "ldapi:///o=home?userBlackList?one?mail=%f" clear
racl whitelist ldapcheck "senderpref"
racl blacklist $userBlackList "%f" msg "you're in recipient's blacklist"

the ldapcheck statement is just the LDAP source configuration.
The first racl cause $userBlackList to be populated as the list of
recipients blakclisted by the user, as stored in the LDAP directory.
There is a syntax horror here, as this "whitelist" will not stop ACL
evaluation. It should really be changed as racl continue ldapcheck
"senderpref", but  this is another story.

Now, we could extend this so that any ACL would be able to set a
property. Something like this:

racl continue from friend@... set($skipdacl,"TRUE")
dacl whitelist $skipdact "TRUE"

And here we get a really powerful solution, where racl can store a lot
of information to be used to make a décision at DATA stage.

If we go that way, we need to introduce the continue action, and I'm in
favor or renaming whitelist/blacklist in accept/tempfail/reject.

--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

On Oct 25, 2009, at 8:23 AM, manu@... wrote:
> Rudy Eschauzier <reschauzier@...> wrote:
>
> > 1. Introduce a new keyword, e.g 'skipdacl'
> >
> > 2. Only have messages that get greylisted by a racl command pass
> through
> > the dacl stage. All globally whitelisted messages (through auth or
> global
> > spf for example) whill skip dacl, as will auto-white listed
> messages.
> >
> > 3. Base the precedence of dacl over racl commands on the line
> number.
>
> I have concerns with options 2 and 3 because they modify the way
> existing setups work. skipdacl looks better to me (I disklike the
> name,
> though, but this is not a big concern)"
>
I'm for option 3, or an way that gets that functionality to mix racls
& dacls, since I think it adds more flexibility, at least for the dkim
& body case as I see it.

Today, or at least then I tested this, racls will greylist all mails
(then using racl greylist default) unless they are whitelisted by the
configuration. When having the philosophy "whitelist all known or
trusted sources, give penalty to known spam sources, and greylist the
rest", the dacl with the rules for checking for example dkim, will
become more or less useless. So, in order to do whitelisting of dkim,
I have to skip the racl stage (racl whitelist default). And then I
loose the major benefit of using greylisting, which of course I don't
want to.

Mixing racl and dacl will be more CPU intensive than doing them in the
current order, but I think it adds more flexibility if it's possible
to be able to do whitelisting (or greylisting and perhaps even
blacklisting) on both RCPT and DATA before falling back on the default.

As an example, I would like to be able to do something similar to this.
(where the acls are parsed in linear order)

racl whitelist list "my network"
racl whitelist list "awl admin recipients"
racl whitelist list "broken mta"
dacl whitelist dkim pass
dacl whitelist body /^-----BEGIN PGP.*MESSAGE-----$/
racl greylist sm_macro "maybe_forged" delay 1h autowhite 3d
racl greylist dnsrbl "SPAMHAUS" delay 24h autowhite 3d
dacl greylist dkim fail delay 1h autowhite 3d
*acl greylist default delay 30m autowhite 30d

*I think "racl greylist default" would be misleading in this
configuration case. Correct me if I'm wrong here, but I also believe
that the way "racl greylist default delay ... autowhite ..." works,
wasn't possible to configure with dacl (see it as another suggestion
for enhancement :-)).

Regards,
/P

Petar Bogdanovic wrote:
  > ------- Original message -------
  > > From: Rudy Eschauzier <reschauzier@...>
  > > Sent: 25.10.'09,  8:54
  > >
  > > Currently, the way milter-greylist processes data stage
  > > commands (dacl lines in greylist.conf) is less than optimal.
  > >
  > > (...)
  >
  > I like 1., dislike 2. and find 3. to be a bit counterintuitive given
  > how SMTP works.

I also prefer 1.

Best regards
    Oliver

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

Python is executable pseudocode.  Perl is executable line noise.

------- Original message -------
> From: Rudy Eschauzier <reschauzier@...>
> Sent: 25.10.'09,  8:54
>
> Currently, the way milter-greylist processes data stage
> commands (dacl lines in greylist.conf) is less than optimal.
>
> (...)

I like 1., dislike 2. and find 3. to be a bit counterintuitive given
how SMTP works.

                  Petar Bogdanovic

I understand where you are coming from, and in most cases I'd agree: backward
compatibility is critical from a user point of view.

In this situation, however, I think the current dacl implementation is so
immature that I'd be surprised that a lot of people are actually using it in
production.

Let's see what the mailing list members have to say. How are they using the dacl
commands right now, and will they be affected by any of the proposals?

Rudy.

--- In milter-greylist@yahoogroups.com, manu@... wrote:
>
> Rudy Eschauzier <reschauzier@...> wrote:
>
> > 1. Introduce a new keyword, e.g 'skipdacl'
> >
> > 2. Only have messages that get greylisted by a racl command pass through
> > the dacl stage. All globally whitelisted messages (through auth or global
> > spf for example) whill skip dacl, as will auto-white listed messages.
> >
> > 3. Base the precedence of dacl over racl commands on the line number.
>
> I have concerns with options 2 and 3 because they modify the way
> existing setups work. skipdacl looks better to me (I disklike the name,
> though, but this is not a big concern)"
>
>
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu@...
>

Rudy Eschauzier <reschauzier@...> wrote:

> 1. Introduce a new keyword, e.g 'skipdacl'
>
> 2. Only have messages that get greylisted by a racl command pass through
> the dacl stage. All globally whitelisted messages (through auth or global
> spf for example) whill skip dacl, as will auto-white listed messages.
>
> 3. Base the precedence of dacl over racl commands on the line number.

I have concerns with options 2 and 3 because they modify the way
existing setups work. skipdacl looks better to me (I disklike the name,
though, but this is not a big concern)"


--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu@...

Currently, the way milter-greylist processes data stage commands (dacl lines in
greylist.conf) is less than optimal.

Data stage commands are commands that activate during the body stage of the
milter and are useful for content scanning (e.g. Spamassasin) and message
signing and verification (e.g. DomainKeys and Dkim).

Dacl commands are expensive, in that the entire message (up to maxpeek, if
defined) is filtered through the milter. Racl commands, on the other hand, base
their decisions on connection information only and are light on the CPU.

In the current setup, milter-greylist will process the dacl commands after the
racl ones and send all messages that do not get rejected (blacklist) by a racl
command through the dacl filter. Even messages that get whitelisted by a racl
line, will have their contents inspected. On top of that, auto-white listed
messages and messages that got globally whitelisted, such as messages from
authenticated users, go through this CPU intensive task.

I have had some email exchange with Emmanuel, and we both agree there is room
for improvement. It is not clear, however, how this improvement should be
implemented. Here are a couple of options:

1. Introduce a new keyword, e.g 'skipdacl'. Placing this keyword in a racl line,
will cause the dacl commands to be skipped if the racl line is matched.

2. Only have messages that get greylisted by a racl command pass through the
dacl stage. All globally whitelisted messages (through auth or global spf for
example) whill skip dacl, as will auto-white listed messages.

This is based on the premise that racl lines will only blacklist or whitelist a
message if the matching rule guarantees nearly 100% certainty that the message
is either ham or spam. Otherwise the result of the line will be either false
positives or false negatives. Only in case of doubt, will a greylist line be
matched. These are the cases that will be forwarded for further evaluation by
the dacl lines.

3. Base the precedence of dacl over racl commands on the line number. Currently,
the order of racl lines determines the precedence of one line over the other.
The first matching line gets executed, all others skipped. By putting whitelist
lines before greylist lines, we can make sure that messages of which it is
certain they are ham get accepted directly, for example.

Dacl lines get processed after racl lines, even if they come first in the conf
file. This is related to the order in which Sendmail invokes the milter
subroutines. It is not very intuitive from a user point of view, however.
Why not have to order of the racl and dacl commands determine if a dacl stage
command needs to be executed? In this case, the user can decide to put all racl
lines before the dacl lines, causing the dacl stage to be skipped whenever is
racl match if found. Or put racl whitelist lines first, than the dacl lines and
then the racl greylist lines. This set up will make sure that messages that get
whitelisted in the racl stage, do not get forwarded to the dacl stage, but racl
greylisted messages do. You get the idea.

To me, option 3 seems like the most elegant one, extending the current
philosophy of milter-greylist seamlessly to the dacl stage processing. Option 2
is not too bad either (and easier to implement). You may have a different
opinion, though. Please let me know what you think. I a prepared to implement a
solution, once we have achieved some consensus.

On Fri, Oct 16, 2009 at 05:34:09PM +0200, Oliver Fromme wrote:
>
> If a tuple is inserted into the greylist and another message
> with the same tuple arrives ten minutes later, then that
> second message could be delayed by as few as 20 minutes.

There is no way for milter-greylist to distinguish two different
messages with an identical ip-from-rcpt combination.

In that particular situation, one message will generate the greylist
tuple while the other will count as a retry.  As soon as the greylisting
period has expired, one message will generate an autowhite tuple while
the other will get a pass.

The behaviour reported by OP is probably a bug and not even a weird one
since I often get ``X-Greylist: Delayed for 00:00:00 (...)'' for dacl
delayed mails.



    Petar Bogdanovic

philippeake wrote:
  > --- In milter-greylist@yahoogroups.com, "d d" <x55k@...> wrote:
  > >
  > > --- In milter-greylist@yahoogroups.com, "philippeake" <philip@> In emails,
I see:
  > > >
  > > >         X-Greylist: Delayed for 00:26:40 by milter-greylist-4.2.3
  > > >         X-Greylist: Delayed for 00:21:04 by milter-greylist-4.2.3
  > > >         etc.
  > > >
  > > > Why?
  > >
  > > That is the same tuple trying to send the email after getting a 30-mins
delay with greylist. They get a 30 mins delay default. If they try to send the
same email after 3 mins 20 secs, they will get: X-Greylist: Delayed for 00:26:40
  > >
  >
  > I think you are misunderstanding - those lines that you quote above are in
the delivered email.

I think you are misunderstanding.  :-)

  > I read them to be saying that "This email was delayed for 00:26:40".
  > No email should be delivered with anything less than 30 miutes.

That's not entirely true.

If a tuple is inserted into the greylist and another message
with the same tuple arrives ten minutes later, then that
second message could be delayed by as few as 20 minutes.

Of course, the exact delays depend on the retry intervals of
the sending MTA.

  > These are not logfile entries, they are email headers.

AFAIK those are the same.

Best regards
    Oliver

PS:  By the way, I think 30 minutes is rather long as a
default greylist delay for _all_ messages.  Using 10 or
even only 5 minutes will catch almost as much spam while
being less annoying for the recipients of legal mail.

Of course, you might chose to use longer delays depending
on blacklists and similar, but that's a different matter.

--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

"Perl will consistently give you what you want,
unless what you want is consistency."
         -- Larry Wall

--- In milter-greylist@yahoogroups.com, "d d" <x55k@...> wrote:
>
> --- In milter-greylist@yahoogroups.com, "philippeake" <philip@> In emails, I
see:
> >
> >         X-Greylist: Delayed for 00:26:40 by milter-greylist-4.2.3
> >         X-Greylist: Delayed for 00:21:04 by milter-greylist-4.2.3
> >         etc.
> >
> > Why?
>
> That is the same tuple trying to send the email after getting a 30-mins delay
with greylist. They get a 30 mins delay default. If they try to send the same
email after 3 mins 20 secs, they will get: X-Greylist: Delayed for 00:26:40
>

I think you are misunderstanding - those lines that you quote above are in the
delivered email.

I read them to be saying that "This email was delayed for 00:26:40".
No email should be delivered with anything less than 30 miutes.

These are not logfile entries, they are email headers.