I'm a bit surprised that the firestorm over AVG Linkscanner hasn't come up
here yet (or did I miss it?)... is anybody else feeling the pain from it?
Or are there so few people here using log file analysis that it has gone
unnoticed by many?
Linkscanner is part of AVG's security package. It pre-fetches links on every
page the user views (or just on SERPs, if the user chooses) and scans them
for malware... and tracking cookies. It masquerades as a normal browser. A
user who normally would have a handful of page views suddenly has hundreds
or thousands (and yes, many users have complainted about what it does to
their browser performance).

On some sites, such as Google, it triggers rate limiters because it is
requesting pages so rapidly.

It is invisible to hosted analytics because (I can tell from my analytics)
99 percent of people using it block the cookies (or the script itself, I
suppose). Log file analytics see a huge increase in page views with no
corresponding increase in visitors. This means it also looks just like a
spambot address harvester.

I was able to identify some of its probable activity because an earlier
version omits a space that's normally in the user agent string (look for
;1813 with no space after the semicolon). That apparently is "fixed" now.

On the good side of things, an AVG executive posted the following on the
Register's site (see
http://www.theregister.co.uk/2008/06/13/avg_scanner_skews_web_traffic_numbers/co\
mments/)
on
Saturday:

"Hi, folks. Pat Bitton from AVG here. This issue has clearly raised some
concerns that we had not anticipated, and we acknowledge that we need to do
something. Our primary purpose with LinkScanner, as Roger Thompson has
pointed out, is to protect users against web-based threats that they cannot
see. These threats are also usually invisible to web site operators, who
presumably also don't wish to be unwittingly passing infections on to their
visitors. This kind of problem can and does affect all types of web sites,
big or small, and is extremely transient - which is why we don't use the
static database approach cited by some as a viable alternative. Over the
next few days, we will be exploring ways in which we can continue to deliver
informed protection as unobtrusively as possible without adversely impacting
site analytics. Any webmaster reading this post who is interested in working
with us constructively to reach this goal is welcome to contact me at
pat.bitton(at)avg.com."

I'd welcome any thoughts on this specific issue or the more general one it
raises.

Has the WAA tried to articulate a position on the tradeoffs between security
and analytics? I don't recall ever seeing anything like that.

Nick


[Non-text portions of this message have been removed]