__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Hi, (I think this is my first post for this mailing list)

I thought about it a little bit and posted to my blog a few days ago.
http://www.witha.jp/blog/archives/2005/04/p2p_feed_client.html
(written in Japanese though..)

the entry talks about possibility of using P2P distributed network as a
means of feeds distribution.

My conclution was that
"Do we really need to move on to P2P network NOW while the good old HTTP
  way manages to work OK(not perfect but, hey) so far."
"Well, if not now, then when? We don't know."
"One thing I'm sure is that a lot of developers would dislike the use of P2P
because developing a P2P client requres a sophisticated skill and knowledge
  of ICP/IP, UDP which means cost maney. The virtue of the current feed
aggregation
  is that it's just a simple HTTP GET."
But, of cource, more people use it more tools created.


that was just random thoughts of mine.
any thoughts?



>
>Hi,
>I've built an RSS/Atom aggregator and was pleasantly surprised to find
>this mailing list.
>
>Obviously one of the main issues facing online feed aggregators is the
>regular polling of feeds. There are various ways to help combat this
>problem (blogs pinging, etags, last-mod headers etc.) but none of them
>perfect or a complete solution.
>Has there been any movement or suggestion to get various aggregators
>to work together and form some type of distributed feed network?
>Such a network could have an open API allowing desktop readers to
>query for updates to feeds and web readers/aggregators could have new
>items for registered feeds sent as they enter the central network.
>Various aggregators using the same system could have responsibility
>for a subset of feeds and only worry about updating those, while still
>having access to all the feeds other aggregators are regularly
>updating. Down the track such a network would have a strong pulling
>power to help update the technology behind web feeds....
>There would definately be various logistical issues to work out (with
>an open world wide communication network), but not unfeasible. It
>could be supported by some of the larger aggregators to help get off
>the ground - although I don't see that as necessary.
>
>Is anyone doing this already? If not, why not?
>
>Regards,
>Chris
>

Hi,
I've built an RSS/Atom aggregator and was pleasantly surprised to find
this mailing list.

Obviously one of the main issues facing online feed aggregators is the
regular polling of feeds. There are various ways to help combat this
problem (blogs pinging, etags, last-mod headers etc.) but none of them
perfect or a complete solution.
Has there been any movement or suggestion to get various aggregators
to work together and form some type of distributed feed network?
Such a network could have an open API allowing desktop readers to
query for updates to feeds and web readers/aggregators could have new
items for registered feeds sent as they enter the central network.
Various aggregators using the same system could have responsibility
for a subset of feeds and only worry about updating those, while still
having access to all the feeds other aggregators are regularly
updating. Down the track such a network would have a strong pulling
power to help update the technology behind web feeds....
There would definately be various logistical issues to work out (with
an open world wide communication network), but not unfeasible. It
could be supported by some of the larger aggregators to help get off
the ground - although I don't see that as necessary.

Is anyone doing this already? If not, why not?

Regards,
Chris

Hi,
Could anyone send me links regarding RSS readsers and
their implementation since i need to write a rss
reader in C

Regards,
me




__________________________________
Do you Yahoo!?
All your favorites on one personal page – Try My Yahoo!
http://my.yahoo.com

I would like to make a RSS Reader, one to read simple 2.0 RSS
Formats.  What should I use? Javascript, Java?

How do I go about it, I mean, how would I go about reading the
formats, and tags?

Sincerely,
         Manakra

I've just finished writing a aggregator that delivers RSS to Outlook:
http://rsspopper.blogspot.com/

Yes, I know such applications exists, but I needed it to have good
support for non-english fonts (and I didn't want to pay...)

bradbury_software <public@...> wrote:
>FeedDemon strips style attributes when displaying a feed, so I'm
>afraid your approach wouldn't work in FeedDemon.

It's not a bit deal. But it has convinced me that it's a good idea to
put the simplest possible html into feed's descriptions. Something like
this.

Start blogging today [Ad by Google]
Share photos and more. Easy to use Sign up for your free trial today

With the first line being two links and the second line plain text.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

FeedDemon strips style attributes when displaying a feed, so I'm
afraid your approach wouldn't work in FeedDemon.

--- In aggregators@yahoogroups.com, Julian Bond <julian_bond@v...>
wrote:
> How aggressive are aggregators now about stripping malicious HTML
from
> the <description> field?
>
> I'd like to add some small css style= alterations to some <a href
> entries. eg
>
> <a href="foo.com" style="text-decoration: none;" >some text</a>
>
> Is this likely to get removed?
>
> BTW. This is all about embedding Adverts in RSS.
>
> --
> Julian Bond Email&MSM: julian.bond at voidstar.com
> Webmaster:                 http://www.ecademy.com/
> Personal WebLog:          http://www.voidstar.com/
> M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

Julian Bond wrote:

>How aggressive are aggregators now about stripping malicious HTML from
>the <description> field?
>
>I'd like to add some small css style= alterations to some <a href
>entries. eg
>
><a href="foo.com" style="text-decoration: none;" >some text</a>
>
>Is this likely to get removed?
>
>
Because of the various attacks that can be propagated that way,
Bloglines strips those declarations.


Mark

BottomFeeder doesn't strip any tags.  It's immune from the various
tag/script based attacks though, since the html component I use doesn't
support any of the features necessary to execute an attack.


At 04:19 AM 11/23/2004, you wrote:
>How aggressive are aggregators now about stripping malicious HTML from
>the <description> field?
>
>I'd like to add some small css style= alterations to some <a href
>entries. eg
>
><a href="foo.com" style="text-decoration: none;" >some text</a>
>
>Is this likely to get removed?
>
>BTW. This is all about embedding Adverts in RSS.
>
>--
>Julian Bond Email&MSM: julian.bond at voidstar.com
>Webmaster:                 <http://www.ecademy.com/>http://www.ecademy.com/
>Personal WebLog:          <http://www.voidstar.com/>http://www.voidstar.com/
>M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433
>
>
><http://groups.yahoo.com/groups/aggregators>http://groups.yahoo.com/groups/aggr\
egators
>
>
>
>
>Yahoo! Groups Sponsor
>ADVERTISEMENT
>
>
>----------
>Yahoo! Groups Links
>    * To visit your group on the web, go to:
>    *
>
<http://groups.yahoo.com/group/aggregators/>http://groups.yahoo.com/group/aggreg\
ators/
>
>    *
>    * To unsubscribe from this group, send an email to:
>    *
>
<mailto:aggregators-unsubscribe@yahoogroups.com?subject=Unsubscribe>aggregators-\
unsubscribe@yahoogroups.com
>
>    *
>    * Your use of Yahoo! Groups is subject to the
> <http://docs.yahoo.com/info/terms/>Yahoo! Terms of Service.

<Talk Small and Carry a Big Class Library>
James Robertson, Product Manager, Cincom Smalltalk
http://www.cincomsmalltalk.com/blog/blogView
Cincom Smalltalk User Conference!
http://www.cincomsmalltalk.com/blog/blogView?entry=3275631428

> How aggressive are aggregators now about stripping malicious
> HTML from the <description> field?
>
> I'd like to add some small css style= alterations to some <a
> href entries. eg
>
> <a href="foo.com" style="text-decoration: none;" >some text</a>
>
> Is this likely to get removed?

Sauce Reader will remove the style attribute, providing the user turn
the 'save' option on (which is on by default).


--
Victor Hadianto
http://synop.com/Products/SauceReader/

How aggressive are aggregators now about stripping malicious HTML from
the <description> field?

I'd like to add some small css style= alterations to some <a href
entries. eg

<a href="foo.com" style="text-decoration: none;" >some text</a>

Is this likely to get removed?

BTW. This is all about embedding Adverts in RSS.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

Bill Kearney <ml_yahoo@...> wrote:
>> - An aggregator with Podcasting really ought to support BitTorrent.
>And other p2p networks like eDonkey/overnet and others.

>A laudable goal, just consider there's more to p2p than BitTorrent.

Is there anything remotely resembling a GUID for files on a generalised
P2P system? I vaguely remember people talking about this in the past,
but I don't recall seeing anything recently. If there was something like
edonkey:foobar.mp3 and this was found in a feed we could just hand it
off to what ever the handler was for edonkey: files. Similarly for
something like soulseek://jbond23/library/chillosophy/mixes/chilled1.mp3

FWIW, I think of BitTorrent rather differently. It may use a swarm to do
the actual delivery, but the .torrent has a completely normal http
location.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

Bill Kearney <ml_yahoo@...> wrote:
>> - If an aggregator with Podcasting has a BitTorrent client built in, it
>> might as well be a full BT client handling any .torrent requests from
>> the OS, not just enclosures.
>
>The aggregator should not be having it's own BT client.

This was all written some time ago. Having now played with Azureus I'd
agree. BT may be a simple protocol, but doing a full client is a project
in it's own right.

seeAlso today's screed: http://www.voidstar.com/node.php?id=2078 What I
think I've learned from podcasting.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

> - Podcasting really needs a resident, local aggregator that runs all the
> time in the tray or as a service so that enclosures are collected in
> background as well as feeds.

I'd rather see the podcasting tool understand how to use a system-wide BT
mechanism, as well as other p2p protocols.  I'd rather not have apps duking
it out trying to be the p2p client and stepping on each other.

> - An aggregator with Podcasting really ought to support BitTorrent.

And other p2p networks like eDonkey/overnet and others.

> - If an aggregator with Podcasting has a BitTorrent client built in, it
> might as well be a full BT client handling any .torrent requests from
> the OS, not just enclosures.

The aggregator should not be having it's own BT client.

> - If it's running all the time, then it will act as a BT seed all the
> time as well.

Yes, a BT client should be running all the time or at least a lot of the
time.

> Now looking at a sample of podcasting feeds, very few of them are using
> BitTorrent. I think there are two reasons for this.
>
> 1) Everyone can read MP3, not everyone can collect BT.

Also not everyone knows about BT or other p2p networks.

> This means that
> we need a way of recommending BT but offering MP3 as an alternative in
> the same feed with some marker to say that the end file is the same. I'm
> not sure how you would code this in RSS/Atom.

RDF's alternate would be a fine candidate for this.  I'd shy away from
assumptions about filename endings and suffixing rules.  Madness lies down
that road.

> 2) Creating a .torrent and getting it on a BT tracker is still too hard.

Yep, keeps me from bothering with it at all.

> I imagine a tracker service with a REST/XMLRPC/SOAP API where you told
> it the location of a file on the public net and it created the .torrent,
> created the tracker entry and then returned the details for inclusion in
> your feed.

Hmmm....

> Once
> the service exists with the API, this could be built into blogging
> systems so that it happens automatically when you post a blog entry with
> an enclosure.

But shouldn't that just be something the client app does on it's own anyway?

> The goal in all this is to make BT ubiquitous but also to have it fade
> into the background where the end user doesn't need to know any more
> about it than they know about http.

A laudable goal, just consider there's more to p2p than BitTorrent.

-Bill Kearney

Hi all,

Anyone coming to DC next week for XML 2004?  Perhaps it'd be good to
coordinate a get-together BoF session.

It's my home town so I'll certainly attend.  I'll be presenting at the
xml.gov town hall on the evening of the 17th so mark your calendars.

-Bill Kearney
Syndic8.com

> Are there other aggregators that allow per-feed checking intervals?

nntp//rss supports per-feed polling intervals.  Its polling manager wakes up
every 30 seconds and determines, based upon the interval setting and the
previous poll time, whether the feed is ready to be polled.

Jason

---
Jason Brome
jason@...
nntp//rss Project: http://www.methodize.org/nntprss/

> > Maybe I didn't explain myself clearly :) What I meant was that most
> > aggregators refreshes its feed in interval between the last refresh
> > and not in the hour. So for example you refreh your feed at
> 2:18PM and
> > you set your aggregator to refresh the feed every 2 hour, the next
> > (automatic) refresh will start at 4:18PM.
> >
> > Most aggregators that I know also refresh the feed
> individually (well
> > at least Sauce Reader does), meaning that it doesn't do automatic
> > 'refresh all' feeds at the same time.
>
> I've hacked at Feed on Feeds a bit and changed the way it
> checks for feeds. The feed checking does happen every hour,
> but it checks the database for each feed before actually
> sending a request, and if enough time has not passed, it
> skips. I came to realize that there are some feeds that I
> would like to check every hour, but there are plenty that I
> don't mind if they are checked every 4, 6, 12, 24, or more
> hours. So with 100 feeds, it might not even check half of
> those hour. Of course it would be ideal to adjust the
> aggregator to determine the real intervals between updates
> and attempt to adjust to that as well.
>
> Are there other aggregators that allow per-feed checking intervals?

Sauce Reader does, and I think SharpReader, RSS Bandit, FeedDemon and
NewzCrawler do this as well.

--
Victor

Pete Prodoehl <pete@...> wrote:
>Of course
>it would be ideal to adjust the aggregator to determine the real
>intervals between updates and attempt to adjust to that as well.
>
>Are there other aggregators that allow per-feed checking intervals?

My home grown one does this.
- Start at 60 minute refresh
- If a new item comes in set it to 60 minutes
- If no new item is present add 30 minutes to the refresh, up to a max
of 8 hours.
- When calculating the nest refresh time, do the collection, then make
the next refresh now() + the new refresh time + rand(5)
- Do a check once a minute

Over time the feeds get spread all over the clock and low volume feeds
get spread out to 8 hour refreshes.

I'm trying to wean myself off checking the aggregator every hour, so I
may make the default 4 hours instead of 1 hour.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

BottomFeeder runs an update loop on a regular (user defined)
interval.  Before firing off an http query, it checks the feed's update
settings (if any) as defined by the feed source - and does not do an update
if it's not time yet.

If it does do an update, it does a conditional-get

At 05:58 AM 11/8/2004, you wrote:
>On Nov 7, 2004, at 3:46 PM, Victor Hadianto wrote:
> >> Victor Hadianto <victor@...> wrote:
> >> Well yes. But what minute past the hour do they fire? It may
> >> be every 2 hours, but still fire at 00:00, 02:00, 04:00. Or
> >> is it 00:23:23, 02:23:23, 04:23:23? If everyone is synced to
> >> mickey mouse time and fires at 00:00 that's still a big hit
> >> at the top of the hour even if every instance only hits every
> >> two hours.
> >
> > Maybe I didn't explain myself clearly :) What I meant was that most
> > aggregators refreshes its feed in interval between the last refresh and
> > not in the hour. So for example you refreh your feed at 2:18PM and you
> > set your aggregator to refresh the feed every 2 hour, the next
> > (automatic) refresh will start at 4:18PM.
> >
> > Most aggregators that I know also refresh the feed individually (well
> > at
> > least Sauce Reader does), meaning that it doesn't do automatic 'refresh
> > all' feeds at the same time.
>
>I've hacked at Feed on Feeds a bit and changed the way it checks for
>feeds. The feed checking does happen every hour, but it checks the
>database for each feed before actually sending a request, and if enough
>time has not passed, it skips. I came to realize that there are some
>feeds that I would like to check every hour, but there are plenty that
>I don't mind if they are checked every 4, 6, 12, 24, or more hours. So
>with 100 feeds, it might not even check half of those hour. Of course
>it would be ideal to adjust the aggregator to determine the real
>intervals between updates and attempt to adjust to that as well.
>
>Are there other aggregators that allow per-feed checking intervals?
>
>Pete
>
>
>
>
>
><http://groups.yahoo.com/groups/aggregators>http://groups.yahoo.com/groups/aggr\
egators
>
>
>
>
>Yahoo! Groups Sponsor
>ADVERTISEMENT
>
>
>----------
>Yahoo! Groups Links
>    * To visit your group on the web, go to:
>    *
>
<http://groups.yahoo.com/group/aggregators/>http://groups.yahoo.com/group/aggreg\
ators/
>
>    *
>    * To unsubscribe from this group, send an email to:
>    *
>
<mailto:aggregators-unsubscribe@yahoogroups.com?subject=Unsubscribe>aggregators-\
unsubscribe@yahoogroups.com
>
>    *
>    * Your use of Yahoo! Groups is subject to the
> <http://docs.yahoo.com/info/terms/>Yahoo! Terms of Service.

<Talk Small and Carry a Big Class Library>
James Robertson, Product Manager, Cincom Smalltalk
http://www.cincomsmalltalk.com/blog/blogView
Cincom Smalltalk User Conference!
http://www.cincomsmalltalk.com/blog/blogView?entry=3275631428

On Nov 7, 2004, at 3:46 PM, Victor Hadianto wrote:
>> Victor Hadianto <victor@...> wrote:
>> Well yes. But what minute past the hour do they fire? It may
>> be every 2 hours, but still fire at 00:00, 02:00, 04:00. Or
>> is it 00:23:23, 02:23:23, 04:23:23? If everyone is synced to
>> mickey mouse time and fires at 00:00 that's still a big hit
>> at the top of the hour even if every instance only hits every
>> two hours.
>
> Maybe I didn't explain myself clearly :) What I meant was that most
> aggregators refreshes its feed in interval between the last refresh and
> not in the hour. So for example you refreh your feed at 2:18PM and you
> set your aggregator to refresh the feed every 2 hour, the next
> (automatic) refresh will start at 4:18PM.
>
> Most aggregators that I know also refresh the feed individually (well
> at
> least Sauce Reader does), meaning that it doesn't do automatic 'refresh
> all' feeds at the same time.

I've hacked at Feed on Feeds a bit and changed the way it checks for
feeds. The feed checking does happen every hour, but it checks the
database for each feed before actually sending a request, and if enough
time has not passed, it skips. I came to realize that there are some
feeds that I would like to check every hour, but there are plenty that
I don't mind if they are checked every 4, 6, 12, 24, or more hours. So
with 100 feeds, it might not even check half of those hour. Of course
it would be ideal to adjust the aggregator to determine the real
intervals between updates and attempt to adjust to that as well.

Are there other aggregators that allow per-feed checking intervals?

Pete

> >Hmm, here's an idea, alter your code to put in a new item asking them to
fix
> >their aggregator.  That way you could provide them content AND the
'public
> >service announcement' about their tool needing to do a better job.
>
> Sadly, I'm inserting a daily item announcing that the service is closing
> down...

Take one last stab at the lazy apps and put a notification of them in there.
Go out in a blaze of Quixotic glory!

-Bill Kearney

> Victor Hadianto <victor@...> wrote:
> >I don't think any aggregators are doing it this way at least
> not that
> >I've seen and I have seen a lot of them :). Aggregators usually use
> >interval which could be customised by users, ie: update feeds every
> >1hr, 2hr, 3hr, 1 day etc.
>
> Well yes. But what minute past the hour do they fire? It may
> be every 2 hours, but still fire at 00:00, 02:00, 04:00. Or
> is it 00:23:23, 02:23:23, 04:23:23? If everyone is synced to
> mickey mouse time and fires at 00:00 that's still a big hit
> at the top of the hour even if every instance only hits every
> two hours.

Maybe I didn't explain myself clearly :) What I meant was that most
aggregators refreshes its feed in interval between the last refresh and
not in the hour. So for example you refreh your feed at 2:18PM and you
set your aggregator to refresh the feed every 2 hour, the next
(automatic) refresh will start at 4:18PM.

Most aggregators that I know also refresh the feed individually (well at
least Sauce Reader does), meaning that it doesn't do automatic 'refresh
all' feeds at the same time.

--
Victor Hadianto

Bill Kearney <ml_yahoo@...> wrote:
>
>> Not reporting a user-agent at all is pretty mean as is leaving referer
>> blank. >99% had no referer
>
>Hmm, here's an idea, alter your code to put in a new item asking them to fix
>their aggregator.  That way you could provide them content AND the 'public
>service announcement' about their tool needing to do a better job.

Sadly, I'm inserting a daily item announcing that the service is closing
down...

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

Julian Bond wrote:
> Graham <dtcd@...com> wrote:
>> Why would an aggregator send a referrer? What would be in it?
> The website of the aggregator manufacturer? Or in the case
> above, the website that is consuming the RSS and then redisplaying it?
	 What you suggest would be an inappropriate use of referrer. This
field is clearly defined in the HTTP 1.1 specification[1] as follows:

"The Referer[sic] request-header field allows the client to specify, for the
server's benefit, the address (URI) of the resource from which the
Request-URI was obtained (the "referrer", although the header field is
misspelled.) The Referer request-header allows a server to generate lists of
back-links to resources for interest, logging, optimized caching, etc. It
also allows obsolete or mistyped links to be traced for maintenance. The
Referer field MUST NOT be sent if the Request-URI was obtained from a source
that does not have its own URI, such as input from the user keyboard."

	 Given this definition, an RSS aggregator should only even consider
providing a referer if it has somehow stored the URI of the resource from
which it originally obtained the URI of the feed. And, if the referer is
provided, it should *only* be the URI of that resource. I am unaware of any
RSS/Atom aggregators that store such URIs. In the rare case when a referer
*does* appear in the log for a fetch of an RSS/Atom file, it would indicate
that the RSS file must have been accessed by following a link in a browser
-- not accessed via an aggregator.

		 bob wyman

[1] http://www.w3.org/Protocols/HTTP/1.1/rfc2616.pdf
See page 86.

At 01:29 AM 11/7/2004, you wrote:

>I don't think any aggregators are doing it this way at least not that
>I've seen and I have seen a lot of them :). Aggregators usually use
>interval which could be customised by users, ie: update feeds every 1hr,
>2hr, 3hr, 1 day etc.

BottomFeeder has a configurable update cycle - the shortest interval it
supports is 30 minutes.  What it does is fire off updates N minutes after
the end of the last update cycle - so the time it does an HTTP query
depends on when you started it up, not on the clock time



>--
>Victor Hadianto
>
>
>-----Original Message-----
>From: Randy Morin [mailto:randy@...]
>Sent: Sunday, 7 November 2004 10:50 AM
>To: aggregators@yahoogroups.com
>Subject: [aggregators] Re: Feed collection timing
>
>
>
>Any ideas which aggregators might be doing this?
>
>
>--- In aggregators@yahoogroups.com, Julian Bond <julian_bond@v...>
>wrote:
> > Which aggregators do a feed collection run at the top of the hour?
> >
> > If you're the author of an aggregator, please don't do this.
>Please add
> > a little randomness. Otherwise, if you're successful and the app
>is
> > widely used, you've created a distributed DOS attack on popular
>feeds.
> >
> > --
> > Julian Bond Email&MSM: julian.bond at voidstar.com
> > Webmaster:                 <http://www.ecademy.com/>http://www.ecademy.com/
> > Personal
> WebLog:          <http://www.voidstar.com/>http://www.voidstar.com/
> > M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433
>
>
>
>
>
>
><http://groups.yahoo.com/groups/aggregators>http://groups.yahoo.com/groups/aggr\
egators
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>
>
>
>
>
>
><http://groups.yahoo.com/groups/aggregators>http://groups.yahoo.com/groups/aggr\
egators
>
>
>
>
>Yahoo! Groups Sponsor
>ADVERTISEMENT
>
>
>----------
>Yahoo! Groups Links
>    * To visit your group on the web, go to:
>    *
>
<http://groups.yahoo.com/group/aggregators/>http://groups.yahoo.com/group/aggreg\
ators/
>
>    *
>    * To unsubscribe from this group, send an email to:
>    *
>
<mailto:aggregators-unsubscribe@yahoogroups.com?subject=Unsubscribe>aggregators-\
unsubscribe@yahoogroups.com
>
>    *
>    * Your use of Yahoo! Groups is subject to the
> <http://docs.yahoo.com/info/terms/>Yahoo! Terms of Service.

<Talk Small and Carry a Big Class Library>
James Robertson, Product Manager, Cincom Smalltalk
http://www.cincomsmalltalk.com/blog/blogView
Cincom Smalltalk User Conference!
http://www.cincomsmalltalk.com/blog/blogView?entry=3275631428

> Not reporting a user-agent at all is pretty mean as is leaving referer
> blank. >99% had no referer

Hmm, here's an idea, alter your code to put in a new item asking them to fix
their aggregator.  That way you could provide them content AND the 'public
service announcement' about their tool needing to do a better job.

-Bill Kearney

Victor Hadianto <victor@...> wrote:
>I don't think any aggregators are doing it this way at least not that
>I've seen and I have seen a lot of them :). Aggregators usually use
>interval which could be customised by users, ie: update feeds every 1hr,
>2hr, 3hr, 1 day etc.

Well yes. But what minute past the hour do they fire? It may be every 2
hours, but still fire at 00:00, 02:00, 04:00. Or is it 00:23:23,
02:23:23, 04:23:23? If everyone is synced to mickey mouse time and fires
at 00:00 that's still a big hit at the top of the hour even if every
instance only hits every two hours.

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

Graham <dtcd@...> wrote:
>On 7 Nov 2004, at 8:00 am, Julian Bond wrote:
>
>> Well in a recent stats run on hits against gnews2rss.php, 90% of the
>>  aggregators reported no User-Agent, so it's impossible to tell.
>
>All popular aggregators I know of send a User-Agent header, as well as
>most bots and most HTTP libraries. Something very strange is happening
>to your site. Check the IP addresses that they're coming from. It could
>also be a bug in your software not logging them correctly.

Here's a typical example.

66.6.223.120 - - [02/Nov/2004:12:21:35 +0000] "GET
/gnews2rss.php?num=number_of_items&q=baby+wont+sleep HTTP/1.0" 200 1030
"-" "-"

It's very standard Apache 2 Combined format. The two hyphens at the end
are referrer and user-agent. The IP seems to be some web hosting.

gnews2rss.php used to scrape a Google news and turn it into RSS. My
quess is that a significant number of the hits were coming from RSS
readers built into websites rather than on the desktop. I can see user
agents for
MagpieRSS
Bloglines
RssReader
NewsGatorOnline
JetBrains
NetNewsWire
YahooFeedSeeker
NetNewsWire
SharpReader
etc

>>  Not reporting a user-agent at all is pretty mean as is leaving referer
>>  blank. >99% had no referer
>Why would an aggregator send a referrer? What would be in it?

The website of the aggregator manufacturer? Or in the case above, the
website that is consuming the RSS and then redisplaying it?

--
Julian Bond Email&MSM: julian.bond at voidstar.com
Webmaster:                 http://www.ecademy.com/
Personal WebLog:          http://www.voidstar.com/
M: +44 (0)77 5907 2173      T: +44 (0)192 0412 433

On 7 Nov 2004, at 8:00 am, Julian Bond wrote:

> Well in a recent stats run on hits against gnews2rss.php, 90% of the
>  aggregators reported no User-Agent, so it's impossible to tell.

All popular aggregators I know of send a User-Agent header, as well as
most bots and most HTTP libraries. Something very strange is happening
to your site. Check the IP addresses that they're coming from. It could
also be a bug in your software not logging them correctly.

>  Not reporting a user-agent at all is pretty mean as is leaving referer
>  blank. >99% had no referer

Why would an aggregator send a referrer? What would be in it?

Graham