Yahoo Groups produces invalid author elements in its RSS 2.0 feeds --
they aren't valid e-mail addresses.
Though this is a minor error for most consumers of the feeds, the RSS
Advisory Board doesn't inspire much confidence by using invalid feeds.
I wrote a PHP script that fixes the error, producing what should be
valid feeds for both of our lists:
http://www.rssboard.org/yahoo-groups-feed/rss-publichttp://www.rssboard.org/yahoo-groups-feed/rss-board
Users can subscribe to these URLs to read feeds that should be valid.
They're written using the Magpie RSS Parser for PHP.
The script caches feeds for 10 minutes and fixes three errors:
1. The author element is transformed into a dc:creator element.
2. If pubDate is empty, it isn't included in an item.
3. If author is empty, it isn't included as dc:creator.
The last two errors occur in a feed Yahoo generates for groups that
don't offer RSS, like this one:
http://www.rssboard.org/yahoo-groups-feed/bobkempfans
If it works, I'll release the code as an example of how to parse and
republish RSS 2.0 feeds in PHP.
> Though this is a minor error for most consumers of the feeds, the RSS
> Advisory Board doesn't inspire much confidence by using invalid feeds.
Nor does it inspire confidence by bodging together a hack instead of working
with the publisher to fix the feed source.
To say nothing of the issues surrounding copyright and republishing of data
without permission. The road to Hell is paved with "good intentions".
-Bill Kearney
Syndic8.com
--- In rss-public@yahoogroups.com, "Bill Kearney" <wkearney@...> wrote:
> Nor does it inspire confidence by bodging together a hack instead of
> working with the publisher to fix the feed source.
I've approached a few people at Yahoo about the issue, but it hasn't
gotten on their radar yet (or more likely, I haven't found the right
person to ask).
In the meantime, it looks bad and the mailing list is a collective
work of its participants, not the copyrighted work of Yahoo. I don't
see a reason we can't publish our own feeds.
Setting a good example comes to mind. If you want a feed produced from a
tool then get that tool to do it right. Fight the good fight, not shortcut
the process.
So if the tool won't take care of itself then use a different tool. Various
other mailing list packages exist, some based on open source.
I think it looks "worse" to cobble together a hack.
-Bill Kearney
Syndic8.com
> --- In rss-public@yahoogroups.com, "Bill Kearney" <wkearney@...> wrote:
> > Nor does it inspire confidence by bodging together a hack instead of
> > working with the publisher to fix the feed source.
>
> I've approached a few people at Yahoo about the issue, but it hasn't
> gotten on their radar yet (or more likely, I haven't found the right
> person to ask).
>
> In the meantime, it looks bad and the mailing list is a collective
> work of its participants, not the copyrighted work of Yahoo. I don't
> see a reason we can't publish our own feeds.
The draft spec for RSS 2.0 has been updated:
http://www.rssboard.org/rss-draft-1
Changes:
1. In item-description, changed "The description MUST be suitable for
presentation as HTML ..." to "The description SHOULD be suitable for
presentation as HTML ..."
2. Removed Sam Ruby's name from the credits at his request.
3. Updated the Trackback Namespace link in the introduction to
http://www.rssboard.org/trackback
4. Converted the document to valid XHTML 1.0.
To my knowledge, this is all of the outstanding issues that needed to
be resolved.
The definition of <author> fails to mention what format of email
address to use. I assume it's a RFC 822 address, with the final
comment after the address giving the person's name.
- Geoffrey Sneddon
The description SHOULD NOT contain relative URLs. When a relative URL is present, an aggregator MAY attempt to resolve it to a full URL using the channel's link as the base.
The first choice of base should instead be the item's <link>, if it has one. That's because that URL is the web page on which the item content appears, so a browser displaying that page will interpret links in the content relative to that page's URL. This happens quite frequently, particularly when an item refers to a per-item image or attachment, or links to another item in the same site.
If the item has no <link>, then the fallback is the channel's <link>.
The description of <lastBuildDate> and <pubDate> is too vague to be
useful, IMHO. In the former, the meaning of "updated" for the content
is unspecified, and in the latter, "the publication date and time"
explains the name but not what that date and time signifies.
The useful information here for an news-reader is a timestamp of the
feed, so it can short-circuit the rest of its processing if it sees
that the timestamp has not changed since the last time it read the
feed (this is a fallback for use with servers that generate feeds
dynamically and never send a 304 even if the feed hasn't changed.)
I believe that lastBuildDate generally represents this timestamp (but
don't quote me on it), meaning that if present, any change to the
content of the feed should cause it to be updated; while pubDate has
meaning to humans but not to the software.
—Jens
Since section 4.1.1.20.6 states that "A publisher SHOULD provide a
guid with each item" (amen to that), I'd like to make this more
visible by adding to the second paragraph of 4.1.1.20 ", and SHOULD
contain a guid".
IMHO, I wish the spec could say that an item MUST contain either a
guid or a link. Feeds whose items have no such identifiers are a pain
in the ass for news-readers, since it becomes almost impossible to
distinguish an edited item (i.e. to fix a typo) from a new item.
Along these lines, it may be worth spelling out that if a guid is
present (without isPermaLink=false) then the link element is
unnecessary since its value is assumed to be the same as the guid. If
the link is given (perhaps for the benefit of old parsers that don't
know about guid) it MUST be identical to the one in the guid element.
—Jens
PS: Is it wise to put quotes from copyrighted material ("Joe Bob Goes
To The Drive-In") in the spec?
On 4 Dec '06, at 11:42 AM, Geoffrey Sneddon wrote:
The definition of <author> fails to mention what format of email address to use. I assume it's a RFC 822 address, with the final comment after the address giving the person's name.
Actually, in the great majority of feeds that I've seen, this element doesn't contain an email address at all. Usually it's just the author's name, either a real name or a username in whatever blog/CMS software generated the feed, e.g.:
<author>Jens Alfke</author>
<author>snej</author>
I suspect this is partly to deter spammers from harvesting email addresses out of feeds.
But since an email address can be present, parsing the <author> element is a bit tricky. A good rule of thumb is that if the text contains an "@" it should be interpreted just like an address in an RFC822 message, else it should (as above) be interpreted as a purely human-readable name. Examples containing email addresses:
On 5 Dec 2006, at 06:00, Jens Alfke wrote:
> Actually, in the great majority of feeds that I've seen, this
> element doesn't contain an email address at all. Usually it's just
> the author's name, either a real name or a username in whatever
> blog/CMS software generated the feed, e.g.:
> <author>Jens Alfke</author>
> <author>snej</author>
> I suspect this is partly to deter spammers from harvesting email
> addresses out of feeds.
However, RSS 2.0.9 is just meant to clear up the existing spec - not
change it in anyway, so what is in common use is irrelevant.
>
> But since an email address can be present, parsing the <author>
> element is a bit tricky. A good rule of thumb is that if the text
> contains an "@" it should be interpreted just like an address in an
> RFC822 message, else it should (as above) be interpreted as a
> purely human-readable name. Examples containing email addresses:
> <author>Jens Alfke <jens@...></author>
> <author>jens@... (Jens Alfke)</author>
> <author>jens@...</author>
What I do is check if it's a valid RFC 822 email address (something
like, in that example, "(Jens Alfke)", is a valid RFC 822 comment),
if so take final comment and count that as the name, and count the
rest as the email address (meaning that anything else must support
RFC 822 correctly, as it may contain further comments); otherwise, I
just treat it as a name.
- Geoffrey Sneddon
Jens,
Great points in both of your posts. As per, link element is
unnecessary when isPermaLink is false, I disagree. In a link feed,
it's common to put a link element to the webpage you are pointing at
and a guid element to the webpage on your blogsite.
Thanks,
Randy Charles Morin
http://www.kbcafe.com/rss/
--- In rss-public@yahoogroups.com, Jens Alfke <jens@...> wrote:
>
> Since section 4.1.1.20.6 states that "A publisher SHOULD provide a
> guid with each item" (amen to that), I'd like to make this more
> visible by adding to the second paragraph of 4.1.1.20 ", and SHOULD
> contain a guid".
>
> IMHO, I wish the spec could say that an item MUST contain either a
> guid or a link. Feeds whose items have no such identifiers are a pain
> in the ass for news-readers, since it becomes almost impossible to
> distinguish an edited item (i.e. to fix a typo) from a new item.
>
> Along these lines, it may be worth spelling out that if a guid is
> present (without isPermaLink=false) then the link element is
> unnecessary since its value is assumed to be the same as the guid. If
> the link is given (perhaps for the benefit of old parsers that don't
> know about guid) it MUST be identical to the one in the guid element.
>
> �Jens
>
> PS: Is it wise to put quotes from copyrighted material ("Joe Bob Goes
> To The Drive-In") in the spec?
>
In an under-the-hood article [1] on how Microsoft's RSS Engine handles
enclosures in Internet Explorer 7 and Windows Vista, program manager
Walter vonKoch offers an important note: If you're publishing a
podcast or another enclosure larger than 15MB, Microsoft only will
download it from servers that support partial downloads with the HTTP
Range header [2].
1: http://masl.to/?M63E2215E
2: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
--- In rss-public@yahoogroups.com, Geoffrey Sneddon <foolistbar@...>
wrote:
> However, RSS 2.0.9 is just meant to clear up the existing spec - not
> change it in anyway, so what is in common use is irrelevant.
The RSS Profile (still under draft) recommends that publishers use
dc:creator when they don't want to expose e-mail addresses in a feed:
http://www.rssboard.org/rss-profile#namespace-elements-dublin
Jens Alfke wrote:
> The first choice of base should instead be the item's <link>, if
> it has one. That's because that URL is the web page on which the
> item content appears, so a browser displaying that page will
> interpret links in the content relative to that page's URL.
The problem with using the item link is that feedburner replaces it - any
feedburner feeds using relative links are guaranteed to break. Using the
channel link isn't perfect, but in general I think it has a better chance of
success. The real solution, of course, is for feed publishers to avoid
relative urls.
Regards
James
On 12/6/06, James Holderness <j4_james@...> wrote:
Jens Alfke wrote: > The first choice of base should instead be the item's <link>, if > it has one. That's because that URL is the web page on which the > item content appears, so a browser displaying that page will
> interpret links in the content relative to that page's URL.
The problem with using the item link is that feedburner replaces it - any feedburner feeds using relative links are guaranteed to break. Using the
channel link isn't perfect, but in general I think it has a better chance of success. The real solution, of course, is for feed publishers to avoid relative urls.
How about then only using the URL in the <link> if the domain of the <link> is the same as that of the <channel>.
AFAICT, enclosures can be anything [1][2]. But are aggregators actually expecting anything? My informal review of aggregators seems to indicate they're expecting audio/video mime types and not much else.
The IEEE's feed for Spectrum (http://spectrum.ieee.org/rss ) uses enclosures, but they're enclosing a gif file that is apparently supposed to be a thumbnail icon for the article:
[item] [title]Tech Start-ups Spurn NASDAQ for London[/title] [link]http://spectrum.ieee.org/dec06/4752[/link] [description]Alternative Investment Market offers faster funds and simpler rules[/description] [pubDate]Fri, 01 Dec 2006 00:00:00 EST[/pubDate] [enclosure url="http://spectrum.ieee.org/images/dec06/images/featurearticles/nnas.gif" length="1" type="image/gif"/] [/item]
None of these aggregators displayed the enclosed image; they all provided only a link (if anything):
Google Reader, NewsGator Online, SharpReader, ThinReader
So my question is: Is this desirable? (I think not.) If not, then it seems that the RSS Profile section on enclosures [3] should address the problem in one of a couple ways:
a) Discourage the use of enclosed images that are intended for display, or
b) Encourage aggregators to explicitly handle them, just as video and audio mime types are often explicitly handled with embedded media players, etc.
From <http://www.rssboard.org/rss-profile#conventions>:
> This profile relies primarily on tests conducted with the
> aggregators Bloglines, BottomFeeder, FeedDemon, Microsoft Internet
> Explorer 7, Mozilla Firefox 1.5, My Yahoo, NewsGator Online and
> Opera 9.
>
> The test of date-time values was conducted on the aggregators
> Blogbridge, Bloglines, BottomFeeder, FeedDemon, FeedReader,
> FeederReader, Google Reader, GreatNews, Internet Explorer 7,
> JetBrains Omea, Mozilla Thunderbird, Newsgator Online, NewzCrawler,
> Pluck, RSSBandit, RSSOwl, Sharpreader and Snarfer.
>
You* should be testing against Apple's Safari browser as well. It's
the third most-used web browser on the Internet with a 4% market share:
http://marketshare.hitslink.com/report.aspx?qprid=0http://arstechnica.com/news.ars/post/20061010-7949.html
It's also the default newsreader for the Mac platform. (Disclaimer: I
work on Apple's Syndication framework which provides the RSS
functionality in Safari.)
It would also be good to see some 3rd party Mac news reader apps in
the list; NetNewsWire is the most popular. (Actually, I've never even
heard of half the aggregators in the second list -- are they Windows-
only?)
—Jens
* Who conducted the testing? If there's a test suite available
somewhere, I'd be glad to run it against Safari and NetNewsWire and
report the results.
The problem with using the item link is that feedburner replaces it - any
feedburner feeds using relative links are guaranteed to break.
Are you sure? I wasn't aware of this, and I took a look at one of the feeds I'm subscribed to via FeedBurner — 37signals' blog — and it doesn't have altered links:
As per, link element is unnecessary when isPermaLink is false, I disagree. In a link feed, it's common to put a link element to the webpage you are pointing at and a guid element to the webpage on your blogsite.
Actually I said the opposite — the link element is unnecessary unless isPermalink=false. Or to avoid double negatives, the link element is unnecessary [redundant] when the guid element is a permalink.
Sorry, my mistake. And I meant the opposite of what I said.
As per, link element is unnecessary when isPermaLink is true, I
disagree. In a link feed, it's common to put a link element to the
webpage you are pointing at and a guid element to the webpage on your
blogsite.
Sorry,
Randy Charles Morin
http://www.kbcafe.com/rss/
--- In rss-public@yahoogroups.com, Jens Alfke <jens@...> wrote:
>
>
> On 6 Dec '06, at 9:09 AM, Randy Morin wrote:
>
> > As per, link element is
> > unnecessary when isPermaLink is false, I disagree. In a link feed,
> > it's common to put a link element to the webpage you are pointing at
> > and a guid element to the webpage on your blogsite.
>
> Actually I said the opposite � the link element is unnecessary unless
> isPermalink=false. Or to avoid double negatives, the link element is
> unnecessary [redundant] when the guid element is a permalink.
>
> �Jens
>
I know of at least one aggregator that displays enclosed images
inline once downloaded -- Egress for Windows Mobile.
That said, the only feed I'm subscribed to that encloses images in
this way, also includes them in an <img> tag in the description, so
oddly enough Egress likes to download the image twice -- once for the
item viewer, and again as an enclosure -- so I have enclosures turned
off for this feed. :-\
Egress is here: http://garishkernels.net/egress.shtml
I'm not affiliated -- I just like the app, despite a few minor
idiosyncrasies.
-Jake
On Dec 7, 2006, at 1:42 PM, Mark Woodman wrote:
> AFAICT, enclosures can be anything [1][2]. But are aggregators
> actually expectinganything? My informal review of aggregators
> seems to indicate they're expecting audio/video mime types and not
> much else.
>
> The IEEE's feed for Spectrum (http://spectrum.ieee.org/rss ) uses
> enclosures, but they're enclosing a gif file that is apparently
> supposed to be a thumbnail icon for the article:
>
> [item]
> [title]Tech Start-ups Spurn NASDAQ for London[/title]
> [link]http://spectrum.ieee.org/dec06/4752[/link]
> [description]Alternative Investment Market offers faster funds and
> simpler rules[/description]
> [pubDate]Fri, 01 Dec 2006 00:00:00 EST[/pubDate]
> [enclosure url="http://spectrum.ieee.org/images/dec06/images/
> featurearticles/nnas.gif" length="1" type="image/gif"/]
> [/item]None of these aggregators displayed the enclosed image; they
> all provided only a link (if anything):
> Google Reader, NewsGator Online, SharpReader, ThinReader
> So my question is: Is this desirable? (I think not.) If not,
> then it seems that the RSS Profile section on enclosures [3] should
> address the problem in one of a couple ways:
>
> a) Discourage the use of enclosed images that are intended for
> display, or
>
> b) Encourage aggregators to explicitly handle them, just as video
> and audio mime types are often explicitly handled with embedded
> media players, etc.
>
> References:
>
> [1] http://www.rssboard.org/rss-
> specification#ltenclosuregtSubelementOfLtitemgt
> [2] http://www.thetwowayweb.com/payloadsforrss
> [3] http://www.rssboard.org/rss-profile#element-channel-item-enclosure
>
>
>
On 7 Dec 2006, at 02:34, rcade wrote:
> The RSS Profile (still under draft) recommends that publishers use
> dc:creator when they don't want to expose e-mail addresses in a feed:
>
> http://www.rssboard.org/rss-profile#namespace-elements-dublin
Surely we should have some formal definition of what the importance
of the final comment in <author> is, though?
- Geoffrey Sneddon
--- In rss-public@yahoogroups.com, Jens Alfke <jens@...> wrote:
>
>
> On 6 Dec '06, at 9:54 PM, James Holderness wrote:
>
> > The problem with using the item link is that feedburner replaces it
> > - any
> > feedburner feeds using relative links are guaranteed to break.
>
> Are you sure?
Heh. As usual, a small number of customers cause most of the support
costs. ;)
<http://www.burningdoor.com/eric/archives/001209.html>
Guess which sorts of feeds get their links rewritten?
-Rob
--- In rss-public@yahoogroups.com, Jens Alfke <jens@...> wrote:
>
> Along these lines, it may be worth spelling out that if a guid is
> present (without isPermaLink=false) then the link element is
> unnecessary since its value is assumed to be the same as the guid. If
> the link is given (perhaps for the benefit of old parsers that don't
> know about guid) it MUST be identical to the one in the guid element.
Hmm. It might be more productive to specify which element consumers
should prefer if they are looking for a link.
I think the spec should instruct consumers to prefer the <link>
element. The attribute defaulting is beyond confusing, and
<guid>D6C514F0-A2AB-451A-99CE-A370A49F3F6E</guid>
and the like can be interpreted as a relative reference. Using <link>
instead of <guid>, even in the presence of an isPermaLink="true"
attribute, has prevented at least one bug I'm aware of
https://bugzilla.mozilla.org/show_bug.cgi?id=356321
* robertsayre2000 <sayrer@...> [2006-12-09 01:05]:
> I think the spec should instruct consumers to prefer the <link>
> element. The attribute defaulting is beyond confusing, and
>
> <guid>D6C514F0-A2AB-451A-99CE-A370A49F3F6E</guid>
>
> and the like can be interpreted as a relative reference. Using
> <link> instead of <guid>, even in the presence of an
> isPermaLink="true" attribute, has prevented at least one bug
> I'm aware of
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=356321
How about instructing consumers to use the guid only if the
attribute is explicitly set to "true", and encouraging producers
to always include an explicit attribute rather than relying on
the defaultification?
Regards,
--
Aristotle Pagaltzis // <http://plasmasturm.org/>
A. Pagaltzis wrote:
>* robertsayre2000 <sayrer@...> [2006-12-09 01:05]:
>> I think the spec should instruct consumers to prefer the <link>
>> element. The attribute defaulting is beyond confusing, and
>>
>> <guid>D6C514F0-A2AB-451A-99CE-A370A49F3F6E</guid>
>>
>> and the like can be interpreted as a relative reference. Using
>> <link> instead of <guid>, even in the presence of an
>> isPermaLink="true" attribute, has prevented at least one bug
>> I'm aware of
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=356321
>
> How about instructing consumers to use the guid only if the
> attribute is explicitly set to "true", and encouraging producers
> to always include an explicit attribute rather than relying on
> the defaultification?
Personally I'd rather just check for a fixed set of schemes (either
blacklist things like tag and uuid. or whitelist things like http, https,
ftp). A lack of scheme is always considered a failure since relative urls
aren't allowed. I don't have the data to back it up, but my guess is that
something like that has the best chance of working with existing feeds.
Any solution that requires the feed producers to make a change seems
impractical to me. If they were capable of following directions we wouldn't
be in this situation in the first place.
Regards
James
--- In rss-public@yahoogroups.com, Geoffrey Sneddon <foolistbar@...>
wrote:
> Surely we should have some formal definition of what the importance
> of the final comment in <author> is, though?
You're right that the definition needs to be reviewed. I've always
assumed that any e-mail address format is OK in RSS 2.0, but I need to
reread the definitions of that element in old versions.
Before I do, there seem like three possibilities:
1. Only the form "username@..." is OK.
2. Only the forms "username@..." and "username@... (real
name)" are OK.
3. Any email address that can be used in email as a From: header is OK.
If people here favor 1 or 3, I'd like to hear your reasoning, since
the current spec only uses "username@... (real name)" in examples.
Send Email
