On Thu, 20 Sep 2001 feit@... wrote:

> this is only one link out of 40 or 50 in the test data. If the
> proxies aren't the problem, then how do I know that once we
> have thousands of links to manage that we won't have dozens
> of such mistakes, which becomes very very annoying for a human
> operator to check...

You will always have erroneous or otherwise unexpected responses if
you use real URLs. There are a lot of odd servers out there. There is
no way around it. Using something like HEAD instead of GET will only
increase the number of servers not being able to respond correctly.
Do the best you can but accept some casualties.

In most cases, if your script works with thousands of links that you
do not control, then you do not care about a few wrong results. If you
control the sites your URLs point to, you should be able to have
more-or-less expected results after all bugs are sorted out.

$0.02,

Alex.