hi tilman,

do you know why Wikipedia links of the kind:
http://en.wikipedia.org/wiki/Alfred_North_Whitehead

are not working for xenu?

i get a "forbidden request" message.

is it becaue the URL has a slightly uncommon format:

http://en.wikipedia.org/wiki/KEYWORD ?

tnx,

frank

I've long had this same experience with Wikipedia using other bots as well as
Xenu.  (returns "Wikimedia error") I have always assumed that Wikipedia blocks
anything that even slightly smells like a bot to keep their content from being
spidered.  Irresponsible bots are becoming so numerous that I'd like to do this
on my site as well but such blocking is more difficult with static pages.  I do
find that my list of sites seeming to block Xenu continually grows (partial list
here http://www.andilinks.com/linkckg.htm#55000).

Andi
   ----- Original Message -----
   From: frank visser
   To: xenu-usergroup@yahoogroups.com
   Sent: Thursday, July 07, 2005 7:04 AM
   Subject: [xenu-usergroup] Wikipedia links not working for Xenu?


   hi tilman,

   do you know why Wikipedia links of the kind:
   http://en.wikipedia.org/wiki/Alfred_North_Whitehead

   are not working for xenu?

   i get a "forbidden request" message.

   is it becaue the URL has a slightly uncommon format:

   http://en.wikipedia.org/wiki/KEYWORD ?

   tnx,

   frank




------------------------------------------------------------------------------
   YAHOO! GROUPS LINKS

     a..  Visit your group "xenu-usergroup" on the web.

     b..  To unsubscribe from this group, send an email to:
      xenu-usergroup-unsubscribe@yahoogroups.com

     c..  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


------------------------------------------------------------------------------



[Non-text portions of this message have been removed]

http://www.andilinks.com/linkckg.htm#55000
   ----- Original Message -----
   From: frank visser
   To: xenu-usergroup@yahoogroups.com
   Sent: Thursday, July 07, 2005 7:04 AM
   Subject: [xenu-usergroup] Wikipedia links not working for Xenu?


   hi tilman,

   do you know why Wikipedia links of the kind:
   http://en.wikipedia.org/wiki/Alfred_North_Whitehead

   are not working for xenu?

   i get a "forbidden request" message.

   is it becaue the URL has a slightly uncommon format:

   http://en.wikipedia.org/wiki/KEYWORD ?

   tnx,

   frank




------------------------------------------------------------------------------
   YAHOO! GROUPS LINKS

     a..  Visit your group "xenu-usergroup" on the web.

     b..  To unsubscribe from this group, send an email to:
      xenu-usergroup-unsubscribe@yahoogroups.com

     c..  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


------------------------------------------------------------------------------



[Non-text portions of this message have been removed]

On Thu, 07 Jul 2005 13:04:39 -0000, frank visser wrote:

>do you know why Wikipedia links of the kind:
>http://en.wikipedia.org/wiki/Alfred_North_Whitehead
>
>are not working for xenu?

http://home.snafu.de/tilman/xenulink.html

See Nr. 20

This is an excellent summary of Professor Whitehead. His work with Bertrand
Russel on the theory of classes was a breakthrough at that time. Both men are of
high intellect. I studied their writings at university and I am pleased that you
cited them.
Thank you.

Ken Lorimer.

Tilman Hausherr <tilman@...> wrote:
On Thu, 07 Jul 2005 13:04:39 -0000, frank visser wrote:

>do you know why Wikipedia links of the kind:
>http://en.wikipedia.org/wiki/Alfred_North_Whitehead
>
>are not working for xenu?

http://home.snafu.de/tilman/xenulink.html

See Nr. 20





---------------------------------
YAHOO! GROUPS LINKS


     Visit your group "xenu-usergroup" on the web.

     To unsubscribe from this group, send an email to:
  xenu-usergroup-unsubscribe@yahoogroups.com

     Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


---------------------------------




---------------------------------
  Start your day with Yahoo! - make it your home page

[Non-text portions of this message have been removed]

hi tilman,

can you elaborate a bit on the "Statistics for Managers" Xenu provides?

shouldn't the results under "Correct internal URLs, by MIME type" add
up to the same total number of URLs as the results under "All pages, by
result type"?

Apparently not all URLs are of the "correct internal URL" variety?

In the MIME type list I also see an "empty" category (no MIME type
listed, only URLs/Bytes/Percentages), so I would assume this covers the
URLs that don't have a correct internal URL -- so my expectation was
that the Total number of URLs should be the same in both tables.

tnx,

frank

On Mon, 25 Jul 2005 09:32:07 -0000, frank visser wrote:

>hi tilman,
>
>can you elaborate a bit on the "Statistics for Managers" Xenu provides?
>
>shouldn't the results under "Correct internal URLs, by MIME type" add
>up to the same total number of URLs as the results under "All pages, by
>result type"?

No.

>Apparently not all URLs are of the "correct internal URL" variety?

No. Some are incorrect, some are external.

>In the MIME type list I also see an "empty" category (no MIME type
>listed, only URLs/Bytes/Percentages), so I would assume this covers the
>URLs that don't have a correct internal URL -- so my expectation was
>that the Total number of URLs should be the same in both tables.

I suspect these are URLs where the server didn't return a proper MIME
type.

Tilman

>
>tnx,
>
>frank
>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>

Andi, I checked your page and noted that you say "There is one major
company URL (http://www.prudential.com/) that will recycle through
dozens of redirects for Xenu yet behaves normally for a browser so I
simply omit it as well."   Out of curiousity, I tried starting with
that URL with Xenu and it only went through 3 redirects before
actually going into the site.  The redirects appear to be the same as
those a browser gets since if you enter http://www.prudential.com/ in
browser, when the page comes up, the URL shows as
http://www.prudential.com/index/.

Investigating further, I determined that
http://www.prudential.com/
first redirects to
http://www.prudential.com/default/
that, in turn, redirects to
http://www.prudential.com/index
and that, again, redirects to
http://www.prudential.com/index/
and that returns a normal page.

Tom
--- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
> I've long had this same experience with Wikipedia using other bots
as well as Xenu.  (returns "Wikimedia error") I have always assumed
that Wikipedia blocks anything that even slightly smells like a bot
to keep their content from being spidered.  Irresponsible bots are
becoming so numerous that I'd like to do this on my site as well but
such blocking is more difficult with static pages.  I do find that my
list of sites seeming to block Xenu continually grows (partial list
here http://www.andilinks.com/linkckg.htm#55000).
>
> Andi

Thanks for this Tom.  When I wrote that (It has been over a year) and put
Prudential on the exeception list it was returning dozens of redirects to the
Xenu result page.  They must have fixed the bug in their rewrite rule (or
whatever, perhaps there's a change in the Xenu version) in the meantime.  I'll
revise the text on my Link checking page.  Thanks again.

Andi
   ----- Original Message -----
   From: adiscretemale
   To: xenu-usergroup@yahoogroups.com
   Sent: Monday, July 25, 2005 3:57 AM
   Subject: [xenu-usergroup] Re: Wikipedia links not working for Xenu?


   Andi, I checked your page and noted that you say "There is one major
   company URL (http://www.prudential.com/) that will recycle through
   dozens of redirects for Xenu yet behaves normally for a browser so I
   simply omit it as well."   Out of curiousity, I tried starting with
   that URL with Xenu and it only went through 3 redirects before
   actually going into the site.  The redirects appear to be the same as
   those a browser gets since if you enter http://www.prudential.com/ in
   browser, when the page comes up, the URL shows as
   http://www.prudential.com/index/.

   Investigating further, I determined that
   http://www.prudential.com/
   first redirects to
   http://www.prudential.com/default/
   that, in turn, redirects to
   http://www.prudential.com/index
   and that, again, redirects to
   http://www.prudential.com/index/
   and that returns a normal page.

   Tom
   --- In xenu-usergroup@yahoogroups.com, "andi" <andi@a...> wrote:
   > I've long had this same experience with Wikipedia using other bots
   as well as Xenu.  (returns "Wikimedia error") I have always assumed
   that Wikipedia blocks anything that even slightly smells like a bot
   to keep their content from being spidered.  Irresponsible bots are
   becoming so numerous that I'd like to do this on my site as well but
   such blocking is more difficult with static pages.  I do find that my
   list of sites seeming to block Xenu continually grows (partial list
   here http://www.andilinks.com/linkckg.htm#55000).
   >
   > Andi





------------------------------------------------------------------------------
   YAHOO! GROUPS LINKS

     a..  Visit your group "xenu-usergroup" on the web.

     b..  To unsubscribe from this group, send an email to:
      xenu-usergroup-unsubscribe@yahoogroups.com

     c..  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


------------------------------------------------------------------------------



[Non-text portions of this message have been removed]

Hi,
I recently got a broadband cable connection to the internet.
Xenu has stopped working. It reports url not found, even though
I had just manually checked the url and had emptied the cache of my
browser to make sure I downloaded the latest version of the site. Does
anyone know why this is happening?
Thanks,
Jack

On Wed, 27 Jul 2005 10:38:51 -0700, wydeboi wrote:

>Hi,
>I recently got a broadband cable connection to the internet.
>Xenu has stopped working. It reports url not found, even though
>I had just manually checked the url and had emptied the cache of my
>browser to make sure I downloaded the latest version of the site. Does
>anyone know why this is happening?

Can you access the URLs with IE ?

Yes.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 10:38:51 -0700, wydeboi wrote:
>
>
>
>>Hi,
>>I recently got a broadband cable connection to the internet.
>>Xenu has stopped working. It reports url not found, even though
>>I had just manually checked the url and had emptied the cache of my
>>browser to make sure I downloaded the latest version of the site. Does
>>anyone know why this is happening?
>>
>>
>
>Can you access the URLs with IE ?
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>
>

In that case, it means that the server "hates" Xenu, and lies by
returning a 404 (= not found) error.

Tilman

On Wed, 27 Jul 2005 11:00:43 -0700, wydeboi wrote:

>Yes.
>
>Tilman Hausherr wrote:
>
>>On Wed, 27 Jul 2005 10:38:51 -0700, wydeboi wrote:
>>
>>
>>
>>>Hi,
>>>I recently got a broadband cable connection to the internet.
>>>Xenu has stopped working. It reports url not found, even though
>>>I had just manually checked the url and had emptied the cache of my
>>>browser to make sure I downloaded the latest version of the site. Does
>>>anyone know why this is happening?
>>>
>>>
>>
>>Can you access the URLs with IE ?
>>
>>
>>
>>
>>Yahoo! Groups Links
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>

I quoted the incorrect error message, I just tried again and the error
returned is file not found.

Tilman Hausherr wrote:

>In that case, it means that the server "hates" Xenu, and lies by
>returning a 404 (= not found) error.
>
>Tilman
>
>On Wed, 27 Jul 2005 11:00:43 -0700, wydeboi wrote:
>
>
>
>>Yes.
>>
>>Tilman Hausherr wrote:
>>
>>
>>
>>>On Wed, 27 Jul 2005 10:38:51 -0700, wydeboi wrote:
>>>
>>>
>>>
>>>
>>>
>>>>Hi,
>>>>I recently got a broadband cable connection to the internet.
>>>>Xenu has stopped working. It reports url not found, even though
>>>>I had just manually checked the url and had emptied the cache of my
>>>>browser to make sure I downloaded the latest version of the site. Does
>>>>anyone know why this is happening?
>>>>
>>>>
>>>>
>>>>
>>>Can you access the URLs with IE ?
>>>
>>>
>>>
>>>
>>>Yahoo! Groups Links
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>Yahoo! Groups Links
>>
>>
>>
>>
>>
>>
>>
>>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 11:15:47 -0700, wydeboi wrote:

>I quoted the incorrect error message, I just tried again and the error
>returned is file not found.

Check your %temp% directory... delete all TGH*.* files. Make sure you
have write rights and enough space on your disk.

Tilman

I just went through all the files on my server and can find no %temp% or
any kind of temp directory or folder or file.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 11:15:47 -0700, wydeboi wrote:
>
>
>
>>I quoted the incorrect error message, I just tried again and the error
>>returned is file not found.
>>
>>
>
>Check your %temp% directory... delete all TGH*.* files. Make sure you
>have write rights and enough space on your disk.
>
>Tilman
>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 11:25:01 -0700, wydeboi wrote:

>I just went through all the files on my server and can find no %temp% or
>any kind of temp directory or folder or file.

Open the Windows Explorer. Then type '%', 't', 'e', 'm', 'p', '%' and
press ENTER. You'll have your temp directory.

I searched the entire c drive for a file or folder with %temp%
nothing found.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 11:25:01 -0700, wydeboi wrote:
>
>
>
>>I just went through all the files on my server and can find no %temp% or
>>any kind of temp directory or folder or file.
>>
>>
>
>Open the Windows Explorer. Then type '%', 't', 'e', 'm', 'p', '%' and
>press ENTER. You'll have your temp directory.
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 11:39:07 -0700, wydeboi wrote:

>I searched the entire c drive for a file or folder with %temp%
>nothing found.

%temp% is a placeholder. Don't search it, type it in the explorer. It
will expand to your actual temp directory. For example, my temp dir is

C:\Dokumente und Einstellungen\XXXX\Lokale Einstellungen\Temp

your explorer must be different from mine on xp, I have no place to type
in anything except in the search box. also, I just tried Xenu with
another site of mine different url and different server, same response,
file not found.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 11:39:07 -0700, wydeboi wrote:
>
>
>
>>I searched the entire c drive for a file or folder with %temp%
>>nothing found.
>>
>>
>
>%temp% is a placeholder. Don't search it, type it in the explorer. It
>will expand to your actual temp directory. For example, my temp dir is
>
>C:\Dokumente und Einstellungen\XXXX\Lokale Einstellungen\Temp
>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 11:47:24 -0700, wydeboi wrote:

>your explorer must be different from mine on xp, I have no place to type
>in anything except in the search box. also, I just tried Xenu with
>another site of mine different url and different server, same response,
>file not found.

Then try this:

Click on START, RUN

type
	 cmd.exe

type
	 cd %temp%
	 del tgh*.*


Alternatively - did you install a (non microsoft) firewall recently? Try
disabling it (install all MS patches first) and run Xenu again.

I have not installed a non-microsoft firewall, I do not use microsofts
firewall, I use zonealarm, for years and it has always worked with Xenu.
I'll try the command line.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 11:47:24 -0700, wydeboi wrote:
>
>
>
>>your explorer must be different from mine on xp, I have no place to type
>>in anything except in the search box. also, I just tried Xenu with
>>another site of mine different url and different server, same response,
>>file not found.
>>
>>
>
>Then try this:
>
>Click on START, RUN
>
>type
> cmd.exe
>
>type
> cd %temp%
> del tgh*.*
>
>
>Alternatively - did you install a (non microsoft) firewall recently? Try
>disabling it (install all MS patches first) and run Xenu again.
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

tried the command line, then ran Xenu, did not work, same error message,
file not found.

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 11:47:24 -0700, wydeboi wrote:
>
>
>
>>your explorer must be different from mine on xp, I have no place to type
>>in anything except in the search box. also, I just tried Xenu with
>>another site of mine different url and different server, same response,
>>file not found.
>>
>>
>
>Then try this:
>
>Click on START, RUN
>
>type
> cmd.exe
>
>type
> cd %temp%
> del tgh*.*
>
>
>Alternatively - did you install a (non microsoft) firewall recently? Try
>disabling it (install all MS patches first) and run Xenu again.
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 12:37:58 -0700, wydeboi wrote:

>tried the command line, then ran Xenu, did not work, same error message,
>file not found.

That's it :-( I have no more ideas (I assume you have the latest
version). If you find a way to let me access your PC through "remote
assistance" (your firewall may prevent this), e-mail me directly.

Tilman

I have version 1.2g, I'll look into remote access.
Thanks for all your help.
Jack

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 12:37:58 -0700, wydeboi wrote:
>
>
>
>>tried the command line, then ran Xenu, did not work, same error message,
>>file not found.
>>
>>
>
>That's it :-( I have no more ideas (I assume you have the latest
>version). If you find a way to let me access your PC through "remote
>assistance" (your firewall may prevent this), e-mail me directly.
>
>Tilman
>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

I work at NASA's Kennedy Space Center for the Joint Base Operations
Support Contractor, Space Gateway Support and have used Xenu for more
than two years now. Thank You, Tilman :)

I am having an issue checking links that are coded as follows:

http://www.ksc.nasa.gov/disclaimer/?target=http://external.web.site/

The purpose of this formatting is to display an intervening
disclaimer page that tells the user that the next page seen is not on
the KSC web site and therefore we are not responsible for the
content :) CYA at it's finest :)

Any ideas about how I could add the ability to check these links
without first stripping them of the disclaimer coding?

Thank You,

Thanks,
Richard Warnock
Space Gateway Support, Associate Software Test Analyst
richard.warnock-1@...

well, I went to your site and read through everything about the download
and use and lo and behold, Xenu will not work unless I have first
launched IE and typed in a url. I never had to do this before, so I did
that and Xenu is now spidering my site. Sorry for the trouble.
Thanks again,
Jack

Tilman Hausherr wrote:

>On Wed, 27 Jul 2005 12:37:58 -0700, wydeboi wrote:
>
>
>
>>tried the command line, then ran Xenu, did not work, same error message,
>>file not found.
>>
>>
>
>That's it :-( I have no more ideas (I assume you have the latest
>version). If you find a way to let me access your PC through "remote
>assistance" (your firewall may prevent this), e-mail me directly.
>
>Tilman
>
>
>
>
>
>
>Yahoo! Groups Links
>
>
>
>
>
>
>
>
>

On Wed, 27 Jul 2005 12:55:59 -0700, wydeboi wrote:

>well, I went to your site and read through everything about the download
>and use and lo and behold, Xenu will not work unless I have first
>launched IE and typed in a url. I never had to do this before, so I did
>that and Xenu is now spidering my site. Sorry for the trouble.

This suggests a setup problem with windows. Maybe you have two
WININET.DLL files, or your DLL isn't in your path... or another
important DLL isn't accessible...

Xenu works for me and I almost never use IE :-)

Tilman

hi richard,

i can answer that one for you.

there's no problem for xenu checking these URLs at all, if your
start URL is just http://www.ksc.nasa.gov/

the URL with the disclaimer code falls within this domain and will
be scanned.

i did it and the url http://external.web.site/ gets a "no such host"
message.

btw, your site also uses popups with the javascript:openNASAwindow()
function, and xenu can scan these as well if you use the regular
expression for this and add it to the .ini file.

i tested this on the NASA site and found some illegal code examples,
such as:

javascript:openNASAWindow('www.spaceflightamerica.org')

this won't work, for the URL needs to start with 'http://....'

since the regular expression looks for URLs starting with http, ftp
or /, this one was not resolved and showed up in the xenu report
under "skip type". that's how i look for illegally coded popups.

any further help needed, let me know!

frank




--- In xenu-usergroup@yahoogroups.com, "rlw3158" <rlw3158@y...>
wrote:
> I work at NASA's Kennedy Space Center for the Joint Base
Operations
> Support Contractor, Space Gateway Support and have used Xenu for
more
> than two years now. Thank You, Tilman :)
>
> I am having an issue checking links that are coded as follows:
>
> http://www.ksc.nasa.gov/disclaimer/?
target=http://external.web.site/
>
> The purpose of this formatting is to display an intervening
> disclaimer page that tells the user that the next page seen is not
on
> the KSC web site and therefore we are not responsible for the
> content :) CYA at it's finest :)
>
> Any ideas about how I could add the ability to check these links
> without first stripping them of the disclaimer coding?
>
> Thank You,
>
> Thanks,
> Richard Warnock
> Space Gateway Support, Associate Software Test Analyst
> richard.warnock-1@k...

hi richard,

on a closer look I see now that the root URL for the NASA site is

http://www.nasa.gov/

so in that case, you have to add this URL to the field "Consider
URLs starting with this as 'internal':

http://www.ksc.nasa.gov/

so Xenu will include this as well.

or alternatively, use the second as start URL, and include the first
as internal.

frank

--- In xenu-usergroup@yahoogroups.com, "frank visser"
<f.visser3@c...> wrote:
> hi richard,
>
> i can answer that one for you.
>
> there's no problem for xenu checking these URLs at all, if your
> start URL is just http://www.ksc.nasa.gov/
>
> the URL with the disclaimer code falls within this domain and will
> be scanned.
>
> i did it and the url http://external.web.site/ gets a "no such
host"
> message.
>
> btw, your site also uses popups with the javascript:openNASAwindow
()
> function, and xenu can scan these as well if you use the regular
> expression for this and add it to the .ini file.
>
> i tested this on the NASA site and found some illegal code
examples,
> such as:
>
> javascript:openNASAWindow('www.spaceflightamerica.org')
>
> this won't work, for the URL needs to start with 'http://....'
>
> since the regular expression looks for URLs starting with http,
ftp
> or /, this one was not resolved and showed up in the xenu report
> under "skip type". that's how i look for illegally coded popups.
>
> any further help needed, let me know!
>
> frank
>
>
>
>
> --- In xenu-usergroup@yahoogroups.com, "rlw3158" <rlw3158@y...>
> wrote:
> > I work at NASA's Kennedy Space Center for the Joint Base
> Operations
> > Support Contractor, Space Gateway Support and have used Xenu for
> more
> > than two years now. Thank You, Tilman :)
> >
> > I am having an issue checking links that are coded as follows:
> >
> > http://www.ksc.nasa.gov/disclaimer/?
> target=http://external.web.site/
> >
> > The purpose of this formatting is to display an intervening
> > disclaimer page that tells the user that the next page seen is
not
> on
> > the KSC web site and therefore we are not responsible for the
> > content :) CYA at it's finest :)
> >
> > Any ideas about how I could add the ability to check these links
> > without first stripping them of the disclaimer coding?
> >
> > Thank You,
> >
> > Thanks,
> > Richard Warnock
> > Space Gateway Support, Associate Software Test Analyst
> > richard.warnock-1@k...