Hi,

I know this has been discussed here before, but I still don't quite
understand...  :-(

I'm running Webalizer 2.01 with Apache2 on a FreeBSD 5.3 box. This
box is the webserver for about a half dozen domains (each with it's
own access/error logs) and I have a separate webalizer script for
each of them.

In the past (before I started using Webalizer several months ago) I
never really worried about rotating Apache logs - when they started
getting  pretty big I'ld just stop Apache, delete them then restart
Apache. I don't think I can do that now that I  am using Webalizer
and still maintain my Webalizer historical data (or can I?).

I read on the Webalizer web site (or in the README) about
incremental webalizer processing. Is this something I can implement
on an already running webalizer setup or would I have to recompile
and start over to do that?

Also, what's the best way to rotate Apache logs on a FreeBSD box?
There's a file in /etc called newsyslog.conf that rotates the system
logs, but I don't think I can do the starting/stopping of Apache
using it. Apache has something called rotatelogs, but it says that
that is for "piped logs" and I really don't know what they mean by
that.

If someone can do it, I'ld sure appreciate info on the best way to
rotate Apache logs without losing Webalizer data.

Thanks,

Lisa

> In the past (before I started using Webalizer several months ago) I
> never really worried about rotating Apache logs - when they started
> getting  pretty big I'ld just stop Apache, delete them then restart
> Apache. I don't think I can do that now that I  am using Webalizer
> and still maintain my Webalizer historical data (or can I?).

Sure you can.. just add an extra step..

Stop apache
Process the web server logs with webalizer
Delete the logs
Restart apache

> I read on the Webalizer web site (or in the README) about
> incremental webalizer processing. Is this something I can implement
> on an already running webalizer setup or would I have to recompile
> and start over to do that?

Just add 'Incremental yes' to your config file or use the '-p' command
line option.  Note: if you do the above procedure at the end of the
month (ie: stop the web server at midnight on the last day of the
month), then you don't need to worry about using the incremental
option.. just do the above and you will be fine.

> Also, what's the best way to rotate Apache logs on a FreeBSD box?

The 'best' way is different for different people and different setups.
Whatever works for you is generally the 'best' way for your server.

> There's a file in /etc called newsyslog.conf that rotates the system
> logs, but I don't think I can do the starting/stopping of Apache
> using it. Apache has something called rotatelogs, but it says that
> that is for "piped logs" and I really don't know what they mean by
> that.
>
> If someone can do it, I'ld sure appreciate info on the best way to
> rotate Apache logs without losing Webalizer data.

As long as you feed all available log records through the webalizer,
you won't lose data.  Be that by the above procedure, by rotating
and then processing, whatever.. (a sample rotate script can be found
in the README file).  Also, most people like to keep at least a month
or more of historic logs though, just in case.  I generally rotate the
log at the end of the month, process it via the webalizer, then compress
and archive it, keeping the last 12 months of logs on the system.  Space
limitations may prevent keeping this may archived logs, so it's really
up to each individual installation as to how they handle them.

--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

> I keep getting a Forbidden error when i try to load n web browser.

As Enric has already stated, that's a problem with your web server, not
the webalizer.  If you get a forbidden error, then either the web server
does not have permissions to get to and display the stats directory, the
web server is configured to prevent access to that directory, or you have
some type of access restriction in place to prevent display.  Keep in mind
that the web server generally runs as a different user than what you log
in as, and must have permissions to enter and read the files/directories
along the whole path (directory tree) to the directory, not just the
directory itself.

> When I check the output dir everything is showning up correctly and I
> have the right permissions on the folders/files am I missing something
> else?

Yup, as otherwise you wouldn't be having the problem :)

--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

Hi Bradford,

> Sure you can.. just add an extra step..
>
> Stop apache
> Process the web server logs with webalizer
> Delete the logs
> Restart apache

Thanks. That's what I needed to know. I set up a cron job to stop
apache, process the log files, delete them then restart apache. Works
great!

Thanks,

Lisa

Virtualmin is a Webmin module to admin virtual servers.
I have imported my main domain.
Then webalizer only display the last four weeks stats.
I found webalizer.current (11Mb) I don't know if it contents previous
months info.  If yes, how can I get webalizer to display complete
information including previous months?
One line of this file contents (directory is not copied)
84-121-12-159.onocable.ono.com
0 11 11 113654 1 1142344042
Another one is
38.red-213-98-206.dynamicip.rima-tde.net
0 7 7 268546 1 1142183927
Thank you

Hi all,

I've noticed that Webalizer crashes whenever it sees a hit from
Googlebot-Mobile.

These look something like this:

66.249.72.232 - - [18/Mar/2006:10:42:26 +0000] "GET /robots.txt
HTTP/1.1" 404 208 "-" "Nokia6820/2.0 (4.83) Profile/MIDP-1.0
Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1;
+http://www.google.com/bot.html)"

This is extremely annoying.

Does anyone know a fix? Is there a patch for Webalizer which fixes this?

Mark

Are you using a MangleAgent of 1 by any chance?

--

On Tue, 21 Mar 2006, Mark Robson wrote:

> Hi all,
>
> I've noticed that Webalizer crashes whenever it sees a hit from
> Googlebot-Mobile.
>
> These look something like this:
>
> 66.249.72.232 - - [18/Mar/2006:10:42:26 +0000] "GET /robots.txt
> HTTP/1.1" 404 208 "-" "Nokia6820/2.0 (4.83) Profile/MIDP-1.0
> Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1;
> +http://www.google.com/bot.html)"
>
> This is extremely annoying.
>
> Does anyone know a fix? Is there a patch for Webalizer which fixes this?
>
> Mark
>
>
> Webalizer homepage: http://www.webalizer.org
>
> Yahoo! Groups Links
>
>
>
>
>
>
>
--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

Hi,

I want to make the URLs in the AllReferrers clickable - where do I need
to edit in the program source to change this?

Cheers :)

I found a huge list of GroupAgent lines that were, I guess, supposed
to make the Agents show up as, say, "Firefox" instead of "Mozilla/5.0
(Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111
Firefox/1.5.0.1" and the 1,001 other variations.  But they're all for
bizarro browsers I've never heard of.  I've been trying to understand
what and how groupAgent works, but I'm drawing a blank.  It doesn't
seem to be like a grep, where if a given value appears anywhere in the
log entry, then it's considered.  I tried:

GroupAgent     Firefox    Firefox

That doesn't work.  So... what would be the cleanest way to group
Firefoxes, Netscapes, Safaris, etc. that will work with the millions
of variations for each (X11 vs. Mac vs. Windows 5.0 vs. Windows 5.1
vs. Win 9x, and then all the languages, and then other variations)?
Or, if there's a way to explain just how GroupAgent works so I can
bang on it some more...

From the file on my signature, where you can also find a bizarro agent
list :) , read this:

# NOTE: You may also set "MangleAgents 4" and comment the list below so
webalizer
#           will kind of group them by itself. Using the list is slower
but has less false positives.

The GroupAgent line you put here won't do anything in a former line
already catches all Firefox entries. For example, in this example both
IE and Firefox will never be grouped, because they are already
"catched" by the Mozilla line (all IE and Firefox agents use the word
"Mozilla" somewhere on its name):

GroupAgent Mozilla  Mozilla
GroupAgent IE       Internet Explorer
GroupAgent Firefox  Firefox

I think that Webalizer currently provides no way to catch the language,
operative system, etc. Also, it won't accept spaces on the browser
name, so you can't try to use things like:

GroupAgent IE 6.0  Internet Explorer version 6.0

because webalizer breaks the browser name on the first space and this
line will catch all entries containing "IE", including "IE 6.0", "IE
5.0" and "thisisapieceofpie" (moreover, it stops when it finds pIEce
and never actually arrives to pIE).



--- John <joliver@...> wrote:

> I found a huge list of GroupAgent lines that were, I guess, supposed
> to make the Agents show up as, say, "Firefox" instead of "Mozilla/5.0
> (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111
> Firefox/1.5.0.1" and the 1,001 other variations.  But they're all for
> bizarro browsers I've never heard of.  I've been trying to understand
> what and how groupAgent works, but I'm drawing a blank.  It doesn't
> seem to be like a grep, where if a given value appears anywhere in
> the
> log entry, then it's considered.  I tried:
>
> GroupAgent     Firefox    Firefox
>
> That doesn't work.  So... what would be the cleanest way to group
> Firefoxes, Netscapes, Safaris, etc. that will work with the millions
> of variations for each (X11 vs. Mac vs. Windows 5.0 vs. Windows 5.1
> vs. Win 9x, and then all the languages, and then other variations)?
> Or, if there's a way to explain just how GroupAgent works so I can
> bang on it some more...
>
>
>
>
>


Enric Naval
Estudiante de Informática de Gestión en la Udl (Lleida)
GRIHO webalizer.conf
http://griho.udl.es/webalizer/webalizer.conf.txt

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

I have Webalizer Version 2.01 installed and running
fine but one of my websites stopped getting stats in
August 2005. Why is this? This site is in the
subdomain stat side on a shared host server.

Please help thanks

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

> I have Webalizer Version 2.01 installed and running
> fine but one of my websites stopped getting stats in
> August 2005. Why is this? This site is in the
> subdomain stat side on a shared host server.

http://www.webalizer.org/faq.html   (See #19)

--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

Nope does not apply. All my other subdomain stats are
fine just this one is not. How do I fix it so it
starts again for this one sub domain?

Thanks

--- "Bradford L. Barrett" <brad@...> wrote:

>
> > I have Webalizer Version 2.01 installed and
> running
> > fine but one of my websites stopped getting stats
> in
> > August 2005. Why is this? This site is in the
> > subdomain stat side on a shared host server.
>
> http://www.webalizer.org/faq.html   (See #19)
>
> --
> Bradford L. Barrett
> brad@...
> A free electron in a sea of neutrons     DoD#1750
> KD4NAW
>
> The only thing Micro$oft has done for society, is
> make people
> believe that computers are inherently unreliable.
>


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

> Nope does not apply.

It sure does!

> All my other subdomain stats are
> fine just this one is not.

Makes no difference.. each domain is run seperate, and there is no
relationship between them unless you have some weird setup.

> How do I fix it so it
> starts again for this one sub domain?

Read the FAQ entry.. can't fix something without any information.
Just saying it 'stopped' doesn't provide much.  Think of it this way,
suppose I call you up and say 'my car stopped, how do I fix it'..
what would your response be?

> --- "Bradford L. Barrett" <brad@...> wrote:
>
> >
> > > I have Webalizer Version 2.01 installed and
> > running
> > > fine but one of my websites stopped getting stats
> > in
> > > August 2005. Why is this? This site is in the
> > > subdomain stat side on a shared host server.
> >
> > http://www.webalizer.org/faq.html   (See #19)
> >
> > --
> > Bradford L. Barrett
> > brad@...
> > A free electron in a sea of neutrons     DoD#1750
> > KD4NAW
> >
> > The only thing Micro$oft has done for society, is
> > make people
> > believe that computers are inherently unreliable.
> >
--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

Those that can, do...  Those that can't... Run Windows!!

Webalizer shows about 3 times as many visits per day to my site as
Google Analytics (both are set to their default 30 minute settings).
Can anyone explain?

Any stats based on inserting javascript or other elements are not as
accurate as log based analysis, which might account for a lot of it.
Also ensure that your PageType setting is set properly for your site.
You should also realize that 'visits' is a metric that cannot be 100%
accurately calculated, even by log analysis (see the documentation for
a description, or the 'simpletons guide to web analysis').


--

On Sat, 15 Apr 2006, djs230548 wrote:

> Webalizer shows about 3 times as many visits per day to my site as
> Google Analytics (both are set to their default 30 minute settings).
> Can anyone explain?
>
--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

--- In webalizer@yahoogroups.com, "richard_laxton"
<richard_laxton@...> wrote:
>
> Hi,
>
> I want to make the URLs in the AllReferrers clickable - where do I
need
> to edit in the program source to change this?
>
> Cheers :)
>

After a lot of searching through code (damn I wish I actually knew C)
I've found the pertinent bit and here it is:

Edit output.c and replace the entire ALL_REFS_PAGE section on line
1405 through to approx 1465 with the following. This will make your
page one line shorter than it was!

Hope its useful to someone :o)

/*********************************************/
/* ALL_REFS_PAGE - HTML page of all refs     */
/*********************************************/

int all_refs_page(u_long r_reg, u_long r_grp)
{
    RNODEPTR rptr, *pointer;
    char     ref_fname[256], buffer[256];
    FILE     *out_fp;
    int      i=(r_grp)?1:0;

    /* generate file name */
    sprintf(ref_fname,"ref_%04d%02d.%s",cur_year,cur_month,html_ext);

    /* open file */
    if ( (out_fp=open_out_file(ref_fname))==NULL ) return 0;

    sprintf(buffer,"%s %d - %s",l_month[cur_month-
1],cur_year,msg_h_ref);
    write_html_head(buffer, out_fp);

    fprintf(out_fp,"<FONT SIZE=\"-1\"></CENTER><TABLE BORDER=\"0\"
CELLSPACING=\"0\" CELLPADDING=\"5\" STYLE=\"font-size:x-small;\">\n");

    fprintf(out_fp,"<TR><TD>%12s</TD><TD> </TD><TD>%
s</TD></TR>\n",msg_h_hits,msg_h_ref);

    /* do groups first (if any) */
    pointer=r_array;
    while(r_grp)
    {
       rptr=*pointer++;
       if (rptr->flag == OBJ_GRP)
       {
          fprintf(out_fp,"<TR><TD ALIGN=\"left\">%-8lu</TD><TD
ALIGN=\"right\">%6.02f%%</TD><TD><A HREF=\"%s\" TARGET=\"_blank\">%
s</A></TD></TR>\n",
             rptr->count,
             (t_hit==0)?0:((float)rptr->count/t_hit)*100.0,rptr-
>string,rptr->string);
          r_grp--;
       }
    }

    if (i) fprintf(out_fp,"\n");

    pointer=r_array;
    while(r_reg)
    {
       rptr=*pointer++;
       if (rptr->flag == OBJ_REG)
       {
          fprintf(out_fp,"<TR><TD ALIGN=\"left\">%-8lu</TD><TD
ALIGN=\"right\">%6.02f%%</TD><TD><A HREF=\"%s\" TARGET=\"_blank\">%
s</A></TD></TR>\n",
             rptr->count,
             (t_hit==0)?0:((float)rptr->count/t_hit)*100.0,rptr-
>string,rptr->string);
          r_reg--;
       }
    }

    fprintf(out_fp,"</TABLE></FONT>\n");
    write_html_tail(out_fp);
    fclose(out_fp);
    return 1;
}

Dear all

Webalizer is resetting value running webalizer 2.01-10
for incremental processing here is changes i have done
in conf file .


Incremental     yes
IncrementalName
/opt/scripts/ht0/webalizer.current

Executing webalizer as follows

./webalizer -p c /opt/scripts/conf/webalizer.conf

Any idea ?

Regards
JH

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Jobs in UK/London/Asia/USA. 

Hot Recruitment , All Categories.
 
Open this : 
 
http://www.hpgroups.com/jobs.asp
 
 
and  Apply Online Today.

> Is there a way to delete a few days' of data from the
> Webalizer.current file so i can re-run all the logs from that day
> till today?

No.

>  or do i have to just delete the file and run the whole month?

Yes.

> Also, if i have to run the whole month, what is the easiest way to do
> that? i have 2 files per day (i rotate at midnight and noon) so they
> are in chronological order, but i wasn't sure if i can tell Webalizer
> to read them all like that or not.

If they are named correctly, you can run the whole batch with something
like the following:

for i in <match_string>; do webalizer $i; done

Replace <match_string> with whatever matches the log files you wish
to process.

--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

Hello again!

I would like to hide/suppress the reporting of a few certain
directories (/images/*, cgi-bin/* and /webalizer/*... I think that
this is correct!) so that they are not included in the "Total URLs"
reports. Is this possible and, if so, how do I do this? I've read and
re-read comments on this discussion board, as well as the read me
docs.... but I'm obviously missing something, because these
directories are still showing up!

Any insight/instruction would be greatly appreciated!

Kind regards,
Jeff

i was thinking if is possible to replace db1 interface
with something more portable (to other databases) like
dbi for c
http://freshmeat.net/projects/libdbi/



__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Hello, Marius:

This dbi project looks a little green yet. It doesn't still support nor
mSQL nor Oracle, and it is in beta state. db1 is very old, mature,
tested, dependable and stable. dbi probably needs a lot more of mileage
to get most bugs sorted. It may even turn out to be a failed system,
and get abandoned in a few years.

Also, I think that the db1 interface is already installed by default on
all linux distributions (and it's also present in SunOs 5.7., I don't
know about other Unix, BSD, etc.), while dbi probably is not, and
should have to be installed.

So, when installing webalizer, you would need to install also dbi. This
may be a problem, because some operative system distributions already
include webalizer and they could refuse to include the new version
because they don't want to install dbi for some reason.

Also, such a change would surely require a lot of changes to the code.
You should give very good reasons to justify it.

Now, as an apart, and as a general rule of thumb, systems don't get
implemented just because they're "more portable" or even "better". You
need to give reasons that appeal to the developers, especially if the
system already works good enough. This is my experience, at least. I
didn't want to critizize you, I'm only pointing why your suggerence
will surely not be accepted, not only in webalizer, but in software
projects in general.



--- Marius Popa <jimi_compot@...> wrote:

> i was thinking if is possible to replace db1 interface
> with something more portable (to other databases) like
> dbi for c
> http://freshmeat.net/projects/libdbi/
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>


Enric Naval
Estudiante de Informática de Gestión en la Udl (Lleida)
GRIHO webalizer.conf
http://griho.udl.es/webalizer/webalizer.conf.txt

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

Well I think I've finally found some more information aboute the seg
faults on RHEL 3&4.  The ones that segfault never have a LogFile line
in the configuration file (I always specifiy one on the command line).
  The ones that don't segfault have this line, but it gets ignored
since I'm specifying it on the command line.

Does this help anyone figure out how to fix the problem?

--- In webalizer@yahoogroups.com, "Tom Cat" <stry_cat@...> wrote:
>
> Well I think I've finally found some more information aboute the seg
> faults on RHEL 3&4.  The ones that segfault never have a LogFile line
> in the configuration file (I always specifiy one on the command line).
>  The ones that don't segfault have this line, but it gets ignored
> since I'm specifying it on the command line.
>
> Does this help anyone figure out how to fix the problem?
>

Ok not good folks.  I've just found two cases that invalidated this
theory.  Going back and repeating the ones that seemed to support this
theory, I've found a number of them segfaults about 25% of the time
but works just fine about 75%.  Just my luck that I didn't get a seg
fault.  *sigh*
Any ideas?

Certain malformed user agent strings will cause a segfaut when you
use a MangleAgent level of one (1).  Has nothing to do with logfile
name or where you specify it.

--

On Mon, 8 May 2006, Tom Cat wrote:

> Well I think I've finally found some more information aboute the seg
> faults on RHEL 3&4.  The ones that segfault never have a LogFile line
> in the configuration file (I always specifiy one on the command line).
>  The ones that don't segfault have this line, but it gets ignored
> since I'm specifying it on the command line.
>
> Does this help anyone figure out how to fix the problem?
>
>
>
>
>
>
>
> Webalizer homepage: http://www.webalizer.org
>
> Yahoo! Groups Links
>
>
>
>
>
>
>
--
Bradford L. Barrett                      brad@...
A free electron in a sea of neutrons     DoD#1750 KD4NAW

The only thing Micro$oft has done for society, is make people
believe that computers are inherently unreliable.

--- In webalizer@yahoogroups.com, "Bradford L. Barrett" <brad@...> wrote:
>
>
> Certain malformed user agent strings will cause a segfaut when you
> use a MangleAgent level of one (1).  Has nothing to do with logfile
> name or where you specify it.
>

Thanks for the answer.  I tried setting MangleAgent to 0 and also to
5.  I'm still getting the segfault. :(

Thinking it could be something in the log, I decided to try it with
just one line in the log:
xxx.xxx.xxx.xxx - - [01/Apr/2006:00:22:30 -0500] "GET /agent/
HTTP/1.1" 200 22473

(I've replaced the real IP address with xxx.xxx.xxx.xxx).  Ran the
following line:

webalizer-2.01-10/webalizer -d -p -c
/export/home/stry_cat/w3logs/conf.test/agent/agent.conf -o
/export/home/stry_cat/w3logs/reports.test/agent -t "agent on " -n
mydomain.com /export/home/stry_cat/w3logs/data.test/agent/agentAPR06a.log

And still got the segfault.

I was about to post the conf file here too, when I noticed that I had
the HTMLHead command on a line all by itself without anything after
it.  I commented it out and ran my test again.  So far it hasn't
segfaulted in 3 tests.  I'm not going to get my hopes up and will post
again if that doesn't seem to fix the problem.



> --
>
> On Mon, 8 May 2006, Tom Cat wrote:
>
> > Well I think I've finally found some more information aboute the seg
> > faults on RHEL 3&4.  The ones that segfault never have a LogFile line
> > in the configuration file (I always specifiy one on the command line).
> >  The ones that don't segfault have this line, but it gets ignored
> > since I'm specifying it on the command line.
> >
> > Does this help anyone figure out how to fix the problem?
> >
> >
> >
> >
> >
> >
> >
> > Webalizer homepage: http://www.webalizer.org
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
> >
> >
> >
> --
> Bradford L. Barrett                      brad@...
> A free electron in a sea of neutrons     DoD#1750 KD4NAW
>
> The only thing Micro$oft has done for society, is make people
> believe that computers are inherently unreliable.
>