Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

AWStats

News Perl HTTP Logs Processing Scripts Recommended Links        
Log rotation Log analyzers  Perl Log Analysis and Management Tools Remote Syslog Tips Cheap Web hosting with SSH access Etc

AWStats is Perl script that can work in standlone (command line) and CGI modes and  generates web stite usage report as well as several other reports (can be used for proxy log processing and several other protocols like ftp and mail). 

It uses a partial information file to be able to process large log files, often and quickly.

Supported are all major servers that produce Apache-style log files (NCSA combined/XLF/ELF log format or common/CLF log format), as well as  WebStar, IIS (W3C log format) and several others. Proxy servers are also supported.

Take a look at this comparison table for an idea on features and differences between most famous statistics tools (AWStats, Analog, Webalizer,...).

AWStats is licensed under the GNU General Public License.

As AWStats works from the command line but also as a CGI, it can work with all web hosting providers which allow Perl, CGI and log access.

Recored conststs of section that can be generated also from the command line:

To see the analysis results, you have several options depending on your security policy.

Note: you must have created a statistics data base for the analysis period by processing your log files before you try to create reports.  See the previous section.

1. The first option is to build the main reports, in a static HTML page, from the command line, using the following syntax (skip to the second option if you only have CGI access):

 
perl awstats.pl -config=mysite -output -staticlinks > awstats.mysite.html

where mysite must be substituted with the domain/virtual host name you selected earlier during AWStats configuration. To create specific individual reports, specify the report name on the command line :
 
perl awstats.pl -config=mysite -output=alldomains -staticlinks > awstats.mysite.alldomains.html
perl awstats.pl -config=mysite -output=allhosts -staticlinks > awstats.mysite.allhosts.html
perl awstats.pl -config=mysite -output=lasthosts -staticlinks > awstats.mysite.lasthosts.html
perl awstats.pl -config=mysite -output=unknownip -staticlinks > awstats.mysite.unknownip.html
perl awstats.pl -config=mysite -output=alllogins -staticlinks > awstats.mysite.alllogins.html
perl awstats.pl -config=mysite -output=lastlogins -staticlinks > awstats.mysite.lastlogins.html
perl awstats.pl -config=mysite -output=allrobots -staticlinks > awstats.mysite.allrobots.html
perl awstats.pl -config=mysite -output=lastrobots -staticlinks > awstats.mysite.lastrobots.html
perl awstats.pl -config=mysite -output=urldetail -staticlinks > awstats.mysite.urldetail.html
perl awstats.pl -config=mysite -output=urlentry -staticlinks > awstats.mysite.urlentry.html
perl awstats.pl -config=mysite -output=urlexit -staticlinks > awstats.mysite.urlexit.html
perl awstats.pl -config=mysite -output=browserdetail -staticlinks > awstats.mysite.browserdetail.html
perl awstats.pl -config=mysite -output=osdetail -staticlinks > awstats.mysite.osdetail.html
perl awstats.pl -config=mysite -output=unknownbrowser -staticlinks > awstats.mysite.unknownbrowser.html
perl awstats.pl -config=mysite -output=unknownos -staticlinks > awstats.mysite.unknownos.html
perl awstats.pl -config=mysite -output=refererse -staticlinks > awstats.mysite.refererse.html
perl awstats.pl -config=mysite -output=refererpages -staticlinks > awstats.mysite.refererpages.html
perl awstats.pl -config=mysite -output=keyphrases -staticlinks > awstats.mysite.keyphrases.html
perl awstats.pl -config=mysite -output=keywords -staticlinks > awstats.mysite.keywords.html
perl awstats.pl -config=mysite -output=errors404 -staticlinks > awstats.mysite.errors404.html
 

When you access AWstats in CGI mode you can enter quieries against each section of the report, for example Navigation/Viewed/full list which contain important statistics and number of vilists for each page.

Query supports regular expressions.


Top Visited
Switchboard
Latest
Past week
Past month

Old News ;-)

How To Install, Secure, And Automate AWStats (CentOS/RHEL)

AWStats is a free and very powerful tool that generates advanced web, streaming, ftp or mail server statistics, graphically. It can analyze log files from all major server tools and convert them into nice graphical display. There are a lot of articles out there for AWStats, the reason I decided to write this one was to consolidate all the different tips and tricks I've learned through my journey into one comprehensive article associating the fantastic efforts of so many out there. There are a few methods of installing AWStats, this article will describe the simpler method of the two utilizing YUM.

This article assumes that you have root/sudo access to achieve all the goals required for a functional setup. This article also assumes you have SELinux set to permissive or disabled and that IPTables configuration is either disabled or customized for AWStats. Let's begin.

 

Install

Add RPMForge Yum Repository:

For x86 (32-bit) systems:

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

from Dag Wieers.

For x64 (64-bit) systems:

rpm -Uhv http://apt.sw.be/redhat/el5/en/x86_64/rpmforge/RPMS//rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm

from Dag Wieers.

 

Install & Configure Prerequisites

Install Apache:

yum install httpd

Configure Apache to start on boot:

/sbin/chkconfig --levels 345 httpd on

Configure iptables to allow Apache traffic:

/sbin/iptables -I INPUT -p tcp --dport 80 -j ACCEPT
/etc/init.d/iptables save
/etc/init.d/iptables restart

 

Install & Configure AWStats

Now that YUM has its additional repository we are ready to install. From the commandline type:

yum install awstats

Modify AWStats Apache Configuration:

Edit /etc/httpd/conf.d/awstats.conf (Note: When putting your conf file in the /etc/httpd/conf.d/ folder it's automatically loaded as part of the Apache configuration. There is no need to add it again into httpd.conf. This setup is usually for one of two reasons; A cleaner approach and separating of different applications in their own configuration files, or you are in a hosted environment that does not allow for direct editing of httpd.conf):

Alias /awstats/icon/ /var/www/awstats/icon/

ScriptAlias /awstats/ /var/www/awstats/
<Directory /var/www/awstats/>
        DirectoryIndex awstats.pl
        Options ExecCGI
        order deny,allow
        allow from all
</Directory>

Alias /awstatsclasses "/var/www/awstats/lib/"
Alias /awstats-icon/ "/var/www/awstats/icon/"
Alias /awstatscss "/var/www/awstats/examples/css"

Note: the mod_cgi module of Apache must be pre-loaded into Apache otherwise Apache will not try to view the file, it will try to execute it. This can be done in two ways, either enable for the entire web server, or utilizing VirtualHosts, enable for AWStats.

Edit the following lines in the default awstats configuration file /etc/awstats/awstats.localhost.localdomain.conf:

SiteDomain="<server name>.<domain>"
HostAliases="<any aliases for the server>"

Rename config file:

mv /etc/awstats/awstats.localhost.localdomain.conf /etc/awstats/awstats.<server name>.<domain>.conf

 

Update Statistics (Note: By default, statistics will be updated every hour.):

/usr/bin/awstats_updateall.pl now -confdir="/etc" -awstatsprog="/var/www/awstats/awstats.pl"

Start Apache:

/etc/init.d/httpd start

To automate startup of Apache on boot up, type

chkconfig --add httpd

 

Verify Install

Go to http://<server name>.<domain>/awstats/awstats.pl?config=<server name>.<domain>

 

Securing AWStats

Setting File System Permissions

The webserver needs only read-access to your files in order for you to be able to access AWStats from the browser. Limiting your own permissions will keep you from accidentally messing with files. Just remember that with this setup you will have to run perl to execute scripts rather than executing the scripts themselves.

$ find ./awstats -type d -exec chmod 701 '{}' \;
$ find ./awstats -not -type d -exec chmod 404 '{}' \;

Apache doesn't need direct access to AWStats configuration files therefore we can secure them tightly and not affect the relationship between them. To ensure that your .htaccess files are not readable via browser:

chmod 400 /etc/awstats/*.conf

 

Protecting The AWStats Directory With And Adding .htaccess

To secure the Awstats folder(s), is a measured process. Ensuring ownership of the awstats folder is owned by the user that needs access to it, creating an htpasswd.users file and adding the corresponding .htaccess file to authenticate against it. Let's first secure the awstats folder by typing the below from the command-line:

find ./awstats -type d -exec chmod 701 '{}' \;
find ./awstats -not -type d -exec chmod 404 '{}' \;

Now that our folders have been secured, we'll need to create the .htpasswd.users file. Go to the /etc/awstats folder and execute the following command:

htpasswd -c /etc/awstats/htpasswd.users user

(Select whatever username you'd like.)

It'll ask you to add a password for the user you've selected, add it and re-type it for confirmation and then save. The final step is to create an .htaccess file pointing to the .htpasswd file for authentication. Go to /var/www/awstats/ and create a new file called .htaccess using your favorite editor, typically nano or vi tend to be the more popular ones. In this example we'll use vi. From the command line type

vi .htaccess

An alternate method of creating an .htaccess file is using the Htaccess Password Generator. Add the following content to your newly created .htaccess file:

AuthName "STOP - Do not continue unless you are authorized to view this site! - Server Access"
AuthType Basic
AuthUserFile /etc/awstats/htpasswd.users
Require valid-user
htpasswd -c /etc/awstat/htpasswd.users awstats_online

Once done, secure the .htaccess file by typing:

chmod 404 awstats/.htaccess

A good and free log analysis tool - awstats

There are a lot of free http log file analysis tools out there that haven't been updated since the mid 90's, awstats however is both free, and up to date. It looks a bit like web trends (though I haven't used web trends in several years). Here's an online demo. awstats can be used on several web servers including IIS, and Apache. You can either have generate static html files, or run with a perl script in the cgi-bin.

Here's a quick rundown of setting it up on unix/apache

Each virtual web site you want to track stats for should have a file /etc/awstats.sitename.conf the directives for the configuration file can be found here: http://awstats.sourceforge.net/docs/awstats_config.html they also provide a default conf file in cgi-bin/awstats.model.conf you can use this as a base.

Make sure your log files are using NCSA combined format, this is usually done in apache by saying CustomLog /logs/access.log combined you can use other formats but you have to customize the conf file.

You will probably want to edit the LogFile directive to point to where your logfile is stored, SiteDomain this is the main domain for the site, HostAliases lets you put in other domains for the site, and the DirData directive lets you specify where the awstats databases will be stored (each site will have its own file in the directory).

Once that is setup you will want to update the database this is done from the command line by running

perl awstats.pl -config=sitename -update 

Now copy everything in the wwwroot folder to a web root, and visit http://sitename.com/cgi-bin/awstats.pl if you want to view other domains use /cgi-bin/awstats.pl?config=othersitename

Where sitename would be the name of your config file awstats.sitename.conf

If you want to generate static html files run the awstats_buildstaticpages.pl script found in the tools folder. You have to give it the path to the awstats.pl perl script, and a directory to put the static html files in.

perl awstats_buildstaticpages.pl -config=sitename -awstatsprog=/web/cgi-bin/awstats.pl 
  -dir=/web/stats/sitename/
More setup info can be found here: http://awstats.sourceforge.net/docs/index.html 

AWstats

AWStats - Free log file analyzer for advanced statistics (GNU GPL). -- extremely flexible and robust Perl=written log analyser. Industrial strength ! Highly recommended..

AWStats is a free powerful and featureful tool that generates advanced web, ftp or mail server statistics, graphically. This log analyzer works as a CGI or from command line and shows you all possible information your log contains, in few graphical web pages. It uses a partial information file to be able to process large log files, often and quickly. It can analyze log files from IIS (W3C log format), Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar and most of all web, proxy, wap, streaming servers, mail servers (and some ftp).

Take a look at this comparison table for an idea on differences between most famous statistics tools (AWStats, Analog, Webalizer,...).

AWStats is a free software distributed under the GNU General Public License. You can have a look at this license chart to know what you can/can't do.

As AWStats works from the command line but also as a CGI, it can work with major web hosting provider that allows CGI and log access.

You can browse AWStats demo (Real-time feature to update stats from web has been disabled on demos) to see a sample of most important information AWStats shows you...

Recommended Links

Softpanorama hot topic of the month

Softpanorama Recommended

AWStats - Wikipedia, the free encyclopedia

AWStats Documentation - Setup page

AWStats Documentation - Configuration directives and parameters

AWStats Installation - DreamHost



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: September, 13, 2011