Hosted and distributed by

Project Page

The Small Time Intranet Logger

Copyright © 2001 Kenneth Earle Fanyo

CVS Repository

Intranet Logger is a suite of programs designed to centralize the parsing and presentation of system logs generated by computers in an intranet. The log data is pushed to the logging server by each client machine. The logging server, in turn, maintains the information in an RDBMS and then responds to queries via a http daemon interfaced to the RDBMS.

How does it work?

The log data is pushed to the logging server using shell scripts and nfs. The data is then parsed at the server, formatted for database loading and loaded to the database. In its present state, this is done using Perl. The RDBMS is MySQL, the httpd is Apache and the two are interfaced via PHP.

What's so special about it?

Mainly the design of the database. It is built of table 'families' consisting of a 'main' table  and 'auxiliary' tables that corresponds to each field in the main table that is not already numeric or date/time in format. Unique text data strings are stored in the auxiliary tables and the index numbers associated with the strings are stored in the main. The result is a drastic reduction in the size of the database on disk and the means to almost eliminate endlessly repetitive text strings in the stored data. There is also the speed payoff that comes from all the data in the large tables being date/time or numeric.

Big Deal! What else?

I think--or hope--it will help create a launch platform for more interest and involvement by folks who would have not historically had much interest in system/network monitoring and administration. Networks are becoming as commonplace as desktop PC's were 10-15 years ago. If we are not going to be killed by endless checking and troubleshooting of individual machines, we need relatively painless ways to monitor network clients.

Blah, blah! Get to the point!

Presently handles all logs generated by TCP Wrappers, Apache, the output of 'last', 'lastlog', and 'w' as well as all logs generated by, or through, the syslog daemon.

It should handle any log data generated under virtually any operating system as long as:

A. The log data is in ASCII text.
B. The message data is at the end of the record, i.e., regardless of how many fields may come prior to the message and regardless of how long the message may be.
C. Lines after the first in multiline log entries begin with whitespace.
D. The clients deliver data to the server, i.e., log data is pushed to the logging server, not pulled from the clients.
E. The format of the log entries is consistent enough, in terms of which data is in which field, to make processing cost effective.
• It minimizes the disk space used by the database by using those table 'families' in the database as mentioned above.

• Does not depend on constant network connectivity to function successfully. IOWs, any client machine or the server itself can go down for almost an indefinite length of time and the logs will simply accumulate on their respective hosts and/or on the logging server until the connection is re-established. The only constraint here is available disk space.

For further reading...

• GNU General Public License
• GNU Free Documentation License
• Program flowcharts for 'acquire', 'parse', and 'dbms' are here.
• Extant program/system documentation is here.
• Sample runtime log/debug output is here.
• Of course, the best source of documentation is the source code. It is expressly intended to be as 'self documenting' as possible. Also, the best documentation is undoubtedly the comments in the source. The comments become much better after I came back to the code from a several month break and discovered I was completely lost. No kidding. Now, that's embarrassment, isn't it?

• announce
• devel
• users