Quest for web log analysis software

I am currently searching for a web log analysis package for our site. I have to say that the more I look at the available options the more disgusted I get. Basically what I am looking for is wel log analysis software with following features:

• Reading data from web server logs (not using custom javascript to record hits)
• Storing log data in a SQL database, so I can use SQL to generate custom reports
• Capable of generating custom reports with custom graphs and charts
• Capable of reading custom log formats (such as Apache LogFormat strings)
• Able to "drill down/zoom in" into the reports for more information
• Running on Linux, BSD or Solaris.
  

It seems that to get all of these is close to impossible.

Blogs are offensive

According to the report created by ScanSafe, 80% of all blogs contain "offensive" and/or "unwanted" content. I haven't read the report myself, but according to the post about it at Ars Technica, it is enough for a blog to have one instance of one of the "bad words" to be considered offensive. I suppose this is one of the rare cases where I prefer to stick with majority. Fuck, fuck, fuck.

Web statistics from the command line

There are a lot of web statistics packages out there. And some of them are good. To name a few, there is Analog (especially when paired with Report Magic), AWStats and Visitors. There are also excellent commercial packages (but they don't pay me to advertise :) ).  Most of these have one particular problem. They generate a number of static reports. So if you just want to see how many hits your site received per day during last week they are excellent. Unfortunately if your question is more like "What are the top 10 pages hit by users with Internet Explorer who were referred to us by Google?" all of these programs become rather useless.

First look at Thunderbird 2

As most of you already know, Thunderbird 2.0 was released today. I have been running the 2.0 release candidate for some time now, so I can share my opinions of the new version, while the going is still hot.

Good Stuff

New default theme and icons

I have found both new icon theme and the new user interface controls theme to be slightly better looking. There are no major changes here, just everything looks a little bit crispier, a little less intrusive, a little better organized and a little aesthetically more pleasing.

Unlimited tags

This is not as much a new feature as a fix of an old bug. Older versions of Thunderbird used to allow you to tag messages using either manual tagging or filters. Tagged message would be colored into particular color, so you can at a glance find out what emails you have received or what is left to do in your inbox. Unfortunately at the same time previous versions of Thunderbird would kill this feature by providing a fixed set of five pre-made tags (you could edit the labels, but you couldn't add your own). The new version still defines the same set of five tags for backward compatibility, but will happily allow you to add any number of your own. You can easily tag your messages by hand with the first nine tags in your list by pressing number keys and you can define message filters to tag messages with particular tags.

New Gecko Engine features

Since new Thunderbird is based on the same version of Gecko (the rendering engine under Mozilla products) as Firefox 2, it inherits some features from it. Spelling checks while you type, auto-completions etc.

New mail notification

The new version is able to notify you about incoming mail by either playing a sound or flashing a small pop-up (self-destructing in a few seconds) with subjects and senders of new messages.

Better support of large IMAP folders

Thunderbird 1.x used to consistently crash on me when I tried to manipulate 10K+ messages IMAP folders with it. Thunderbird 2 seems not to notice the difference between a 15K messages in a folder and 15 messages in a folder.

Bad Stuff

Finer customizations (they are there... but they are not)

Something bit me to customize the "such and such wrote" message that appears on the top of quoted message in your replies. And to my surprise, to do this you need edit some obscure configuration files in Thunderbird profile directory. Yes, it is documented extensively on the Tips and Tricks page, but I think this would not sit well with a casual user. Same goes for many other features that Thunderbird has, but you will never find out about them unless somebody tells you.

Some icons are inconsistent with previous releases

Took me some time to get used to the new junk mail icon. Not a big deal though.

Still no "Reply to All" shortcut of any sort

This is especially annoying when you are trying to CC on some of your business correspondence to some people (say your boss and your team) and every time you reply to a message you cannot just hit CTRL-R or some other key, but actually need to go through the menu to catch all the addresses in the original message. I suppose there has to be an extension for this somewhere, but so far I couldn't find it.
Update: Ctrl-Shift-R does reply all. I should have RTFM'd more

Conclusions

• If you are already using Thunderbird, you should strongly consider upgrading. The new Thunderbird is leaner, meaner, faster and with sharper teeth :) The only reason to wait is if you are using some specific extensions not yet available for the new version


• If you are not using Thunderbird and you do not require Outlook-like abilities such as calendar, to do lists, exchange compatibility etc., but only use your mail client to send and read email you should definitely consider giving Thunderbird a try.


• The general feeling about the new Thunderbird is that it is not a huge leap forward, compared to previous versions, but a lot of small useful improvements making the overall experience of using it a much more pleasant one.

Freedom vs. accountability in system administration

One of the standard security measures on a contemporary UNIX system is sudo command. For those unfamiliar with it sudo allows a user to run commands under privileges of another user, so for example a regular user can run a command as root. This, at the first glance, seems very similar to su, but sudo allows a very fine configuration of what exact commands are allowed to be run by what user and coming from what host and sudo, as opposed to su, doesn't require the user to know root password. Also, sudo will log every use of itself, weather succesful or failed therefore leaving an audit trail of administration command used on the system. Sudo is exceptionally good, for giving regular users fragments of root power where they need it. For example using sudo you can give your developers rights to restart development database server or development web server or give them rights to use network sniffers etc. One of the other things sudo seems to be good for is to record actions taken by system administrators, for accountability purposes. It all seems very simple

• Create regular users for every administrator
• Configure sudo to allow administrators run any command as rot using sudo
• Disable the actual root logon

And voile, every time one of the administrators does something that requires root privileges, he is forced to use sudo and his exact command line is logged for potential future audit. Or that would be the idea. Unfortunately there are two things that prevent this from being an administration audit panacea. Namely,

sudo /bin/bash

and

sudo vim /var/log/secure

, where the first one will run interactive root shell (allowing one to start running commands as root directly from the shell without any logging) and the second one starts editor on the sudo audit log (log name may be different on different systems) allowing to delete or edit any audit lines one deems unsightly (for example change your user name to somebody else's in that line that says rm -rf /oracle :) ). What are the ways to prevent this?

• Exclude potentially dangerous commands such as command shell and editor without arguments from the sudo config
  
• Set a strict list of administration commands that is allowed for execution by administrators
• Use external auditing mechanisms such as auditd daemon
  
• Use external privilege restriction mechanisms such as SELinux.

The first way is obviously bad. This is a classic example of "enumerating badness" where you are trying to enumerate every pattern you are trying to catch instead of enumerating every pattern you do not want to catch. Also, this approach is just plain impossible to implement, since there are too many ways to run a shell or an editor without triggering the sudo restrictions you might impose. The second way might work somewhat in a big shop where each administrator is given a particular piece of the system to work with, so web administrator is setup to run web server administration commands and nothing else and database administrator only has access to database administartion etc. Unfortunately this approach also has its faults. For one, somebody has to have full access to the system, at least so that sudo configuration can be changed when staff moves around. Also, in situations such as debugging a difficult to catch problem on the server an administrator may benefit greatly from access to unusual tools and such use can be difficult to predict. Third and fourth way are definitely worth loking at and probably worth implementing, but discussion is a bit out of scope of this article. I will make write another article someday on administration of SELinux and auditing with auditd some other day. Returning to uses of sudo, the question is where you want to draw the line between the convenience and freedom of action of your system administration staff and having a trustworthy audit trail. In big companies this question has only one answer and that is "we want to have a trusted audit information no matter at what cost" while in smaller shops, accountability may be less of a concern due to more trustful relationships between the staff and sudo logs may be enough for a basic "who did what to the system" logging.

Firefox extensions to install first

It happens to all of us sooner or later. My Firefox profile could not bear my continuous abuse and committed suicide without even writing a note. This event, albeit unfortunate, was not unforeseen. I knew, that if I keep switching back and forth between Firefox 1.x and 2.x, install and remove all sorts of suspicious extensions and tinker with about:config settings, I will eventually be punished. So, I assessed the situation and figured that if I am careful I will not lose anything important. I have backed up my corrupted profile, started and shutdown Firefox to create a new one, copied my bookmarks, stored passwords and saved sessions and called it a day. Once I started Firefox again though it still didn't look friendly, so I started adding extensions. Here is my list ordered by importance.

What did I install:

• Tab Mix Plus - Is only the best tab manager extension I have seen so far. It makes tab switching behave in a logical manner (like windows on alt-tab and not in a dumb loop) it adds a lot of useful tab related functions such as lock tab or duplicate tab. Locking is a way to make sure that wherever you click this tab stays on the same page and links are opened in new tabs, this is highly useful for browsing lists of things, be that google search results, bookmarks or craigslist.org listings. Also Tab MIx Plus replaces the built-in Firefox two feature of crash recovery and turns it into a complete session management. You can save and restore multiple sessions including closed tabs and windows (oh, did I mention that you can undo tab close with Tab Mix Plus?) and other information.
• Adblock Plus and Adblock Filterset.G Updater - Unless you are a masochist and enjoy intrusive advertising you need these extensions. Yes, you really do. This extension effectively bloxk most forms of banners, flash ads, popups (even the ones built-in popup blocker doesn't catch) etc. The updater will download current set of patterns, so you don't have to train the blocker yourself and will keep it updated.
  
• del.icio.us firefox extension - A very convenient way to keep your bookmarks online. Includes a "Bookmark This" button that will open a new window allowing you to tag, describe and save current page.
  
• Deepest Sender - There are a few blogging extensions out there that allow you to post blog entries in a comfortable (or not so comfortable in some cases) way. I have chosen Deepest Sender as my personal favorite. It supports all the major blog engines (in my case Live Journal, Blogger and WordPress), allows for simple formating, allows direct source editing and has a simple preview. I guess I would prefer a few more WordPress specific options, but I have yet to find a better blogging solution.
• Colorful Tabs - All this extension does is paint your tabs carious semi-random colors (the colors cannot be assigned, but it will make sure that no two neighboring tabs are the same color) and slightly fades away tabs which are out of focus. You cannot imagine without trying just how much easier it is to navigate multiple tabs with this extension. Albeit your tab bar starts to look much less officious.
• GreaseMonkey - a generic extension allowing you to execute custom JavaScript scripts on pages you choose. Using these scripts you can enhance usability of popular sites, add missing features, change look and feel etc. Pre-made scripts can be downloaded from UserScripts.Org site.
  
• Web Developer and FireBug - The first one is the web developer's multi-tool. It is a tool bar that includes all features that you could possibly want when testing the web site you are working on. Cache disabling, headers, authentication, security and other information, window resizes for different resolution simulation, element outlines etc. etc. etc. And where Web Developer leaves off, FireBug comes in. Normally hiding in the status bar icon FireBug will tell you exact lines in CSS that affect particular tag, tag that corresponds to particular element, how long it took to load and render any of the page requirements, what scripts have been loaded and much, much more.

There are several extensions I didn't install because I personally didn't find them useful, but which should still be mentioned.

What I didn't install:

• Sage - is the most popular RSS reader extension. I do not use it, because I don't like side bars and I am quite happy with my external RSS reader which happens to be Liferea
• All-in-One Sidebar - is a great tool for people who use side bar a lot. It integrates downloads, extensions, source view and other features into the side bar and allows for custom side bar panels.
• ScribeFire - is another popular blogger extension. It even supports some WordPress features better than Deepest Sender, but the interface is a little cumbersome and the Live Journal support is very buggy.

So, at this my browser is ready for action again. I will be back soon with Firefox extensions for web site testing.

A FreeBSD experiment

About a year back, there have been some activity around a post by one of the FreeBSD developers  regarding FreeBSD being ready to compete with Linux (and I suppose by proxy with Windows) as a desktop system. Back then I wanted to play around with FreeBSD once again (my friendship with UNIX started with installing FreeBSD 2.2.4 on my home computer), but found some features lacking for a proper support of my favorite UNIX desktop (that would be GNOME). A few days ago I figured it was a good time to take a look at what the BSD people came up with in the desktop department. I have done some probe installs in VMware, so now I am ready to try it on my home computer. So far (after those test installs) I figured out two main things about FreeBSD.

• A lot of things are very different from Linux.

Well, this would be natural and expected, since FreeBSD is not Linux. But it has been a long time since a new system confused me and now I am refreshingly confused. The aspect I found especially confusing is disk allocation. I still hope to find a reasonable documentation on what the relation is between partitions, slices and labels is and how the information about the layout is stored etc. Since all of base system in FreeBSD is developed as part of the FreeBSD project, a lot of basic commands work in unexpected ways. This is not a problem though, I was ready for it and now I seem to cope well with the differences.

• The community is extremely rude to new users.

This, unfortunately, is a problem I didn't expect. For years of working with Linux, I have gotten used to people being willing to help and if not at least not being outright evil. Not so in FreeBSD world. On one of the test installs, I messed up my disks by trying to switch to a different boot manager. I couldn't boot my system and I didn't want to reinstall, since I have configured and installed and compiled a lot of stuff on it. So, being a newbie I went to #freebsd channel and asked for help. To my surprise, I was immediately told that the only way for me was to reinstall entire system. I have expressed some doubts about this, since I was pretty sure that my data was still intact on the system, but was told again, that the only way was to reinstall and restore from backup if I had one. At this point I figured that this was a big usability hole for a modern operating system, but I figured that I will get a second opinion before I destroy my data. Some 10-15 minutes later, some other channel member took pity on me and told me that the reinstall was only suggested because I was on a wrong channel. I was supposed to ask for help on #freebsdhelp. I went to that channel and while my question was ignored for a while, I kept digging through man pages and mailing lists and other documentation and found my answer. By that time, someone on #freebsdhelp told me to shut up because I didn't use proper terms for disk allocation units. If I wasn't stubborn and didn't have enough prior computer knowledge, at this point I would be reinstalling my system from scratch. Why? Because I asked a question on a wrong channel. Mind you that the "right" channel jst plain ignored my question, which, while being better than the previous experience, also didn't help much. I am still going to try FreeBSD. Albeit I doubt I will ever ask for help from anybody in FreeBSD community.