To recap, a useful system logging solution consists of four components: generation, transport, storage and analysis.
I will argue if you already have any logs at all, that your first step should be to build an analysis capability. This will let you begin to analyze the logs you already have, become familiar with your analysis tool on a smaller dataset and use the analysis tool to help debug any problems that you encounter while building the rest of the system.
I’ve been a big Splunk fan for years. The Splunk folks understand system and network administration and that shows in the design and capabilities of the product. The free “home” license is a great contribution to the community, too.
There is a lot of good documentation out there on getting started with Splunk, so I’ll focus on what it allowed me to find instead of the details of using it. I encourage you to experiment and try different kinds of searches, you’ll be surprised at what you find.
After starting Splunk, I pointed it at my /var/log directory, which has all the usual system logs, and also all my Apache logs. Splunk indexed about 2 million log events in less than 8 minutes, on my low-power Atom CPU with only 2G RAM and a single 150G IDE laptop disk.
In the 30 minutes or so, I found (all on a single host, all in the last 30 days)
- 935 root SSH root login attempts
- 838 attempts to exploit PHP bugs in my web server
- 20 attempts to buffer overflow my web server
- over 100K attempts to delivery SPAM or use my hosts as a mail relay
- 40 attempts to use MyAdmin scripts (which I don’t have)
So, less than 30 minutes to install Splunk and 30 minutes of playing with the search tool has already paid off 🙂
Next steps: get the home router sending its logs to the log server and setting up some Splunk “canned” searches.