Find a great logging system

I wanted to get more detailed logging and have it available to my team and I so I can do postmortem analyzing of transactions after they were completed.  This could help me discover what data was actually sent by the user, what the SQL query was (if any), the resource requested, and what the status code returned was.  I needed this both for my web app code as well as for my API, and I needed to be able to search through both of them to find specific transactions.

First step was to choose a logging platform.  I had several criteria that I used while evaluating each logging product.

1.  Ease of Install
2.  Ease of Maintenance
3.  Low cost of entry
4.  Great Search functionality
5.  API to pull out data from.

I would like to stress here that I am in no way a sysadmin and only know enough to be dangerous!  I narrowed my choices to 2, Splunk and Loggly.  Here is what I found:



Splunk

Pros:  Great support backed by a company that has been in business for years.  They have a free product and an enterprise product so you can start small for free and when you out grow the free edition, you can pay for the enterprise edition.  Can install on virtually any platform.  You are in total control of your data and how it is accessed.  Great search functionality and browser based tools.

Cons:  Free version is missing some key features.  You need your own hardware to run this product, as well as doing any upgrades or maintenance yourself.  They didn't list pricing for enterprise edition so I can only assume that it is pricey!

Note:  Splunk is now has a cloud based service that is by invite only that I didn't evaluate.

Loggly

Pros:  Small client install process but after that maintenance is not needed.  Free usage tier which includes 200MB daily volume limit and stores those logs for 1 week.  Unlimited scalability which can be easily upgraded.  Search functionality and developer API to get at your data.

Cons:  Access to your logs is reliant on their availability.  So if their service is down for maintenance (or any other reason) you aren't able to view your information.  Can get pricey if your app gets huge.  For instance if you wanted to keep 12GB a day of data for 90days, you are looking at $1,799 a month.  Maybe if you are logging that many transactions, that price might not make you blink! 


So what did I choose?  I went with Loggly.  Their initial setup cost ($0) and ongoing maintenance costs ($0) works well for my small team.  We don't have time/resources/skills to manage another server so this takes a load off of us and let's us do what we do best, code!  If I was part of  a well funded start up and had a person who all he did sysadmin work for us, I probably would have gone with Splunk.  There would be advantages to retaining all our data in house and in the long run, could be more cost effective.

I will chronicle my adventures in logging.  I hope to show web engineers the importance of logging and how a little extra setup in the beginning could save you time tracking down customer issues.