Centralized Logging with Codeigniter

Background

In a previous article I spoke about the search for a logging system.  Having decided on Loggly, I needed a way to integrate logging in both my API and application.

Setup

All of my servers needed to be set up to pass syslog messages onto Loggly.  I was using Ubuntu and there was an easy walk through on the Loggly wiki here.  I grouped all of my servers in the "Production" group so that all logs were searchable under one group.  I could have separated the groups (or inputs as Loggly calls them) into an "Application" and "API" group but thought that combining them together would help me search for issues more efficiently.

After the initial setup,  I found that cron messages were immediately filling up my logs with unnecessary information. Messages similar to the one below were common.

pam_unix(cron:session): session opened for user root by (uid=0)



I wanted to remove those messages from Loggly.  I had to go into  /etc/syslog-ng/syslog-ng.conf and edit the filters to ignore cron messages.

log {
  source(s_all);filter(f_cron); destination(d_loggly);
};



Integrating with the APP/API

We use Codeigniter as our framework for our app.  There were not any libraries to help us implement our app with syslog-ng, so we wrote our own here.  We set the library up to log as much information about the request as possible.  We also extended the CI_Log class so that all errors in it were being forwarded to Loggly.

The API is written mainly in Python but we had already been logging messages to syslog.  That was an easy switch as they were automatically forwarded on to Loggly.  We just added logging of the request header information so we could monitor those as well.

Why is Logging Great?

Troubleshooting user errors became much easier.  We only do email support, so now instead of always needing to ask for more information and wait for a reply, we can now just look at their activity and see a picture of what they were doing.  What pages they visited.  What information they submitted in forms.  What browser they were using.  If they encountered any errors.   Were there API request headers accurate.  Things of that nature.

All of our application errors were logged into Loggly and could be queried for by any of our team without having to ssh into each box and search for errors in all the various logs.  They were now centralized, and anyone of our team members could now find and debug errors easily.

What's Next

We need to centralize our logging for everything.  We want to centralize MySQL queries and logs and our Apache access/error logs.   We also are looking into Loggly's API to provide some automated reporting on errors and MySQL slow queries.

Conclusion

I think using centralized logging is a must.  Information is a valuable commodity and having that data easily accessible makes locating and resolving issues more manageable.