Creating a Go Library: Part 1

Intro

This tutorial will follow my progress working on a library that would handle working with dynamic JSON data. Go is a great language for decoding JSON when the data layout is fixed, but since Go is a static language, dealing with JSON that is dynamic can be less intuitive than when working with dynamic languages like PHP, Ruby, or Python.  This tutorial assumes that you have some working knowledge of Go.  If you have no prior knowledge of Go, go here and here.

<rant>
I think Go is lacking in regards to the amount of intermediate and advanced examples that show how Go works solving actual problems instead of a "10000 foot view"of Go.  I have great respect for the Go team and in particular, I think Andrew Gerrand is doing a great job of representing Go accross the globe, but if I see one more video of Andrew demoing chat roulette, I will commit Seppuku. 
</rant>

Taking a whack at it

This is the sample JSON data from the Github API I am working with.  And here is a sample JSON record.

Looking at the events data, I found that the payload object is different depending on the "type" of event.  Immediately, I wanted to go back to my PHP roots and use $event = json_decode($source) and this problem would be solved, but the following example uses the most basic Go code.



The problem with this is you can only access top level data items.  For instance, I can access the payload by using m["type"] but trying to get m["payload"]["description"] will not compile because an interface does not have any indexes.  So what can we do to clean this up?  I created a couple of structs and some basic functions to help me decode the JSON.  I wanted a JSONCollection struct which contained the entire document and JSONObject which would be the struct used to read the results.


This still has the same limitation as before.  I can use "Get" to find a top level attribute, but If I want a nested attribute, I have to type cast the interface to a map again and then grab the description out of that map.  I need to come up with a better solution. The goal is to have syntax like the following.





This will give us a cleaner syntax to get at nested variables. So let's fix the code to allow this.


This gives a start to our library.  Now we can retrieve strings of data (and nested data) with a minimal amount of Go code.  Here is a an example as to how we would use this library.


Current Limitations

The library uses only strings, so if you have an integer or float, those values will have trouble converting to strings (I will fix this in a later blog post).  You will have trouble traversing nested slices as that this will cause an error.  Currently the example is read-only, as that was all that was initially needed.

Coming Up on Personal Ramblings

I want to make this a feature complete library, so here is a list of things that I want to add to this library.  

• Get JSON Object by Position
• Get JSON Objects by using limit/offset
• Filter JSON Results
• Handling different variable types
• Creating Tests
• Creating a better Iterator
• Using Godoc
• Using GoFmt

If you feel anything is missing, feel free to comment below.  When completed I will be releasing the source for this library on Github.

Chromebook - Road Warrior or Road Block

Introduction

I recently received the arm Chromebook and was looking at ways to use the versatile laptop for work as well as play.  First a quick overview of the Chromebook.

Pros

Boots up in ~10 seconds
Fast Internet enabled device
Very affordable ($249)
Great battery life (Can get you through an 8 hour day)
12 Free airplane wireless passes and 100GB storage for 2 years although I have not able to get deal yet :(
The trackpad works well and has two finger scrolling
Very mobile

Cons

Not very useful without internet access
No "full blown" shell without switching into developer mode
Smallish Screen
Not able to run many desktop apps
I hate track pads

The Chromebook is a great secondary device, especially when the wife wants to do shopping online or plot a new vacation while you are doing work on the main computer.  But what about making the device work for taking my work on the road ( or to Starbucks)?

My Requirements:

Secure(er) access to the internet using public WiFi
Ability to access work VPN
Ability to use Vim and code in PHP, Python, Ruby, and Javascript
Need to be able to access company apps

I can easily access apps like Zendesk and Campfire without any issues.  The full chrome browser is the same browser that I have on my desktop.  Very fast and without issues.  Started noticing a performance hit when I have more than 10 tabs open, but nothing terrible.

Using VPN is easy as well.  Under settings you can add a new VPN connection.  The office uses an OpenVPN provider so this works well.

Secure access on a public WiFi is difficult to achieve.  First you need to make sure you are on the right  network, so ask the business what their network name is before accessing the network to avoid MiM attack. I have a box that I use to host some websites that I have root access to, so I decided to use that box as a SOCKS proxy to access the internet.  This is actually pretty simple.  Just access the chrome shell (cleverly named crosh) by pressing ctrl + alt + t.  This is a simple shell with only a handful of commands  (Type in "help" to see them all listed).  The one we are interested in is ssh.  The ssh command has an option "dynamic-forward" that will help establish a secure tunnel for your browser traffic.  You can add all your other options like port, host, user and various others.

You first need to go to your settings (chrome://chrome/settings) and check the option "Allow proxies for shared networks".  Then type chrome://proxy-settings in your address bar, and set the options in the SOCKS bar that you set up up using the ssh command.  This will proxy all your browser traffic through the VPS.

If you don't have a spare box as I do, there is a website that always has some deals on cheap VPS hosting, Lowendbox.com.

What about Vim?  Well, since I already have the box, I can just use the crosh shell to ssh into my spare box and just fire up Vim.  Every once and a while there is key typing delays but not enough to really bother me.  I was able to turn crosh into bash this way.  To make things easier, I also set up a new virtual host entry in Apache to give me an easy to remember domain to do web programming.

Conclusion

This Chromebook provides great value for $249.  I find myself being able to use the Chromebook instead of a traditional laptop.  I have a Asus TF101 which I now find little or no use for.  This computer can perform all necessary tasks well and there is a huge list of chrome extensions that fill in some missing gaps.  The only scenarios where this computer would not be effective is gaming and if you need to store massive amounts of data (music, ebooks, or whatever).  Yes I would rather have a MacBook, but since the MacBook could cost me 4-5 times the price of the Chromebook, I'm satisfied with the Chromebook.

Dumping PHP For Go

Background

It has been a long time coming for me to switch out of PHP for my main programming language.  I have been working with PHP for years and have grown tired of the limitations/issues.  It has been hard for me to convert, being that there are always jobs for PHP programmers out there and I gotta eat! For me, the aggravation coming with working with PHP has forced me to switch.


Why do you hate PHP?

I don't hate PHP at all.  The language itself has many advantages like a great community, numerous libraries (although how many different MVC frameworks can you have?), great documentation, and ease of entry.  There are libraries to connect PHP to virtually every database, REST API, or other third party software.  These are all reasons I began using the language in the first place.

PHP is supported on virtually every web hosting company on the planet, making PHP a useful skill to have.  There are some great open source frameworks and apps like WordPress, Zend, Joomla and too many others to list.

Why no more PHP?  First, I should probably clarify that I will continue to use PHP as many of the projects I'm working on still use PHP.  However, there are many quirks about PHP that have been bothering me.  I won't go into detail because I will not be saying anything new about the subject. If you want to read the most epic of rants on PHP issues, see this post.

Centralized Logging with Codeigniter

Background

In a previous article I spoke about the search for a logging system.  Having decided on Loggly, I needed a way to integrate logging in both my API and application.

Setup

All of my servers needed to be set up to pass syslog messages onto Loggly.  I was using Ubuntu and there was an easy walk through on the Loggly wiki here.  I grouped all of my servers in the "Production" group so that all logs were searchable under one group.  I could have separated the groups (or inputs as Loggly calls them) into an "Application" and "API" group but thought that combining them together would help me search for issues more efficiently.

After the initial setup,  I found that cron messages were immediately filling up my logs with unnecessary information. Messages similar to the one below were common.

pam_unix(cron:session): session opened for user root by (uid=0)



I wanted to remove those messages from Loggly.  I had to go into  /etc/syslog-ng/syslog-ng.conf and edit the filters to ignore cron messages.

log {
  source(s_all);filter(f_cron); destination(d_loggly);
};

Go and XML

Background

After months of lurking in the Go forums and following #golang and Go's reddit feed,  I decided to dive in with my first "real" Go project creating a API client library for Recurly.  I'm still in the middle of finishing the project but already I find Go easy to use and I find myself being very productive. One particular area of productivity, is with processing XML.  Recurly only returns xml  at the time of this writing, so marshalling/unmarshalling XML to structs was going to be very important for this project.

Find a great logging system

I wanted to get more detailed logging and have it available to my team and I so I can do postmortem analyzing of transactions after they were completed.  This could help me discover what data was actually sent by the user, what the SQL query was (if any), the resource requested, and what the status code returned was.  I needed this both for my web app code as well as for my API, and I needed to be able to search through both of them to find specific transactions.

First step was to choose a logging platform.  I had several criteria that I used while evaluating each logging product.

1.  Ease of Install
2.  Ease of Maintenance
3.  Low cost of entry
4.  Great Search functionality
5.  API to pull out data from.

I would like to stress here that I am in no way a sysadmin and only know enough to be dangerous!  I narrowed my choices to 2, Splunk and Loggly.  Here is what I found: