This tutorial will follow my progress working on a library that would handle working with dynamic JSON data. Go is a great language for decoding JSON when the data layout is fixed, but since Go is a static language, dealing with JSON that is dynamic can be less intuitive than when working with dynamic languages like PHP, Ruby, or Python. This tutorial assumes that you have some working knowledge of Go. If you have no prior knowledge of Go, go here and here.
<rant>
I think Go is lacking in regards to the amount of intermediate and advanced examples that show how Go works solving actual problems instead of a "10000 foot view"of Go. I have great respect for the Go team and in particular, I think Andrew Gerrand is doing a great job of representing Go accross the globe, but if I see one more video of Andrew demoing chat roulette, I will commit Seppuku.
</rant>
Taking a whack at it
This is the sample JSON data from the Github API I am working with. And here is a sample JSON record.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "type" : "CreateEvent", | |
| "actor" : { | |
| "avatar_url" : "https://secure.gravatar.com/avatar/05abda697c2654a02391248bed3a5f3e?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-user-420.png", | |
| "url" : "https://api.github.com/users/mbeale", | |
| "id" : 1507647, | |
| "gravatar_id" : "05abda697c2654a02391248bed3a5f3e", | |
| "login" : "mbeale" | |
| }, | |
| "repo" : { | |
| "url" : "https://api.github.com/repos/mbeale/gorecurly", | |
| "id" : 4352662, | |
| "name" : "mbeale/gorecurly" | |
| }, | |
| "public" : true, | |
| "payload" : { | |
| "master_branch" : "master", | |
| "description" : "gorecurly", | |
| "ref" : null, | |
| "ref_type" : "repository", | |
| "deeper_nesting" : { | |
| "return_value" : "success" | |
| } | |
| }, | |
| "created_at" : "2012-05-16T22:52:24Z", | |
| "id" : "1552916813" | |
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| var jsondata []interface{} | |
| err := json.Unmarshal(rawData, &jsondata) | |
| if err == nil{ | |
| m := v.(map[string]interface{}) | |
| //the following prints out the type | |
| fmt.Printf("%s \n",m["type"]) | |
| } else { | |
| fmt.Println("error:", err) | |
| } |
The problem with this is you can only access top level data items. For instance, I can access the payload by using m["type"] but trying to get m["payload"]["description"] will not compile because an interface does not have any indexes. So what can we do to clean this up? I created a couple of structs and some basic functions to help me decode the JSON. I wanted a JSONCollection struct which contained the entire document and JSONObject which would be the struct used to read the results.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| //Document Container | |
| type JSONContainer struct { | |
| data []interface{} | |
| } | |
| //Return all objects | |
| func (j *JSONContainer) All() (objects []JSONObject) { | |
| for _, v := range j.data { | |
| newJSONObject := JSONObject{data: v} | |
| objects = append(objects, newJSONObject) | |
| } | |
| return | |
| } | |
| //JSON Object container | |
| type JSONObject struct { | |
| data interface{} | |
| } | |
| //Search for a attribute | |
| func (j *JSONObject) Get(val string) (string,error) { | |
| m := j.createMap(); | |
| if val, success := m[val]; success { | |
| return fmt.Sprintf("%s",val), nil | |
| } | |
| return "", errors.New("Key does not exist") | |
| } | |
| //Type casts the integer as a map | |
| func (j *JSONObject) createMap() (map[string]interface{}) { | |
| return j.data.(map[string]interface{}) | |
| } | |
| //Function to create and intialize a container with a document | |
| func DynamicJSON(rawData []byte) (j JSONContainer, e error){ | |
| if err := json.Unmarshal(rawData, &j.data); err != nil { | |
| return j, err | |
| } | |
| return | |
| } |
This still has the same limitation as before. I can use "Get" to find a top level attribute, but If I want a nested attribute, I have to type cast the interface to a map again and then grab the description out of that map. I need to come up with a better solution. The goal is to have syntax like the following.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val, _ := v.Get("payload.deeper_nesting.return_value") |
This will give us a cleaner syntax to get at nested variables. So let's fix the code to allow this.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| type JSONContainer struct { | |
| data []interface{} | |
| } | |
| func (j *JSONContainer) All() (objects []JSONObject) { | |
| for _, v := range j.data { | |
| objects = append(objects, JSONObject{data: v}) | |
| } | |
| return | |
| } | |
| func (j *JSONContainer) GetByKey(val int) (json JSONObject, e error) { | |
| return | |
| } | |
| type JSONObject struct { | |
| data interface{} | |
| } | |
| func (j *JSONObject) Get(value string) (string,error) { | |
| //get the root map | |
| m := j.createMap(); | |
| //get all the segments | |
| segments := strings.Split(value,".") | |
| for i,v := range segments { | |
| if val, success := m[v]; success { | |
| //if this is the ultimate segment | |
| if i + 1 == len(explode) { | |
| if s, isstring := val.(string); isstring { | |
| return s, nil | |
| } else { | |
| return "", errors.New("Value is not a string"); | |
| } | |
| } else { | |
| //type cast the interface into a map | |
| m, success = m[v].(map[string]interface{}) | |
| if !success { | |
| return "", errors.New("Can not convert value to map") | |
| } | |
| } | |
| } | |
| } | |
| return "", errors.New("Key does not exist") | |
| } | |
| func (j *JSONObject) createMap() (map[string]interface{}) { | |
| return j.data.(map[string]interface{}) | |
| } | |
| func MakeJSONContainer(rawData []byte) (j JSONContainer, e error){ | |
| if err := json.Unmarshal(rawData, &j.data); err != nil { | |
| return j, err | |
| } | |
| return | |
| } |
This gives a start to our library. Now we can retrieve strings of data (and nested data) with a minimal amount of Go code. Here is a an example as to how we would use this library.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| func main(){ | |
| dyn, err := MakeJSONContainer(rawData) | |
| if err != nil { | |
| fmt.Println("error:", err) | |
| } else { | |
| for _, v := range dyn.All() { | |
| if val ,err := v.Get("payload.deeper_nesting.return_value");err == nil { | |
| println(val) | |
| } else { | |
| fmt.Println("error:", err) | |
| } | |
| } | |
| } | |
| } |
The library uses only strings, so if you have an integer or float, those values will have trouble converting to strings (I will fix this in a later blog post). You will have trouble traversing nested slices as that this will cause an error. Currently the example is read-only, as that was all that was initially needed.
Coming Up on Personal Ramblings
I want to make this a feature complete library, so here is a list of things that I want to add to this library.
- Get JSON Object by Position
- Get JSON Objects by using limit/offset
- Filter JSON Results
- Handling different variable types
- Creating Tests
- Creating a better Iterator
- Using Godoc
- Using GoFmt
