Back to top
Discuss  February 15, 2014

Fun With Data & Maps

This actually happened in July of 2013, and I finally got around to learning more of the parts, storing the actual map data in Riak somewhere, and hosting the map bits on real servers so I could see how to properly deal with CORS and CSRF related issues.

The hackathon was focused on using Chigaco city data. During some downtime (my job was to help the attendees with technical questions), myself and another coach grabbed a bunch of the data people were working with and started seeing what we could do with it. After watching my partner convert the data from tabular format into JSON, and noticing geo spacial coordinates within, we decides to just make it into GeoJSON.

I then grabbed angular js, leaflet.js, marker cluster, and open street map.

A few ten's of minutes later, I had mashed these together with the data the city gave us, and ended up with the simple map control below. Afterward, we had a few conversations with people on the Internet of Things, that left me feeling like I should know the client side application stuff better.

I particularly like the marker cluster combination effect at max zoom levels. This particular data set shows all the green spaces in the city. With a bit more effort it may be possible to reduce the amount of data pulled across if I can map it to access patterns that make sense for the changing zoom levels.

Will have to think & learn more on this.

Discuss  February 14, 2014

Happy Valentines Day!

Next week I am presenting a talk on some of the new features within the upcoming Riak 2.0 release as well as something currently called Riak JSON.

Wait, What is Riak?

Riak 2.0

Riak is a open source distributed database which is made by the company I work for. Riak has been around for a while and there are a number of great introductions found all over the internet.

If you think you know me, you had best learn what Riak is and how to use it if you want to continue being friends.

I kid.

Ok, so what is new in Riak?

Lots of things! Riak 2.0 comes with better search capability, new built in distributed data types, options for stronger consistency, and improved configuration management for deployments in the cloud.

Of all the features, I am most excited about both Search 2.0 and the new built in Conflict-free replicated data types otherwise known as CRDT. These are already documented, and I may write about these later.

Great, so what is Riak JSON?

Riak JSON is a open source document query interface which is built on top of Riak 2.0 that uses Solr to index document data.

Riak JSON focuses on JSON documents. Why JSON? Because (lets be honest here) it is currently gaining in popularity and people want to use it. JSON is now popping up all the time for me, and I want better tooling to help my work be more efficient (while still being correct) while working with distributed databases.

Are you not like me and have a specific serialized format you actually care about? Riak JSON will show you how to build out such a specific interface on top of Riak, or you can use the lower level Solr / Yokozuna API directly which is underneath Riak JSON. This is also something we (at Basho) can help you with.

Diving In

Ok, so to get started with Riak JSON we start off with the branch of Riak that has everything put together. If need a pre-built package please yell at me (nicely) on twitter and I may accomodate you. Riak JSON will work on any Riak 2.0 install.

Note, the content below is the same material within the the slides prepared for the talk.

Building from source:

git clone https://github.com/basho/riak.git
git checkout ack-riak-json
make rel
make devrel

Enable both search and riak json within the etc/riak.conf file.

search = on
riak_json_http = on

Fire up Riak

ulimit -n 4096
bin/riak start
bin/riak ping

The Java client has the following dependencies to deal with the HTTP, serialization to JSON, and logging concerns:

  • Apache HTTP Client
  • Jackson
  • SLF4J

Using the Java / Scala library:

import com.basho.riak.json._
import com.basho.riak.json.Field.Type._

import scala.beans.BeanProperty
import com.fasterxml.jackson.annotation.JsonIgnore;

val client = new Client("localhost", 10018)
val collection = client.createCollection("squares")

Define a document class:

class MySquare (l:Int, w:Int) extends Document {
  def this() =  this(0, 0)
  @BeanProperty var key: String = _
  @BeanProperty var length: Int = l
  @BeanProperty var width: Int = w

  /* don't serialize this tuple */
  @JsonIgnore def getSize = Tuple2(length, width)
}

Define a schema:

val schema = new Schema.Builder()
  .addField(new Field("length", INTEGER).setRequired(true))
  .addField(new Field("width", INTEGER).setRequired(true))
  .addField(new Field("owner", STRING))
  .build()

collection.setSchema(schema)

Add some data, riak will generate the keys for you if you wish, or you may assign your own.

val large_square = new MySquare(1024, 768)
val normal_square = new MySquare(640, 480)
collection.insert(large_square)
collection.insert(normal_square)

Query by key:

val result = collection.findByKey("9tT49FHJoQImObmPYgVPRcB56T2", classOf[MySquare])
result.getLength() => 1024

Search! Find One Thing:

val q_string = "{\"length\": {\"$gt\": 300}}"
val query = new Query(q_string, classOf[MySquare]);
val single_result = collection.findOne(query)

Find ALL the things!

val many_results = collection.findAll(query)

// documents are a java Collection
many_results.getDocuments().size() => 2

// info about the result set
many_results.numPages(); => 1 -- total pages in result set
many_results.getPage(); => 0 -- current page (zero-indexed)
many_results.perPage(); => results per page, defaults to 100

// extract into a more malable scala list if needed:
many_results.getDocuments.toArray(
  new Array[MySquare](many_results.getDocuments.size)
)

Requery with: "$per_page":10 and "$page":1 to control the pagination set.

Lastly, you can query Riak using a few different approaches:

KV Style:

'http://localhost:10018/types/squaresRJType/buckets/squares/keys/<key>'

Riak Search 2.0 (Solr / Yokozuna) Style:

'http://localhost:10018/search/squaresRJIndex?q=length:[640%20TO%20*]&wt=json'

And finally cleanup:

collection.remove(large_square)
collection.remove(normal_square)
collection.deleteSchema()

Conclusion

A key interest of mine is making Riak more developer friendly, and while there will continue to be gaps as we iterate, I feel that the 2.0 is a step in the right direction to achieve this.

Basho Technologies

Discuss  February 13, 2014

I recently re-discovered my github user page and looked upon it in horror. If you somehow saw this attrocity, please try to unsee it.

Long ago, I ran my own web server at home and spent a lot of time doing a mix of varied consulting work that involved both the front and back end. After college, I focused more on the backend aspect of software development and forgot there was such a thing as HTML or CSS. In a big company, that is perfectly fine, there is always someone else besides me that does the UI job, usually using a nice editor that I would simply not shell out the money for unless I did it every day.

As this is the first time I have written anything public for a while that is not on twitter or facebook, things you can expect to find on here are my thoughts on things that interest me. I have been a long time believer in doing projects like this on the side, it helps me keep a whole host of skills fresh.

Interests I have are:

  1. Written Language (English or 한국어)
  2. Making useful things using ...
    1. Erlang
    2. Java / Scala
    3. C / Objective C
    4. Ruby
    5. Arduino
    6. Raspberry PI

If I add more functionality, I may write more about random thoughts around the latest Peter Balis papers, but only once there is a mechanism on site to interact with all of you. You can always find me on the twitterz.

Until then, I hope you find what you are looking for.

Copyright © 2014 randy.secrist.io >