sean mcdaniel

Drying Out Your Crack

Sat, 09 Jul 2011 10:41:57 -0400

A Little Powder Goes a Long Way

Look, I spend a lot of time commuting a couple of times a week into downtown DC via the ‘fine’ public transportation system here. There is no doubt that during these hot summer months, while standing ‘nuts to butt’ with my fellow commuters in a metro car packed like sardines, sweating buckets. It’s an uncomfortable feeling to put it mildly. Which brings me to my point, and that is-sometimes code imitates life. To be honest, I spend the majority of my time during my commute reviewing code, which is oftentimes my own.

On to the Real Show

I know, I know, lame blog bait. Ok, fair enough. But I promise the title really makes sense within the context of what I want to talk about. So I’m on the train today reviewing some code I’ve been working on here and there over the last week. It’s a Ruby gem that provides an abstraction to a REST API provided by my current client as part of the data.gov initiative. This particular API uses an XMsmeLs, also know as XML, protocol for interacting with its resources. Needing a way to parse and deal with XML, I took a look around at the available gems in the wild. Nokogiri looked awesome, but it was more that what I needed. I ended up using the Crack gem which, from what I gathered, had a lot of fans in the Ruby community.

Crack

Crack is a XML and JSON parsing gem. Actually, it is a simple XML to Hash marshaling framework that is shamelessly a copy and paste job from source found in both Merb and Rails. The gist is feed Crack, an XML document, and it provides you with a Hash representation of that document. This is great, because a Hash can easily and quite naturally represent hierarchical data.

xml = Crack::XML.parse("<foo><bar><baz>contents</baz></bar></foo>")
 => {"foo"=>{"bar"=>{"baz"=>"contents"}}}

puts xml['foo']['bar']['baz']
 => "contents"

Awesome, this is a straight forward and an easy way to deal with (dreadful) XML. So I moved forward with it and wrote a lot of code using this nested Hash key style even though it always made me feel a little ‘dirty’. So this morning at 6:30am while on the train with 300 of my closest friends I revisited this code and had the same ‘dirty’ feeling. So I decided to do something about it.

My first inclination was that I needed a nice way to navigate the document using attribute-like ‘.’ notation. Attribute access just feels better since there is less syntactic noise and it is just as expressive as doing Hash navigation. So I knew what I wanted about half way through my 1 1/2 hour-long commute. Unfortunately, the data service was pretty spotty because the train snakes its way underground into the nation’s capital. Luckily, as the train entered into Metro Center Station, I got a signal and hit the Googs with a search for “recursive openstruct”. Bam! Once again Google provided me with exactly what I was looking for to “dry” out this Crack.

“Drying Out Your Crack” with Recursive Open Struct

Ruby’s stdlib OpenStruct class gives a nice abstraction (syntactic sweetener) to Hashes. It’s weakness is that it only supports a single level of abstraction.

better = OpenStruct.new({"foo"=>{"bar"=>{"baz"=>"contents"}}})
 => #<OpenStruct foo={"bar"=>{"baz"=>"contents"}}> 

puts better.foo
 => {"bar"=>{"baz"=>"contents"}}

Better, but foo is a Hash, and therefore I obviously can’t navigate to better.foo.bar, which is the goal. Enter the recursive open struct gem/class, which does exactly what the name suggests.

mo_better = RecursiveOpenStruct.new({"foo"=>{"bar"=>{"baz"=>"contents"}}})
 => #<RecursiveOpenStruct foo={"bar"=>{"baz"=>"contents"}}>

looks similar but thanks to Ruby’s ability to dynamically define methods, this now works

puts mo_better.foo.bar.baz
 => "contents"

Now that is “Mo Better”. Mission accomplished.

Summary

I wouldn’t use Crack if the XML documents that I was dealing were large, and required XPath to effectively navigate. Nokogiri would be a much more appropriate tool to use. But for my particular case, the documents that I had to deal with made Crack a nice fit. I’m not sure if there is a rule about Hash depth, but to me Hash navigation higher than two starts to make the code smell. RecursiveOpenStruct gave me the syntactic style I was looking for and dried out my Crack quite well. Now I need to find some talcum power to dry out my other problem.

NodeJS - From "Yea Right" to "Hell Ya"

Mon, 13 Jun 2011 22:27:00 -0400

Yea Right

If someone had told me 2 years ago that I would be implementing server-side logic in Javascript, I would have laughed him out of the room. It was about a year ago I came across an article on a framework called, node.JS. At the time, the idea of building networked servers/services in Javascript seemed very foreign to me, but I remember the sense of excitement I felt about the thought of it. When I tried to explain what node was to my colleagues and it’s potential to solve some of our specific problems, they gave me a dumbass look followed by a resonating, ‘Yea right’.

What follows is a very high level view of a few of the pieces in the node puzzle, but this should be enough to get you inspired to craft some server-side Javascript.

Non-blocking I/O

Node is a set of asynchronous libraries that are built on top of Google’s high performance V8 Javascript engine. The asynchronous nature comes from the fact that it takes a non-blocking approach to all I/O. In traditional runtime environments, threads are used as a means to support concurrency and scale. In a threaded model I/O is a wait operation. This simply means that operations required to access an external resource, a database request for instance, will result in the thread waiting until the I/O completes. Oftentimes, from my experience at least, the majority of the time spent processing client requests is spent waiting on I/O.

Node works quite differently using a single-threaded/event loop design. “In node everything runs in parallel except your code”. Yep, that’s right. The best way to explain this is to use an analogy. Let’s say your code is the manager for a courier service and node is the manager’s fleet of couriers. When the day starts, the manager assigns jobs to the couriers and the couriers go to work delivering packages throughout the city. Once all the work has been assigned, the manager decides it’s time for a little nap. After a short snooze, the manager is awakened by a knock at the door. His couriers have just begun to return after completing their tasks and are lining up at the door. The manager lets each courier in one at a time and gathers the relevant paperwork associated with the job. As each courier leaves the manager’s office, he closes the door and the manager tries to get back to his nap. Unfortunately for the manager, as long as there are couriers lined up outside of his door, the knocking will continue. This flow continues throughout the day until there is no more work left and all the couriers have reported back with the necessary paperwork from their jobs.

While this is a contrived example, it should give you a general sense of how things work in node. Your code issues async function calls to the node API, and node calls back (knocks), as results become available. There are a couple of interesting things to note about the node runtime environment. First, when the application code is running, it is the only thing that is running in the event loop,(it’s single threaded remember), and will run until there is no more logic to execute. When the event loop starts, it checks for the presence of async call results and calls back to the application code if one is available. One interesting behavior to be aware of is that the order in which the callbacks are received by the application code may or may not be the order in which the async operation completed. This detail will/should not have any observable side effects as your application code ‘should’ run blazingly fast. (Stay away from CPU intensive operations in your script).

Intermission break => roflscale

CommonJS & Node Package Manager (NPM)

The node environment includes a module system base on CommonJS. The CommonJS initiative was started in order to address many of the needs that would be required to use Javascript as a language outside of the browser. Platform like things such as a module system, file system access, binary I/O, process management, sockets, etc. The important thing for this context is the module system that allows us to require other modules via the require() function defined in node’s global namespace. It will become more apparent in the code examples later.

NPM is to node what RubyGems is to Ruby or Maven is to Java (yuck btw). You can use it to publish your node modules for others to consume but you will primarily use it to fetch 3rd party packages for use in your scripts.

Conventions & Patterns

The programming model can be summed up as ‘callback driven’. This shouldn’t be a big surprise, it’s just Javascript. A large majority of your code will exist in callback functions.

Comparison between sync and async programming paradigms.

Sync

function do(something) {
  if (!something) throw new Error('nothing to do!');
  return 'done';  
}

try {   
  var result = do(something);
  console.log(result);
} catch err {
  throw err
}

Async

function do(something, callback) {
  if (!something) callback(new Error('nothing to do!'));
  return callback(null, result);  
}

do('work', function(err, result) {
  if (err) throw err;
  console.log(result);
});

Patterns

Callback parameter

The convention in node is to pass the callback function as the last parameter to an async function.

function doAsync(something, callback) {
  callback(null, result);
}

doAsync(work, function(err, result) {
  if (err) throw err;
  console.log(result);
});

Callback result

The convention in node is to pass the result to the callback as the 2nd argument.

function doAsync(something, callback) {
  ...
  callback(null, result);
}

Error Propagation aka Errorback

The convention in node is to pass the callback function an error as the first argument or null if an error wasn’t encountered.

function doAsnyc(something, callback) {
  if (!something) callback(new Error('nothing to do!'));
  ...
}

Codes using node - taken from the node docs

Filesystem

var fs = require('fs');

fs.readFile('/etc/passwd', function (err, data) {
  if (err) throw err;
  console.log(data);
});

Http

var http = require('http');

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, "127.0.0.1");

console.log('Server running at http://127.0.0.1:1337/')

Sweet Spot

The ecosystem around node is pretty incredible and there are an amazing number of 3rd party modules in the NPM repo. It seems that just about everything under the moon is being developed or ported for the node platform. I cut my teeth writing a JSONP middleware module for the connect framework which is to node what rack is to Ruby. Connect is the foundation for the popular Express web application framework. While Express is pretty solid, I’m not convinced that I would write web applications on top of node. I don’t see the advantage over mature web frameworks.

Another area where I wouldn’t use node is for CPU intensive logic. This actually goes against what node is trying to solve, efficiency through offloading latency intensive operations outside of the process / event loop. If each iteration of the event loop takes a considerable amount of time performing application logic, the benefit of non-blocking I/O gets negated.

I have found node to be very well suited for I/O intensive batch processing. In particular, I’m using node as a lightweight, scalable and high performant means for shelling out to CPU hungry Unix command line tools using node to manage intensive I/O. I built Mimeograph as a simple OCR/text extraction batching solution to orchestrate tools such as ghostscript, image magick and tesseract. Mimeograph uses a simple utility called RedisFS, to move files to be processed in and out of Redis and to and from a Unix filesystem. Mimeograph uses coffee-resque as the batching framework, a node port of the popular Resque framework. With the architecture, I can massively scale out mimeograph processes across virtual machines to support a very impressive amount of OCR/text extraction processing required by the business to support upcoming legal proceedings as well as process multiple years worth of indexing backlog. I’ll post a focused discussion of Mimeograph in the near future and how to best utilize coffee-resque to distribute CPU intensive parallel problems.

I recently watched a vid in which Isaac Z. Schlueter of NPM fame in which he explains why node is well suited for Data-Intensive Real-Time (DIRT) applications. Pretty good stuff, take a look.

Hell Ya

Javascript is definitely not just for the browser anymore, thanks to node. I’ve been working with it for the last year and have had great success with it in production. Mimeograph impressed my client so much that we are targeting more opportunities within the organization where it is a good fit. Things have changed quite a bit over the last year, the dumbass looks are no longer, and the ‘yeah right’ responses are now a resonating ‘hell ya!’.

Happy Noding.

Introduction to node.js

Shiny New Toys

Wed, 25 May 2011 22:33:00 -0400

Holy shit, I have a blog (again)! I previously maintained a blog ~ 3 years ago that focused on ExtJS and integration solutions for various Java based MVC frameworks. Since that material completely is wack and out of date I simply flushed it from the tubes. Out with the old, in with the new.

In any case for those of you who don’t know me my name is Sean McDaniel. I’m a freelance developer working out of the greater Washington DC area. I like to consider myself a generalist in my field and have had the privilege of working with a lot of interesting technologies, clients and problem domains. I’ve spent a lot of my free time over the last 6 months building tools using a slew of relatively new technologies. Implementation wise these tools/utils are based on node.js and other frameworks surrounding node.js. In addition I’ve done quite a bit with the Redis key/value store.

To help me better understand these new technologies I’ve been building open source tools, as well as contributing to existing ones. For me as a developer open source serves a couple of purposes besides providing great frameworks upon which I build solutions on. First it gives means by which I can dig into preexisting source code to understand concepts in a concrete way and second it provides a way for me to scratch and itch. Therefore, many of these node.js based projects I work on are inspired by real world problems I’ve been trying to help solve for my current client. In fact the tools that I’m going to be talking about over the next few posts have been in production for over a month now and purring like a kitty.

So here are my shiny new toys that I will be covering shortly:

connect-jsonp - JSONP middleware for sencha labs popular connect framework. Think ruby rack for node.js.
redisfs - Utility to pump files in & out of redis.
coffee-resque - a CoffeeScript version of the popular ruby resque project.
mimeograph - CoffeeScript batching library, based on coffee-resque, to extract text and create searchable pdf files, OCRing where necessary.

Me

Wed, 25 May 2011 21:30:48 -0400

Nice Virginia evening.

Thu, 19 May 2011 19:46:25 -0400

Nice Virginia evening.