Pages

Saturday, October 11, 2014

How to Write Hello World in Go (in Go)

Let's write "hello world" in go.

Sure, you can do it this way:

package main

import "fmt"

func main() {
fmt.Printf("Hello world.\n")
}

But that's no fun, is it?
"Give a clever engineer a straightforward problem and they'll add complexity until it's interesting enough to solve."
Let's write a program that writes hello.go for us!  That's right: write go, using go.

(Sure, you could just write a program that echos the above source text as a quoted string to stdout, but that's not much fun. We must add more complexity to make it interesting.)

The "go/ast" package is used by gofmt and gofix (and various other tools) to parse and manipulate go source code in the form of abstract syntax trees.

We're going to build a new go program from scratch, in go, so we're not really interested in parsing so much as constructing an AST and printing the result as human-readable source code.

Here's a skeleton that uses go/ast to construct an AST for a very minimal go program that compiles, but doesn't actually do anything:

package main

import (
"bytes"
"fmt"
"go/ast"
"go/printer"
"go/token"
)

func main() {
// Start with a file
f := &ast.File{
Name: &ast.Ident{
// The package name is "main"
Name: "main",
},
// Top-level declarations in this file:
Decls: []ast.Decl{
// A basic func declaration with no receiver:
&ast.FuncDecl{
Name: &ast.Ident{
// This func is named "main"
Name: "main",
},
// With an empty func type (no params, no returns)
Type: &ast.FuncType{},
// And an empty body.
Body: &ast.BlockStmt{},
},
},
}

fset := token.NewFileSet()

var buf bytes.Buffer
printer.Fprint(&buf, fset, f)
fmt.Printf("%s\n", buf.String())
}

Try it out on play.golang.org here.  It produces the following:

package main

func main() {
}

Which does compile, but doesn't actually do anything.  Let's add the next pieces: The import statement for "fmt" and the fmt.Printf statement that actually prints "Hello world."

To add the import statement, add a new element to f.Decls:

// Start an "import" declaration
&ast.GenDecl{
Tok: token.IMPORT,
Specs: []ast.Spec{
&ast.ImportSpec{
// With a string literal of "fmt"
Path: &ast.BasicLit{
Kind:  token.STRING,
// Note the "" contained in ``
Value: `"fmt"`,
},
},
},
},

If you leave f.Decls as it is here and run it, it will produce the following:

package main

import "fmt"

func main() {
}

Which will fail to compile because "fmt" is unused.  So let's use it by adding the fmt.Printf statement inside the body of main(). Change the empty Body: &ast.BlockStmt{} in the above skeleton to include some ast.Stmts:

Body: &ast.BlockStmt{
List: []ast.Stmt{
// Start a stand-alone expression statement
&ast.ExprStmt{
// Representing a function call to "fmt"
X: &ast.CallExpr{
Fun: &ast.SelectorExpr{
X: &ast.Ident{
Name: "fmt",
},
// With a selector for Printf
Sel: &ast.Ident{
Name: "Printf",
},
},
// And a single-element arg list consisting of a string literal
Args: []ast.Expr{
&ast.BasicLit{
Kind:  token.STRING,
Value: `"Hello world.\n"`,
},
},
},
},
},
},

This will finally produce a runnable hello world:

package main

import "fmt"

func main() {
fmt.Printf("Hello world.\n")
}

Try the final product out on play.golang.org here.

In conclusion, if you'd like to do some code generation with go, this might not be a bad place to start. You can explore other parts of go/ast by adding some extra function declarations with actual parameter lists, return values and even receiver types, and then calling them from main.

If I was really bored, I'd write a further iteration of this program that constructs itself :)


Tuesday, September 21, 2010

CSS3 Star Wars Tweet Scroller

I've been playing around with some new CSS3 3-D transforms and whipped up this Star Wars scroller for Tweets:


Caveat: This currently only works on Safari.  It may work on a bleeding edge Chrome dev build but I haven't tried that.

There are some other examples of this type of scrolling effect out there but I wanted to make one for tweets.  I also noticed that there are some odd effects if you naively keep appending content to a div that's been rotated in 3-D.  As it reaches a certain height, webkit apparently starts downsampling (I guess to save memory in the rendering pipeline?  A limitation of hardware?  Who knows) and you get this pixelated effect:

This isn't a downsampled down screen shot.  It actually looks like this if you let the rotated div get too large.
As you can see, it becomes unreadable after a while.  It also starts eating up an increasing amount of CPU: it got to 100% (out of 400%) on my quad core iMac after a few minutes.  Not good.

So instead of putting all of the tweets into the same rotated div, I rotate each tweet element individually and that seems to fix the problem.  Also, I start deleting tweets at the top (where you can no longer see them anyways) after a while, just to keep the dom size down.  They are now readable and the CPU is happy.

Other stuff:

Instead of using CSS3 animations, I opted for javascript because I was loading the tweets with JS already and I'd have to manipulate the animations with JS anyways.

Also, the background is generated dynamically using the Canvas element.  I wanted a more realistic looking star field so I did some research on the distribution of star sizes and colors to see if I could perhaps simulate it with a Pareto distribution.  Turns out the distribution of star colors and brightness is some other weird distribution so I just made it more or less random, with smaller stars slightly more frequent than larger stars.

Saturday, August 21, 2010

λ as a variable name in JavaScript

JavaScript is a functional language, or at least has very functional roots we could all probably agree.

Functional languages are based on lambda calculus and the CS literature often uses examples where a lambda expression is called simply, "lambda."

In JavaScript, one common use of lambdas is when you register a callback function with some asynchronous operation like an ajax request. Another is for operations on collections of objects like visiting every item in an array.

I wondered if one could use the lambda character, λ, as a variable name in javascript. It would be more compact (one character instead of five six) and (more importantly) would look cool. Turns out you can do this!

See it run here.



Notice that the character encoding is set to UTF-16. If you don't do this you'll get errors. Since the encoding is set that way, I don't even use html entities like λ to write λ :)

Note: I haven't tried this in any version of IE but it appears to work in Chrome, Safari and Firefox.

Now, how do you actually type λ into a text editor? That took some digging. Here's what I did on OSX:

  1. Open System Preferences -> Language and Text -> Input Sources
  2. Scroll down to "Unicode Hex Input" and make sure that checkbox is checked.
  3. Open the US Flag (or nationality of your locale :) icon in the upper right of the OSX menu bar and switch to "Unicode Hex Input" (the "U+" icon)
  4. Now whenever I want a λ I can just hold down the Option key while I type 03bb and blammo I get a λ. Which by the way is still five key presses but looks way cooler than "lambda" and takes up less space on the screen.

I have no idea what you have to do on linux or windows.

You're right, this isn't very practical. And setting your encoding from UTF-8 to UTF-16 like, doubles the bytes you have to transmit. But you ARE gzipping all of your http responses aren't you? AREN'T YOU?

Thursday, February 25, 2010

PubSubHubhub for NodeJS: Callbacks All the Way Down

NodeJS is a callback-based Javascript server API.

PubSubHubbub is a callback-based web protocol.

I put them together and the result is a PubSubHubbub client for NodeJS:

node-pshb on github


This project only includes a PubSubHubbub client interface, but to me that's the interesting part. You can specify an atom feed url, and functions to call when events happen on that feed due to PubSubHubbub.

The client library takes care of identifying the hub for that feed, requesting a subscription, and listening for subscription confirmation requests and feed updates from the hub.

It provides callback hooks for "subscribed", "update" and "error" events.

A simple client app looks like this:
var callbackPort = 4443;
var subscriber = new pshb.Subscriber(callbackPort);

// Start listening for subscription confirmation callbacks.
subscriber.startCallbackServer(); 

var topicUri = url.parse("http://localhost/foo"); // Dummy feed, always has updates

var feedEvents = subscriber.listen(topicUri);

feedEvents.addListener('subscribed', 
  function(atomFeed) {
    sys.puts('subscribed: ' + atomFeed.id);
  });

feedEvents.addListener('error', 
  function(error) {
    sys.puts('ERROR: ' + error);
  });

feedEvents.addListener('update',
  function(atomFeed) {
    sys.puts('got a PubSubHubub update: ' + atomFeed.id);
  });

I tested this out with the Demo Hub running on a local AppEngine launcher.

The demo app creates a a second server to host a dummy feed on port 80, so http://localhost/foo always returns a feed with the current time as it's "last updated." This is so the test hub always thinks there's an update ready for you.

So start the appengine with the hub running locally (in this demo it's assumed to be on port 8086), then run the test.js app, then go to your hub with your browser and manually update http://localhost/foo.

I noticed that I had to manually run some tasks in the hub's work queue so if you don't see any updates try checking the Task Queues in the app console for the hub. Run any pending "feed-pulls" and "event-delivery" tasks. I imagine there's a way to make them do that automatically but I haven't dug around enough in there to find it.

So there you go, NodeJS and PubSubHubbub: it's callbacks all the way down.

Friday, February 19, 2010

Webfinger Client for Node.JS

In a previous post, I demonstrated how you could use webfinger with nothing more than curl.   This post is about how you can use webfinger from nodejs with a non-blocking webfinger client.

Code for node-webfinger is here on github.

The project contains a simple webfinger-buzz.js command line app that demonstrates the webfinger client. It uses webfinger to find a google buzz feed based on a gmail address, then fetches the updates as an Atom feed, and then prints out the latest entry from that feed.

This could be generalized to support any other webfinger-enabled site like yahoo (though it looks like they're using an older version of XRD which my code can't parse :/).

The webfinger-buzz.js client looks something like this:
var sys = require('sys'),
  http = require("http"),
  url = require("url"),
  atom = require("./lib/atom"),
  webfinger = require('./lib/webfinger-client');

if (process.argv.length < 3) {
  sys.puts("usage: " + process.argv[0] + " " + process.argv[1] + " <user uri>");
  process.exit();
}
 
var userUri =   process.argv[2];
 
sys.puts("fingering " + userUri);

var wf = new webfinger.WebFingerClient();
var fingerPromise = wf.finger(userUri);
fingerPromise.addCallback(function(xrdObj) {
  var statusLinks = xrdObj.getLinksByRel("http://schemas.google.com/g/2010#updates-from");
  var statusUrl = url.parse(statusLinks[0].getAttrValues('href')[0]);
  var httpClient = http.createClient(80, statusUrl.hostname);
  var path = statusUrl.pathname;
  if (statusUrl.search) {
    path += statusUrl.search;
  }
 
  var request = httpClient.request("GET", path, {"host": statusUrl.hostname});
 
  request.addListener('response', function (response) {
    response.setBodyEncoding("utf8");
    var body = "";
    response.addListener("data", function (chunk) {
      body += chunk;
    });
    response.addListener("end", function() {
      var atomParser = new atom.AtomParser(false);
      var atomPromise = atomParser.parse(body);
      atomPromise.addCallback(function(atomFeed) {
        sys.puts("Feed: " + atomFeed.title);
        sys.puts(atomFeed.entries.length + " entries");
        sys.puts("Updated: " + atomFeed.entries[0].updated);
        sys.puts(atomFeed.entries[0].title + ": " + atomFeed.entries[0].summary);
      });
    });
  });
  request.close();
});

hehe fingerPromise. Is that a generalization of pinkySwear?

In the process of writing this webfinger client I used a couple of libraries I found on teh internets: sax-js by Isaac Z. Schlueter - a SAX parser for nodejs, and this URI Template library by James Snell. Both worked well and I recommend them.

The remaining non-webfinger-specific pieces I needed were an XRD parser and an Atom parser, both for javascript and SAX (as opposed to DOM). I couldn't find much in the way of those, so I rolled my own. They are included in the node-webfinger project in the lib/ directory. They're pretty crude parsers but they worked for this example. I'll probably use them in other projects in the future and make improvements as necessary. Unless something better comes along. That seems inevitable.

Monday, February 15, 2010

Bring the (Perlin) Noise

If you just want the source code: The JavaScript Perlin noise generator code is here.

I mentioned in a previous post that I was working on a Perlin noise generator for Art Evolver.

Perlin noise is a function of (x, y) that produces a random-ish pattern. It's not completely random because it has smooth hills and valleys, but the distribution of those hills and valleys is random.

A side note about this algorithm: Usually if you're a computer scientist and you come up with a clever algorithm to solve a particular problem, you get an award from a university, or a CS-centric professional organization like the ACM or IEEE. Perlin got an Academy Award for this noise function. As in, the Oscar kind of Academy Award.  For an algorithm, something not usually consider artsy.  I found that interesting.

Anyways, rather than dive into Perlin's impenetrable description of how the algorithm works, I set out to find an existing JavaScript implementation. That let me to this message board, and specifically this example.

Unfortunately that implementation has some pretty serious directional artifacts:

Note the horizontal and vertical stripes.  There's almost an upside down cross in the lower right. SAAAATAAAAN!

Rather than try to fix that source code (which the author apparently closureized (making it very difficult to understand)) I kept searching.

From the main Wikipedia entry on Perlin noise, I ran across a variant called Simplex noise.  This is an improvement on the original algorithm, also written by Ken Perlin, in 2001.  That Wikipedia page linked to a paper by Stefan Gustavson(pdf) that explains both classical and Simplex Perlin noise in a much easier to grok way than anything I've read by Perlin himself.  I highly recommend Gustavson's paper if you found Perlin difficult.

I took the Java source code in Gustavson's paper and ported it to JavaScript, and the results are here on github.

I ran some performance comparisons between the classical and Simplex algorithms, and for 2-D I only saw a ~10% improvement with Simplex.  Granted, the latter is supposed to be faster in higher dimensions (classical is O(N^2) vs. simplex O(N) where N is the number of dimensions) so it doesn't matter much for my purposes.

Classical Perlin noise

Simplex Perlin noise

Subjectively I think I prefer the Simplex noise to classical, so I'll probably go with that for Art Evolver.

Again, the source code is here.

Saturday, February 13, 2010

Curl-ing up with WebFinger and PubSubHubub

This morning I've been playing around with WebFinger and PubSubHubub. One of the great things about open web APIs is that you can tinker around with them without even writing an application. Just use curl!

Let's start with WebFinger. First, we need to figure out how to get my WebFinger data from gmail. There's standard place to look for that explanation, given a domain name:
http[s]://{domain-name}/.well-known/host-meta
So for gmail we get the explanation of how to get my WebFinger data like so:
$ curl http://gmail.com/.well-known/host-meta
Out pops an XRD doc that contains a URI template (in bold, below):
<?xml version='1.0' encoding='UTF-8'?>
<!-- NOTE: this host-meta end-point is a pre-alpha work in progress.   Don't rely on it. -->
<!-- Please follow the list at http://groups.google.com/group/webfinger -->
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0' 
     xmlns:hm='http://host-meta.net/xrd/1.0'>
  <hm:Host xmlns='http://host-meta.net/xrd/1.0'>gmail.com</hm:Host>
  <Link rel='lrdd' 
        template='http://www.google.com/s2/webfinger/?q={uri}'>
    <Title>Resource Descriptor</Title>
  </Link>
</XRD>
Substitute my email address for {uri} and curl it:
$ curl http://www.google.com/s2/webfinger/?q=banksean@gmail.com
That spits out another XRD that describes some other resources associated with my email address:
<?xml version='1.0'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
 <Subject>acct:banksean@gmail.com</Subject>
 <Alias>http://www.google.com/profiles/banksean</Alias>
 <Link rel='http://portablecontacts.net/spec/1.0'
href='http://www-opensocial.googleusercontent.com/api/people/'/>
 <Link rel='http://webfinger.net/rel/profile-page' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://microformats.org/profile/hcard' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://gmpg.org/xfn/11' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://specs.openid.net/auth/2.0/provider' 
href='http://www.google.com/profiles/banksean'/>
 <Link rel='describedby' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='describedby' 
href='http://s2.googleusercontent.com/webfinger/?q=banksean%40gmail.com&amp;fmt=foaf'
type='application/rdf+xml'/>
 <Link rel='http://schemas.google.com/g/2010#updates-from' 
href='http://buzz.googleapis.com/feeds/103419049256232792514/public/posted' 
type='application/atom+xml'/>
</XRD>
Bolded above is the rel='http://schemas.google.com/g/2010#updates-from' URI for my public status updates. Let's fetch that:
$ curl http://buzz.googleapis.com/feeds/103419049256232792514/public/posted
<?xml version='1.0' encoding='utf-8'?>
<feed xmlns='http://www.w3.org/2005/Atom' 
xmlns:thr='http://purl.org/syndication/thread/1.0' 
xmlns:media='http://search.yahoo.com/mrss' 
xmlns:activity='http://activitystrea.ms/spec/1.0/'>
<link rel='self' type='application/atom+xml'
href='http://buzz.googleapis.com/feeds/103419049256232792514/public/posted'/>
<link rel='hub' href='http://pubsubhubbub.appspot.com/'/>
<!-- lots more feed data not relevant to this discussion -->
It's a standard Atom feed. Amongst a lot of other stuff in the beginning is a rel="hub" link. This is where PubSubHubub comes in. Suppose that I want some other service to be notified whenever I post a status update (for instance, I have an app that reposts it in the sidebar of my blog). I could poll this Atom feed but polling is pretty janky. With PuSH I can register a callback to be notified whenever I post an update. Since this feed has a rel="hub" link set to http://pubsubhubbub.appspot.com/, that's where I go to do register a callback.

If you just navigate to http://pubsubhubbub.appspot.com/ with your browser you get a form you can fill out to create a subscription. One of the fields is for a "Callback" url. I don't run any websites that know how to handle PuSH callbacks (or subscription confirmation, for that matter). Luckily there is a test subscriber on appspot that accepts subscription requests for anything: http://pubsubhubbub-subscriber.appspot.com/. To create your own callback URL for testing, just add /subscriber.{some_unique_identifier} to the end of it.

According to the PuSH spec, I should POST some form fields to the hub URL like so:
$ curl -v http://pubsubhubbub.appspot.com/subscribe \
-d hub.callback=http://pubsubhubbub-subscriber.appspot.com/subscriber.banksean\&\
hub.topic=http://buzz.googleapis.com/feeds/103419049256232792514/public/posted\&\
hub.verify=sync\&hub.mode=subscribe\&hub.verify_token=\&hub.secret= 
And that creates the subscription. Here's the verbose output:
* About to connect() to pubsubhubbub.appspot.com port 80 (#0)
*   Trying 74.125.19.141... connected
* Connected to pubsubhubbub.appspot.com (74.125.19.141) port 80 (#0)
> POST /subscribe HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: pubsubhubbub.appspot.com
> Accept: */*
> Content-Length: 219
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 204 No Content
< Cache-Control: no-cache
< Content-Type: text/plain
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Sat, 13 Feb 2010 17:41:29 GMT
< Server: Google Frontend
< Content-Length: 0
< X-XSS-Protection: 0
< 
* Connection #0 to host pubsubhubbub.appspot.com left intact
* Closing connection #0

The 204 No Content response indicates the subscription was created and is active, according to the spec.

And if you go to http://pubsubhubbub-subscriber.appspot.com/ right now (Saturday morning, October 13th 2009), you'll indeed see a bunch of my posts on it.

Ta Da. No code. Just curl. I love the internet.

Wednesday, February 10, 2010

Mersenne Twister to the Rescue

For Art Evolver I'd like to add a Perlin noise function. The problem with doing this in javascript is the noise is unstable from generation to generation because you can't specify a seed value for Math.random().

Luckily there was an existing implementation of a pseudorandom number generator in javascript here (it's an implementation of Mersenne Twister. The problem with this code is that its functions and state variables are all in the global namespace. Meaning you can only have one generator. I need an arbitrary number of them at any given time, so I wrapped Makoto Matsumoto and Takuji Nishimura's code in a namespace.

Now I can use it like so:
var m = new MersenneTwister(123);

// now calling m.random() four times should return
// the following sequence:
// 2991312382
// 3062119789
// 1228959102
// 1840268610

The namespaced Mersenne Twister code is here.

Sunday, February 7, 2010

Logging With HTML5 WebWorkers

I'm converting some of my code for Art Evolver to use HTML5 WebWorkers instead of doing CPU-intensive operations in the main UI thread (cardinal sin, that is). It went pretty smoothly but I ran into a small problem while debugging: you can't log messages to console.log from a WebWorker. I assume this is for security reasons.

I worked around this by calling postMessage() from my WebWorker to get the log message into the browser window where it could be logged:

// inside render-worker.js
function log(msg) {
  postMessage("log: " + msg);
}
and in the code that calls the worker, I added this to the onmessage callback:
// inside UI code running in the main browser window
worker.onmessage = function(e) {
  if (e.data.indexOf("log:") == 0) {
    window.console.log(e.data);
    return;
  }

  // assume it's not a log message, JSON.parse() it,
  // or do other stuff with it here.
}

Voila- logging from inside a WebWorker. Totally a hack, but it worked for me in a pinch.

NodeJS + WebSockets = Stoopid Easy Comet Chat

A few weeks ago, amix wrote a blog post about Plurk's use of NodeJS for Comet chat.

He didn't post source code (as far as I could tell, after searching for two minutes :) but I was able to cobble something together yesterday along the same lines.

Here's the client:

and server:


This works with Guille's node.websocket.js library.  Start with that, then add chat.js into ./modules.

One last tweak is you have to add onDisconnect to node.websocket.js.  I did that by adding the following:


if (this.module.onConnect) {
    this.module.onConnect(this);
}


to the end of Connection.prototype._handshake. I suppose it could be called from elsewhere but that seems to work.

Check out amix's blog - lots of interesting details about how Plurk scales out to over 100,000 open connections. Hint: you don't do it with one thread per connection. The Servlet/CGI model is starting to look pretty old and creaky these days.

Update: bru has a node.websocket.js fork that includes a comet chat app. It's more complete than what I've posted here and he appears to have worked around the missing onConnect by just adding new connection objects in the onData callback.

Monday, November 30, 2009

Closure Templates and Node.JS: Server-Side Soy

Another NodeJS experiment: using Google's Closure Templates with NodeJS.

Closure Templates (aka Soy) can be used either on the server or client-side. The recommended way to use them server-side is with SoyTofu, but there is a way to use them in pure javascript on the server with our new pal, NodeJS.

Suppose we have this blog.soy template to render a simple blog post with some comments:

{namespace blog}

/**
 * Renders a post with comments.
 * @param post
 * @param comments
 */
{template .postPage}
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
   "http://www.w3.org/TR/html4/strict.dtd">

<html lang="en">
<head>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
 <title>blog</title>
 <meta name="generator" content="TextMate http://macromates.com/">
 <meta name="author" content="Sean McCullough">
 <!-- Date: 2009-11-28 -->
</head>
<body>
{call .post}
  {param post: $post /}
{/call}
{call .comments}
  {param comments: $comments /}
{/call}
</body>
</html>
{/template}

/**
 * Renders a Post.
 * @param post
 */
{template .post}
<h2>{$post.title}</h2>
{$post.body}
{/template}

/**
 * Renders a list of comments.
 * @param comments
 */
{template .comments}
<h3>Comments:</h3>
<ul>
  {foreach $comment in $comments}
 <li>{$comment}</li>
  {/foreach}
</ul>
{/template}

The Closure template compiler will take this input .soy file and create an output .js file that contains functions corresponding to the {template .functionName} sections above.

To compile:
java -jar SoyToJsSrcCompiler.jar --outputPathFormat templates-compiled/blog.js templates/blog.soy


The generated functions in templates-compiled/blog.js look like this:

blog.post = function(opt_data, opt_sb) {
  var output = opt_sb || new soy.StringBuilder();
  output.append('<h2>', soy.$$escapeHtml(opt_data.post.title), '</h2>', soy.$$escapeHtml(opt_data.post.body));
  if (!opt_sb) return output.toString();
};

Now, if you just try to require() this generated .js file, Node will complain because it doesn't know what soy.StringBuilder() is. We can fix that by shoehorning soyutils.js into node, of course.

First we need to make soyutils.js work with Node's require mechanism. require works in conjunction with process.mixin(), so you make soy require()-able by adding this to the bottom of soyutils.js (copied into your application code directory from the closure templates distribution):

process.mixin(exports, soy); 

Then we need to require soyutils in the blog.js file (you can just paste these into the bottom of the file but it's probably better to implement this as a post-soy-compile step in a build script so you don't have to keep pasting every time you recompile the template)

var soy = require('../soyutils');

process.mixin(exports, blog);

That last process.mixin call will make the blog template functions available to other source files via require.

Now we're ready to use the soy template with our nodejs server code. You'd just require templates-compiled/blog.js and call the functions that it provides from within your event handlers (again, building on the blogging example from a previous post):

var sys = require("sys"), http = require("http"),
  blogTemplates = require("./templates-compiled/blog");

var handlers = {
  '/posts/{postId}' : {
      GET : function(request, response, args) {
        response.sendHeader(200, {"Content-Type": "text/html"});
        var commentsPromise = getCommentsPromise(args.postId);
        var postPromise = getPostPromise(args.postId);
        var templateVars = {};
        commentsPromise.addCallback(function(comments) {
          templateVars.comments = comments;
        });
        postPromise.addCallback(function(post) {
          templateVars.post = post;
        })

        var joinedPromise = join([commentsPromise, postPromise]);
        
        joinedPromise.addCallback(function() {
          var pageHtml = blogTemplates.postPage(templateVars);
          response.sendBody(pageHtml);
          response.finish();
        });
      }
    }
  }
};

This is awfully clunky. I'd like to write a directory watcher that automatically compiles recently updated .soy files, appends the require and process.mixin calls, and reloads the result into Node.

Thoughts on Closure and NodeJS


I spent a little time trying to get the closure compiler to work with Node so that you could for instance, statically verify that the template function invocation parameters match up with the declared parameter types in the .soy file. Haven't gotten enough working there to blog about yet though.

I don't know if the closure compiler optimizations would help NodeJS much, but the static analysis would probably help catch a lot of easy-to-introduce but too-tedious-to-unit-test problems that crop up when you have lots of people working on the same code base.

Also, the Closure Library contains a lot of useful packages that could be applied server-side as well.

Wednesday, November 25, 2009

Joining Promises for Parallel RPCs in Node.JS

I've been playing around a bit more with Node.JS since my last post and I decided to experiment with the asynchronous process.Promise object this time around. In other languages I believe this concept is sometimes referred to as a future.

Diving in, suppose the following:

  • You're writing a blogging engine.
  • Blog Posts are kept in one data store, and Comments are in another.
  • A request for a Post object takes 1 second to return.
  • A request for a list of Comments on a Post takes 2 seconds to return (the comments data store is run by a bunch of slackers who don't care about latency)
  • You want to have /posts/{postId} return an html page that renders both a Post and all all the Comment objects on it.

When you make an RPC (or any I/O call) in Node.JS you should wrap it in a Promise so the process doesn't block on your HTTP request.

So our getPost and getComments RPCs (faked out) look like this:

var getCommentsPromise = function(postId) {
  var promise = new process.Promise();
  var comments = ["Comment 1 on " + postId, "Comment 2 on " + postId];
  setTimeout(function() { promise.emitSuccess(comments); }, 2000);
  return promise;
}

var getPostPromise = function(postId) {
  var promise = new process.Promise();
  setTimeout(function() { promise.emitSuccess({title: "Post Title " + postId, body: "Post Body " + postId}); }, 1000);
  return promise;
}

Now, if all you had to render on /posts/{postId} was the Post object and not the comments, you could just put the rendering code inside the handler for the Post RPC and be done with it, like so (building on the URI template router from my last post):
var handlers = {
  '/posts/{postId}' : {
      GET : function(request, response, args) {
        var postPromise = getPostPromise(postId);

        postPromise.addCallback(function(post) {
          templateVars.post = post;
          var pageHtml = postTemplate(templateVars);
          response.sendBody(pageHtml);
          response.finish();
        });
      }
    }
  }
}


But life is never that simple, and /posts/{postId} has to make two RPCs to get the data required to render a page. This is complicated because you can't render the page until both RPCs are complete.

There are at least two ways to deal with this situation. One sucks and the other doesn't suck as much.

Teh Suck: Serialize the RPCs, then render.

You can serialize the RPCs by nesting the call to the second one inside the handler for the first:

'/slowposts/{postId}' : {
      GET : function(request, response, args) {
        response.sendHeader(200, {"Content-Type": "text/html"});
        var postPromise = getPostPromise(postId);
        postPromise.addCallback(function(post) {
          var commentsPromise = getCommentsPromise(postId);
          commentsPromise.addCallback(function(comments) {
            var postTemplate = tmpl['post-template.html'];            
            var pageHtml = postTemplate({'post': post, 'comments': comments});
            response.sendBody(pageHtml);
            response.finish();
          });
        });
      }
    }
  }

This takes 3 seconds to complete: 2 for fetching comments, then 1 more for fetching the post.

Teh Not So Suck: Parallelize the RPCs, join them and render when the join is complete.

'/fasterposts/{postId}' : {
      GET : function(request, response, args) {
        response.sendHeader(200, {"Content-Type": "text/html"});
        var commentsPromise = getCommentsPromise(args.postId);
        var postPromise = getPostPromise(args.postId);
        var templateVars = {};
        commentsPromise.addCallback(function(comments) {
          templateVars.comments = comments;
        });
        postPromise.addCallback(function(post) {
          templateVars.post = post;
        })

        var joinedPromise = join([commentsPromise, postPromise]);
        
        joinedPromise.addCallback(function() {
          var postTemplate = tmpl['post-template.html'];            
          var pageHtml = postTemplate(templateVars);
          response.sendBody(pageHtml);
          response.finish();
        });
      }
    }
  }

This takes 2 seconds to complete since the RPCs are made in parallel, and the total time is just the slowest RPC (2 for fetching comments).

This method takes a special function, join(), to make it work.   join takes a bunch of promise objects and returns another promise that fires once all the other promises are complete:

function join(promises) {
  var count = promises.length;
  var p = new process.Promise();
  for (var i=0; i<promises.length; i++) {
    promises[i].addCallback(function() {
      if (--count == 0) { p.emitSuccess(); }
    });
  }
  
  return p;
}

Note that this example ignores stuff like errors, which make things even more complicated.  What to do with join when one of the promise objects fires an error instead of success?  Probably a good topic for another post in the future.

Also, I've been using Jed Schmidt's tmpl-node engine to render html in this example. Templating in Node.JS appears to be an active area of debate, but this one works fine for my purposes.

Note that one could also parallelize the rendering of the template as well, so the postPromise handler renders the html for the Post while commentsPromise is fetching/rendering comments. Then the join handler would stitch together the final html.

Sunday, November 22, 2009

Request Routing With URI Templates in Node.JS

I've been playing around with node.js, an asynchronous JavaScript server built on V8.

Node.js itself is pretty bare bones.  It's not a framework like Rails, but rather plain request-response handling.  It's sort of like Python's Twisted framework, from what I gather.

There are more full-featured frameworks for node.js if you look around on github, but I'm bored and feel like committing the sin of writing yet more framework code.

This morning I started with a request router for Node.JS that leverages URI templates*.  You specify the application as a series of request templates, paired with the functions that handle them.

For instance if you take the typical blogging application example, you might have a path like /posts/1234 - and the URI template would look like /posts/{postId}.

The magic is in turning {param}s in the URI template into parameters in the handler call.

Here's an example app that routes blog-like requests:

var handlers = {
  '/posts/{postId}' : {
      GET : function(postId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("GET Post ID: " + postId);
        this.response.finish();
      },
      POST : function(postId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("POST Post ID: " + postId);
        this.response.finish();        
      }
  },
  '/comments/{postId}/{commentId}' : {
      GET : function(postId, commentId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("GET Post ID: " + postId + " Comment ID: " + commentId);    
        this.response.finish();
      }
  }
};

As you can see the individual handler functions are further distinguished by HTTP method.

I'm not sure how to pass POST bodies to the handlers. They could just be attached to the handler's this I suppose.

Here's the full source:

var sys = require("sys"), http = require("http");

var handlers = {
  '/posts/{postId}' : {
      GET : function(postId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("GET Post ID: " + postId);
        this.response.finish();
      },
      POST : function(postId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("POST Post ID: " + postId);
        this.response.finish();        
      }
  },
  '/comments/{postId}/{commentId}' : {
      GET : function(postId, commentId) {
        this.response.sendHeader(200, {"Content-Type": "text/plain"});
        this.response.sendBody("GET Post ID: " + postId + " Comment ID: " + commentId);    
        this.response.finish();
      }
  }
};

var Route = function(uriTemplate) {
 this.uriTemplate = uriTemplate;
 var nameMatcher = new RegExp('{([^}]+)}', 'g');
 
 this.paramNames = this.uriTemplate.match(nameMatcher);
 // the regex keeps the {} on the param names for some reason. TODO: fix this.
 for (var i = 0; i < this.paramNames.length; i++) {
  this.paramNames[i] = this.paramNames[i].replace('{', '').replace('}', '');
 }

 this.matcherRegex = this.uriTemplate.replace('?', "\\?").replace(/{([^}]+)}/g, '([^/?&]+)');
 this.matcher = new RegExp(this.matcherRegex);
};

Route.prototype.parse = function(path) {
 if (this.matcher.test(path)) {
  var result = {};
  var paramValues = this.matcher.exec(path);
  // assert: paramValues.length == paramNames.length
  for (var i = 1; i < paramValues.length; i++) {
      result[this.paramNames[i-1]] = paramValues[i];
    }
  return result;
 }
 return null; //throw exception?
};

http.createServer(function (request, response) {
   var handled = false;

   for (pathTemplate in handlers) {
     var route = new Route(pathTemplate);
     var params = route.parse(request.uri.full);
     if (params) {
       // Convert the results to an array so we can pass them in via apply().
       var values = [];
       for (name in params) {
         values[values.length] = params[name];
       }

       var handler = handlers[pathTemplate][request.method];
       // So you can call this.request and this.response in the handlers.
       handler.apply({'request' : request, 'response' : response}, values);
       handled = true;
     }
   }

   if (!handled) {
     response.sendHeader(404, {"Content-Type": "text/plain"});
     var output = "Couldn't route: " + request.uri.full + "\n";
     for (name in request) {
       output += name + ": " + request[name] + "\n";
     }
     response.sendBody(output);
     response.finish();
   }
}).listen(8000);

sys.puts("Server running at http://127.0.0.1:8000/");
The route lookup in the http.createServer could be a lot more efficient, like memoizing Route objects for instance.

Anyways, NodeJS looks pretty exciting. Combined with CouchDB you could have a full JavaScript application stack: from storage to app server to client.

*Yes, I realize this is not a full implementation of the URI template spec.  It's just a proof of concept.

Saturday, September 12, 2009

Thoughts on Processing for the Web

Mozilla's Processing for the Web aims to leverage processing.org's low learning curve and expressiveness, but abandon the JVM dependency.  This is probably for the best as Processing is a great learning and experimentation environment, but applets are still clunky today well over ten years since they were introduced.

My personal felling is that processing's Java roots hinder it slightly in the learning curve department due to its static typing system.  Collections are particularly messy since processing doesn't do parameterized types, so you end up with lots of explicit casting if you're doing anything interesting at all.  JavaScript is much more lax in this regard and has language-level associative array support.  I know my own sketches would be much easier to work with in JS than in the current Java based Processing language because of this.  I create lots of classes to represent objects in my sketches and it's a pain to stuff them into and cast them back out of java.util.ArrayList and java.util.HashMap.

I met John Resig (author of jQuery and processing.js) when he came to speak at Google about JavaScript performance several months back.  I mentioned my backport of processing.js to processing to him, and he chuckled and said something to the effect of "Yeah, processing wasn't ready for the web."

It really wasn't, but I'm not sure that was a mistake.

You can do some really neat things with processing, specifically hardware controller interfaces, that you can't do in a browser without some serious compromises in the browser security model.  It would be a shame if Processing for the Web drew all the attention and developer resources away from the original Processing, but I doubt that will happen (soon, if ever).  P4Web will probably bring in new developers who wouldn't have touched the original Processing in the first place.

What I would like to see from this development is an evolution of the (JVM based) Processing language itself to be more JavaScripty. Perhaps P4Web will help nudge it in that direction.

Thursday, July 16, 2009

Thoughts on Leaked Twitter Docs

I couldn't stop myself from reading Tech Crunch's leaked Twitter docs. [disclaimer: Arrington is a total douche, but you knew that already.]

The more I read, the more I wanted to know. At the same time, I feel like I really shouldn't be looking at it. A delicious but guilt-inducing infosnack.

The feelings of uncontrollable curiosity reminded me of another incident involving leaked data: the AOL Search Data Scandal of 2006.

Fun Fact: Abdur Chowdhury, the researcher responsible for the AOL data leak back in 2006, is now Chief Scientist at ... Twitter.

Not that that's anything more than a coincidence, of course.

Wednesday, April 22, 2009

How to MapReduce in JavaScript Like a Pr0n Star With CouchDB

<soapbox> While I can appreciate the attempt to add some color to a technical presentation, at the risk of sounding prude there are ways to entertain a professional audience that don't involve pr0n references. </soapbox>

Here's a possibly NSFW presentation on CouchDB (Blogger *really* needs an after-the-jump feature :)



The tidbit that caught my eye though is the slide on how you build Views on CouchDB: CouchDB's Views are specified by JavaScript map/reduce functions, which is kinda cool. Here's a nifty interactive demo seasoned with jQuery.

jQuery meets MapReduce. The JavaScript Singularity approaches.

Sunday, April 19, 2009

Alex Payne Speaks at Stanford

Watch the whole video here

Random notes, taken as I watched and listened to Twitter's API Lead Alex Payne talk about API as UI, and various Twitter details:

The fact that in many schools, a student's first programming experience is with VB6 might explain why so many kids are turned off by by CS. If they were exposed to APIs with more thought put into usability they might be more likely to pursue CS.

API to HCI: millions of computer users are developers, and the way they interact with their computers is via APIs. API == super geeky UI.

100's of Millions of requests per day to the Twitter API.

Oooh Tweetie for OSX. Nice.

Twitter API support collaborates on one twitter account using CoTweet.

StockTwits: many of these users never go to Twitter.com after registering. StockTwits is how they interact with twitter.

Hardware Hack: BakerTweet: a baker in london turns the dial to "cupcakes" and hits a button to tweet that cupcakes are ready.
Hardware Hack: Kill-a-watt

inconsistency upsets developers
- actions that can be interpreted multiple ways
- don't provide UI conventions - devs have to use the twitter website first to understand what to do with the data.

java.util.Calendar/Date is indeed a fucking awful API.

import in Scala lets you do "import java.util.{Calendar, Date}" - Mmmmm love that syntactic sugar.

Yelp has an API? hehe aww how cute.

Yelp Docs don't match the actual API. Examples use single-quoted strings in JSON, which isn't valid JSON.

Yelp API puts response codes into the JSON payload instead of using HTTP headers to indicate OK, ERROR and so on.

! in Scala can mean either the traditional boolean NOT operator, or "send the following message to this actor": actorObj ! "message"
This is somewhat ambiguous. Al3x: "You get used to it." If it's okay for Scala why is it not okay for Win32 APIs? Many developers have just gotten "Used to it" in Win32. This is a nit, and I see his point.

REST vs. SOAP, WS-*: SOAP does a lot for the programmer: machine readable API definition -> auto-code generation.
REST doesn't write any code for you, but is intuitive enough not to require that.

"In my experience programmers don't like having details hidden from them" - but encapsulation and information hiding are fundamental to OO, if not API design.

"Smalltalk never really took off as a programming language" I know some pros who sold a shit ton of Smalltalk to banks in the 1980/90s who would disagree.

High traffic Twitter API users have requested Thrift access instead of REST API access. This makes total sense and I expect to see more of it.

"what reason do you have for not offering your own url shortening service, or not including URLs in the message itself?"
Al3x: "we don't have a good reason, which is why how we deal with urls is going to change" This sounds like twitter is going to offer its own URL shortener. Can't tell from his comments, but they are just dropping a lot of out-click data on the floor currently.

Al3x: The people who build the most interesting stuff bug us the least on mailing lists.

I have to wonder if in the future, as wifi becomes more prevalent, will Twitter : SMS :: Blogger : FTP ?

SignUp API coming soon?

how much did CNN pay for @cnnbrk? (would indicate $ value of followers to CNN)

>90% of their traffic is to the API rather than the web UI.

Wednesday, March 4, 2009

Map Reduce in JavaScript pt. 2

About two months ago I discussed how to run map reduce in JavaScript, closing with:

If only there was a vast sea of computers, all running javascript interpreters, all connected to the internet, all capable of downloading and running your m/r jobs. :)


Seems I wasn't the only one thinking along these lines. See igvita.com: Collaborative Map-Reduce in the Browser.

While I still like the idea (it makes use of vast untapped resources), there are some fundamental problems.

Running this kind of job over the open internet instead of a fast local (and secure) network is asking for trouble.

Sabotage: Forget accidental corruption. Workers can intentionally poison your jobs if they have an incentive to. Suppose you want to use this m/r setup to produce a spam classifier. Spammers could set up "workers" that submit bogus results that bias the filter to let their spam in.

How do you know you can trust a worker? (It's much easier to answer this if you're running your map reduce on a fast, secure local network.)

Economics and Speed: Map Reduce works on large data sets. Datasets that will cost significant amounts of money to move back and forth across the internet. Even if you use Amazon S3, the JavaScript has to post the interim results back to your server (unless you want to post your s3 key inside the worker JS, which I doubt ;) so you're paying for that bandwidth from your hosting provider in addition to whatever S3 charges. Even if the dollar cost is not an issue, you're talking about some really slow total run times since your slowest worker is much slower on the open internet than it is on a set of machines you control.

How do you decide if the cost (in time and dollars) of running the job justifies the value of obtaining its results? (It's much easier to answer this if you're running your map reduce on a fast, secure local network.)

So until someone does a lot of legwork to sort out the basic m/r infrastructure and then tackles the additional problems introduced by running on an open, slow, expensive network connection, the JavaScript MapReduce over HTTP idea is just a (admittedly fun) toy.

Recommendation: if you need to crunch a data set, use Hadoop. If you want to demonstrate Feats of Technical Strength Regardless of Utility (as I often do), try JSMRHTTP.

JSMRHTTP is not very catchy. There must be some other name for this concept.

Wednesday, January 28, 2009

CS4 Master Collection Dead Drop

Adobe Platform Evangelist Lee Brimelow presented a pretty awesome challenge, CS4 Master Collection dead drop:

This is your chance to get a free copy of CS4 Master Collection which is valued at over $2500. I have long been a fan of spy movies and the various aspects of tradecraft that intelligence agencies use. With that being said I have created a dead drop which now contains the software. Watch the video below to get all the clues you’ll need to find the drop.

I just heard about this a from my friend, E. He's the one who actually picked up the drop last night. Awesome. What are the odds? He just happened to be:

  • Geeky enough to be reading flash blogs

  • Reading the post soon after it went live

  • Sitting 5 miles away from the drop site when he read about it

  • an Eagle Scout, totally at home in the great outdoors at night


The last one is important because at least one of the commenters got close, but then got lost.

That's like the definition of opportunity: When luck meets preparation.