Pages

Tuesday, September 21, 2010

CSS3 Star Wars Tweet Scroller

I've been playing around with some new CSS3 3-D transforms and whipped up this Star Wars scroller for Tweets:


Caveat: This currently only works on Safari.  It may work on a bleeding edge Chrome dev build but I haven't tried that.

There are some other examples of this type of scrolling effect out there but I wanted to make one for tweets.  I also noticed that there are some odd effects if you naively keep appending content to a div that's been rotated in 3-D.  As it reaches a certain height, webkit apparently starts downsampling (I guess to save memory in the rendering pipeline?  A limitation of hardware?  Who knows) and you get this pixelated effect:

This isn't a downsampled down screen shot.  It actually looks like this if you let the rotated div get too large.
As you can see, it becomes unreadable after a while.  It also starts eating up an increasing amount of CPU: it got to 100% (out of 400%) on my quad core iMac after a few minutes.  Not good.

So instead of putting all of the tweets into the same rotated div, I rotate each tweet element individually and that seems to fix the problem.  Also, I start deleting tweets at the top (where you can no longer see them anyways) after a while, just to keep the dom size down.  They are now readable and the CPU is happy.

Other stuff:

Instead of using CSS3 animations, I opted for javascript because I was loading the tweets with JS already and I'd have to manipulate the animations with JS anyways.

Also, the background is generated dynamically using the Canvas element.  I wanted a more realistic looking star field so I did some research on the distribution of star sizes and colors to see if I could perhaps simulate it with a Pareto distribution.  Turns out the distribution of star colors and brightness is some other weird distribution so I just made it more or less random, with smaller stars slightly more frequent than larger stars.

Saturday, August 21, 2010

λ as a variable name in JavaScript

JavaScript is a functional language, or at least has very functional roots we could all probably agree.

Functional languages are based on lambda calculus and the CS literature often uses examples where a lambda expression is called simply, "lambda."

In JavaScript, one common use of lambdas is when you register a callback function with some asynchronous operation like an ajax request. Another is for operations on collections of objects like visiting every item in an array.

I wondered if one could use the lambda character, λ, as a variable name in javascript. It would be more compact (one character instead of five six) and (more importantly) would look cool. Turns out you can do this!

See it run here.



Notice that the character encoding is set to UTF-16. If you don't do this you'll get errors. Since the encoding is set that way, I don't even use html entities like λ to write λ :)

Note: I haven't tried this in any version of IE but it appears to work in Chrome, Safari and Firefox.

Now, how do you actually type λ into a text editor? That took some digging. Here's what I did on OSX:

  1. Open System Preferences -> Language and Text -> Input Sources
  2. Scroll down to "Unicode Hex Input" and make sure that checkbox is checked.
  3. Open the US Flag (or nationality of your locale :) icon in the upper right of the OSX menu bar and switch to "Unicode Hex Input" (the "U+" icon)
  4. Now whenever I want a λ I can just hold down the Option key while I type 03bb and blammo I get a λ. Which by the way is still five key presses but looks way cooler than "lambda" and takes up less space on the screen.

I have no idea what you have to do on linux or windows.

You're right, this isn't very practical. And setting your encoding from UTF-8 to UTF-16 like, doubles the bytes you have to transmit. But you ARE gzipping all of your http responses aren't you? AREN'T YOU?

Thursday, February 25, 2010

PubSubHubhub for NodeJS: Callbacks All the Way Down

NodeJS is a callback-based Javascript server API.

PubSubHubbub is a callback-based web protocol.

I put them together and the result is a PubSubHubbub client for NodeJS:

node-pshb on github


This project only includes a PubSubHubbub client interface, but to me that's the interesting part. You can specify an atom feed url, and functions to call when events happen on that feed due to PubSubHubbub.

The client library takes care of identifying the hub for that feed, requesting a subscription, and listening for subscription confirmation requests and feed updates from the hub.

It provides callback hooks for "subscribed", "update" and "error" events.

A simple client app looks like this:
var callbackPort = 4443;
var subscriber = new pshb.Subscriber(callbackPort);

// Start listening for subscription confirmation callbacks.
subscriber.startCallbackServer(); 

var topicUri = url.parse("http://localhost/foo"); // Dummy feed, always has updates

var feedEvents = subscriber.listen(topicUri);

feedEvents.addListener('subscribed', 
  function(atomFeed) {
    sys.puts('subscribed: ' + atomFeed.id);
  });

feedEvents.addListener('error', 
  function(error) {
    sys.puts('ERROR: ' + error);
  });

feedEvents.addListener('update',
  function(atomFeed) {
    sys.puts('got a PubSubHubub update: ' + atomFeed.id);
  });

I tested this out with the Demo Hub running on a local AppEngine launcher.

The demo app creates a a second server to host a dummy feed on port 80, so http://localhost/foo always returns a feed with the current time as it's "last updated." This is so the test hub always thinks there's an update ready for you.

So start the appengine with the hub running locally (in this demo it's assumed to be on port 8086), then run the test.js app, then go to your hub with your browser and manually update http://localhost/foo.

I noticed that I had to manually run some tasks in the hub's work queue so if you don't see any updates try checking the Task Queues in the app console for the hub. Run any pending "feed-pulls" and "event-delivery" tasks. I imagine there's a way to make them do that automatically but I haven't dug around enough in there to find it.

So there you go, NodeJS and PubSubHubbub: it's callbacks all the way down.

Friday, February 19, 2010

Webfinger Client for Node.JS

In a previous post, I demonstrated how you could use webfinger with nothing more than curl.   This post is about how you can use webfinger from nodejs with a non-blocking webfinger client.

Code for node-webfinger is here on github.

The project contains a simple webfinger-buzz.js command line app that demonstrates the webfinger client. It uses webfinger to find a google buzz feed based on a gmail address, then fetches the updates as an Atom feed, and then prints out the latest entry from that feed.

This could be generalized to support any other webfinger-enabled site like yahoo (though it looks like they're using an older version of XRD which my code can't parse :/).

The webfinger-buzz.js client looks something like this:
var sys = require('sys'),
  http = require("http"),
  url = require("url"),
  atom = require("./lib/atom"),
  webfinger = require('./lib/webfinger-client');

if (process.argv.length < 3) {
  sys.puts("usage: " + process.argv[0] + " " + process.argv[1] + " <user uri>");
  process.exit();
}
 
var userUri =   process.argv[2];
 
sys.puts("fingering " + userUri);

var wf = new webfinger.WebFingerClient();
var fingerPromise = wf.finger(userUri);
fingerPromise.addCallback(function(xrdObj) {
  var statusLinks = xrdObj.getLinksByRel("http://schemas.google.com/g/2010#updates-from");
  var statusUrl = url.parse(statusLinks[0].getAttrValues('href')[0]);
  var httpClient = http.createClient(80, statusUrl.hostname);
  var path = statusUrl.pathname;
  if (statusUrl.search) {
    path += statusUrl.search;
  }
 
  var request = httpClient.request("GET", path, {"host": statusUrl.hostname});
 
  request.addListener('response', function (response) {
    response.setBodyEncoding("utf8");
    var body = "";
    response.addListener("data", function (chunk) {
      body += chunk;
    });
    response.addListener("end", function() {
      var atomParser = new atom.AtomParser(false);
      var atomPromise = atomParser.parse(body);
      atomPromise.addCallback(function(atomFeed) {
        sys.puts("Feed: " + atomFeed.title);
        sys.puts(atomFeed.entries.length + " entries");
        sys.puts("Updated: " + atomFeed.entries[0].updated);
        sys.puts(atomFeed.entries[0].title + ": " + atomFeed.entries[0].summary);
      });
    });
  });
  request.close();
});

hehe fingerPromise. Is that a generalization of pinkySwear?

In the process of writing this webfinger client I used a couple of libraries I found on teh internets: sax-js by Isaac Z. Schlueter - a SAX parser for nodejs, and this URI Template library by James Snell. Both worked well and I recommend them.

The remaining non-webfinger-specific pieces I needed were an XRD parser and an Atom parser, both for javascript and SAX (as opposed to DOM). I couldn't find much in the way of those, so I rolled my own. They are included in the node-webfinger project in the lib/ directory. They're pretty crude parsers but they worked for this example. I'll probably use them in other projects in the future and make improvements as necessary. Unless something better comes along. That seems inevitable.

Monday, February 15, 2010

Bring the (Perlin) Noise

If you just want the source code: The JavaScript Perlin noise generator code is here.

I mentioned in a previous post that I was working on a Perlin noise generator for Art Evolver.

Perlin noise is a function of (x, y) that produces a random-ish pattern. It's not completely random because it has smooth hills and valleys, but the distribution of those hills and valleys is random.

A side note about this algorithm: Usually if you're a computer scientist and you come up with a clever algorithm to solve a particular problem, you get an award from a university, or a CS-centric professional organization like the ACM or IEEE. Perlin got an Academy Award for this noise function. As in, the Oscar kind of Academy Award.  For an algorithm, something not usually consider artsy.  I found that interesting.

Anyways, rather than dive into Perlin's impenetrable description of how the algorithm works, I set out to find an existing JavaScript implementation. That let me to this message board, and specifically this example.

Unfortunately that implementation has some pretty serious directional artifacts:

Note the horizontal and vertical stripes.  There's almost an upside down cross in the lower right. SAAAATAAAAN!

Rather than try to fix that source code (which the author apparently closureized (making it very difficult to understand)) I kept searching.

From the main Wikipedia entry on Perlin noise, I ran across a variant called Simplex noise.  This is an improvement on the original algorithm, also written by Ken Perlin, in 2001.  That Wikipedia page linked to a paper by Stefan Gustavson(pdf) that explains both classical and Simplex Perlin noise in a much easier to grok way than anything I've read by Perlin himself.  I highly recommend Gustavson's paper if you found Perlin difficult.

I took the Java source code in Gustavson's paper and ported it to JavaScript, and the results are here on github.

I ran some performance comparisons between the classical and Simplex algorithms, and for 2-D I only saw a ~10% improvement with Simplex.  Granted, the latter is supposed to be faster in higher dimensions (classical is O(N^2) vs. simplex O(N) where N is the number of dimensions) so it doesn't matter much for my purposes.

Classical Perlin noise

Simplex Perlin noise

Subjectively I think I prefer the Simplex noise to classical, so I'll probably go with that for Art Evolver.

Again, the source code is here.

Saturday, February 13, 2010

Curl-ing up with WebFinger and PubSubHubub

This morning I've been playing around with WebFinger and PubSubHubub. One of the great things about open web APIs is that you can tinker around with them without even writing an application. Just use curl!

Let's start with WebFinger. First, we need to figure out how to get my WebFinger data from gmail. There's standard place to look for that explanation, given a domain name:
http[s]://{domain-name}/.well-known/host-meta
So for gmail we get the explanation of how to get my WebFinger data like so:
$ curl http://gmail.com/.well-known/host-meta
Out pops an XRD doc that contains a URI template (in bold, below):
<?xml version='1.0' encoding='UTF-8'?>
<!-- NOTE: this host-meta end-point is a pre-alpha work in progress.   Don't rely on it. -->
<!-- Please follow the list at http://groups.google.com/group/webfinger -->
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0' 
     xmlns:hm='http://host-meta.net/xrd/1.0'>
  <hm:Host xmlns='http://host-meta.net/xrd/1.0'>gmail.com</hm:Host>
  <Link rel='lrdd' 
        template='http://www.google.com/s2/webfinger/?q={uri}'>
    <Title>Resource Descriptor</Title>
  </Link>
</XRD>
Substitute my email address for {uri} and curl it:
$ curl http://www.google.com/s2/webfinger/?q=banksean@gmail.com
That spits out another XRD that describes some other resources associated with my email address:
<?xml version='1.0'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>
 <Subject>acct:banksean@gmail.com</Subject>
 <Alias>http://www.google.com/profiles/banksean</Alias>
 <Link rel='http://portablecontacts.net/spec/1.0'
href='http://www-opensocial.googleusercontent.com/api/people/'/>
 <Link rel='http://webfinger.net/rel/profile-page' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://microformats.org/profile/hcard' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://gmpg.org/xfn/11' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='http://specs.openid.net/auth/2.0/provider' 
href='http://www.google.com/profiles/banksean'/>
 <Link rel='describedby' 
href='http://www.google.com/profiles/banksean' type='text/html'/>
 <Link rel='describedby' 
href='http://s2.googleusercontent.com/webfinger/?q=banksean%40gmail.com&amp;fmt=foaf'
type='application/rdf+xml'/>
 <Link rel='http://schemas.google.com/g/2010#updates-from' 
href='http://buzz.googleapis.com/feeds/103419049256232792514/public/posted' 
type='application/atom+xml'/>
</XRD>
Bolded above is the rel='http://schemas.google.com/g/2010#updates-from' URI for my public status updates. Let's fetch that:
$ curl http://buzz.googleapis.com/feeds/103419049256232792514/public/posted
<?xml version='1.0' encoding='utf-8'?>
<feed xmlns='http://www.w3.org/2005/Atom' 
xmlns:thr='http://purl.org/syndication/thread/1.0' 
xmlns:media='http://search.yahoo.com/mrss' 
xmlns:activity='http://activitystrea.ms/spec/1.0/'>
<link rel='self' type='application/atom+xml'
href='http://buzz.googleapis.com/feeds/103419049256232792514/public/posted'/>
<link rel='hub' href='http://pubsubhubbub.appspot.com/'/>
<!-- lots more feed data not relevant to this discussion -->
It's a standard Atom feed. Amongst a lot of other stuff in the beginning is a rel="hub" link. This is where PubSubHubub comes in. Suppose that I want some other service to be notified whenever I post a status update (for instance, I have an app that reposts it in the sidebar of my blog). I could poll this Atom feed but polling is pretty janky. With PuSH I can register a callback to be notified whenever I post an update. Since this feed has a rel="hub" link set to http://pubsubhubbub.appspot.com/, that's where I go to do register a callback.

If you just navigate to http://pubsubhubbub.appspot.com/ with your browser you get a form you can fill out to create a subscription. One of the fields is for a "Callback" url. I don't run any websites that know how to handle PuSH callbacks (or subscription confirmation, for that matter). Luckily there is a test subscriber on appspot that accepts subscription requests for anything: http://pubsubhubbub-subscriber.appspot.com/. To create your own callback URL for testing, just add /subscriber.{some_unique_identifier} to the end of it.

According to the PuSH spec, I should POST some form fields to the hub URL like so:
$ curl -v http://pubsubhubbub.appspot.com/subscribe \
-d hub.callback=http://pubsubhubbub-subscriber.appspot.com/subscriber.banksean\&\
hub.topic=http://buzz.googleapis.com/feeds/103419049256232792514/public/posted\&\
hub.verify=sync\&hub.mode=subscribe\&hub.verify_token=\&hub.secret= 
And that creates the subscription. Here's the verbose output:
* About to connect() to pubsubhubbub.appspot.com port 80 (#0)
*   Trying 74.125.19.141... connected
* Connected to pubsubhubbub.appspot.com (74.125.19.141) port 80 (#0)
> POST /subscribe HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: pubsubhubbub.appspot.com
> Accept: */*
> Content-Length: 219
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 204 No Content
< Cache-Control: no-cache
< Content-Type: text/plain
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Date: Sat, 13 Feb 2010 17:41:29 GMT
< Server: Google Frontend
< Content-Length: 0
< X-XSS-Protection: 0
< 
* Connection #0 to host pubsubhubbub.appspot.com left intact
* Closing connection #0

The 204 No Content response indicates the subscription was created and is active, according to the spec.

And if you go to http://pubsubhubbub-subscriber.appspot.com/ right now (Saturday morning, October 13th 2009), you'll indeed see a bunch of my posts on it.

Ta Da. No code. Just curl. I love the internet.