Presenting our CFO:

Animoto's Webby acceptance speech

Do I work for the greatest company on earth or what?

Animoto was asked to create a highlight clip for the Webby's (which we also got to accept an award from) and it turned out really well:

Highlight Reel 2009

The awesome team that I'm lucky to be a part of at Animoto has taken home an 'oscar of the internet' in this year's Webby Awards in the Services & Applications category. Go Animoto!

I can't thank Barry Hess and Andi Schacke enough for writing and commenting on this post and saving me from all the darkness and despair ;) I've experienced over the past few months as my Rails test times plummetted on this shiny new MacBook Pro. I recompiled as per Andi's comment and my times are now about a fourth what they were. Thank you both.

Revised Hivelogic instructions:

curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
tar xzvf ruby-1.8.7-p72.tar.gz
cd ruby-1.8.7-p72
./configure
make
sudo make install
cd ..

Jeremy McAnally - Machine Learning

Works for entp. His latest app: tweekapp

My Summary

Jeremy was out to convert everyone to doing DSLs the 'right way'. Some salient points were:

• the point of a DSL ought to be to make the DSL easier to use for the experts.
• 'external dsls' (i.e., you make up your own language) can be great for simple project, like config-writing code
• 'internal dsls' (e.g., ruby-code-dsls) get a lot of stuff for free (loop structures, etc)
• dsl composition (using more than one dsl at a time) can cause namespace issues (he didn't really offer much in the way of solutions here as far as I can remember)

DSLs

Used by experts, for specific problems.

Domain

Different problems have different domains, and they each have their own language.

Specific

DSLs that are very generic or look a lot like english don't make sense to him. DSLs are supposed to make the brain-to-code jump much smaller. Also, why waste the keystrokes?

External DSLs

These are DSLs that don't use an existing language (e.g. ruby). Sounds like he doesn't like them as much in general, though there are specific cases where the language structure is simple enough that it makes sense (e.g. puppet).

Internal DSLs

Implemented in an existing language (e.g. ruby). Loops & basic things like that are baked-in and already understood by end-users. External DSLs have to go to a lot of work to create these structures & etc. When to use? Look at the possible complexity of your code, look at what your 'experts' will use.

Examples:

• Rails Templates (written by him).
• Ambition - 'Take the suck out of SQL'
• Foundry (Another of his).

Designing the DSL

1. Think about the problem domain
2. Deconstruct -- remove the cruft. DSLs are not fake English. No Finglish allowed.
3. Implementation

Method chaining

Approach 1: return a new object every time

Approach 2: return self after changing state

Choosing an approach -- think about which is more concise in your case, also think about how #2 can get ugly & unwieldy.

Dynamic dispatch

He likes using closures. Some extra chrome -- if that's bad maybe look at an external DSL.

Compositon - meshing DSLs. Presents namespace issues.

Decoupled DSLs

In his opinion, the DSL ought to be decoupled from the library behind it. The DSL is just the interface to the logic. Thus, when testing you test them separately.

Kirk Haines - Vertebra

Kirk is working for EngineYard on Vertebra

Jordan's Summary

Vertebra is a solution from EngineYard to manage processes in the cloud. Still in development (no production usage so far). Sounds interesting, will probably take a look at their source code if for nothing else than an interesting architecture. Uses XMPP, which I seem keep hearing about these days.

Cloud Computing

Clouds are great b/c it's many, many machines waiting to do you bidding.

Clouds are hard b/c there are many, many machines to manage.

Scripted SSH works, but he says it has its limits. Vertebra built to be 'fault-tolerant operations among autonomous agents'.

Enter Vertebra

Built on XMPP.

• very chatty protocol
• upside to chatty-ness is fault tolerance. back-and-forth paints the picture of message reception.
• goal is: if network dies, operation in progress should be able to pick right up where it left off

For an XMPP Server they use EJabberd

The pieces of Vertebra are agents. E.g. a logger agent.

Special Vertebra Agent: Herault

• originally written in ruby, rewritten in erlang for performance & scalability
• new agents coming on board advertise their capabilities to Herault
• when looking for capabilities ('show me all loggers') you ask Herault
• also handles authorization for dispatching operations
• when asking Herault to dispatch an operation, can specify :all or :single (more options in the works)

Concurrency API

Talking about using vertebra to schedule future units of work that may or may not execute depending on a given set of conditions.

Yehuda Katz - The Great Rails Refactor

Yehuda is working for EngineYard, was working on merb, is now working on rails.

My Summary

He says he's talking more about the future of rails & less about what-came-from-merb these days.

Yehuda's presentation really seemed to be centered on componentization of the entire Rails stack

• Pluggable persistence layers (ActiveRecord, DataMapper, Sequel, CouchDB, etc)
• Using and becoming middleware
• Making rails' ruby modules less tightly coupled with each other
• Replaceable rails components (ActiveRecord, ActiveSupport, Action)

He ended talking about how all these areas of componentization will open up new possibilities of competition (and thus greater health & growth) for the whole Rails ecosystem. An interesting thought in juxtaposed with the elimination of Merb vs Rails competiton. I'm a bit skeptical of whether this can be really pulled off, but I'm excited to see what the brilliant minds behind all this produce.

ORM Agnosticism

Possible in rails right now, but things like form_for ought to work also.

There were two interfaces that merb switched on: ActiveRecord/Datamapper or Sequel. Not too bad but not ideal.

The solution: ActionORM (Lori Holden) - a way of making any ORM look like ActiveRecord.

Two options going forward for compliance for ORMs

• mimick AR
• provide a proxy that supports the key methods

A few core things will be covered (like form_for) but it will be up to the community to create extensions for the rest. It's hard for them to ascertain what's going to be useful up front and they want to move forward quickly.

Rack

Some middleware they used in merb: PathPrefix, ConditionalGet, Static

Static - can use for tweaking static asset handling for dev mode (and turn off in prod)

Rails middleware usage: Session middleware, Param parsing, Failsafe

This is all working toward amore unix-like setup, requests come in and get routed to pieces (Sinatra, Rails, etc) which are not special -- they just know how to handle the same thing.

Callbacks

He worked on improving callbacks a lot. His first analysis of Rails showed callbacks taking 20% of the time for a request (admittedly a simple request). A bit on the history of callbacks in Rails: there was no unified callback system originally -- lots of different ones. Then it was refactored into callbacks.rb. But it didn't solve all the needs of all the pieces and thus there was a lot of reaching-into-and changing going on that added complexity and slowness. Yehuda went through & made a single calback system that served all needs. Improved actual calling of callbacks - went to 'compilation' instead.

On complex code: Sometimes it's ok to have a complicated bit of code, as long as your abstraction is leak proof & well-abstracted. E.g., nobody cares that ruby is complicated internally because it's done well enough that nobody has to go digging through the C code.

One big problem w/ current rails code -- poorly named (confusing) instance variables make it hard to read & understand.

Upcoming: Abstract controller. Will be module based (truly module based, not tightly coupled modules like rails currently has).

Ruby tip: If you include a module in a class you can super from a module method. Much better than alias method chaining, which can get really messed up if things aren't chained in the right order.

About the plugin api -- there are lots of methods in Rails that are not for everyday use, but are available for plugin authors. These are methods that they don't want to clutter the API docs with, but they will stand by them as far as keeping them working the same way. Sort of a 'hidden' but valid public API. Still thinking about how to provide plugin API for the future of Rails to avoid all the crazy monkey patching that currently goes on.

Upcoming: Debug Toolbar like Django's, but more pluggable.

Orchestra -- gets the pieces working together. Hopefully will make it a lot easier for people to add instrumentation (especially specialized instumentation) to Rails, not only for fiveruns & etc, but for normal end users.

On Competition

Yes, the competition between Rails & Merb is gone. However, Yehuda says that they've really eliminated a less-useful competition in order to open up a more useful one. That is, competition on the component level. The want to make the pieces of Rails compete instead of the entire framework compete. E.g., Sequel vs ActiveRecord, your own ActionController, etc.

Jon Crosby - Middleware

getcloudkit.com
contributor to rack-contrib project
working at Engine Yard on Engine Yard Solo

CGI

Remembering the bad old days...

Things we maybe could pull out of our apps

• authentication
• caching
• openid + oauth

Rack

http://github.com/rack/rack-contrib
based on http model of having layers
based on wsgi (web server gateway interace)
Interesting -- Rack::CSSHTTPRequest
Rack::Cache
Check out his cloudkit - intersting out it builds up rack items w/ oauth, openid items & etc
Middleware announcing itself - Middleware can announce it's presence via env headers
Adding sinatra as middleware before Rails

Comment from Yehuda: they are currently working on rails as middleware stuff

James Edward Gray, II

If you want to learn Ruby you should read code, lots of code. He estimates that an 'expert' ruby programmer would read more code than he writes.

Examples of good code libraries

Restclient

• clean interface to restful services
• this is networking code done right
• quote he likes: 'Everything in a method should be written to the same level of abstraction'
• logs from restclient are executable ruby code

bj

background priority queue for rails

terminator

robust thread/pipe/etc level code written by an expert

slave

trivial multiprocessing with built-in ipc

The art of code reading

• when you start reading, relax. not all code sucks & you probably won't understand the author's reasons at first
• don't start with Rails (too big & complex)
• have a goal
• get the source code open in your standard text editor (e.g. use gem unpack or get a copy from a vcs)

Two videos from our family to yours. :)

(from Maik Schmidt's Enterprise Recipes with Ruby and Rails)

"Back in the mid-90s, an experiment started as a way to make average developers more effective, because the demand continued (as it does today) to outstrip the supply of good developers. If the software industry can figure out a way to make mediocre developers productive, software development can expand to enterprise scales. Thus, we saw the rise of languages like Visual Basic and Java and later C#. These languages were specifically made less powerful than alternatives (like Smalltalk). The goal of the Lockdown Experiment: make tools to keep average developers out of trouble while still being able to write code. But then a couple of interesting things happened. First, creating restrictive tools and languages didn’t really keep average developers out of trouble, because average developers sometimes apply great ingenuity to coming up with ridiculously complex solutions to problems. But while this didn’t really make the average developers better, it put a serious governor on the best developers. The whole industry seemed to be optimizing for the wrong thing: safety at the expense of power, with the stated goal of creating software faster. Yet, we didn’t produce software faster; we just annoyed the best developers. The second effect was this new wave of languages was so restrictive that they immediately had to start supplementing them to get real work done. For example, in the Java world, the second version added a bunch of new features (like anonymous inner classes), and eventually some limited metaprogramming was added to Java via aspect-oriented programming.

The real underlying problem with lots of "enterprise languages" is one that Stuart Halloway of Relevance software summed up brilliantly: ceremony vs. essence. Languages that require you to jump through hoops to achieve results are highly ceremonious, whereas languages that make it easy to do sophisticated things are more essential. At the end of the day, you have to solve problems. You want languages and frameworks that lessen the distance from intent to result. Ceremonious languages sometimes make that distance quite far, requiring lots of work that doesn’t really move your solution forward. More essential languages get out of your way, making the distance from intent to result shorter. "

"True happiness comes only by making others happy—the practical application of the Savior’s doctrine of losing one’s life to gain it. In short, the Christmas spirit is the Christ spirit, that makes our hearts glow in brotherly love and friendship and prompts us to kind deeds of service." ¹

Had a lovely time trying to figure out why my flex widget couldn't talk to our sever via https. Turns out that (even with a crossdomain.xml file) a swf served from http cannot access https, unless you add an extra special parameter of 'secure="false"' to the crossdomain file. I really wish flash returned more helpful error messages than 'Security Error'.

http://livedocs.adobe.com/flex/3/html/help.html?content=security2_15.html

We're using this to allow secure communication from our non-https page for some ajax login & fetch behavior. Using the flash widget as a proxy since same origin policy for javascript prohibits just about everything if you need a secure communication w/o having the whole page in https. Ajax requests are prohibited, the script-tag hack doesn't work (login params would have to go (unencrypted) in the url), and iframes suffer from the same problem. Google uses the iframe trick on some of it's pages --

http://www.google.com/shoppinglist?hl=en

(make sure you're not logged in)

but it seems that that only works because they redirect the whole page when successful, which we didn't want to do. Looks like the flash widget will work.

Doing some fun stuff with named_scope this morning led me to some surprising discoveries in Ruby's handling of blocks/closures/etc.

In Ruby, I had thought that lambda was supposed to do arity enforcement and that Proc.new wasn't (and also that return behaves differently in Proc's vs lambda's) and that this was the extent of the differences. See here.

However, I was getting a "warning: multiple values for a block parameter (0 for 1)" warning when doing this:

  named_scope :ordered, Proc.new {|o| {:order => o || "created_at DESC"}}

and I then also discovered that the following also worked (with the same warning) whereas I thought it would have raised an exception because it was lambda:

  named_scope :ordered, lambda {|o| {:order => o || "created_at DESC"}}

My coworker Jeff found this fascinating and revealing investigation into Ruby closures where he reveals "four types of closures and near-closures, expressible in seven syntactic variants." Yikes.

However, I think the chart he has about half way down is either outdated or a little wrong. Here's what I came up with after a little investigation of my own. Perhaps I'll write down a test for these at some point to test future Ruby versions (and to make sure I got the chart right). But for now --

                                                    "return" returns from closure
                                   True closure?    or declaring context...?         Mismatched arity behavior
                                   ---------------  -----------------------------    -------------------
1. block (called with yield)       N                declaring                        Behavior 1
2. block (&b => f(&b) => yield)    N                declaring                        Behavior 1
3. block (&b => b.call)            Y except return  declaring                        Behavior 1
4. Proc.new                        Y except return  declaring                        Behavior 1
5. proc                                    <<< alias for lambda in 1.8, Proc.new in 1.9 >>>
6. lambda                          Y                closure                          Behavior 3
7. method                          Y                closure                          Behavior 2

* Behavior 1: Succeed (supply nil for missing args and ignore extra args).  Exception: When block is defined to accept exactly 1 parameter then warn on any mismatch
* Behavior 2: raise ArgumentError on mismatch 
* Behavior 3: same as Behavior 2 except for the following:
    - Same as Behavior 1 if block has exactly one parameter, e.g. -- lambda{|a| puts a}.call # warn and succeed
    - Same as Behavior 1 if block was defined w/o argument pipes, e.g. -- lambda{puts 'hi'}.call(1,2) # succeed

It's a crazy Ruby world :)

I'm working on an Adobe AIR cross-platform app at work. We're providing a physical CD for distributing the application and I wanted the CD to work like a real Mac CD should look like -- Background image in the finder, hide all extraneous files, auto-open the finder window when the CD is inserted, etc.

Getting just this to happen requires some trickery, and getting it to happen while also putting an auto-launching Windows installer on the same disc proved to be even more of a game.

After a lot of googling I think I've got a pretty good process down and wanted to share, so here are the details.

Core Concepts

• The standard ISO CD filesystem doesn't let you do all the good stuff that you'll want to do for Mac (finder background image, hiding files, etc). You need to use some variant of HFS. However, Windows can't read HFS.
• 'Hybrid CDs' solve the previous problem. From what I understand, a hybrid CD contains an HFS volume and a standard ISO volume. Windows machines can't see the HFS volume and OS X machines ignore the ISO volume when there is an HFS volume present.
• You'll need the OS X Developer tools for these steps
• From what I understand, Roxio Toast 7 & above make this much easier, but this is more fun and it's free. :)
• It is much easier to test all this out in Windows if you have Parallels (or presumably Fusion) because Parallels let's you mount an iso in Windows as if it were a CD, so you don't have to waste your precious blank CDs creating actual copies. :)

Steps

1. Create a standard folder on your harddrive and put all the install files (windows and mac, including an autorun.inf for your Windows installer, presumably) in it. We'll call it 'install folder' for now.

2. Make a user-friendly finder window

   To make a nice OS X finder window for a CD (or dmg or whatever) we want to have a friendly background image that helps the user understand what they're supposed to do. Go grab AdiumX to see an example of what I mean. We have to include this image on the CD, but we also want to hide it from the end user so that it doesn't clutter up our install window. Here are the steps:

   1. Create your background image.
   2. Create a folder called 'background' inside 'install folder'.
   3. Put your background image into this folder.
   4. Use terminal to rename this folder to '.background'. e.g.: mv background .background (The folder should then disappear from view in the Finder)
   5. Open 'install folder' in the Finder and hit Command+J. Make sure that 'This window only' is selected. Click 'Picture' for the background. In the dialog that comes up, hit command+shift+g to bring up a box where you should enter the path to your '.background' folder. This allows us to navigate there even though it is hidden. Select your background image and click OK.
   6. Click 'Picture' again for the background. For some reason it gets reset after selecting the picture.

   7. For each file that you'd like hidden in both windows & mac (e.g. perhaps the window autorun file), run this command in Terminal:

      /Developer/Tools/SetFile -a V /path/to/your/folder/name-of-file-to-hide

3. Get rid of the finder toolbar (click that top-right oval button) and resize your window & arrange your icons, leaving the windows icons clumped together somewhere for now (we'll hide them in a minute)

4. Create the hybrid ISO using hdiutil Run this command in Terminal to turn your folder into a hybrid ISO:

   hdiutil makehybrid /source/folder/name/ -o outputfile.iso

5. Mount the ISO in read-write mode: Run this in terminal:

   hdiutil attach -readwrite outputfile.iso

6. Make files invisible to Mac ONLY (e.g., your windows installer file) by setting the invisible bit on them: Again in terminal:

   /Developer/Tools/SetFile -a V /path/to/your/folder/name-of-file-to-hide

7. Make the finder window auto-open when the cd is inserted/image is mounted: In Terminal:

   sudo bless -folder "/Volumes/discName" -openfolder "/Volumes/discName"

8. Test it out. Mount your ISO and unmount it a couple times to see the wonderous installer. If you have Parallels you can try it out in Windows too by using 'Connect Image' to simulate it as a CD in Windows

9. Open Disk Utility & drag your iso file into the left-hand pane. Click 'Burn' in the toolbar above. Insert a blank CD and let it burn.

Done! Try inserting it into a Mac and a Windows machine and be amazed by the beauty of it all.

Derek Wischusen showed a cool technique for simulating Ruby's Mixin's in Flex here. I started trying it out and was loving it until I found out some details about event listeners that were show-stoppers for me. Well, after learning more about Prototype recently (with it's very closely related javascript/ecmascript) I solved the problem and am throwing this code up here in case anyone else finds it useful.

The Problem

Derek's code allows you to add code to your classes by 'mixing' in other code. One way to add this 'other code' is by adding functions to the prototype. (see Derek's post for details) Unfortunately however, when functions are added in this way and used as event listeners, the 'this' keyword inside the function will refer to the global object instead of the object you meant it to be attached to. (see 'Programming Actionscript 3.0' pg 361 http://www.adobe.com/go/programmingAS3_pdf). This means that your listener function becomes essentially static to you and can't reference all the data you want it to.

The solution, it turns out, is to resurrect Mike Chamber's old idea of Delegate.create. This static method was used like so:

myComboBox.addEventListener("change", Delegate.create(this,onMyComboBoxChange));

And it ensured that when 'onMyComboBoxChange' was called, the 'this' keyword would refer to the 'this' from the 'create' call (or whatever obejct you placed there). Delegate.create was apparently dropped in ActionScript 3.0 because they thought we didn't need it anymore (because they added method closures which are automatically bound to their object and they don't like using the prototype anymore I guess, silly them).

However, since actionscript anonymous functions are closures, you can do the same thing that the Prototype framework does (which almost the exact same code) to add a 'bind' method to the Function class so that you can get the same result as above with this code:

myComboBox.addEventListener("change", onMyComboBoxChange.bind(this));

Viola! Back to the land of having fun with Derek's mixin class.

Here is the code (someday I will extend it to be more general like Prototype's code, but for now all I need it for is simple event listeners):

Function.prototype.bind = function():Function {
  var __method:Function = this;
  var object:Object = arguments[0];
  return function():void {
    __method.apply(object, arguments);
  }
}

Just wanted to thank and put out a link to RadSoftware's awesome (and free) Regular Expression Designer. Best free .NET RegEx tool I've tried.

Regular Expression Designer

I noticed an error in my rails log today that said:

"WARNING: You're using the Ruby-based MySQL library that ships with Rails. This library is not suited for production. Please install the C-based MySQL library instead (gem install mysql)."

Unfortunately, when I went to install the gem I got an error going something like this:

Failed to build native gem extension ... * extconf.rb failed * Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers.

I am using the MacPorts install of MySQL. It took a little leg work but I found a post that led me to use the following command, which installed the gem on my Intel Mac successfully.

sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

90%, 90%, 90%!!!

Background:

Windley on DSLs

Atwood on Languages Inside Languages (e.g.: Using multiple DSLs at the same time)

Rob Conery in defense of (his) ORM

Jeff, why the SubSonic hate? Rob, why the Regex hate? Why not use them where they're appropriate?

1) I want SubSonic to turn 90% of my Database needs into easy one-liners.

2) For everything else, I want SubSonic to get out of my way.

SubSonic does a pretty good job of both things.

Beautiful:

Order o = Order.FetchByID(orderID);
foreach (Product p in order.Products)
  doSomeCoolThing(p.price, p.weight, p.title);

(plus intellisense and compile-time errors)

Waste-of-my-time:

IDataReader rdr = QueryDb("SELECT * FROM Products WHERE OrderID = '" + " sqlEscaper(orderID) " + "');
while (reader.Read())
  doSomeCoolThing(
    reader.GetFloat(reader.GetOrdinal("price")),
    reader.GetInt32(reader.GetOrdinal("weight")),
    reader.GetTitle(reader.GetOrdinal("title"))
  );

(yikes.)

That right there is 90% of the SQL queries for 90% of DB applications out there. Look at how much more beautiful SubSonic can make the world.

Jeff is right though, people get their hands on a tool like SubSonic and suddenly think it will do their laundry and solve world hunger too. SubSonic's fluent interface doesn't cut it for complex data queries (and is often ugly when you try). But it shouldn't try to cut it, and it doesn't.

When you want to do complex data queries, drop down to SQL, or wrap your queries in Views, etc. Either way, you gotta know SQL, so don't try to hide it in a Kingdom of Nouns.

Jeff shouldn't be so down on SubSonic -- it is actually more of what he's talking about: It's a Domain Specific Language itself (for the most common queries a DB application makes). Why not let it play along with SQL & Regex too? Just don't try to use it for what it's not good at, just like with SQL & Regex.

Domain Specific Languages are fun and useful. SubSonic is fun and useful. They both need to stay out of each other's territories. Why the hate?

Last note on Regex's: When you want to parse something out of a string, use a short, concise Regex. People usually hate Regex because they get so excited by it that they try to write a single indecipherable Regex string that encapsulates the complete parsing structure of a entire programming language. Bad idea, but they blame it on Regex. You don't need an IDE for Regex's, you need to divide and conquer and separate your concerns, just like you approach every programming challenge in your favorite language: Wrap your whole program into one method and you will die. Period.

My previous post about not having profiles load is now thankfully obsolete, due to the latest Capistrano 2.1 update.

See here

Say the notes:

No default PTY. Prior to 2.1, Capistrano would request a pseudo-tty for each command that it executed. This had the side-effect of causing the profile scripts for the user to not be loaded. Well, no more! As of 2.1, Capistrano no longer requests a pty on each command, which means your .profile (or .bashrc, or whatever) will be properly loaded on each command! Note, however, that some have reported on some systems, when a pty is not allocated, some commands will go into non-interactive mode automatically. If you’re not seeing commands prompt like they used to, like svn or passwd, you can return to the previous behavior by adding the following line to your capfile:

default_run_options[:pty] = true