While trying to verify some Google Analytics tracking on Animoto's iOS app I discovered that the GA iOS SDK (at least version 1.3 that we're using) makes itself hard to monitor because it doesn't use the HTTP Proxy configured for the iOS device when it makes its tracking calls, so I can't use a debugging proxy like Charles or Fiddler to observe the traffic. i.e., this method doesn't work.

Instead, I have to share my ethernet internet connection to my iOS device wirelessly and then use a tool like ngrep or Wireshark to monitor the traffic going through the wireless interface (usually "en1").

How I verified:

Using two laptops, I shared my ethernet connection from my first laptop to both my iPod Touch and to my second laptop. I started Charles Proxy on my second laptop and configured my iPod Touch to use it as its HTTP Proxy. Like this:

Using Wireshark to inspect network traffic on both laptops I saw most http traffic coming through on both laptops. However, the GA traffic was showing up only on the first laptop. So, for some reason the GA iOS SDK isn't using the iOS-configured HTTP proxy to send its traffic. I'm not sure if this is a bug or by design, but it does make things harder to monitor.

Interestingly, even when you run the app through the iOS simulator via Xcode, the iOS simulator also refuses to use the OSX-configured proxy for GA traffic (i.e., you still can't use Charles).

So to monitor GA traffic from an iPhone app you have two options:

1. Using one Mac and one iOS device, share your ethernet internet connection from your Mac to your iOS device
2. If you have the source code, you can use the iOS simulator via Xcode and the simulator will be using your Mac's internet connection.

And in both cases you need to use ngrep or Wireshark on your Mac to monitor the traffic.

Some discussion on StackOverflow.

example:

cap download_file FILE=/some/remote/file.ext DEST=/local/save/path.ext

Downloads files from the remote servers and saves them as "/local/save/path.ext-#{hostname}" locally.

code:

desc 'Download a file.  Local filenames are postfixed with the origin server hostname.'
task :download_file do
  abort "Please specify a file or directory to download (via the FILE environment variable)" if ENV["FILE"].to_s.empty?
  abort "Please specify a destination file or directory (via the DEST environment variable)" if ENV["DEST"].to_s.empty?

  file = ENV["FILE"]
  dest = "#{ENV["DEST"]}-$CAPISTRANO:HOST$"

  download file, dest
end

Handy function for inspecting (counting) array-like data. Usage:

> [9,10,11,11,11,10].count_by(&:odd?)
=> {true=>4, false=>2}

>> class Foo < Struct.new(:x); end
>> [Foo.new("a"), Foo.new("b"), Foo.new("a")].count_by(&:x)
=> {"a"=>2, "b"=>1}

Source: (I put the first of these in my ~/.irbrc file)

# Ruby >= 1.8.7 (incl 1.9.x)

module Enumerable
  def count_by(&block)
    Hash[group_by(&block).map { |key,vals| [key, vals.size] }]
  end
end

OR

# Ruby <= 1.8.6 (Hash#[] behaves differently in <=1.8.6.  note: breaks when group_by key is an array)

module Enumerable
  def count_by(&block)
    Hash[*group_by(&block).map { |key,vals| [key, vals.size] }.flatten]
  end
end

Here's a way to sum up multi-valued fields within events in Splunk.

Splunk provides "mvcount" and "mvjoin" but doesn't provide anything like "mvsum". So I wrote a custom search command (splunk reference docs here) that can be plugged into Splunk to provide functionality like this:

* | rex max_match=10 "(?<nums>\d+)" | mvsum nums as total_nums | table nums total_nums

Which generates output like this:

Installation is as easy as installing this source code and restarting Splunk. Tested on Splunk 4.3.2.

Linking to a URL on github like

github.com/someaccount/somerepo/blob/master/somefile

is fragile because "master" moves around. It's much better to link to a specific version of the file like

github.com/someaccount/somerepo/blob/somehash/somefile

But, github makes it easy to arrive at the former and hard to transition to the latter. I wish they would make it easy, but until that happens here's a handy bookmarklet I crafted that:

• Redirects you to the associated permalink page
• Shrinks the commit hash to the standard 7 character abbreviation
• Maintains any urls hashes (e.g., highlighted line numbers)

The bookmarklet: github permalink (drag to your bookmark bar)

Use on any "blob" page, such as https://github.com/rails/rails/blob/master/version.rb#L1-2

To be able to run spec tests from within Sublime Text 2:

Menu choice: Tools > Build System > New Build System...

Then enter:

    {
      "cmd": ["bundle", "exec", "rspec", "$file"],
      "working_dir": "${project_path:${folder}}",
      "file_regex": "^(...*?):([0-9]*):?([0-9]*)",
      "selector": "source.ruby"
    }

Thanks to http://wesbos.com/sublime-text-build-scripts/

Hilarious enough to get me to post on my blog. :)

Funny comment from notch8 following Github's production database slip-up.

These are instructions for setting up 64-bit postgres on Mac OS X 10.6, Snow Leopard using MacPorts. My purpose was to be able to run Rails' tests when creating patches for ActiveRecord.

This is a pure Snow Leopard install (not upgraded from 10.5 Leopard). There may be some complications if you upgraded. Good luck. ;)

Install with macports:

sudo port install postgresql84 postgresql84-server

While that's happening, update PATH in the appropriate file

(mine is ~/.bash_login, yours may be ~/.profile, ~/.bash_profile, or something else)

(in the correct file)
export PATH="/opt/local/lib/postgresql84/bin:$PATH"

Follow the instructions that 'port install' printed out:

"To create a database instance, after install do"

sudo mkdir -p /opt/local/var/db/postgresql84/defaultdb  
sudo chown postgres:postgres /opt/local/var/db/postgresql84/defaultdb  
sudo su postgres -c '/opt/local/lib/postgresql84/bin/initdb -D  /opt/local/var/db/postgresql84/defaultdb'

"Execute the following command to start it, and to cause it to launch at startup:"

sudo port load postgresql84-server

Create a postgres superuser for your username:

sudo -u postgres createuser your-username-here
Shall the new role be a superuser? (y/n) y

Install the ruby gem:

sudo gem install pg

Set up the rails activerecord test db:

cd your-rails-checkout-dir/activerecord
rake postgresql:build_databases

Verify test passage:

rake test_postgresql

It works! Hopefully. :)

Some possibly helpful references

• general flow
• know which postgres gem to use
• possibly relevant 64-bit info
• create a pg superuser for your username

Discovering the sleekness of wkhtmltopdf + wicked_pdf and recalling old wounds incurred in past PDF generation coding made me smile so much I just had to make a sample app out of it. What better way to do so than a plug-and-play Heroku app?

Voilá heroku-pdf. Working example.

Presenting our CFO:

Animoto's Webby acceptance speech

Surely this is the greatest company on earth :)

Animoto was asked to create a highlight clip for the Webby's (which we also got to accept an award from) --

Highlight Reel 2009

The awesome team here at Animoto has taken home an 'oscar of the internet' in this year's Webby Awards in the Services & Applications category. Go Animoto!

I can't thank Barry Hess and Andi Schacke enough for writing and commenting on this post and saving me from all the darkness and despair ;) I've experienced over the past few months as my Rails test times plummetted on this shiny new MacBook Pro. I recompiled as per Andi's comment and my times are now about a fourth what they were. Thank you both.

Revised Hivelogic instructions:

curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
tar xzvf ruby-1.8.7-p72.tar.gz
cd ruby-1.8.7-p72
./configure
make
sudo make install
cd ..

Jeremy McAnally - Machine Learning

Works for entp. His latest app: tweekapp

My Summary

Jeremy was out to convert everyone to doing DSLs the 'right way'. Some salient points were:

• the point of a DSL ought to be to make the DSL easier to use for the experts.
• 'external dsls' (i.e., you make up your own language) can be great for simple project, like config-writing code
• 'internal dsls' (e.g., ruby-code-dsls) get a lot of stuff for free (loop structures, etc)
• dsl composition (using more than one dsl at a time) can cause namespace issues (he didn't really offer much in the way of solutions here as far as I can remember)

DSLs

Used by experts, for specific problems.

Domain

Different problems have different domains, and they each have their own language.

Specific

DSLs that are very generic or look a lot like english don't make sense to him. DSLs are supposed to make the brain-to-code jump much smaller. Also, why waste the keystrokes?

External DSLs

These are DSLs that don't use an existing language (e.g. ruby). Sounds like he doesn't like them as much in general, though there are specific cases where the language structure is simple enough that it makes sense (e.g. puppet).

Internal DSLs

Implemented in an existing language (e.g. ruby). Loops & basic things like that are baked-in and already understood by end-users. External DSLs have to go to a lot of work to create these structures & etc. When to use? Look at the possible complexity of your code, look at what your 'experts' will use.

Examples:

• Rails Templates (written by him).
• Ambition - 'Take the suck out of SQL'
• Foundry (Another of his).

Designing the DSL

1. Think about the problem domain
2. Deconstruct -- remove the cruft. DSLs are not fake English. No Finglish allowed.
3. Implementation

Method chaining

Approach 1: return a new object every time

Approach 2: return self after changing state

Choosing an approach -- think about which is more concise in your case, also think about how #2 can get ugly & unwieldy.

Dynamic dispatch

He likes using closures. Some extra chrome -- if that's bad maybe look at an external DSL.

Compositon - meshing DSLs. Presents namespace issues.

Decoupled DSLs

In his opinion, the DSL ought to be decoupled from the library behind it. The DSL is just the interface to the logic. Thus, when testing you test them separately.

Kirk Haines - Vertebra

Kirk is working for EngineYard on Vertebra

Jordan's Summary

Vertebra is a solution from EngineYard to manage processes in the cloud. Still in development (no production usage so far). Sounds interesting, will probably take a look at their source code if for nothing else than an interesting architecture. Uses XMPP, which I seem keep hearing about these days.

Cloud Computing

Clouds are great b/c it's many, many machines waiting to do you bidding.

Clouds are hard b/c there are many, many machines to manage.

Scripted SSH works, but he says it has its limits. Vertebra built to be 'fault-tolerant operations among autonomous agents'.

Enter Vertebra

Built on XMPP.

• very chatty protocol
• upside to chatty-ness is fault tolerance. back-and-forth paints the picture of message reception.
• goal is: if network dies, operation in progress should be able to pick right up where it left off

For an XMPP Server they use EJabberd

The pieces of Vertebra are agents. E.g. a logger agent.

Special Vertebra Agent: Herault

• originally written in ruby, rewritten in erlang for performance & scalability
• new agents coming on board advertise their capabilities to Herault
• when looking for capabilities ('show me all loggers') you ask Herault
• also handles authorization for dispatching operations
• when asking Herault to dispatch an operation, can specify :all or :single (more options in the works)

Concurrency API

Talking about using vertebra to schedule future units of work that may or may not execute depending on a given set of conditions.

Yehuda Katz - The Great Rails Refactor

Yehuda is working for EngineYard, was working on merb, is now working on rails.

My Summary

He says he's talking more about the future of rails & less about what-came-from-merb these days.

Yehuda's presentation really seemed to be centered on componentization of the entire Rails stack

• Pluggable persistence layers (ActiveRecord, DataMapper, Sequel, CouchDB, etc)
• Using and becoming middleware
• Making rails' ruby modules less tightly coupled with each other
• Replaceable rails components (ActiveRecord, ActiveSupport, Action)

He ended talking about how all these areas of componentization will open up new possibilities of competition (and thus greater health & growth) for the whole Rails ecosystem. An interesting thought in juxtaposed with the elimination of Merb vs Rails competiton. I'm a bit skeptical of whether this can be really pulled off, but I'm excited to see what the brilliant minds behind all this produce.

ORM Agnosticism

Possible in rails right now, but things like form_for ought to work also.

There were two interfaces that merb switched on: ActiveRecord/Datamapper or Sequel. Not too bad but not ideal.

The solution: ActionORM (Lori Holden) - a way of making any ORM look like ActiveRecord.

Two options going forward for compliance for ORMs

• mimick AR
• provide a proxy that supports the key methods

A few core things will be covered (like form_for) but it will be up to the community to create extensions for the rest. It's hard for them to ascertain what's going to be useful up front and they want to move forward quickly.

Rack

Some middleware they used in merb: PathPrefix, ConditionalGet, Static

Static - can use for tweaking static asset handling for dev mode (and turn off in prod)

Rails middleware usage: Session middleware, Param parsing, Failsafe

This is all working toward amore unix-like setup, requests come in and get routed to pieces (Sinatra, Rails, etc) which are not special -- they just know how to handle the same thing.

Callbacks

He worked on improving callbacks a lot. His first analysis of Rails showed callbacks taking 20% of the time for a request (admittedly a simple request). A bit on the history of callbacks in Rails: there was no unified callback system originally -- lots of different ones. Then it was refactored into callbacks.rb. But it didn't solve all the needs of all the pieces and thus there was a lot of reaching-into-and changing going on that added complexity and slowness. Yehuda went through & made a single calback system that served all needs. Improved actual calling of callbacks - went to 'compilation' instead.

On complex code: Sometimes it's ok to have a complicated bit of code, as long as your abstraction is leak proof & well-abstracted. E.g., nobody cares that ruby is complicated internally because it's done well enough that nobody has to go digging through the C code.

One big problem w/ current rails code -- poorly named (confusing) instance variables make it hard to read & understand.

Upcoming: Abstract controller. Will be module based (truly module based, not tightly coupled modules like rails currently has).

Ruby tip: If you include a module in a class you can super from a module method. Much better than alias method chaining, which can get really messed up if things aren't chained in the right order.

About the plugin api -- there are lots of methods in Rails that are not for everyday use, but are available for plugin authors. These are methods that they don't want to clutter the API docs with, but they will stand by them as far as keeping them working the same way. Sort of a 'hidden' but valid public API. Still thinking about how to provide plugin API for the future of Rails to avoid all the crazy monkey patching that currently goes on.

Upcoming: Debug Toolbar like Django's, but more pluggable.

Orchestra -- gets the pieces working together. Hopefully will make it a lot easier for people to add instrumentation (especially specialized instumentation) to Rails, not only for fiveruns & etc, but for normal end users.

On Competition

Yes, the competition between Rails & Merb is gone. However, Yehuda says that they've really eliminated a less-useful competition in order to open up a more useful one. That is, competition on the component level. The want to make the pieces of Rails compete instead of the entire framework compete. E.g., Sequel vs ActiveRecord, your own ActionController, etc.

Jon Crosby - Middleware

getcloudkit.com
contributor to rack-contrib project
working at Engine Yard on Engine Yard Solo

CGI

Remembering the bad old days...

Things we maybe could pull out of our apps

• authentication
• caching
• openid + oauth

Rack

http://github.com/rack/rack-contrib
based on http model of having layers
based on wsgi (web server gateway interace)
Interesting -- Rack::CSSHTTPRequest
Rack::Cache
Check out his cloudkit - intersting out it builds up rack items w/ oauth, openid items & etc
Middleware announcing itself - Middleware can announce it's presence via env headers
Adding sinatra as middleware before Rails

Comment from Yehuda: they are currently working on rails as middleware stuff

James Edward Gray, II

If you want to learn Ruby you should read code, lots of code. He estimates that an 'expert' ruby programmer would read more code than he writes.

Examples of good code libraries

Restclient

• clean interface to restful services
• this is networking code done right
• quote he likes: 'Everything in a method should be written to the same level of abstraction'
• logs from restclient are executable ruby code

bj

background priority queue for rails

terminator

robust thread/pipe/etc level code written by an expert

slave

trivial multiprocessing with built-in ipc

The art of code reading

• when you start reading, relax. not all code sucks & you probably won't understand the author's reasons at first
• don't start with Rails (too big & complex)
• have a goal
• get the source code open in your standard text editor (e.g. use gem unpack or get a copy from a vcs)

Two videos from our family to yours. :)