Sunday, July 10, 2011

Why it's the late Cretaceous for server side MVC frameworks

We're at an inflection point with web application development.  I've been developing web applications for 15+ years, and it seems like we're at a familiar place.  The last time I remember being here was in the late 90s when the first server side MVC frameworks started becoming popular.  Up til that point, most of the web applications were big lumps of code that handled a request and returned some html.  For the most part, apps glommed together strings of html and sent back.  Next we got to html with embedded code,  and that was an improvement but still things were a mass.  We had no concept of separation of concerns, no good way to organize our code.  SQL was spread willy nilly amongst the angle brackets.  It was baaaaad.

And then someone had the idea to apply the Model View Controller pattern to web development.  Frameworks to do so abounded.  I wrote my own, and then abandoned it when the world standardized on Struts, which was, at the time, A Good Thing.  Web browsers in this era were fairly dumb, so it was clear where the code need to be for such a framework: on the server.  The interaction with web browsers was strictly limited to a request/response cycle.

Over the past 10 years or so, the basic idea of a server side MVC framework has remained the predominant way to develop web applications.  Variations such as component oriented frameworks like Tapestry and WebObjects have emerged, but the core idea of a server side framework that handles requests and returns responses and manages the UI and navigation for the application remains pretty much unchanged.  If you look at Struts from 2000, you easily recognize many of the same pieces as you will find in ruby on rails of 2011.  This for a good reason: server side MVC has been a great way to develop web applications for most of this time.

In the last few years, I've felt this start to shift.  Web application UIs have gotten more interactive, dynamic, and sophisticated.  The limitation of the request/response cycle during an application has started to feel positively archaic.  Having to reload the whole page to see what happened when I attempt to perform an action just isn't cutting it when so many apps have proved we can do better.  Server side frameworks have done their best to adapt and allow for richer UIs.  But as clients get richer, it becomes clear that a lot more UI logic has to be on the client.  Eventually it becomes unclear where to put what.  Our clean separation of concerns begins to break down.  We are back to a mess.

I'm going to suggest that it's time to do something radical: Let's stop.

Server side MVC frameworks have served us nobly, and we should thank them for their years of service.  But it's time to move on.  We need to embrace a different architecture for web application development.  UI logic needs to be on the client.  Communication with back end services and persistence stores can take place via RESTful HTTP calls with JSON payloads.  Server side code doesn't go away, but it assumes a new place with more limited responsibilities.

This is a big change.  A lot of our skills as web developers will need to be updated or replaced.  But the ecosystem for building these kinds of application is booming.  Many of the things that made developing applications in this manner painful a few years ago are dramatically different.  Unit testing tools like Jasmine abound.  A better javascript syntax thanks to coffeescript makes our code much more enjoyable to write.  And frameworks like Backbone.js give us a clear separation of concerns while we build applications in this new way.

As I've started developing apps this way, I've been really excited by the results.  The frustrations and foreboding sense of "this doesn't feel right" I found developing rich client apps with server side MVC is completely gone.  It definitely took some time to change over, but I wouldn't go back.

There's a lot to learn though.  To give you a jump start we've put together a two day training class so that you can jump in and be productive immediately.  We hope that you'll join us in Cincinnati on September 8th and 9th.

Monday, May 30, 2011

Managing coffeescript/js dependencies with the Rails 3.1 asset pipeline

We've been using and loving coffeescript and backbone on my current project.  In the last couple days we've upgraded our project to rails 3.1rc1 and been able to take advantage of the new asset pipeline therein.  This feature elegantly solves some super annoying problems we've had with managing dependencies in our app, but is not yet well documented.  I thought I'd take a few minutes and share what I've learned.

Where the assets live

Before rails 3.1, we'd lump all our javascript and stylesheets in the "junk drawer" that is public.  No more.  Now they can go in 3 different dirs depending on their purpose:

  • app/assets for things that our your application code
  • lib/assets for shared-ish things (not sure I grok what I'd put here yet)
  • vendor/assets to put things that your app uses but is provided by others
These directories are magically put on the asset pipeline for you.  In your views, you can reference things in any of these directories under the prefix /assets, eg. if foo.js was in app/assets (or lib or vendor) you would refer to it as /assets/foo.js.  The rails helpers such as javascript_include_tag will do this for you.

So this gives places to put stuff, which is nice, but only the beginning.  The killer features (for me) are the ability to package assets and manage dependencies.  To see how this works, take a look the app/assets/application.js file you get when generate a new rails 3.1 app.

Nothing but comments, weird huh. That's because this is what we're calling a "manifest file" which is basically just a file that requires in other files. When you include /assets/application.js at runtime, rails will package all the files you required and concatenate (and optionally minimize) them. Let's look at those last 3 lines, as there's magic in them thar comments. The first 2 require in jquery and jquery_ujs. It seems like this must mean that there are jquery.js and jquery_ujs.js files somewhere in one of the assets directories, but this isn't so. That's because gems can contribute to the asset pipeline as well. More on this later. The last line says to include all files in this directory or subdirectories as well.  In our apps so far we find it nice to have 2 such manifest files, vendor/assets/javascripts/vendor.js and app/assets/javascripts/application.js.  Not sure I can call this "best practice" or not yet, just that it seems nice to us so far.

It's also worth pointing out that files that need to processed (eg coffeescript and sass) will be handled automatically as well. Simply drop a file named whatever.coffee into the /app/assets directory and it will be compiled and included into application.js

Where this all gets more interesting to me is where we have files in the application that depend on each other. Files that are required can have requires of their own. The example I have in my app is coffeescript class inheritance. Imagine a couple classes like so:


When you're building apps with a lot of front end code, it's to be able to organize code with each class in it's own file. But this means you have to make sure to have the script that brings in fruit before apple. With the ability to require in rails 3.1, this problem is nicely solved for us. If we add a require statement to apple.coffee like so, the asset pipeline will take care of making sure things are in the correct order:

This is a huge win for me, as before this I was naming files with numbers such as 1_fruit.coffee to get around this problem. Yuck.

I mentioned earlier that I'd talk about assets living in gems.  This is a feature that allows you to take front end code and easily share it between multiple projects.  In my next post I'll talk about my experience building a rails 3.1 asset containing gem.

Sunday, May 15, 2011

2 easy things you can screw up in backbone

First off let's be clear.  Backbone is awesome.  It's totally changed the way I develop web applications for the better.  Before backbone, working on javascript code for a rich client web app was a miserable experience.  With backbone (and coffeescript) I'm now enjoyably writing code I can be proud of.  But because we are now developing our apps in a new way, we've found new ways to mess things up ;)  Here are a couple ways we found that could save you some hours of frustration if you avoid doing them.

Don't have multiple views using the same element

This one I did early on in my backbone days the first time I had one view that created another view.  Imagine some code like the following:



Seems ok at first. When we click a button in FooView it creates a BarView giving it an element and telling it to render. The problem happens the second time the button gets clicked. At that point you have 2 instances of BarView which each use the same element. And if those BarViews are listening to events on (or within) said element, mayhem ensues. I've found it to work out better to write the code like so:



In this version the BarView is created during the render method, right after his element has been created presumably. Then the BarView instance is displayed when needed. I've found this to be a good rule of thumb: backbone view objects should have the same life cycle as their elements. They should be created when their elements are created and destroyed when they are removed.

Beware model defaults that initialize complex attributes

The second gotcha I've run into cost us a good few hours today actually.  Backbone models have a nice feature for allowing default attributes.  It's a handy feature, but by using it to initialize array or object properties we found it caused a subtle (to us) but pretty terrible bug.  In our model we had code like so:



The problem here is what happens when you create a second Foo. Turns out backbone does a shallow clone of defaults, which means the same array object in memory is used for both. This spec describes the problem nicely:



Turns out there is a pretty easy fix for this too. Defaults don't have to be an object literal, they can also be a function and backbone will be smart enough to invoke it to build the model's attributes. In coffeescript, this fix is exactly 2 characters:



Hopefully these two tips will save you some frustration as you dig deeper into backbone. In a future post I'll share some of the code we've refactored out of our app that we feel like is generally useful for other backbone + coffeescript + rails apps.

Sunday, March 13, 2011

CoffeeScript + Backbone.js + Rails = Superfantasticalness

Lately I've been working on application by writing the front end in CoffeeScript using Backbone.js and the back end in Rails. A few people have asked me about it and so I thought I'd share my experience so far.

Backbone.js

I've been looking for a front end javascript framework for quite awhile, ever since I realized that server side MVC frameworks were probably coming to the end of their time in the sun.  I've even tried several aborted attempts to write one. I've spent good amounts of time playing around with both Sproutcore and JavascriptMVC, but when I picked up Backbone.js I didn't want to put it down again. I don't really want to spend the time right now to delve into critiques of the other two, so I'm going to focus on what I like about Backbone.js

For me, the big attraction of Backbone is there's so little to it.  Earlier in my career, I was enamored by frameworks and tools that did "everything I would ever need".  The idea of learning one environment where I could do everything was appealing.  I was attracted by neato whiz bang features even I didn't need them just yet.  But as I've grown older and more curmudgeonly, I now prefer tools that just barely do what I need and then stay out of the way.  In my mind pivotal tracker is such a tool.  Backbone.js  feels like this too.  It provides a nice structure for the code of a rich client web MVC application.  It allows me to continue to use html, css, and jquery without insisting that I give up one of these or do all my design and layout in javascript.  I prefer to work a designer who can create visually appealing UI in html/css, so frameworks that don't accomodate this workflow are a non-starter for me.

I don't intend this to be a Backbone.js guide, the excellent documentation is where to go for that.  But I'll briefly touch on my experience with the different pieces I've used thus far.

Views

The first Backbone code I wrote was a view.  I had started building my app in the usual (for me) way of just adding a some jquery to make my client side UI more dynamic.  But this time, when it started to get unmanagable as it always does, I refactored it into a Backbone View.  This is just a "class" that extends from Backbone.View.  The only thing this does for you is allow you to specify an element that the view contains, and a set of events scoped within this element that bind to functions of your view.  But it turns out this feels just right for organizing my code.  In my case I ended with a tree view component that uses jstree to do a lot of the heavy lifting.  Later on, I actually extracted it into a superclass and reused it on a couple other pages.  Over time I've found that my view classes tend to end up as nicely reusable components.

Interestingly, backbone views don't mandate any particular choice for producing the html.  After some experimentation, we came up with a convention we like on our project.  Each of our views generally has a template file, we're using mustache right now so the files are named things like foo_view.mustache.html.  Originally we were just embedding our templates into our rails views as hidden divs or in script tags.  But we found ourselves repeating the same markup in our jquery-jasmine fixtures, and this made us sad so we searched for a better way.  We settled on having templates in a separate file, and then wrote a rails helper method that renders the views template on the page.  We also wrote a jasmine jquery method that loads the template into a fixture so we can use them in our specs as well.  Here's what that code looks like, if anyone else wants it:


There are a couple of other things worth passing along we've learned working with Backbone views.  The first is that you really want to have your view's element have the same lifespan as your view object.  I ran into this issue by creating new view objects inside the listener method of another view.  There's nothing inherently wrong with this (that I know of), but in my case the element for the view I was creating already existed.  The net effect was that if if the listener method that created the view was called multiple times you could end up with a multiple views listening to events on the same DOM element.  This caused no end of confusion.  It was a much better idea to create the view once and just reuse-it as needed with different models (as necessary).

The other idea I feel like we're learning is how to communicate between models and views.  We regularly have views talk directly to models, getting and setting properties, telling models to persist themselves, etc.  But when it comes time for models to communicate with views, it really feels right to use the Backbone event framework.  There are a set of nice built in events on models, and it is trivial to add your own.  It's worked out well for models to trigger events when interesting things happen and let the views listen to them and do what they will.  So our best practice is shaping up that views can interact with models, including listening to events from them, but models should never talk to views directly.  It would probably seem obvious to do it this way for anyone coming from rails-land, but I thought it might be worth mentioning anyways.

Models

Backbone.js models know how to speak json to a restful back end out with little to no coding required.  This ended up being a pretty big selling point for me, as it made it trivially to integrate Backbone with my rails app and it pretty much Just Worked.  The only gotcha here is that rails would prefer to have the attributes for a model be grouped into a name parameter, but telling backbone to do this as as trivial as the following code:



I guess the biggest difference working with Backbone models than Rails models is that everything is asynchronous.  Calling a fetch (think find in rails) or a save in backbone means you also have to specify a function to invoke on success or failure, or listen to an event.  This would seem to add up to a lot of extra code, were it not for...

Coffeescript

I've been searching for Coffeescript without realizing it for many years.  I've been treating javascript as a first class language for a few years, test driving, trying to use clearly communicative object oriented design, etc.  This has made the experience of writing javascript immensely more satisfying, and I hope, helped me produce higher quality code.  But it's always chafed a bit.  I didn't get to choose javascript.  There's a lot of syntax is in it that I don't like, and feels noisier than I wish it was.  I spent a good amount of time looking for solutions to let me use a language of my choosing in the browser, but in the end they've all fallen short.

Until now.  Coffeescript is a lovely language that compiles into readable, debuggable, javascript.  It's documentation is fantastic and it's source code is beautifully documented.  The most annoying bits of javascript go away, and the coffescript code I've written is so much cleaner and less noisy it's been a joy to use.  I'm finding it difficult to convey how big a deal coffeescript is.  Someone smart I know responded to coffeescript allegedly by saying "But isn't it just nicer syntax?".  Yes it is!  And it turns out this is incredibly valuable and important.  I chose ruby because I can express my intent, in elegant, readable code.  Coffeescript feels the same way to me.  But I can use Coffeescript in all the places I would have had to use javascript before.  It's been huge to me.  I'm pretty much at the point where I'll go back to writing javascript when you pry Coffeescript from out of my hands.

Though I dig backbone a lot, it's still possible one of the other frameworks might mature into something awesome and be the way to go.  Coffeescript, on the other hand, doesn't appear to have anything close.

Thursday, November 18, 2010

Highlights of rubyconf

So here is my rubyconf recap, in no particular order:

As always, I enjoyed tenderloves presentation. He's always hilarious, and this year was no exception. This year he really hit the technical material hard and did an excellent job. It was all about the performance tuning and eventual rewrite of Arel. It was a practical deep dive into how to improve the performance of your ruby code. Awesome stuff, definitely worth catching on confreaks.

I was also inspired by DHH's talk. He was basically talk about why he loved ruby. I really liked his thoughts about rejecting the "best tool for the job" idea as if we choose a programming language based on a feature matrix. For me it was the same: ruby code made me happy. What got me inspired though, is that I feel a lot of the same kind of almost giddy happiness about my coffeescript code.

So I've spent time trying to learn write the best javascript I can. It's certainly got some nice features and is not as terrible as people make it out to be. I can write code that I'm pleased with, and gets the job done. But the fact is, I didn't choose the language, the browsers did. With coffeescript on the other hand, all the most annoying things about javascript are just gone. The code I've written (not enough yet, mind you) comes out just so sparse, elegant, and communicative that it makes me feel all warm and fuzzy. It really is much like how I first felt (and still feel) about ruby.

I gave a very lame 3 minute lightning talk on coffeescript on Sunday, and got to talk to a few other people that seem as excited about it as I am. It also felt very reassuring to talk to DHH and one of the other 37 signals guys, Sam Stephenson, about their use of it and how impressed we both are with backbone.js as well. It's still early time here but I can't wait to see how things develop. It's exciting.

Another session I really enjoyed was the redcar session. I was already geeked about redcar before, but after seeing Dan crank out a plugin live in a few minutes I got excited and have been coding on it ever since in my spare hacking moments. I managed to fix a bug (turns out it only happened because I was a version of date with Safari) and then coded up a plugin for the feature I wanted the most, running a single test by name based on where the cursor is. It only talk an hour or two while waiting for a flight. I don't think I've ever had an editor where coding in it was actually FUN. It's quite empowering. There's still lots and lots to do. Hope I can keep finding some spare to time to hack. It's also worth noting that I've been productively using redcar for real coding for all of this week. Rubymine, I loved you for a time, but in the end the performance woes made you too frustrating. Best wishes and fond regards, we wrote some lovely codes together.

And of course, the Polite Programmers session. There were definitely some issues. The banter back and forth didn't seem to work as well as I hoped. Jim left me my openings, I just got nervous and had trouble coming up with natural sounding responses. But we got lots of positive reviews, and it's always a privilege to present and learn from Jim. Ed did a fine job as Mr. Manners as well, what with his inimitable cultured accent and all :)

Can't wait for next year!



Wednesday, July 14, 2010

JSpec tests running in CI

A lot of people, myself included, have been TDDing our javascript for awhile but it seems like it's still fairly rare for the tests to be run in the CI build. I've taken some time today to get our existing suite of JSpec tests to run in our CI server (Hudson in our case) and thought I would share what I did. My general approach should be useful regardless of which CI server you are using.

It's actually fairly simple: I'm using the capybara selenium driver to run the JSpec tests and grab the results from the DOM when they finish. Here's the testcase:

I define an array of all my jspec tests, loop over them, munge them into a legal test method name and define a method that delegates to run_jspec. run_spec opens a file url (had to go underneath capy here so its a little icky) and then waits for the .failures div to appear. It should be noted that capybara will wait for an element to appear when you call page.has_css?

From here, running it under CI is trivial. In fact I didn't have to do anything at all, the tests was found by rake test:integrations and ran (and passed!) in the next build. Most of the heavy lifting is done by capybara and selenium-webdriver.

Obviously there is a lot that could be done to improve this code. I could grab the jspec file from a FileList, and grab some better failure messages from the DOM. I could also probably write a custom JSpec formatter to make this easier on myself even. I'm thinking if there is interest I could finish this up and turn it into a gem. Please comment if you'd use such a thing.

Also, one more thing you will need is the JSpec runner file that can run a single test. I did that like so:

Enjoy!

Saturday, January 31, 2009

Why don't I pair more?

I've been watching a video of the hashrocket guys just now that Corey Haines twittered about. I've been amazed and inspired by Corey's pair programming tour: talk about turning lemons into lemonade. What an awesome idea. I've paired with Corey on some open source stuff at conferences, so I naturally wanted to get in on his tour. As a result I had the good fortune to have Corey stay with me on his tour. Unfortunately the timing of his stop happened on a day I had to spend dealing with client issues and didn't get to spend time pairing with him. And this leads into my topic: Why don't I pair more?

This isn't another post about why pair programming doesn't work, isn't a good idea, or any other such nonsense. I'm a true believer in pair programming. I've been sold on the concept since I first read about it in the first edition of Kent Beck's Extreme Programming late in the prior millenium. I've experienced pair programming and so I know the benefits first hand:
  • It consistently produces the best code, the code I'm proudest of in my entire career. And not a little prouder of. I mean, like, an order of magnitude more proud of.
  • I'm operating at about twice the velocity I would normally, which is a rush, and incidentally, means there is no negative effect on team velocity. I've actually measured this on a real project.
  • I learn way more
  • I enjoy my job way more
And on an on. But that's not what I actually want to talk about. This post is a way for me to put down in writing the reasons that I don't pair nearly as much as I want to. I'm doing this chiefly as an exercise so I can better understand the issues and change them. If it helps someone else, hey, that's pure bonus. The first few are about me. Which is good news, because it means I can change them.

  1. Part of me is afraid to
    I'll just go ahead and lead with this one since it's the hardest to talk about. I really like how the hashrocket guy put it: "Pairing is a burning crucible". Heat reveals impurities and exposes the true nature of things. If I'm having a self-doubting, insecure kind of day, I might not want this. Intellectually I'm totally convinced this is BS and the benefits far outweigh the minor pain of revealing what's obvious to everyone: I don't know everything. I like to think this reason is not a major factor preventing me from pairing, but in the interest of self-disclosure I'm putting it out there.
  2. But I might not get my stuff done
    This one is a bit more insidious, but like the prior reason begins with self-centered fear. If I pair on your task, I might not get mine done. And conversely if you help me you might not get yours done. There are at least two ways to address this. The first is to get over it: I believe that the software I produce will be way better if I pair. And obviously producing better software is what's best for me and whoever I work for. I have to be clear on this and act on it, which in my experience is not always easy. And of course the other members of my team do as well.
    The other way to fix this is organizationally: don't assign tasks to individuals. I've been in situations where this was out of my control. Some clients like having a single "go to" person for a given feature or issue. But this doesn't have to stop pair programming if the team doesn't let it.
  3. Inertia
    This one is a little more nebulous but I don't think it's less real. Sometimes I don't pair because I don't put forth enough effort to find someone to pair with. Other times I might be "in the middle of something". Either of these fall into the same general category of inertia: an object at rest (me, by myself, in front of my computer) tends to stay at rest.
The next few are harder to change, because they are about other people. Obviously I can't pair program if no one will pair with me. And no matter how much I would like to and am sure I know what's best for them, I can't actually control other people. But I'm going to list these issues anyways in the hopes that solutions will become clearer if we understand the problems.

  1. Odd number of people on your team
    This is so obvious an issue it seems silly to put down, but I've seen it make a big impact. I know the song says "One is the loneliest number" but three is also not very good. What tends to happen is pairing becomes awkward as one gal or guy is left out and sometimes leads to little or no pairing at all. I suppose you could address this by making a pair your smallest unit of staffing a project. But it's also true that even if you have 3 people on your team you could still pair 2/3 of the time if you are intentional about it.
  2. Distributed team
    Remote pair programming is a whole nother topic and I have some experience with it that perhaps I'll share in another post. Suffice to say: it's not as good as pairing in person, but better than not pairing at all. Because there are some technical details to work out and do, there is that much more barrier to overcome and it requires even more sustained effort to keep up.
  3. They won't let me
    This is supposed to be the elephant in the room: management won't let me pair program. My thinking is that this one is mostly a myth. I've worked in a lot of places, some of them quite draconian. The truth however is that pair programming can start so innocuously that it's hard to have a mandate against it. Imagine management saying: you're not allowed to help other people. Even the pointiest of pointy haired bosses would see how stupid that is. I'm sure someone will comment that "No, that really does happen". Fine. I just don't think it's all that common. Most people, in my experience, actually do want to be useful and contribute something positive. They just may (wildly) disagree about how best to do so.
  4. They code in X and X sucks
    This one requires compromise, but can be a a positive sometimes if you let it. My first ruby project I got to pair with my friend Jim, a dedicated emacs guy. As a result I had to learn emacs, something which I probably wouldn't have decided to do if I hadn't see what a productive environment it is first hand. There are other times where I pair with someone and they use something and even after trying it I still think it sucks. In that situation I try to get them to always pair on my machine. Err, I mean, switch back and forth between each others machines in the interest of fairness ;)
  5. No one wants to pair (except me)
    Alas, this has been the hardest to solve. You can be fortunate enough to work for (or start) a company that mandates pair programming. Or else you can attempt to influence people. My only advice, based on experience, is that less is more. Don't talk about pair programming at all, just offer to help people. And then ask for help when you're stuck. I've sadly tried the approach of trying to convert people to pair programming, unit testing, or any other number of quite good ideas by incessantly talking to people about them. Don't do this.