Last night I gave a talk on pressing the big scary red reset button at the NY CTO School. The first talk before me was by @malcolmcasey, CTO of Skillshare, another awesome NYC startup. It’s a great meet-up for early team leads that want to learn from other technologists that have been doing this for a while.

I’ve reset a few projects in my life. None as big as Microsoft Windows Longhorn (Vista), which was a fun example of failure of monumental proportion. In 2006 Jim Alchin wrote in the Wall Street Journal: “Longhorn was crashing into the ground.” While he was credited with ultimately shipping Vista following a successful reset, it was one of Microsoft’s worse operating systems. Developers just no longer cared. You can see how a reset is by no means a guarantee of success. In fact, anecdotal evidence shows that 7 out of 10 projects that are reset still fail. And when projects fail there’s plenty of wrongs to be pointed out. Marketers will blame the product, product managers the technology and everybody will blame other people. A few will cite failure all around or an unstoppable death spiral.

I interviewed several CTOs when I was preparing this talk. It was really fun to hear about their experiences. They would usually start with a gasp and use a particular project reset vocabulary. “We slammed the brakes.” “Changed the wings on an airplane.” This might actually be a silly analogy – have you ever seen anyone attempt such a feat? No wonder most projects fail after a reset – taking the wings off in-flight means crashing into the ground, guaranteed. But if your reset is going too smoothly, feel free to add some drama as in this amazing video.

I think that pressing the reset button on a clearly failing project is always the right thing to do. Put some numbers in perspective by measuring what value the project is yielding vs. what it costs. You may have sunken a ton of money into it, but you’re just bleeding more. To execute, get rid of all the chickens (people who aren’t committed to the project with their flesh and bones), trim some fat and time-box the reset. Use all the brownie points that you have earned until now with the rest of the company (that’s why it’s so important to invest in relationships inside an organization when things are going well, both up and down). Successful resets are very rewarding and have a long tail, but people tend to block out negative emotions associated with violence and doom that occurred in the past, so few will actually remember. You’re not a hero, you’re just doing your job.

Finally, things don’t need to reach the dire straits. Personally, I try to foster a culture where we experiment with everything. Want to build the new system with a different data store? Try it. Want to rewrite our entire UI with anotherframework.js? Go for it. You don’t need buy-in from me – you just need to be transparent and let everybody know what you’re doing. You’ll get plenty constructive criticism from others - make your own decisions. The job of a tech lead is to keep the output of the entire team massively positive, not to micromanage individual’s minutes. For every one developer stepping out of doing visibly productive feature work for a sprint another will step out to find a multiplying factor of 10 for something you’re building next. It’s a small price to pay for being awesome, as a team.

Slides: http://www.slideshare.net/dblockdotorg/pressing-the-big-scary-red-button

I wrote a blog post yesterday about How to Start Small With Big Data and Google Analytics. Essentially, it’s a jumpstart for fetching Google Analytics pageviews and merging the daily data with some domain knowledge. But it also asks an important question:

Why do so many companies write a homegrown pageviews tracking system? Between Google Analytics, Kissmetrics and many others, isn’t that a completely solved problem?

this is a solved problem, too

I look back at my early years in software development and find myself rewriting a lot of software that has already been written by other people. It’s a typical overconfident engineer problem, where my initial reaction to anything created by other people is that it’s insufficient in too many ways. It started with early C++ code, where achieving performance and portability between many *nix systems and Windows was a nightmare – STL wasn’t nearly as good as today. I ended up rewriting everything, including String or Vector classes (an entire C++ library open-sourced as https://github.com/dblock/baseclasses). I then went on to work for Microsoft where we were trying to build a 24/7 service that would not die from memory allocation failures. .NET didn’t exist then, MFC was out of the question because of it’s legendary reliability and so was the Microsoft version of the STL that was just merging with MFC. Open-source was not even part of Microsoft lingo. We built CoreSDK, a C++ library where I ended up rewriting everything, including String or Vector classes, again. I think I implemented at least a dozen versions of String in my life that went into some production code. Those were the days!

This decade’s software is way too complicated to rewrite everything. When I choose a platform or framework for a project, I like open-source ecosystems that encourage reuse. Adding third-party components in the .NET world is not commonplace. Adding dependencies in Java with Maven is better, until you need to make changes to the code in your dependencies. The Ruby world with Rubygems and Bundler gets it right: there’s virtually no overhead for using a third-party library. Node.js is a bit younger, but also gets it right.

Now that I can easily reuse third-party systems, I must try to remember that I should write less lines of code, introduce less operational overhead and create more positive product impact in a reasonable amount of time. It doesn’t mean I just want to slap together someone else’s components - things worth focusing on should be elements that serve my company’s core competencies or be educational for me, useful to the open-source community at large or just plain fun. While this sounds obvious, sticking to the guidelines and keeping myself accountable is a daily effort.

How do you avoid rewriting everything?

I want to talk about your scope of influence as a Software Engineer and how to expand it.

Your scope of influence in your first piece of software is a function. You are parachuted in a well isolated world and are told to implement a simple algorithm. As you find your marks, you develop a component. Eventually they let you take on a small project. The entire company now knows about you. You switch jobs and become a familiar face in the city. You talk at a big conference and people from all over the world follow you on Twitter. You might, one day, invent something major and make an impact on humanity. Your life as a Software Engineer has just flashed in front of you.

I am somewhere in a shade of green in the graphic above. Crossing the lines between these circles is a difficult affair. If that is part of your goals, I want to suggest a few practical ways of expanding your scope of influence. Circles, that I feel I have crossed.

From Component to Project

Your component can be thrown away and replaced by another in no time. So can you. In order to cross this chasm, your commits must have global project impact.

Start with trivial tasks of standardizing the number of white spaces or tabs or by removing trailing spaces after meaningful code. Touch as many files as you can. I used to make commits of thousands of files because I was irritated by the lack of periods in code comments. Am I crazy? The truth is that I was terrified about modifying so many files at once and that prevented me from refactoring large chunks of architecture that was very broken. So changing whitespaces or adding periods was a useless exercise that helped me. When you’re comfortable with modifying code left and right, find a major piece of infrastructure to rip out and replace by something better. I remember swapping a C++ core library from underneath a huge system, a seminal moment.

From Project to Company

Nobody knows who you are until you’ve built something that many people want. In order to cross this chasm, seek a need that goes beyond your team.

Around 2001 I needed a tool to send a massive amount of e-mail to stress test some service. I couldn’t find a decent one, so I wrote something new. I put it on an intranet site and kept updating it. Eventually a lot of people with similar needs in the company used it. Then I  started another larger tool to help my team with our complicated builds. I made an intranet site for that too, and it took off. Six months later I was doing that project full time  with executive support and a team of five developers.

When building something that your entire company needs you learn much more than code. You have actual customers and you suddenly find yourself needing to talk to people, presenting the project to them and arguing with skeptics. Working with people is hard and providing a useful service within a corporation is a good place to start.

From Company to City

You’ll find yourself solving the same type of problems as your friends working at other companies. In order to cross this chasm, organize a visit to a friendly business. Go with a million questions and drag your fellow coworkers. Bring pastries.

Every time I lift my head from the computer and visit another tech startup in NYC I come out pumped. I learn so much and meet new Engineers! The first company I visited with my current team was Foursquare about a year and a half ago and the last as of last week was Rent-The-Runway. I met dozens of new techies in between. Every single time, by brainstorming ideas I changed my mind on how to build something or at least it made me think.

When you have built a solid network that includes technologists from other companies, you have crossed this chasm. It will help you hire coworkers and find a new job. You’re no longer behind artificial corporate barriers, you’re free.

From City to World

What did you do when you crossed the chasm from project to company? You sought a company-wide need and made a solution public within the walls of your business. Time to do it again, but in public. Scared of so many eyes? The truth is that nobody cares until you build something really good. So my philosophy is that anything that is not core intellectual property can be open-sourced and is always a great learning experience. When you build something good, you’ll remember the moment when a developer from another continent makes a pull request into your project on Github!

You must also write. Start a technical blog. And don’t forget to give a 3 minute lightning talk about your open-source project at a meet-up. Standing and talking in front of people is hard, but they need to put a face to your name. Slowly graduate to big conferences. On to conquer the world!

World to Humanity

Don’t ask me. Most people that had an impact on humanity have written books, given TED talks and won Nobel prizes. Honestly, I have no idea and am certainly very far from any of these. Help?

I talked to a developer recently. He said:

“I cannot talk about this project, because it’s under NDA.” … followed by silence

First, you’re confusing “NDA” and “Classified” or “Top Secret”. I worked on Classified projects. Or have I? You’ll never know. But I have a t-shirt that says: “my project is so secret, I don’t even know what I am doing”. And that’s precisely how you sound.

Can you please tell me something useful?

Start by describing your project in abstract terms without violating any of these commitments. For example, my uncle worked in a classified nuclear facility near Petyagorsk in former Soviet Union. At least I think he has. He was smuggled to Israel where all that classified stuff was quickly declassified and translated to Hebrew. I am guessing that he worked on some weapons systems that could destroy half of the planet or a small satellite. I know he could talk about his work in very abstract terms. And more specifically, he could definitely explain general issues with building explosives from small to nuclear, without revealing anything classified. So can you.

“Our project was in the daily deals space. I cannot describe the exact purpose of the project, but it involved deals … daily ones. One of the interesting problems I worked consisted of moving a million instances of data between 3 and 5Kb each from a node located in South America to a node in the U.S. I used MongoDB to store the data and a Redis queue for processing.”

Now we’re talking.

After almost a year of active development with 200+ commits, Grape 0.2.0 has finally been released. It’s the code from the “frontier” branch that we’ve been using in production for quite a while.

Grape is an API DSL. If you’re building a RESTful API today, this is a great place to start. Grape lives at https://github.com/intridea/grape.

Here’re the 0.2.0 highlights:

• Vendor-based versioning
• Model exposures presentation layer
• PATCH and OPTIONS HTTP methods
• API description blocks, reflection and introspection
• Modules
• Cookies
• Anchoring

Grape is a great project. It’s always an honor to contribute to systems written by much stronger Ruby developers. Many of my pull requests would get rejected and Michael would then rewrite the feature with a much improved architecture (API modules is one of those). I feel like I really learned something in this process. Open-source FTW!

Sometimes you need to iterate over a large MongoDB collection. The biggest issue is that, by default, cursors timeout after 10 minutes of inactivity. For very large collections it’s not uncommon to take longer than that to process results and you get an exception half way through the iteration. A cursor is a server-side construct, how about a client-side cursor?

Here’s a Mongo Ruby iterator that will call Mongo::Collection.find in increments.

And a Mongoid iterator built into Mongoid::Criteria.

Of course you must be careful that the collection doesn’t change during the iteration. If you add or remove an item before you, or will skip elements or process some elements twice.

You’d think someone has implemented simple conversion of numbers into English by now? For example 42 becomes “forty-two”.

I found a few interesting posts, a couple of gems and even a competition. All had issues that fit in two buckets: they could only do English or their implementation was scary. While I didn’t really *need* a converter that worked for different languages, none of the libraries inspired much confidence. Finally, I ran into a I18n implementation that was generic, well implemented and properly tested for Russian. It needed some minor English work, but you can be sure that the Russian version is *much* more complicated.

A few pull requests later, please welcome a new gem, numbers_and_words. Kirill designed the library with conversion strategies and proper localization in-mind.

These produce "forty-two" and "сорок два".

And a fun one.

This is obviously “two octillion nine hundred thirty-five septillion one hundred seventy-four sextillion three hundred fifteen quintillion one hundred nineteen quadrillion six hundred fifty-four trillion six hundred fifty-four billion six hundred fifty-four million six hundred fifty-four thousand six hundred fifty-four”.

Can someone please contribute a Spanish version next?

https://github.com/kslazarev/numbers_and_words

I moved the popular VMWareTasks C# library from CodePlex to Github.

The new home is: https://github.com/dblock/vmwaretasks. There’s also a new Google Group for discussions.

I haven’t been using VMWare ESXi since their new licensing scheme started milking customers, even for API access, but it remains a great product and as far as I know most people who write C# code to talk to ESXi are using VMWareTasks. Several people have been sending patches recently, Github will make this process a lot easier.

I am not a patient person. But years of software practice have taught me how to take my time while coding or how to, otherwise, remove time from the equation.

Manual Tasks Become Features

Otherwise known as process becomes automation.

You need a lot of patience when operating a manual change on a live system. You’re always this close to dropping the users table. So I learned a decent way to prevent mistakes: always run commands in a test environment, first. But that’s the same as walking on a cable between two chairs, before trying it between two skyscrapers – there’s still no safety net.

“first, he tried it at home between two chairs”

Instead, transform any manual task into a feature. In the Ruby world we write Rake tasks.

The cost of transforming a task into a feature is about half a day of work. But it’s still less than one production disaster for every 100 such instances.

Automated Tasks Become Business Logic with Tests

Automation is good, but testing is better. Consider the following task that sends some kind of reminder e-mail.

It’s pretty concise. Unfortunately, user.reminded? has a bug and you now have a communication disaster on your hands.

"he just reminded a million users that their delinquent account is about to be suspended”

We can move this entire logic into the User model and write a test.

The test can cover other important aspects, such as the actual source and destination of the e-mail.

And the task is simpler.

The cost of this change depends on the complexity of the task, but it tends toward zero. Tests take time to implement, but save debugging and regression time. Operational, or “machine” cost is roughly half a day, because the changes must go through continuous integration and a deploy before they can be run.

Friday Changes Become Monday Changes

Why do something today when you can do it tomorrow?

It’s Friday night and happy hour is about to start. Plan that next deploy for Monday morning, when everybody is in the office.

“she just spent her week-end undoing Friday’s mess”

Spending a week-end in the office fixing what you (or someone else) broke on a Friday shoots your Monday’s productivity. You’ll feel like exhausted heroes, at the cost of three days of work (week-end included) multiplied by the number of people involved.

This Week Becomes This or Next Sprint

I firmly believe in under-promising and over-delivering.

“he over-promised and under-delivered”

Never promise anything for tomorrow, day-after-tomorrow or even this week.

I see the best engineers learn to answer requests with “this sprint” and working really hard and getting it done for the next day. I love that. They don’t have an impossible deadline, and aren’t required to cut corners and can do things patiently (implement the request as a feature, write tests and get a code review).

The person on the receiving end is juggling a hundred things too, and generally they don’t expect anything to be done immediately, either. They also never remember when something was done in a day vs. two or five, but they remember well when something made in a hurry produced a production outage.

Asking a Person Becomes Asking The Team

As the team via the team lead when you need a feature.

“she will gladly implement your feature request, right now”

We’re creatures of habit. So we ask the same person who implemented our previous request to do it again. Asking a team lead is really not a matter of process, – I don’t get offended when someone goes directly to a developer to ask for something (in fact, I love when this happens in some cases) – it’s a matter of actually getting you what’s important, first. A team lead has a complete view of everything that’s going on and will be able to find hands for your request. It will get done much sooner.

***

It’s Friday afternoon and you’re about to make a clean break and grab a beer. Someone wants you, a developer, to e-mail a million users and remind them about their account being suspended. Rewind through my blog post.

Nice job from @gschrader.

This is an example of using WAFFLE to authenticate a client to a server using single sign on (SSO) using the Negotiate Security Support Provider.  It passes messages back and forth using Netty.

https://github.com/gschrader/ssoexample

My family counts many teachers. My grandparents were teachers, my father started as a school English teacher and is now a University professor. My wife changed her successful career to become a teacher.

My own first encounter with education as an instructor was teaching elder kids the proper pronunciation and the meaning of the Beatles songs in the depths of the winter coat check in Moscow school No. 80 (now 1266). I was probably 10. Those kids were 8th graders, probably 15. Why me? I could speak English and my dad had a few rare Beatles tapes. I was also a terrible singer, but I knew what they were singing. Without realizing it, I encountered a magical desire to learn from a group that bullied me into teaching them.

Sgt. Pepper's Lonely Hearts Club Band …

the school I went to and Gagarin Square nearby

I got an opportunity to teach a Ruby-on-Rails class to a group of 15 developers at GeneralAssemb.ly around September last year. I have given a few 2-4 hour talks at GA to various groups and I was excited by this prospect. The audience in my talks was always eager to learn and I met a large number of smart people through the exercise. Matt, who works full time for GA, would produce a properly structured, six week affair, paid and supported, as with all GA educational events. So we settled on a good size for the class, took lots of student applications, found an experienced RoR teaching assistant, Dimitri, secured a space, arranged office hours and a lot more. The big missing piece was a curriculum that I began working on.

Learning in a classroom is not the same as learning with a tutorial. An online document doesn’t look you in the eyes or help you when you’re stuck. It can’t monitor the room for blank stares of forgotten students. It doesn’t even sweat when live coding doesn’t translate into working software. And a tutorial can’t help you with issues unrelated to the immediate task, such as setting up a working local database or editor. But I learned with a tutorial and I always wondered whether I could be better at what I do had I learned wit ha mentor. I decided to build a new curriculum that would transmit my working RoR knowledge and deeper understanding of the system over the six weeks of tutorship, instead of six months of copying, pasting, stumbling and googling.

How long does it take to create a 12-lecture, 8-week course?

I spent between two and four hours per lecture researching and writing it, then about two hours preparing exercises and doing a dry-run through my own material. This is essential to avoid stumbling during live coding - just like for an important demo or talk, the last thing you want when presenting a course is to type some code and fail in front of an audience. I also created, assigned, reviewed and graded exercises after the first six lectures, which usually takes fifteen to thirty minutes per student, lets say another four hours. And I prepped a talk at NYC.rb too :) This adds up to about 60 hours of preparing for the lectures, about 25 hours of total homework, 24 hours of actual lecturing, 8 hours of project collaborative coding in a classroom (we extended the class by two weeks for this). Grand total 117 hours, which is very roughly three weeks of full time work.

Thanks to GA, the curriculum is now open-source and free for non-commercial use. If you’re teaching a class, consider using it! You can find it at https://github.com/generalassembly/ga-ruby-on-rails-for-devs. And, while we had our first public pull request this morning to fix a couple of broken links, I can’t wait for some feedback and substantial contributions.

Today I gave a talk to a packed NYC.rb about crafting a Ruby-on-Rails course for developers. I’m working on a blog post about classroom education, but in the meantime, here’re a few links.

The Open-Source Curriculum @ https://github.com/generalassembly/ga-ruby-on-rails-for-devs

Slides from the Presentation on SlideShare

Demo Stashboards, Students’ Projects

https://github.com/jameslin101/stashboard: a clean Stashboard clone

https://github.com/MedText/stashfu: Stashboard as a service

https://github.com/generalassembly/boardroom: Stashboard with voice-over-ip-activated control

I want to thank the audience for attending en-masse, the students who demoed their Stashboard clones, GA and the course sponsors. Some of the students are looking for a Ruby-on-Rails programmer job, reach out to Matthew Owens at GeneralAssemb.ly (mowens[at]generalassemb.ly) if you’re interested and hiring.

Sometimes you type a hash-bang URL too fast you send many e-mails with an important URL having the bang first. No big deal. Read about it how to redirect it on the Art.sy Engineering blog.

I read Bertrand’s post on “Tales from the Evil Empire” about him leaving Microsoft this morning. Congratulations. We have never met, but your blog was in my RSS feed for a very long time. I would regularly learn valuable .NET-related lessons - my blog happened to be hand-written in .NET and I was still writing quite a bit of it myself with a few open-source projects like dotNetInstaller or ResourceLib.

I left Microsoft in October 2004 to move to New York. If you have worked at Microsoft for over 5 years, and your management, peers and reports value and consider you above average, you should leave too. In fact, if you have worked in any large company, including Google or Goldman Sachs for as long, it may be that time.

Open Source Changed Everything

Bertrand writes:

Almost all sectors of human activity have started to move away from a hierarchical, top-down model, to a distributed one. It takes the form of open source, of peer-to-peer, or of social networks.

Agreed 100%. That’s pretty huge.

Team heads, including myself, are making open-source their foundation. This means building non-core intellectual property components as open source. That’s easily 2/3 of the code you write and we all want to focus on our core competencies. Hence open-source is a better way to develop software, it’s like working for a company of the size of Microsoft, without the centralized bureaucracy and true competition.

Break free and start getting paid for contributing to open-source.

Big Companies Are Always Looking Inward

The amount of accumulated tools and legacy force big corporations to look inward, a lot. Which, in turn, changes the meta-level of your skills. This is a syndrome of  a lot of Google engineers: the amount of infrastructure available to you at Google looks impressive. To their credit Google does open-source a lot of code, but between existing internal distributed file systems and execution grids you actually aren’t capable of building a working large scale service from scratch. It’s like living in a bubble.

In reality, you have everything to learn.

Your Skills Don’t Match New Tech Jobs

Enterprises keep what works to make money. How much overlap do your skills have with a typical startup of 2012? Lets make a list: RoR, Python, PHP, Javascript MVCs? Have you ever deployed an EC2 instance or even heard of Heroku? Have you made a single REST APIs or are you writing SOAP RPC? What kind of Java are you writing today? That’s actually a lot like me a year and a half ago – I am still bewildered anyone wanted to hire me!

If you don’t press the reset button new hires at your company in another five years will obsolete you.

Your Resume Will Look Identical in a Year or Another Five

How much do you read into a resume when you interview people? Your resume will read the same in two years. If you’re an individual contributor it will have a minor technology refresh and maybe a new product, but it will read the same. If you’re a team lead, you might get a few more direct reports. You’ll get that next promotion level – I left MSFT at 65 or 66, can’t remember. Nobody outside of MSFT cared about that.

Five vs. seven years makes no difference, but 2 years at a startup is everything.

You Could Work with an A-Team

Corporations don’t fire idiots as fast and aren’t able to always retain the best, especially entrepreneurial kind. You end up with a lot of average people that breed more average people. That, in turn, demotivates you. Even if you score a great sub-team, you are still ten levels below decision makers that will can it the first time they get a chance.

Finding a new team or starting your own is the most straightforward way to work with an A-Team.

You Can Always Return

Large corporations with good finances will always want to take you back. They are constantly starved for talent and even more starved for former and successful employees that know how to navigate the company. It’s a huge win for a manager to get someone who left by their own will, back. I get a concrete solicitation from Microsoft every three months. It’s flattering.

In my experience, few people return to their mother ship. But it’s there.

Want to Jump?

None of this may be for you. But, as always, my offer to help you stands. If you want to work for any of the companies in the NYTM.org list after reading this post, drop me a note. I’ll help you with direct introductions to the hiring managers.

You should take a serious look at your application and write some tests, first thing Monday.

I would write integration tests with real data that attempt to exploit the issues that were exposed by the Github hack. Even if you’re sure of your code, sit down and write a few tests, just to be double-sure. Don’t do a code review, write some code that will tell you, 100%, whether you have problems or you don’t.

I see two major attack vectors.

Mass Assignment

Read Homakov’s post. If it’s not clear, read it again until it’s clear.

Given models Parent and Child where children belong to parents - can I post a parent’s ID to a form that updates a child and therefore change which parent a child belongs to? If so, you have a problem. Go fix it first thing in the morning in a systematic way, by writing a test that reproduces the issue, then by protecting the attributes with an attr_accessible method. This will filter out everything that’s not in the list when you call update_attributes. Make sure you just use this on all models, all the time.

A variation of this problem is garbage in, garbage out. This affects systems backed by NoSQL document databases. Make sure you aren’t writing random attributes that come from a form into your model. In a relational database you get an exception because the field doesn’t match the schema. In a document store you have just stored junk. It may be harmless or harmful, but you’d rather not find out the hard way.

We use a home grown hash map to whitelist attributes for historical reasons, but attr_accessible does the job just fine.

Identity Confusion

Whitelisting attributes only works when you actually don’t need to assign relationships. Do you pass an identity for a Widget as a parameter, maybe in a URL? Do widgets belong to different users? If so, write a test that ensures that a user that doesn’t have access to this Widget cannot modify it.

My recommendation is to use something like CanCan and to check authorization in a single layer. You spell out who can create/retrieve/update/delete models and enforce this with a single has_authorization_to?. We do this in our API layer systematically. We also learned to key off current_user as much as possible. So if you’re modifying widgets that belong to current_user, you won’t find a widget with a rogue ID by doing current_user.widgets.find(someone_elses_widget_id).

Dear Github

I still love you. This happens to the best people out there. Shameless plug for my former Team SHATTER, if you want a list. Move on and learn from it.

Waffle is a native C# and Java library that does everything Windows authentication (Negotiate, NTLM and Kerberos), on Windows - http://waffle.codeplex.com/.

Slides from my presentation here.

Oh, and yes, if you’re still writing JNI code, stop and switch to JNA.

One of the frequently requested features in Grape to to automatically reload code changes, much like Rails does (#131). This is actually pretty difficult to implement and requires Grape support where an API::reload! method would blow away all of the mounted features, procs, settings, etc.

For bare Rack-mounted applications we can make our life easy with Guard. Introducing guard-rack, which will rackup your Rack application on change. On a decent developer box this just takes a second or two.  The implementation is heavily inspired (aka stolen) from guard-rails.

For a typical Rack application, including one that uses Grape, add guard, guard-bundler (to watch Gemfile changes) and guard-rack to Gemfile.

Create a Guardfile. Watch the folders of your application.

Run with bundle exec guard. Watch rackup happen every time you change files.

I’ve added this to my grape-on-rack demo.

Expanding from my previous post on a Grape API mounted on RACK.

Refactoring the Application Instance

Instead of sticking all of the Rack application code into config.ru, lets build a cleaner Acme::App (in app/acme_app.rb). We’re going to drop Rack::TryStatic and build this logic ourselves, since we might need to deal with other error codes than 404 (depending on your URL strategy you may be tripping over a 405). The logic remains the same: we try a bunch of static files and delegate to the API otherwise. You can also build primitive routing instead, so that everything requesting /api goes to the API and everything else goes to Rack::Static. Your mileage will vary.

RSpec API Tests

Now that we have an application class, we can add API and Capybara integration tests. We start with RSpec and Rack test gems in Gemfile.

The spec/spec_helper.rb adds Rack::Test.

Testing an API involves making requests on the Rack application, pretty straightforward.

RSpec Capybara Integration Tests

Notice that in the tests above we’re mounting the Rack application and making requests directly to it. Does it actually work in a browser? Do we see the public/index.html page?

We start by adding capybara into Gemfile. At the time of the writing we need to use the code from Capybara head, since it adds support for Capybara.app.

The spec/spec_helper.rb requires capybara/rspec, which brings in methods like page.visit and assigns an instance of the application to Capybara.app. Capybara will launch the application for us.

An integration test can go into spec/integration and must be marked with request: true and js: true (the latter forces the use of the Selenium driver that will popup a browser). Lets look for a proper title on the homepage.

A POST, PUT and Some JQuery

The sample source in https://github.com/dblock/grape-on-rack also adds JQuery, extends the API to simulate a persisted counter, and makes PUT requests to it. Complete with an integration test. Run bundle install and bundle exec rackup to see it and bundle exec rspec spec to run the tests.

Backbone.js w/ Mongo

@knewter put together a neat Backbone.js + MongoDB w/ Mongoid demo using Grape that is built in a similar manner, https://github.com/knewter/grape_demo.

(this Capybara is quite sure of himself, photo from here)

Last week I wrote an article on the Art.sy Enginering blog about testing with RSpec and Capybara.

This wasn’t the first time I dealt with UI testing. And I am definitely getting a hang of it. I believe it goes in hand with the fact that I detest using mocks in tests and prefer to test the real thing. Sacrificing test purity for better coverage is better, IMHO.

I’ve done quite a bit of Win32 UI testing: read this post. In fact, the entire dotNetInstaller InstallEditor test suite manipulates UI – check out the source of this test. For the anecdote I was writing a DNI feature on the Subway and the guy sitting next to me looked at my computer as if it were possessed. He proceeded telling me a story about how he thought his computer was hijacked, what the best Anti-Virus was and how he got rid of unusual Windows problems. He was incredible and spoke with great passion. I gave him my card and asked him to come and give a motivational talk on the topic of “computers have a life of their own”. Too bad he never did.

I’ve done a bit of GWT UI testing with Selenium. That was a major pain. Capybara definitely takes a lot of that pain away.

I know the test automation community is always complaining about brittle and slow tests that involve a real browser. I refuse to be a hater – UI test automation, otherwise known as integration testing, has saved my product’s behind many, many times.

I recently read “The Darker Side of Lean” by Darius Mehri. It is a fascinating account of the evil aspects of the much celebrated Toyota Production System. The author of the paper is a mechanical engineer. Yet, the way he describes Japanese approach to choosing technology struck me as déjà vu.

To arrive at the best design, the engineers would gather huge amounts of information, comparing new designs with previous designs. If the technology unearthed by their research could benefit the product in any way, they would include it in the many alternatives they were considering. Sobek, Ward and Liker’s research compare Toyota’s approach to product development as set-based design where “designers think and reason about sets of design alternatives. Over time, these sets are gradually narrowed as the designers eliminate inferior alternatives until they find a final solution.” They claim this method differs from the conventional practice of choosing a single design early on and iterating to improve it until a solution is obtained.

We often think of software architects as experienced developers whose job is to consider all aspects of a future system, including scalability and performance. At extremes this causes architects to lock themselves in a room for days and think about all possible alternatives by making long lists or drawing endless architecture diagrams. Vendors are invited and feature matrices are compared. It’s a very similar process to the one described in this paper, heavily adopted by the Enterprise.

A simple solution is to ensure that architects write code, at least 40% of their time. This forces us to “choose a single design early and iterate to improve it”. The consequences are felt first hand. And when we feel pain with our own skin, we know when it’s time to throw out a component, technology or system and when to go back to the drawing board.

Devise has a :trackable strategy which updates the user’s last sign-in time, remote IP and increments a counter in the User model upon successful logon. This is implemented in Devise::Models::Trackable.update_tracked_fields! and invoked as a Warden callback in devise/hooks/trackable.rb.

Warden will invoke the callback every time warden.set_user is called, which is done once per logon with :event => :authenticate in the options. But when used with the :rememberable strategy, a returning user is not logging on – he continues a previous session. The callback is invoked with an :event => :fetch in the options, which is explicitly excluded in the code above.

Of course, we might not see things in the same eye. A user returning 24 hours later might as well be logging in again. Lets add a callback that will update tracked fields in this case (to config/initializers/devise_trackable.rb).

Is there a better way to do this? Maybe something worth contributing in one way or another to Devise?

I just released AnsiColor 0.2.0 that now supports the ANSI conceal code. Conceal code 0x8 is a hacky way to hide data inside text using ANSI sequences. Jenkins uses it to annotate output with something that looks like ha:<long base64 string>. You can see this implemented in ConsoleNote.java. The previous version of AnsiColor produced garbage by stripping the conceal sequence, therefore revealing the content inside the conceal block (issue #3). Fixed in this commit. Also updated Jansi to 1.7 – Jansi makes this plugin possible.

Update your Jenkins.

I’ve written about delay-processing certain image versions in this article. We’ve made some progress since with a much more robust implementation. Check out this new post on the Art.sy Engineering blog.

We use the Analytical gem to include various thirdparty Javascripts in our Rails application. Our test environment was configured with dummy values in config/analytical.yml.

We wrote a few simple tests to make sure the analytical code is being included properly. For example, we can check whether Google Analytics is included on the splash page.

Or, more complicated, test whether a user ID is sent to Kissmetrics.

The drawback of this approach is that the Analytical code is included with every other test that doesn’t need it. With every Capybara test that runs in a browser, this hits DNS, then makes an HTTP request to Google and Kissmetrics. We cannot stub that with VCR or another gem because we’re using a real browser. But we can selectively enable Analytics without a configuration file by emptying the :test block in config/analytical.yml by configuring it before any test that needs it.

It helps to be open-source. Looking at the internals of Analytical, its options are processed inside each controller on load, then merged with request options on every request before anything is rendered. The code above updates the configuration between those two steps. Note that since we’re modifying a static we have to make sure to cleanup after ourselves.

tl;dr – the source is here

Here’s how I mount a Grape API on Rack and also serve static pages. Most useful for building a service with documentation.

Setup bundler with the required gems. I am using the next version of Grape (frontier branch and the next version of rack-contrib which contains Rack::TryStatic that we’re going to want to use).

Static content goes to a new public folder, for example public/index.html.

Our application will need to boot with all these gems, we’ll do it Rails-style by starting with a config/boot.rb file. It brings in Bundler.

Lets define our Acme Grape API that will have a system resource that answers ping requests with “pong”. This goes into api/api.rb.

The application requires Bundler, all the gems in Gemfile and our API. Lets define it in config/application.rb.

Note the odd File.expand_path construct, borrowed from Rails, - it translates a relative path to the current file into an absolute path, there’re allowing us to run the application from any directory – useful for hosting where you never know who boots the application.

Continuing to borrow from Rails, we will want different environments (development, production, etc.), so it’s a good idea to keep things organized. Setup the environment in config/environment.rb and then load the application.

Finally, we need to “rackup” this whole thing in config.ru. We will use Rack::TryStatic to serve static pages when available and pass through to the Acme API otherwise.

Run the application with bundle exec rackup.

Full source on Github.