I gave a talk last week at AppNexus on building and scaling a test-driven culture.

http://www.youtube.com/watch?v=QvHf94hxzRc

We talk a lot about error handling in Ruby. But we rarely talk about raising errors and creating helpful error messages that are actionable. A good error should tell the developer what went wrong and what to do about it.

One library is known for its excellent error messages. Consider a typical validation error from Mongoid.

This error describes the problem, offers a detailed summary and provides a possible resolution!

Lets implement a similar system for the ruby-enum gem that I live-coded at NYC.rb.

First, add a dependency on i18n and require "i18n". Then, create a lib/config/locales folder and an en.yml file in it. English error messages will go there. This file will need to be loaded by our library, specifically in ruby-enum.rb.

Error descriptions inside en.yml contain the problem, summary and resolution. The YAML format supports multi-lines with \_\_ and can include values from parameters using the %{name} syntax.

The base error class, Ruby::Enum::Errors::Base takes care of the translation. I stripped the implementation details below – the important parts is the BASE_KEY value for localized error messages and the compose_message method. Get the full implementation here and modify it for your project.

Specific errors derive from this class.

When raising an UninitializedConstantError, pass the values of key and name used in the en.yml file above.

Here’s the result.

Beautiful.

I live-coded a new gem called Ruby::Enum at NYC.rb on Tuesday. This library adds enum-like functionality to Ruby.

Here’s my checklist for creating a new gem.

Check the Name

Ruby gems are registered with Rubygems and managed in your projects using Bundler. Search for the new gem name on Rubygems to make sure it's not taken.

Create a Folder

I organize all my code in source and since I often fork code from others to contribute, create a subfolder with the Github username, including mine.

Initialize a Git Repository

Create a Github Repo

Create a new repository on Github.

Add it as a remote.

Create a Readme

Documentation is written in Markdown. Create and commit a README.md.

Push the README to Github.

Add a License

Every project needs a license. I use the MIT license because it’s short and nobody has time to read licenses. Add a copyright notice to the README, don't forget future contributors.

Gemfile

A Gemfile is something that comes with Bundler and declares gem dependencies.

Install Bundler.

Create a Gemfile. For now it just says where to get other gems from.

Run bundle install.

.gitignore

The generated Gemfile.lock should not be included, create a .gitignore.

Enum Library

Create lib/ruby-enum.rb and lib/ruby-enum/version.rb.

Tests

You. Must. Test.

Add RSpec to Gemfile.

Tests need some setup, specifically to load the code in lib. Create spec/spec_helper.rb.

Create a test in spec/ruby-enum/version.rb.

Add .rspec to pretty-print test output.

Gem Declaration

A ruby-enum.gemspec is a gem declaration.

The declaration can be loaded in Gemfile, so that we can list dependencies in one place.

When running under Bundler, the Gemfile will automatically be loaded, which will automatically load the gem specification.

Rakefile

Bundler comes with a number of Rake tasks to release a gem. Add Rake to Gemfile.

Create a Rakefile.

Add pkg to .gitignore.

Default Rakefile to Running Tests

Travis-CI

Add .travis.yml, register the project on travis-ci.org and add a badge.

Enum Library Code

Include ruby-enum/enum. See enum.rb and enum_spec.rb for the implementation.

Changelog

Create a CHANGELOG to list current and future updates.

Contributing

Add a contributing section to README.

Release the Gem

Prepare for Next Release

Bump the version, add a Next Release section to CHANGELOG.md.

Source Code

All source code for the above can be found at https://github.com/dblock/ruby-enum.

There’re a number of excellent micro-frameworks to author RESTful APIs in Ruby. Let’s take a look at the newcomer, Poncho, from the Stripe team, brought up on the Grape mailing list. For those that don’t know me, I’m the maintainer of Grape and use it very heavily at Artsy. I’ll do my best not to be biased.

With Poncho you start by declaring resources that inherit from Poncho::Resource. A resource is a prescriptive way of declaring what parameters to expect and what results to render from the API. In many ways resources are similar to Grape’s entities, except that the latter are no longer part of Grape and you can use roar, rabl or any other framework to render results - you choose. Poncho resources include parameter validation, which makes sense. Personally, language matters, so I would not call values that can be assigned to the resource parameters, but maybe fields or attributes.

You then declare a Poncho::JSONMethod, which inherits from Poncho::Method, which is a piece of middleware. The former is equivalent to a Grape formatter, and the latter is equivalent to the Grape endpoint Rack middleware. You have to override the invoke method and implement your business logic.

How does all this come together?

Grape could easily support the Poncho syntax with a few lines of code. Internally it creates a Proc from the body of the block.

I think that Poncho is in the very early days of development and has made some choices that Grape already made a year or two ago and decided undo more recently. Poncho hasn’t dealt with more complicated problems like versioning or content-negotiation, yet. Once it does, poncho contributors will find that infrastructure begins to bloat and will likely split it the same way we split grape and grape-entity. My current view is that the API DSL layer should be providing functionality closer to the HTTP protocol and a separate entities-like library should deal with data representation. That’s because I have strong opinions that a modern API should be a Hypermedia one, but I would not want to impose this on anyone, which is incidentally a similar philosophy that lead to  Webmachine (check out my earlier Grape vs. Webmachine post).

That said, none of the above is difficult to accomplish.

I would certainly have preferred if Stripe chose Grape as the API DSL and implemented their style of data presenters if they felt strongly about that. But we all want our own micro frameworks, don’t we?

I visited a friend who lives in a remote, tropical region of the United States. He’s a doctor and a practicing eye surgeon. The island has both the very rich and the extremely poor. It’s known for both the beautiful beaches and gun violence in poorer neighborhoods. There’re no Starbucks for over 50K residents. There’re a few richer patients that have good insurance or pay cash, but the majority are older natives with a large proportion on Medicare. A dozen surgeries or 60 patients a day is considered uneventful. A full night operating in the ER only happened once during our stay. There’re talks about closing the only hospital.

The phone rang in the middle of a dinner with a fellow doctor on the other end of the line, and our side of the conversation went something like this: “It sounds like this patient really needs to be seen by a doctor first thing tomorrow morning, so if she doesn’t have a Medicare coupon for a specialist, we’ll see her for free.” I asked him how that’s even an option? He replied that the money made with so many patients is good enough, and the health of the resident population is so precarious, that it just makes sense to be helpful as much as possible.

I think there’s a valuable lesson to be learned here for everyone writing software, too.

By all accounts we’re paid extremely well and we’re as much in demand as doctors. An even smaller percentage of us are the kind of specialists capable of building the software equivalent of performing eye surgery. Of course, there’s no true comparison with being able to directly affect one’s vision with a surgical instrument and the next website, but how can we use our privileged position to, at least, be more helpful? Here’re a few ideas.

If you can afford it, work for a non-profit. If you’re in NYC, I would start with DonorsChoose. Their Engineering team is lead by a very experienced techie and an amazing guy. They promise you’ll never get rich, but you’ll wake up every day with a purpose and heartwarming “thank you” emails from thousands of children in the public school system around the US.

For the socially-conscious that are looking for a startup experience, check out NYC’s Amicus. We shared an office at GeneralAssembly with their founder and CEO and the company has labored relentless helping non-profits be a lot more efficient in their fundraising. This means more money goes to the cause.

Continuing the theme of education, consider Neverware, which makes old computers run like new and who’s CEO is a volunteer firefighter.

Don’t work for anyone evil. One way to ensure that is to join a company that encourages, promotes and writes open-source. I try to practice this as much as possible by going the extra mile of creating new open-source projects, contributing to existing ones, answering questions on mailing lists and educating rising technologists without expecting anything in return. Software has eaten the world, so even if you don’t completely side with Richard Stallman’s vision of freedom activism, you could, maybe, compare the little bit of your open-source work you do to an occasional free visit to the doctor.

A resume is what you did, it’s not who you are.

In the past few years I’ve collected dozens of resumes from junior Engineering candidates and have provided written feedback to those who wanted to improve. I found that the vast majority of resumes have similar anti-patterns, repeating mistakes that are a quick turn off for any hiring manager. If you are in school or a soon to be graduating computer science major and don’t want your resume to go to the garbage, read the following 5 anti-patterns.

Objectives Are Too Broad

“To obtain a full time position that matches my enthusiasm, constant effort and efficiency.”

An objectives paragraph creates unnecessary barriers in hiring you, at best. The above example limits you to full time offers, then just moves some air around. Don’t limit your search to full time positions, especially if you’re trying to relocate. Most companies I know will extend a full time offer if you do an amazing job as an intern. Remove this text and repeat after me: “Seeking a paid internship or full time programming position with an experienced Engineering team.”

Github is Your New Resume

“I am an open-source contributor …”

I wrote an article that went viral, Github is Your New Resume. When evaluating someone’s Github profile, I usually click on Repositories and then Sources. Do you have a project that you have started that has been heavily forked and contributed to? Do you have any activity that’s longer than a year ago or did you just throw some code up there to beef up your resume? Does your contributions graph really look like this?

I don’t expect a junior developer to have any of these things. Displaying a link to an empty Github profile is a terrible idea. Do link to your personal website.

You have Lead, Driven, Managed, Advised or Evangelized

“Advised xyz.com founders in technology selection.”

It reads like your head can barely squeeze through your chateau’s living room French doors! You are not the project’s lead because you’re the only programmer. You’re not an advisor because your friends started a website and wanted to know which language is cool. Downplay and humble-down every single word on paper. Let your results and references speak for your value, not your resume. Remove every single adjective or verb that isn’t “did”, “implemented” or “built”.

You Have Expert Skills

“Expert in C++, Photoshop and Excel.”

I recommend removing a list of skills altogether and writing, in tiniest font possible, the technologies used within your work experience.  Should you talk about source control that you have worked with on the resume? My answer is no: make sure to avoid the impedance mismatch between the tools and technologies that you describe. My rule of thumb – delete anything that’s not a programming language.

You Took Many Courses and Excelled in Clubs

“2nd Place in the Campus Green Computing Initiative”

Most of us who left college many years ago will roll our eyes when you ask about those college years.

I think your resume should demonstrate that you can make an effective transition from college to the workforce, which means leaving as much “college” behind as possible. Being a Teaching Assistant should go into your work experience, but your participation in chess club should be left out. I would also not include a mention of your GPA. And any work experience that you may have should be concise and explain what the project was and what you did within that project.

***

If you like the advice above, and are in college or a recent grad, send me a copy of your resume to dblock[at]dblock[dot]org. I promise to give it 15 minutes and email you a few bullet points of feedback.

I’m very excited to Release Grape 0.4.0 today. Again, thanks to the 10 (!) contributors to this release.

Is this a solid release? You bet. I’ve upgraded our largest API project at artsy.net to Grape 0.4.0 and all of the 1870 API specs have passed.

The complete CHANGELOG is here. There’re a few backwards incompatible changes that you might want to check out while you’re upgrading.

JSON and XML Formatter Changes

Until the 0.4.0 release Grape took a pragmatic approach to JSON response and request data. It was also incorrect.

If your API returned a an Object, it was translated to JSON via to_json or a MultiJson.dump call. This worked well for both Hash and Array, meanwhile Grape left objects of type String alone. This friendly method ensured that you could translate an object into a String, skipping all the hard work Grape’s formatter would otherwise have to do. Similarly, Grape tried to be helpful by ignoring any data that couldn’t be parsed into a hash and therefore retrieved via a call to params from a PUT or POST.

I have fixed this in @6fa8f59, released with Grape 0.4.0, and it’s now possible to return proper JSON representations of NilClass or String, as well as to parse any valid JSON, such as Array. This means that if your API implementation returned a String or a nil, the output will be different.

You can monkey-patch back the old behavior as follows.

Similar changes apply to the XML formatter.

Reading Input

With the above changes its now possible to read JSON input that’s not a hash. This was reported as a bug in #347. Hash parameters continue to be merged into params, while everything else is now available in env[‘api.request.body’], you don’t have to parse env[‘rack.input’] again, a notable performance optimization.

Default Log Format

Somewhere in version 0.3.2 the default logger format was inadvertently changed to be something like this.

This was a side-effect of requiring all of ActiveSupport, reported and fixed in #353. The logger is now back to be its undecorated version without all those timestamps, levels or process IDs.

If you want the above behavior, require active_support/all.

https://github.com/dblock/resourcelib

Download: http://code.dblock.org/downloads/resourcelib/Vestris.ResourceLib.1.4.zip

• #2: Added VersionResource.FileFlags - @alaendle.
• Updated solution to Visual Studio 2010 - @redwyre.
• Added NUGet references for NUnit, NUnit.Runners, and MSBuildTasks - @redwyre.
• Added more options to build.cmd: code, code_and_test and run_test_only - @redwyre.
• Documentation rewritten in markdown - @dblock.

ResourceLib is a framework that enumerates resources and implements both read and write of the file version VS_VERSIONINFO, string resources such as company, copyright and product information, RT_GROUP_ICON and RT_ICON, RT_CURSOR, RT_BITMAP,RT_MENU, RT_DIALOG, RT_STRING, RT_ACCELERATOR, RT_FONT and RT_FONTDIR and RT_MANIFEST resources.

My talk from JUG NYC from last summer has finally been uploaded to YouTube!

There’s also a summary with all the links in this blog post and the slides here.

Le Clash of Booba, Rohff and La Fouine have a new player, RapGenius.

James, who I have met, and who is a super nice guy wrote a post called “Heroku’s Ugly Secret”, describing the changes in the Heroku’s routing mesh on the Cedar stack and how much it sucks. It’s a bit sensationalist, and there was some hint on the evil overcharging overlords. It’s the price of getting up there on HN, I guess. I was going to just watch this quietly, but I just can’t.

This is unproductive. This sensational news has been discussed here and here a year ago. Man, you tore. Them. UP.

Heroku is not an evil organization and it provides an amazing service to millions of applications. It has enabled so much in my world! It has very smart developers that try to be as helpful as they can and, I am sure, do their best to improve the said routing. They don’t always succeed, but they certainly don’t deserve “serious internet beef”, which in Russian we would call “полемика высосанная из пальца”.  Lets all get back to making technology a better, more civil place, please.

I am increasingly convinced that those who blame MongoDB for any kind of data corruption are waving hands out of desperation. Every single time I had an issue with data I ended up finding code that I’ve written or code that was written in a library that I used. The hardest bugs involve some kind of concurrent operation, including one I’ve described in a new post on the Artsy Engineering blog: Data Corruption and Concurrent Updates to Embedded Objects w/ MongoDB. Read it, especially if you are using an ODM to manipulate embedded objects.

I also want to point out two magical lines of code that I use all the time.

Knowing what Mongoid is doing is half the battle.

Heroku and MongoDB enable the kind of frictionless devops workflow every mature software organization wants and spends mountains of money on. It begins with a 1-box developer setup in which I can checkout source code and be ready to start development after a bundle install, and continues with a git push to a Heroku staging environment that deploys the application for last minute integration testing  and, finally, production. Each separate Heroku instance has its own configuration and resources. My code also needs sensible defaults for development. For example, mongoid.yml lists localhost:27017 as the development database host and ENV['MONGOHQ_HOST_LIST'] for production.

However, in true devops, I want to be able to reach into a remote environment programmatically, run commands remotely or connect two environments together, as long as we have access to them. In this post I’ll propose an implementation of three tasks that have become part of the daily routine of one of my projects.

1. Execute a Rake task with local code modifications and the configuration of a production environment.
2. Open a shell to the primary node of the MongoDB on my development Heroku environment.
3. Dump a single table from a production database before doing something scary.

We’ll accomplish the above with help from two new gems: mongoid-shell and heroku-commander. Add those to your Gemfile in the :development section.

And, please remember, that with great power (and the word “production” in much of what follows) comes a lot of responsibility.

Execute a Rake task with local code modifications and the configuration of a production environment.

This is made possible by heroku-commander. The library wraps the Heroku CLI (intro here) and will run heroku config –s. It’s now easy to reach out to a Heroku application and retrieve its configuration programmatically without worrying about API keys (by default it will use the Heroku app defined via the “heroku” GIT remote). We will also need a bit of code to apply our application’s naming convention. This lets me change the execution environment to the one of a remote Heroku application, in Ruby.

So how do I run a task locally, but configured as production? With the following Rake task.

Run RAILS_ENV=production rake heroku:config_from_env my:task.

Open a shell to the primary node of the MongoDB on my development Heroku environment.

First, figure out the remote MongoDB configuration, then execute the mongo shell command. It’s important to know that the built-in system command doesn’t raise an error when the process returns a non-zero status code. Lets add a system! function that fixes that.

Instead of making MongoDB command lines manually, I’ve used a new gem called mongoid-shell (intro here).

Run rake db:production:shell.

Dump a single table from a production database before doing something scary.

We all do backups and other important things, daily. But when manipulating production data I want to have the last safeguard with the freshest data from the collection I am about to drop accidentally update. Dump a MongoDB collection locally.

Points

Thanks to @joeyAghion and @fancyremarker who are responsible for many of the core concepts above.

There’re two kinds of developers: those who use should in RSpec examples and those who use present tense.

Should you use should?

My short answer is no, you shouldn’t. Write your examples in present tense. Two reasons.

• BDD often refers to examples as documentation. Imagine that your fridge’s documentation said “when plugged in, it should keep food cold”. IMO it should say “when plugged in, it keeps food cold”.
• Using “should” makes the text longer by a few characters. We have over 3000 specs in one projects, probably saving a megabyte worth of test descriptions by not using “should”. Shorter is better.

(@mzikherman has heavily contributed to this thought)

Pure. Evil.

New York Times has just published “New Cornell Technology School Will Foster Commerce Amid Education”. I am super excited about Cornell Tech, and not just because Ariel Kaminer chose to quote me on the modest Artsy partnership with the University :)

In 1994 I lived in Switzerland and was contemplating going to either the University of Geneva or the Polytechnic of Lausanne (EPFL). I was convinced by the former, which offered a much more theoretical education in mathematics with world-wide known professors, had strong ties to the CERN and frequent contacts with some of early Internet legends. I felt strongly about not wanting an education of an applied computer programmer, which I thought as inferior. I imagined UniGE producing scientists and the EPFL, the equivalent of car mechanics. So I spent three painful years sucking at math, finally finishing with a year of trivial computer-related courses, and generally coming out of the experience wanting to put it all behind me.

15 years ago your ability to memorize 4 whiteboards worth of proof for each of the 120 theorems of Algebra II was the ultimate benchmark for computer science students. Today’s measure is your Github public activity, your ability to translate theoretical knowledge into working systems, excellent social and networking skills, clear communication and mad skills of learning and adaptation. Any educational program that can produce well-rounded individuals that can enter the workforce without feeling like they have been hit but a bus is on the right track.

With people like Daniel Huttenlocher and Greg Pass New York’s Cornell Tech is going to succeed at this. I am excited because they are doing all the right things, including partnering with smaller startups and seeking out exciting projects for their students, integrating them with the industry early, right here in my favorite city in the world.

Aside of all the posts about recruiting (recruiter spam dropped by 75%, amazing, thank you very much), website performance and the importance of source code, I wrote a few posts recently on the Artsy Engineering Blog and made some open-source gem releases and contributions. It feels really good and there’s a whole new week ahead!

I got very excited with getting my hands into em-proxy and writing Beat Heroku’s 60 Seconds Application Boot Timeout with a Proxy using the heroku-forward gem. I even contributed to em-proxy itself, adding support for unix domain sockets. It’s exciting because em-proxy is written by @igrigorik, who is one of those impressive people that I still don’t have the pleasure to know personally. The post was accompanied by a more philosophical piece on Losing Control to the Heroku PaaS. Speaking of losing control, I annoyed the crap out of Heroku developers trying to figure out why an application was timing out during bundle install on Heroku, which turned into an already well trafficked blog post on Debugging Bundler Issues on Heroku. Sorry everyone.

Today I released mongoid-cached-json 1.4 and garner 0.2 with some significant performance optimizations for the former, described in today’s blog post. This was some pretty exciting, although relatively basic CS work, that didn’t always turn out the way you’d expect from performance improvements. My first optimization ran 20% slower than the original – you cannot make assumptions about the runtime characteristics of theoretically better algorithms until you actually benchmark them.

Grape got a lot of love with the release of 0.2.6 after a ton of formatter-related work and security fixes via multi_xml (upgrade!) and I plan to push 3.0 really soon. Generally, Grape is getting much skinnier, faster and lighter: I simplified a lot of internals and entities were taken out into the grape-entity gem by @agileanimal.

Last week I released guard-rack 1.3.1 in which @viking got rid of .pid files and I finished by making it compatible with all versions of Ruby using the awesome spoon gem (solid replacement for not-so-cross-platform posix-spawn).

Finally, I posted an amazing study of a horse by George Stubbs on Artsy. You can follow me there for all the art, and for my tech stuff, on Twitter.

everything is in your own hands by Nele Brönner and Luca Bogoni

Recruiting is broken and a plethora of startups are attempting to fix it with better mouse traps, in which you, the magical engineering unicorn that everybody wants, continue being a mouse. Trapped. The basic problem is the same: no matter what recruiters tell you, however genuinely helpful they try to be, they don’t have your best interest at heart, except in a single case where you want to maximize your money. More on that in the final words of this post.

To fix the problem we must replace the recruiters that suck by recruiters that don’t, with only the candidate’s best interest in mind. This is not a problem of tools. We must replace recruiters with ourselves.

Building Industry Relationships

Engineering doesn’t happen in a vacuum behind closed doors anymore. Deliberately maintaining long lasting relationships with other technologists is part of your job.

My first contacts were made with people I went to college with or worked with in the 90s. I have great friends in Seattle and San Francisco and amongst those I hung out with after moving to New York ten years ago. I made some amazing connections through open-source coordination work on popular projects such as JNA or Grape.

For me, being deliberate about building industry relationships means that in every conference, event or social function I set myself the objective of meeting and establishing a contact with just one single peer who I respect for technical achievement and who is currently or often be in a hiring manager position. I’ve invited all these people to visit me during work hours, showed them what we’re working on and often bought them coffee at La Colombe around the corner. It’s amazing to brainstorm with these smart technologists, so I get a ton out of it. I’m booked almost every early afternoon for the sole purpose of meeting people.

I don’t want anything from these connections except an opportunity to learn from them and to refer engineers to them.

Opening Your Doors

When I worked at Microsoft my manager used to talk about an “open door policy”. Everyone literally had a small personal office and keeping the door open meant allowing anyone to walk in and talk to you. Today I still keep a wide open virtual door.

I’ve written several blog posts that have gone viral, including Your Commute is Killing You, Move to New York, followed by  Moving to New York? A Guide for Software Engineers and even a more comical Sitting is Killing You? Move to New York. (You get the idea – move to New York!) Since then I’ve received hundreds of emails and have invited countless engineers to visit me at work. A disproportionate amount of these are talented developers stuck in a cubicle working for the BORG or generally unhappy with their current situation. But more than a new job, they want a first, personal, connection to the New York startup scene. Someone helped me with this a long time ago – I owe it to them to return the favor.

I’ve offered at least 3 personal connections to every one of these engineers visiting me. I’ve directed others to NYTM and offered more personal help if cold connections failed. I’ve hooked up people with startup CEOs running tiny teams at GeneralAssemb.ly where we used to have our office and that was my entry in the NYC startup scene, engineering managers that are actively hiring at other NYC startups or other technologists with similar technical interests at large.

I don’t want anything from these referrals except an opportunity to help them. I am very emotionally attached to New York and helping a single engineer helps the NYC tech scene rise.

Becoming a Connector

I am a connector. I take the left hand side of my relationships with hiring technologists and the right hand side of my connections with engineers and make introductions between the two. I don’t necessarily enjoy being part of their conversation and I have no incentive other than helping the two individuals involved. I am a recruiter, the kind you actually want, because I have your interest at heart. I once read an article that said that casual professional acquaintances are much more likely to help you find a new job than your close friends – I am that acquaintance.

If all engineers in a similar position to mine became recruiters, the first reaction of someone looking for a job would not be to post their resume online or update their LinkedIn profile and get spammed. It would be to quietly reach out to 3 people, which would make 9 direct connections, which is plenty to get a good view at what’s going on in your city right now with job opportunities in your industry, line up a few interviews and land your next gig.

Exceptions and Money

So what are actual recruiters good for?

There’s always an exception to the rule. Some engineers just want the best paying job possible, anywhere. They will compromise on the excitement, the interesting project or the view. While I may have your happiness at heart I probably don’t have as much stake in your financial interests. So by all means use a recruiter who will do their best to maximize your money. A good recruiter is like a salesman and will negotiate your pay to get a nice 25% cut of your yearly salary when you get hired. To every connector I recommend establishing a business relationship with a single professional recruiter you have used in the past and that you trust not to screw you or the candidate over. Recommending a good established recruiter to a candidate that wants it is good for our industry. This will force natural selection and help weed bad recruiters out of the system.

For Those That Love Lists

1. Establish a referral network with your peers.
2. Offer help to those seeking opportunities.
3. Make connections and help in any way you can.

It’s obvious. When was the last time you did this? Fixing recruiting is in your own hands.

An investor, a startup CEO, and an engineer walk into a bar to discuss “Page Weight Matters”. They naturally end up talking about their own slow website.

***

The CEO’s friend works in finance. She invested a bit of cash in her friend’s startup. She came home late, made herself dinner and read her personal email (her bank won’t let her do that at work). The website’s newsletter had something interesting, so she clicked. She stared at a white page for a bit, then it finally loaded after 8 seconds. She was already bored and figured it’s not worth it. “Something’s wrong with my internet”, she thought. Google came up instantly, though, so she emailed her CEO friend.

This morning the CEO had to make the difficult decision to invest into a new kind of reticulated spline, the 3-dimensional kind. After looking at the data gathered for the past two weeks by the early prototypes of the 2-dimensional spline, it became clear that users want that other, much more esoteric, 3rd dimension. Tomorrow he’ll gather the team to show them the user data. Also, a friend and early investor emailed him that his website was slow. Going to the website right now on his iPad confirms that, indeed, that 2D spline page takes 4 seconds to show up. Trying again was fast, though. Weird.

The engineer was watching an API response graph that oscillated abnormally with 4 seconds peaks fetching the 2D spline when he received the investor’s email forwarded by the CEO. A large code refactor that enables a much faster 2D spline page rendering finally had a successful build and was now ready to be merged onto the master branch, pending code review. He replied to the CEO’s email with the good news of progress being made before leaving the office.

They all meet at Tom & Jerry’s. Where else?

***

If you can find yourself in my story, you’re in a good place. You have the CEO doing his job having visions of the future based on the experiences of the past. You have engaged users that want you to succeed and are willing to provide feedback. You have engineers doing their job building new features as fast as they can and fixing issues, including performance, as they are found incrementally. You are in a bar in New York with cheap drinks.

Generally, though, the website is slow, and that’s a problem. What should you do?

The answer, in my opinion, lies in a basic equation. You can focus on performance, but your features won’t move forward. Build more features, and performance will suffer. To have both takes much more time. You want to find the right balance. Here’re 5 ideas for engineers and non-engineers for improving website performance.

For engineers:

1. Observe humans using your product and don’t make excuses, your system is slow until it’s not.
2. Optimize for simplicity, complex software is very hard to optimize for performance.
3. Measure your slowest components. As an individual contributor, address one a week.
4. Discuss alternate solutions to experiences that will perform poorly with the product and design team.
5. Regularly undertake large projects that address fundamental bottlenecks by your own initiative.

For non-engineers:

1. Learn about the engineering process and technical debt.
2. Avoid asking “When is it going to be done?”. Organize features in a backlog rather than milestones with deadlines.
3. Be helpful, but don’t try to suggest technical or external solutions to performance problems, such as experts or consultants.
4. Give improvements time and assume that a good engineering team knows what they are doing.
5. Be constructive when reporting slow parts of the system and avoid broad “it’s too slow” statements.

It’s incredible to see talented people succeed at building great things.

I met Dmitri through a mentorship program and brainstormed with him during his early prototyping of Backspaces. I told him to feel free to ignore most or all of what I said, too. So, needless to say I was pleasantly surprised when he attributed an excellent piece advice to me in his recent blog post – Treat Every Line of Code as If It's Going to be Thrown Away One Day – and put it in the context of programming as an art form.

Many have drawn parallels between software and art, but I think that while programming is most certainly a creative endeavor, most of its output is mastery and technique. A piece of art may live forever and offer new generations an updated perspective in their context, whereas a program is ephemeral by design, and is quickly made obsolete with rapidly advancing technology. And, unlike art, the large part of elegance of a program is perceived at the time of the writing  (replaced by a new, less beautiful and less buggy implementation the next morning). So, how long should code live, anyway?

For a startup, the answer to that question is not long. A true startup is experimenting and looking for a business model. In that context code is more dead weight than intellectual property. The best we can do is not to get attached to it and treat every line of code as if it’s going to be thrown away one day tomorrow.

Just this week I deleted a huge and solid piece of functionality that lived for barely three months and took weeks to develop. The complexity of the feature was staggering and the implications of the change reverberated through every part of the system. But I cannot feel bad about it, it has served its purpose and show must go on. We all have to remember that the real goal of software is to empower or entertains an individual, a group, or, in rare cases, society. The editor I am using now is helping me put my thoughts on this blog. The website I work on every day brings art to millions of people. Software makes it possible for an airplane to fly across the ocean to visit your family. It brings planet Mars a bit closer – I hope to visit one day.

Code may be important, but everything else is more valuable.

I love Heroku and we have had great success with it. Yet, today, I find myself releasing heroku-forward, a gem born from frustration, that works around a seemingly simple limitation of the platform - a 60 seconds application boot timeout.

Why did I need to do all that work? How hard can it be for Heroku to raise a timeout limit? The truth is that it’s probably very hard. I don’t know anything about Heroku’s internals, but I bet that there’s some math going on on for resource allocation as well as a system that runs on a 1-minute cycle. Changing a number impacts everything, making some customers happy and others upset, adding risk.

We all make similar design choices that hard-code numbers all the time. I’ve deliberately created a 1:1 relationship between an object and the object’s owner knowing that in some future an object may have multiple owners. I’ve probably saved countless hours of minor pain doing so, but I am about to pay the full price. The entire system must be turned on its head.

Are those two issues comparable? The biggest difference is control. In the Heroku PaaS example, someone else has it. In my domain model, it’s me.  But after years of tinkering with every single layer of hardware and software, I find myself increasingly comfortable delegating control of those areas outside of my domain expertise. I guess it’s called focusing exclusively on your core competencies.

I was recently quoted in a TechRepublic article, making rather sweeping general statements about how the free open-source NoSQL databases are the best thing since both SQL and sliced bread. I think some context may have been lost there, so it’s worth talking about it.

Two years ago I was looking at the Art.sy prototype implemented with MySQL. I found out that it was doing a JOIN to compose the hundreds of genes with their values for each of N artworks at startup time, where N > 10000. That cannot scale, but it’s a dumb problem with many solutions. The best approach I could think of was to choose a store that natively supported a hash-like data structure – therefore I chose MongoDB. Learning how to use it was easy – much easier than even installing Oracle 11g. But even I can argue today that a relational database would have continued to pay dividends by helping me with data integrity, while a NoSQL store required a lot of programming discipline. Do you trust a programmer or the machine? The jury is out.

The article’s conclusion begins with “Foursquare’s and Art.sy’s applications may be somewhat unique." I would want to stop there. Every single application that you’re building is unique and deserves individualized attention to its data. Furthermore, database technology makes giant leaps every year in both SQL and NoSQL spaces - when I started, Heroku didn’t have a PostgreSQL offering that can do 1TB storage with a 68GB cache layer and fork-and-follow, and Amazon did not have DynamoDB that runs off SSDs. Personally, I don’t make such a black and white distinction between SQL and NoSQL anymore – these are powerful tools available to the developers – I choose the best tool for the job in front of me.

PS: we use MongoDB in production and I strongly recommend it, but if you find me talking excitedly about the next big thing in databases today, it’s likely to be Datomic.

When I joined Microsoft in ‘99 I was taught how to properly interview candidates. I was shown the wheel of competencies, a kind of a wheel of fortune where a color represents the candidate’s technical skill, ability to solve complicated problems or to communicate with their peers. Each  slice included broad interviewing suggestions, which often gave birth to elaborate puzzles. What could possibly be the best way to figure out whether the candidate is capable of thinking out of the box? Ask them why the potholes are round. Can they crank complex working code on a deadline? Ask them to implement a memory allocator in 30 minutes or less in C.

As the interviewee I’ve been given all these problems myself. I hated the puzzles, but I loved the code, even though I failed my share of coding interviews - one that I still remember was for a team within the Windows division, where I was asked to implement “malloc”, which I think went well, and then some other simple string manipulation in C that I totally missed because I didn’t listen to the problem carefully enough. I also bombed an interview at Google because I’m still not capable of doing a power of 2 after 4 years of hardcore college math.

Below are some well-researched and widely practiced techniques of breaking even the best interview candidate.

***

1. Whiteboarding

Ask an expert C++ level candidate to reverse a string in C.

This is guaranteed to be insulting to someone who has demonstrable success developing software in any relevant technology. It’s the first thing taught in any C class and should take you about 45 seconds. The fact that many just cannot do it is irrelevant, those should never make it past a phone screen. By giving this problem to someone you’re telling them that you don’t trust their resume and that their demonstrable experience is completely irrelevant. It’s a single, efficient blow to their ego.

2. Dehydration, Starvation and Solitary Confinement

Create a system in which the candidate does not have access to food, water or bathroom for extended hours and spends significant lone time in a small, preferably windowless space.

After each 1 hour interview, grab the next interviewer and interrogate the previous interviewer on the strengths and weaknesses of the candidate. There’s a series of canned questions to ask: What did you ask the candidate? What did they answer? How well did they do? What are the areas that they failed at and that need to be drilled further by the next interviewer?

This is a clever management technique that extends the interview schedule by about an hour. The candidate is left in a conference room alone, while observing their incomplete solution for the previous problem on the whiteboard. I’ve never seen a candidate walk out to find a bathroom or a glass of water - they don’t want the next interviewer to come to an empty room. The bladder will speak in the middle of the next round, at which point it’s not polite to ask to take a break either. Lunch will be skipped since everything is running so late and the CEO only has a 2:00-2:01 PM window to meet.

3. Pitchcapping

Ask the candidate to solve a simple problem in a completely different way, until you run out of time.

This is akin to pouring hot pitch on a candidate’s head, letting it cool and then restarting again. It’s theoretically a good exercise if the question is complex, but it’s guaranteed to stump even the best programmer on the fifth “Can you please implement fib(x) in yet another completely different way?” iteration. You’ve succeeded when the candidate gives up and says that there’s just no other reasonable way to do this, at which point you end the interview, shake their hand and walk out.

4. Keelhauling

Lead the candidate to a stopping point where they are drowning in a problem, then maintain them underwater.

Pretend to be helpful. Finally, give them a few minutes to concentrate and solve the problem alone. Have a smoke outside. Come back 15 minutes later to see no real progress. Ask them to explain what they wrote and keep a concerned look. Finally, shoot them with a “this is good, although pretty far from working, and we’re out of time”.

5. Flaying

Ask an executive-level, non-programming interviewee to implement a sort algorithm of their choosing.

Guaranteed to remove any kind of skin that one developed over their 20 year long career. In small companies low rank developers interview their future executive-level bosses. Guide the candidate to talk about how they got started and wait for the story about their small, yet relevant programming experience. Mention with a smile that you’re impressed by their solid CS foundation and that “they must surely be able to implement a sort function with a bit of help, even today”. The candidate will definitely bite. The next step is a whiteboard.

About 10 years ago, one CTO candidate told me that he would walk out of the room if I was actually seriously about asking him to code a sort. He complied though, got sweaty and nervous and finally broke down with the next interrogator after me. His interview was cut short.

***

In a future post I will talk about great ways to interview people. In the meantime, I would like to apologize to all my past interview detainees and ask you to post your torture stories in the comments!

If you’re not using NewRelic, here’s a pretty picture to convince you to try it out. And a blog post on measuring performance in Grape APIs with NewRelic RPM on the Art.sy Engineering blog, with a Grape middleware I rewrote last week.

You got to love debugging JavaScript errors. Here’s one from a MongoDB map/reduce. The operation fails with this:

Examining the MongoDB log we notice that the map/reduce completes to about 70%, so this is caused by some data inside the database. In this case the issue is in the spline object being accessed by the map function. Indeed, the map tries to fetch spline.reticulated, which is hopefully a Boolean value. The error “spline has no properties” is, generally, another way of saying “spline is null”. In my case splines live in an embedded collection inside bones, and aren’t expected to be null, so I need to track down a null spline inside bone.splines.

While we can just look for a null embedded object, more generally, we can find whether a JS object has no properties with this isEmpty function from SO. Declare it in the mongodb console (use a 1-liner):

Iterate over all bones records.

Here’s the culprit, notice the null embedded spline.

You can get rid of it in different ways on the mongo console. In this case we have two splines, one to keep and another to erase.

How did we get into having this null record anyway? See https://jira.mongodb.org/browse/SERVER-831 and https://github.com/mongoid/mongoid/issues/2545.

I’ve been working on the the tax_cloud gem for the past couple of days and am happy to announce version 0.2.0, released today. The gem was started by @tempelmeyer and is now a mature wrapper for the TaxCloud US Sales Tax calculation service.

This library is also a nice example of a generic SOAP client wrapper in Ruby. I wanted to point out several successful patterns for this integration, which I cannot take credit for, for the most part.

Error Handling

I borrowed error handling from @modetojoy’s Mongoid. Today someone said: “I had a bug in a spec and Durran told me how to fix it in an error message.” True story. To accomplish this we define a base error that holds the problem, summary and resolution. In tax_cloud’s case this is  TaxCloud::Errors::TaxCloudError paired with config/locales/en.yml, a locale file that does the error formatting. There’re two things to do in order for the error code to find the message: add the locale file to the load path, in tax_cloud.rb, and do a bit of formatting with I18n.

What does an error in tax_cloud look like?

Pretty awesome.

Safe SOAP Requests

The tax_cloud gem uses Savon to make SOAP requests. “Savon” is French for “Soap”, which confuses the French speakers like myself trying to explain that SOAP is Savon. Anyway, a client is initialized with its WSDL.

First, we need to add authentication to every request, which is required by the API. We can override request.

Second, we want to handle SOAP errors, and give a detailed explanation for SOAP faults, Mongoid-style. This is your typical block with yield.

The complete code can be found in client.rb. The error itself is parsed in soap_error.rb – SOAP faults come in standard format.

Parsing Responses

We will now raise a good-looking exception on SOAP failures, but we still must protect ourselves from unexpected data or successful SOAP requests that return API errors. That possibility is the thing I detest most about SOAP (vs. REST) – it makes programming a client unnecessarily complicated. The TaxCloud service returns a SOAP body with different values in key_response/key_result/response_type, where the key will be the name of the method invoked (eg. ping_response). A bit of meta-programming can make a base class, which can parse a response and match an XML path, raising errors where appropriate. It can be subclassed into a generic response type and, finally, into specific declarative implementations such as ping or authorized.

Most services have a common response pattern, generalizing it yields a very productive framework where adding support for new calls requires very little to no code. And you must never, ever expose to the user that you’re making SOAP requests and return any kind of raw SOAP object. Return domain-specific classes with attributes on success and raise exceptions otherwise.

Testing SOAP Requests

The tax_cloud gem uses VCR to test SOAP requests. It’s surprisingly easy: use a cassette (a YAML file), which records it the first time you make a request. Second time around the file contents are used and no HTTP requests are made. You can filter out sensitive keys in the configuration.

You can see the rest of the tests here.

Finally

Let me know if you use some of these ideas, post your comments and suggestions and please help improve the tax_cloud gem on Github.