Last night I gave a talk on pressing the big scary red reset button at the NY CTO School. The first talk before me was by @malcolmcasey, CTO of Skillshare, another awesome NYC startup. It’s a great meet-up for early team leads that want to learn from other technologists that have been doing this for a while.

I’ve reset a few projects in my life. None as big as Microsoft Windows Longhorn (Vista), which was a fun example of failure of monumental proportion. In 2006 Jim Alchin wrote in the Wall Street Journal: “Longhorn was crashing into the ground.” While he was credited with ultimately shipping Vista following a successful reset, it was one of Microsoft’s worse operating systems. Developers just no longer cared. You can see how a reset is by no means a guarantee of success. In fact, anecdotal evidence shows that 7 out of 10 projects that are reset still fail. And when projects fail there’s plenty of wrongs to be pointed out. Marketers will blame the product, product managers the technology and everybody will blame other people. A few will cite failure all around or an unstoppable death spiral.

I interviewed several CTOs when I was preparing this talk. It was really fun to hear about their experiences. They would usually start with a gasp and use a particular project reset vocabulary. “We slammed the brakes.” “Changed the wings on an airplane.” This might actually be a silly analogy – have you ever seen anyone attempt such a feat? No wonder most projects fail after a reset – taking the wings off in-flight means crashing into the ground, guaranteed. But if your reset is going too smoothly, feel free to add some drama as in this amazing video.

I think that pressing the reset button on a clearly failing project is always the right thing to do. Put some numbers in perspective by measuring what value the project is yielding vs. what it costs. You may have sunken a ton of money into it, but you’re just bleeding more. To execute, get rid of all the chickens (people who aren’t committed to the project with their flesh and bones), trim some fat and time-box the reset. Use all the brownie points that you have earned until now with the rest of the company (that’s why it’s so important to invest in relationships inside an organization when things are going well, both up and down). Successful resets are very rewarding and have a long tail, but people tend to block out negative emotions associated with violence and doom that occurred in the past, so few will actually remember. You’re not a hero, you’re just doing your job.

Finally, things don’t need to reach the dire straits. Personally, I try to foster a culture where we experiment with everything. Want to build the new system with a different data store? Try it. Want to rewrite our entire UI with anotherframework.js? Go for it. You don’t need buy-in from me – you just need to be transparent and let everybody know what you’re doing. You’ll get plenty constructive criticism from others - make your own decisions. The job of a tech lead is to keep the output of the entire team massively positive, not to micromanage individual’s minutes. For every one developer stepping out of doing visibly productive feature work for a sprint another will step out to find a multiplying factor of 10 for something you’re building next. It’s a small price to pay for being awesome, as a team.

Slides: http://www.slideshare.net/dblockdotorg/pressing-the-big-scary-red-button

I wrote a blog post yesterday about How to Start Small With Big Data and Google Analytics. Essentially, it’s a jumpstart for fetching Google Analytics pageviews and merging the daily data with some domain knowledge. But it also asks an important question:

Why do so many companies write a homegrown pageviews tracking system? Between Google Analytics, Kissmetrics and many others, isn’t that a completely solved problem?

this is a solved problem, too

I look back at my early years in software development and find myself rewriting a lot of software that has already been written by other people. It’s a typical overconfident engineer problem, where my initial reaction to anything created by other people is that it’s insufficient in too many ways. It started with early C++ code, where achieving performance and portability between many *nix systems and Windows was a nightmare – STL wasn’t nearly as good as today. I ended up rewriting everything, including String or Vector classes (an entire C++ library open-sourced as https://github.com/dblock/baseclasses). I then went on to work for Microsoft where we were trying to build a 24/7 service that would not die from memory allocation failures. .NET didn’t exist then, MFC was out of the question because of it’s legendary reliability and so was the Microsoft version of the STL that was just merging with MFC. Open-source was not even part of Microsoft lingo. We built CoreSDK, a C++ library where I ended up rewriting everything, including String or Vector classes, again. I think I implemented at least a dozen versions of String in my life that went into some production code. Those were the days!

This decade’s software is way too complicated to rewrite everything. When I choose a platform or framework for a project, I like open-source ecosystems that encourage reuse. Adding third-party components in the .NET world is not commonplace. Adding dependencies in Java with Maven is better, until you need to make changes to the code in your dependencies. The Ruby world with Rubygems and Bundler gets it right: there’s virtually no overhead for using a third-party library. Node.js is a bit younger, but also gets it right.

Now that I can easily reuse third-party systems, I must try to remember that I should write less lines of code, introduce less operational overhead and create more positive product impact in a reasonable amount of time. It doesn’t mean I just want to slap together someone else’s components - things worth focusing on should be elements that serve my company’s core competencies or be educational for me, useful to the open-source community at large or just plain fun. While this sounds obvious, sticking to the guidelines and keeping myself accountable is a daily effort.

How do you avoid rewriting everything?

I want to talk about your scope of influence as a Software Engineer and how to expand it.

Your scope of influence in your first piece of software is a function. You are parachuted in a well isolated world and are told to implement a simple algorithm. As you find your marks, you develop a component. Eventually they let you take on a small project. The entire company now knows about you. You switch jobs and become a familiar face in the city. You talk at a big conference and people from all over the world follow you on Twitter. You might, one day, invent something major and make an impact on humanity. Your life as a Software Engineer has just flashed in front of you.

I am somewhere in a shade of green in the graphic above. Crossing the lines between these circles is a difficult affair. If that is part of your goals, I want to suggest a few practical ways of expanding your scope of influence. Circles, that I feel I have crossed.

From Component to Project

Your component can be thrown away and replaced by another in no time. So can you. In order to cross this chasm, your commits must have global project impact.

Start with trivial tasks of standardizing the number of white spaces or tabs or by removing trailing spaces after meaningful code. Touch as many files as you can. I used to make commits of thousands of files because I was irritated by the lack of periods in code comments. Am I crazy? The truth is that I was terrified about modifying so many files at once and that prevented me from refactoring large chunks of architecture that was very broken. So changing whitespaces or adding periods was a useless exercise that helped me. When you’re comfortable with modifying code left and right, find a major piece of infrastructure to rip out and replace by something better. I remember swapping a C++ core library from underneath a huge system, a seminal moment.

From Project to Company

Nobody knows who you are until you’ve built something that many people want. In order to cross this chasm, seek a need that goes beyond your team.

Around 2001 I needed a tool to send a massive amount of e-mail to stress test some service. I couldn’t find a decent one, so I wrote something new. I put it on an intranet site and kept updating it. Eventually a lot of people with similar needs in the company used it. Then I  started another larger tool to help my team with our complicated builds. I made an intranet site for that too, and it took off. Six months later I was doing that project full time  with executive support and a team of five developers.

When building something that your entire company needs you learn much more than code. You have actual customers and you suddenly find yourself needing to talk to people, presenting the project to them and arguing with skeptics. Working with people is hard and providing a useful service within a corporation is a good place to start.

From Company to City

You’ll find yourself solving the same type of problems as your friends working at other companies. In order to cross this chasm, organize a visit to a friendly business. Go with a million questions and drag your fellow coworkers. Bring pastries.

Every time I lift my head from the computer and visit another tech startup in NYC I come out pumped. I learn so much and meet new Engineers! The first company I visited with my current team was Foursquare about a year and a half ago and the last as of last week was Rent-The-Runway. I met dozens of new techies in between. Every single time, by brainstorming ideas I changed my mind on how to build something or at least it made me think.

When you have built a solid network that includes technologists from other companies, you have crossed this chasm. It will help you hire coworkers and find a new job. You’re no longer behind artificial corporate barriers, you’re free.

From City to World

What did you do when you crossed the chasm from project to company? You sought a company-wide need and made a solution public within the walls of your business. Time to do it again, but in public. Scared of so many eyes? The truth is that nobody cares until you build something really good. So my philosophy is that anything that is not core intellectual property can be open-sourced and is always a great learning experience. When you build something good, you’ll remember the moment when a developer from another continent makes a pull request into your project on Github!

You must also write. Start a technical blog. And don’t forget to give a 3 minute lightning talk about your open-source project at a meet-up. Standing and talking in front of people is hard, but they need to put a face to your name. Slowly graduate to big conferences. On to conquer the world!

World to Humanity

Don’t ask me. Most people that had an impact on humanity have written books, given TED talks and won Nobel prizes. Honestly, I have no idea and am certainly very far from any of these. Help?

I talked to a developer recently. He said:

“I cannot talk about this project, because it’s under NDA.” … followed by silence

First, you’re confusing “NDA” and “Classified” or “Top Secret”. I worked on Classified projects. Or have I? You’ll never know. But I have a t-shirt that says: “my project is so secret, I don’t even know what I am doing”. And that’s precisely how you sound.

Can you please tell me something useful?

Start by describing your project in abstract terms without violating any of these commitments. For example, my uncle worked in a classified nuclear facility near Petyagorsk in former Soviet Union. At least I think he has. He was smuggled to Israel where all that classified stuff was quickly declassified and translated to Hebrew. I am guessing that he worked on some weapons systems that could destroy half of the planet or a small satellite. I know he could talk about his work in very abstract terms. And more specifically, he could definitely explain general issues with building explosives from small to nuclear, without revealing anything classified. So can you.

“Our project was in the daily deals space. I cannot describe the exact purpose of the project, but it involved deals … daily ones. One of the interesting problems I worked consisted of moving a million instances of data between 3 and 5Kb each from a node located in South America to a node in the U.S. I used MongoDB to store the data and a Redis queue for processing.”

Now we’re talking.

After almost a year of active development with 200+ commits, Grape 0.2.0 has finally been released. It’s the code from the “frontier” branch that we’ve been using in production for quite a while.

Grape is an API DSL. If you’re building a RESTful API today, this is a great place to start. Grape lives at https://github.com/intridea/grape.

Here’re the 0.2.0 highlights:

• Vendor-based versioning
• Model exposures presentation layer
• PATCH and OPTIONS HTTP methods
• API description blocks, reflection and introspection
• Modules
• Cookies
• Anchoring

Grape is a great project. It’s always an honor to contribute to systems written by much stronger Ruby developers. Many of my pull requests would get rejected and Michael would then rewrite the feature with a much improved architecture (API modules is one of those). I feel like I really learned something in this process. Open-source FTW!

Sometimes you need to iterate over a large MongoDB collection. The biggest issue is that, by default, cursors timeout after 10 minutes of inactivity. For very large collections it’s not uncommon to take longer than that to process results and you get an exception half way through the iteration. A cursor is a server-side construct, how about a client-side cursor?

Here’s a Mongo Ruby iterator that will call Mongo::Collection.find in increments.

And a Mongoid iterator built into Mongoid::Criteria.

Of course you must be careful that the collection doesn’t change during the iteration. If you add or remove an item before you, or will skip elements or process some elements twice.

You’d think someone has implemented simple conversion of numbers into English by now? For example 42 becomes “forty-two”.

I found a few interesting posts, a couple of gems and even a competition. All had issues that fit in two buckets: they could only do English or their implementation was scary. While I didn’t really *need* a converter that worked for different languages, none of the libraries inspired much confidence. Finally, I ran into a I18n implementation that was generic, well implemented and properly tested for Russian. It needed some minor English work, but you can be sure that the Russian version is *much* more complicated.

A few pull requests later, please welcome a new gem, numbers_and_words. Kirill designed the library with conversion strategies and proper localization in-mind.

These produce "forty-two" and "сорок два".

And a fun one.

This is obviously “two octillion nine hundred thirty-five septillion one hundred seventy-four sextillion three hundred fifteen quintillion one hundred nineteen quadrillion six hundred fifty-four trillion six hundred fifty-four billion six hundred fifty-four million six hundred fifty-four thousand six hundred fifty-four”.

Can someone please contribute a Spanish version next?

https://github.com/kslazarev/numbers_and_words

I moved the popular VMWareTasks C# library from CodePlex to Github.

The new home is: https://github.com/dblock/vmwaretasks. There’s also a new Google Group for discussions.

I haven’t been using VMWare ESXi since their new licensing scheme started milking customers, even for API access, but it remains a great product and as far as I know most people who write C# code to talk to ESXi are using VMWareTasks. Several people have been sending patches recently, Github will make this process a lot easier.

I am not a patient person. But years of software practice have taught me how to take my time while coding or how to, otherwise, remove time from the equation.

Manual Tasks Become Features

Otherwise known as process becomes automation.

You need a lot of patience when operating a manual change on a live system. You’re always this close to dropping the users table. So I learned a decent way to prevent mistakes: always run commands in a test environment, first. But that’s the same as walking on a cable between two chairs, before trying it between two skyscrapers – there’s still no safety net.

“first, he tried it at home between two chairs”

Instead, transform any manual task into a feature. In the Ruby world we write Rake tasks.

The cost of transforming a task into a feature is about half a day of work. But it’s still less than one production disaster for every 100 such instances.

Automated Tasks Become Business Logic with Tests

Automation is good, but testing is better. Consider the following task that sends some kind of reminder e-mail.

It’s pretty concise. Unfortunately, user.reminded? has a bug and you now have a communication disaster on your hands.

"he just reminded a million users that their delinquent account is about to be suspended”

We can move this entire logic into the User model and write a test.

The test can cover other important aspects, such as the actual source and destination of the e-mail.

And the task is simpler.

The cost of this change depends on the complexity of the task, but it tends toward zero. Tests take time to implement, but save debugging and regression time. Operational, or “machine” cost is roughly half a day, because the changes must go through continuous integration and a deploy before they can be run.

Friday Changes Become Monday Changes

Why do something today when you can do it tomorrow?

It’s Friday night and happy hour is about to start. Plan that next deploy for Monday morning, when everybody is in the office.

“she just spent her week-end undoing Friday’s mess”

Spending a week-end in the office fixing what you (or someone else) broke on a Friday shoots your Monday’s productivity. You’ll feel like exhausted heroes, at the cost of three days of work (week-end included) multiplied by the number of people involved.

This Week Becomes This or Next Sprint

I firmly believe in under-promising and over-delivering.

“he over-promised and under-delivered”

Never promise anything for tomorrow, day-after-tomorrow or even this week.

I see the best engineers learn to answer requests with “this sprint” and working really hard and getting it done for the next day. I love that. They don’t have an impossible deadline, and aren’t required to cut corners and can do things patiently (implement the request as a feature, write tests and get a code review).

The person on the receiving end is juggling a hundred things too, and generally they don’t expect anything to be done immediately, either. They also never remember when something was done in a day vs. two or five, but they remember well when something made in a hurry produced a production outage.

Asking a Person Becomes Asking The Team

As the team via the team lead when you need a feature.

“she will gladly implement your feature request, right now”

We’re creatures of habit. So we ask the same person who implemented our previous request to do it again. Asking a team lead is really not a matter of process, – I don’t get offended when someone goes directly to a developer to ask for something (in fact, I love when this happens in some cases) – it’s a matter of actually getting you what’s important, first. A team lead has a complete view of everything that’s going on and will be able to find hands for your request. It will get done much sooner.

***

It’s Friday afternoon and you’re about to make a clean break and grab a beer. Someone wants you, a developer, to e-mail a million users and remind them about their account being suspended. Rewind through my blog post.

Nice job from @gschrader.

This is an example of using WAFFLE to authenticate a client to a server using single sign on (SSO) using the Negotiate Security Support Provider.  It passes messages back and forth using Netty.

https://github.com/gschrader/ssoexample