Software Development is Hard

This is something that's been bothering me for the longest time. And NOT because I want to feel good about myself telling others how hard my job/work is. The reason is quite opposite. Our company has gone through expansion and shrinking cycle for several times. Each time we change a few things, achieve better results but frankly nowhere near the ideal situation. The last time round we came up with some important structural changes, but it doesn't really stop there. Given an appropriate and workable structure, the next question is: what kind of people are we looking for? What are the hard skills and what are the soft skills they should have? What should their motivation be? 

Long time ago I wrote about 10 books to start learning software development. It was an attempt to address the fundamental areas of software development - to take a side in plethora of different possible approaches to them and say this is how we do it here. I really didn't mean to say read those 10 books and you're a top class software developer. I used to start my Ruby on Rails trainings with the following picture:

I found the picture somewhere on the net - but as an ex-java developer I owned and read every single book from the stack (even though I ditched Struts for Spring in 2003 so my stack would have Spring books instead of struts). The ruby stack is pretty much all that was available back then. It would be very nice to think that after reading 2 books you'll know everything you need to know :-). Just for the record, my ruby and rails collection today has exactly 25 books plus a Ruby on Rails Essential Training from Linda.com. Regardless of which stack you're looking at, however, the fundamental problem with the picture is that those books will teach you only very small portion of what you need to know - they only cover 1 area of software development (to be fair the java stack has books on some of the other aspects). They only teach you the language but they are not making you a programmer - just like knowing all 12 notes doesn't make you a musician and being able to write doesn't make you a writer. You still have to add a stack on patterns and principles, databases, testing, basic design, HTML+CSS, object oriented analysis and design, agile techniques and practices, meta-programming, deployment and other aspects. It's extremely important to say that those books are not novels you don't just read them - you really have to work with them, do the examples, play with it try it in different scenarios - basically deep practice with them. Reading in a few hours is totally useless and waste of time. 

This is what you have to spend your first 10,000 hours on.

The problem is that after those 10,000 hours you only know HOW ... you still don't know WHAT - a.k.a. the domain. Depending on your job this can very narrow or very broad or your every project may be in a different domain. That's not to say that you have to become a domain expert in every domain - but you can hardly write an accounting software if you refuse to learn about invoices, receivables, GL postings, financial statements and so on. While in traditional model there's system architect (or designer) that is expected to define everything to the last step with screen layouts and function of the last button and programmer simply translates the UML diagrams to code agile models require a very opposite approach. Software developer is no longer just shoveling coal from one pile to another. Instead, he/she is an active part of the development process - sitting with customers, communicating and shaping the final product. 

For example, when I was working on my first inventory control system I spent 7 months in the warehouse doing every role from unloading the deliveries from suppliers, preparing deliveries to customers, picking the stock for walk-in customers, through weekly stock checks representing the logistics department on management meetings. I was even a delivery boy for several days to understand what the drivers go through. Doing accounting systems I was a part of accounting department from data entry clerk, going through the end of the month salary preparation stress, working on financial statements for management and going through audits - working with auditors to understand what they need and what's the best way to present it to them. You may say it's totally crazy - and to be perfectly frank many of my (ex)employees do - you did not finish university to be some store clerk and if you wanted to be an auditor you would have studied finance - not programing. As one of my ex-employee (he used to liken himself to Douglas Crockford) put it - I am a luminary programmer - I cannot care about invoices and whatever other domain nonsense. 

The problem is that our customers don't give a damn about how luminary programmers we are. They don't know the difference between Java or Ruby, Rails or Spring, Apache or Tomcat. In fact, a few years back when I said I was programming in Java one of my customer asked if it's not too expensive to fly to Java every week and asked if it's because of the environment or cheap massages. They don't need a drill - they need holes. I have long 3 - 5 hours meetings with customers every day but we don't talk about programming - we talk about their issues and their problems. It's about understanding those problems and finding a solution - yet I could hardly find a solution if I had no idea what they're talking about. Problem is that many programmers I know don't even have any interest to understand.

To say it differently - not so long ago I spent 9 months in a restaurant - doing everything from serving, cashier, bar-tending to helping in the kitchen. Kitchen is a single THE most stressful place in the restaurant. Surprisingly the difference between great chefs, mediocre chefs and fired-in-3-days chefs wasn't how well they could fry the fish or roast the beef or how they're goulash tasted - in one word how well they could COOK. They all had to be able and were able to cook. The difference was in preparation. Great chefs would come in as early as 7am and started by cleaning the kitchen, they personally received all the deliveries checking the quality of everything - pretty much smelling every tomato, fish or pork chop. They would check they tools, go through the dishes, cutlery, prepare the condiments and just before opening for lunch sweep the restaurant floor one more time and check every single table one more time just to make sure that it's all perfect and ready. In the afternoon break they would talk to suppliers prepare samples and work on improvements. Mediocre chefs would turn up around 11:00 just in time  to defrost the fish heat up the soup and have a cigarette. In the afternoon break they'd be sleeping in the office - because legally, it was they're break.

The software development is very hard - but not because of programming. Being able to program doesn't make you a great programmer just like being able to cook doesn't make you a great chef. It's a given that programmer should be able to write a clean code with tests. What sets them apart is how they approach the other aspects - especially the low glam aspects. I realize that all that I just said goes right against the core values of Rails framework and general perception of Rails. After four years in rails I am very sure I am not a COOL developer working on the next, much COOLER Google or Facebook or Web 3.75 webapp. We do clinically uncool systems - dare I say the word enterprise systems and only a part of the work is actual programming.

I started by asking about motivation - now looking back at what I said - one would have to be crazy just to do all that stuff once - what kind of an insane person would suffer through all this learning and then suffer again on every project (not even mentioning finish stress and starting stress and stress during every iteration - considering a good project is never finished it's like a living organism). Now imagine you find this kind of person, after all there all sorts of crazy people around (like climbers, actors, musicians, etc.) that go through a similar ordeal - how would you pay them? If a fresh grad without any experience asks for $6,000 because he has an option of stress free MNC job (at least here they do) - what can you offer? Martin Fowler once said that they start charging clients on 1 million - when it runs out they charge again. We're almost there - but not quite yet :-) 

Last week I read Talent is Overrated by Geoff Colvin. There is a section dedicated to the the motivation of top performers which is very interesting - so I'll come back with more on this topic (hopefully) soon.

The Talent Code

I read an interesting book couple of weeks ago. The Talent Code by Daniel Coyle. While the whole book is really good I particularly liked the section explaining the "sudden" appearance of extraordinary talents - like Brazilian football players, Bronte's sisters, Mozart, venetian sculptors and painters and so on. How it's believed that all those people were simply born with great talent and their great work just happened. And how the reality was almost a total opposite. The main idea behind the book is that no matter what you do, in order to achieve a world class skill you need to spend 10,000 hours in practice. The skill is basically "neural connections" in the brain. Those connections are created during practice but what determines the level of the skill is the "quality" of those connections. What increases the quality of the connection is myelin by insulating nerve fibers. The 10,000 hours is necessary to create sufficient myelin. What's interesting is that it's not just any practice - the best results are achieved in "deep practice".

The author goes on to explain how any of the great talents already had their 10,000 hours clocked in by the time they were "discovered". How bronte's sisters had written tens of training books before Jane Eyre or Wuthering Heights, how Mozart had his 10,000 hours of practice in very early age and how venetian painters and sculptors got their hours of training in apprenticeship. They were not born great but started from scratch and achieved their greatness by increasingly improving their skills. Interesting thing is that the apprenticeship didn't mean that they would simply paint the whole day and after 10 years became masters. No - they had to do all sorts of things - especially all sorts of "low level things" - like setting the canvas, preparing chisels and so on for their masters. Only after they learnt the basics could they move on to more difficult things. And this is what basically constitutes the "deep training" - choose a goal just outside your comfort zone and keep on failing until you achieve it. Then repeat the process. 

To sidetrack a bit to IT - this is where I think today's system breaks down. Software development is hard. And it only starts with the technical skills - you still need to understand the domain (when working on accounting system you have to have a very solid understanding of accounting), you have to be able to work with people and translate their ideas and feelings about something they've never seen to reality. And there is no magic shortcut - you have to have your 10,000 hours clocked in before you can really do something.  Programming is like writing - just like bronte's sisters - you have to write a lot a lot a lot and about everything you can find - only then your writing will start making sense. Unfortunately, I wouldn't really count any hours spent at school. Maybe it's an unfortunate situation around here - but the IT schools here are set to producing CIO's making decisions on a golf course rather than doing programing or any other actual IT work. To their defense that's pretty much what market here wants - most of the fresh grads from IT will find a good shake-leg work in a bank or MNC paying at least $10,000 per month. I meet quite a few of those people in my trainings or recently, as the banks scale down certain areas, in interviews. 

Anyway, my point here was that only very very few people in IT are willing (or forced) to go through the whole journey - from setting up networks, installing computers, moving to servers, maintaining different operating systems, doing backup / recovery, working with databases, optimizing databases, writing SQL, programming in several languages for several years, understanding patterns going through several releases and so on. And none of that means "I used MS Access in 3rd semester".

Another interesting thing was the difference between masters and beginners. The difference was explained on chess players. The difference between chess masters and beginners is that masters can remember the whole board set up on one look. There is a twist, though. They could only remember the set up from an actual game. If the figures were in random order the chess masters photographic memory vanished. The reason why chess masters could remember the board setup from actual games is because they were not seeing the individual pieces - they were seeing the patterns. 

I picked up the book because I wanted to know more about how people learn - I wanted to understand why after a year of training and working some of my employees are not able to do even a simple task, why is it that they spent a week installing a server without any success while somebody else can do the same work in 1 hour. Why some spend months and months programming a piece of functionality creating a total mess and somehow not "seeing" the simple solution that can be finished in a few hours or days. I used to be especially puzzled about the not seeing part. Even after showing them the simple solution they just couldn't understand it and had to go a big round of all possible wrong ways to arrive at the same solution. I guess some of them see patterns while others see thousands of lines of commands. As a result, in the light of the book's 10,000 hours I start to look very differently at this kind of things. The good news is: it's learnable. The bad news is: it may take 10 years for someone to learn it.