Passionate Programmer

It's really been some time since I wrote something. I've been really busy - more than usual busy with all the projects but additionally busy trying to workout a solution to some of my perennial problems.

I finally managed to read The Passionate Programmer. It almost felt like reading my own notes. Over the past few years I've been talking and writing about quite a number of issues from this book. It's almost unbelievable that somebody has so similar views. Starting with the saxophone experience - I played saxophone for 10 years - I can relate really well to Chad's stories and parallels. Of course, that's not to say that because some things work in music they will work in programming as well. As Martin Fowler once said software development is software development and we shouldn't be comparing ourselves to other industries and drawing misleading or wrong parallels. Anyway, regardless of whether the conclusions are right or wrong the flashbacks to saxophone playing experience added extra level of depth (at least for me).

Out of the many ideas that struck a cord with me...here are a few I noted down

Invest in Learning
The nerve!!! Judging by the reactions I receive whenever I talk about importance of proper education and practice for programmers - it's ranking among the worst insults you can ever make. Every programmer worth his salt knows that learning is for dummies. We were born better than them (basically everybody else). Sarcasm aside, I believe that a lot of programmers would benefit from experience of playing a musical instrument. Daily (i.e. every day) practice and performances. When you're performing on stage without practice - okay let me rephrase that - when you're on stage without hundreds of hours of practice - you're very likely to screw up - and if you do you're like naked. There's nowhere to hide. The whole audience looking at you and everybody knows it was you who screwed up. Believe me, the next time you'll think more than twice before coming on stage without proper practice. When you program without practice (or learn on the job) you're just as likely to screw up...the only difference is that you can blame it on the customer, on the specs, on the technology, or whatever or whoever else around you. Many times you can just comfortably hide behind your boss as he/she is the one facing the customer.

Work expands so as to fill the time available for its completion
This is an interesting concept. In the industry where everybody complains about way too tight deadlines - Chad goes on to say that just by making the deadlines even tighter you can easily double or even triple your productivity. While I am still to meet an employee that would confirm this - it's definitely true for me and all my ex-staff that went independent. It's really fun to see that things that took them several months to finish while being employed can suddenly be done in a month when they're on their own.

Writing Skills
"If you can’t organize your thoughts in your mother tongue so that others can clearly understand them, how can we expect that you can do it in a programming language? The ability to shape an idea and lead a reader through a thought process to a logical conclusion is not much different from the ability to create a clear design and system implementation that future maintainers will be able to understand." While this politically incorrect statement would get you to prison in a number of countries it's surprisingly true. And quite opposite to the geek stereotype. Interestingly enough, ability to speak and organize thoughts clearly has become a very important factor for us. Try to notice the similarities between way people speak and code - if you have a programmer that is very verbose - talking, talking, talking and still not making the point clear - chances are his code will be very similar - long winded, using unnecessary constructs and never quite clear on what it's supposed to do. On the other hand, if you have an introverted programmer that says 10 words a day, chances are his code will be very terse, taking one liners to the whole new level.

Businesses and those who run them are interested in business results
A thing you learn really fast when you're on your own but it's your biggest enemy when you're an employee. I remember a fight I had with one of my employees - his conclusion was that he just wanted to have a life. And he was right. Why should he worry about the deadlines or delivering what the customers needed? His goal was to have a life and the work was one of the main constrains - there was no direct link between his performance and quality of life. Not seeing and understanding this was my biggest failure running the business earlier on. While we had the basic performance based structure in place it was so detached from the company's culture and pretty much left for everybody to make their own interpretation. The challenge is to reorganize from a black box to a concert stage where everybody is crystal clear on how his performance is linked to the achievements of the whole team and how that translates to the distribution of rewards. Chad puts it very bluntly: "Sure, you got it done, but what was it? Why did it matter? How was this so-called accomplishment not just a waste of company time?" He couldn't be more right. I cannot even count how many times people told me that they had been making sacrifices, sitting in the office 8 or 9 hours every day and I still complain about their performance. The issue here was that while they counted hours I was counting closed iterations and customer satisfaction.

It's easy to get hung up on technology choices
Especially when our technology of choice is the underdog... I believe this idea comes from the Pragmatic Programmer - something along the lines of when all you have is a hammer everything starts to look like a nail. I've always tried not to be attached to technology, however, having a 50 projects investment in a technology makes you very close to being dependent on whatever choices you made. Yet, an investment in wrong technology - no matter how much you like it is usually much higher than investment in a different technology that is actually right for the job.

Go independent
This is quite a dangerous point. While definitely true - there are uncountable advantages to running your own show - just go for any motivational seminar by T. Harv Eker, Robert Kiyosaki and the likes - but being the best technician is only a start. Even though, based on the book 50 Great eBusinesses there's a number of teenagers without any experience that are managing billion dollar businesses (or one man shows) out of they're parents living rooms - vast majority of businesses and people are not so lucky. While Chad describes this as natural next step and spends only a few pages explaining the benefits and some of the obvious challenges it's a whole new ball game - one that has very little to do with programming.

Finally, the book got me really thinking. I've been running the software business for 7 years now and while we've been fighting with lots of problems, the single most difficult thing has been to get the right developers. I wrote about different aspects of it in a number of posts analyzing it from many different angles. This books is/says exactly what I've been trying to say and find for all this time. So many people told me that my ideal developer is a mix of too many and too different skills, and that such animals are more rare than a unicorn. On one hand it was very encouraging to see that somebody else has the same ideas and expectations, on the other it's quite a gamble to base your business on something so rare. Yes, it's possible to find one or two and get started. It may even be feasible to hack a new Web 4.5 killer app and sell it to Google for millions of dollars, but it may be much harder create a sustainable business out of it. It may work for huge multinationals that Chad is talking about as they can afford $300,000 packages but hardly any SME can do that. In fact, I couldn't sleep for almost 2 weeks trying to find a solution questioning my every belief and experience with this business. I tried to unhung my self from the technology, the product, the people and try to weigh the price and the risk of each option today and 10 years from now. It's been quite an exercise - I am now iteratively implementing the changes - if at least 25% of it works out this could be a book that totally changed my business and life. We'll see in half a year :-)

Ext JS Image Field

Yesterday I spent way too much time on another simple thing in EXT that I would have expected to work out of the box. What I needed was to upload a company logo to a profile and, of course, display that logo in the form. While it's very simple to display an image in a panel or box, it's a totally different thing to display the in image in the form and update it together with other form fields. What I needed is something like this:

and after form submit it should automatically load the new image:

For image upload I used the image upload plugin from EXT examples. To display the image I needed to extend the field component like this:

Ext.form.ImageField = Ext.extend(Ext.form.Field, {
    autoCreate: {tag: 'img'}
    ,setValue: function(new_value){
        if (new_value == undefined || new_value == null) {
            this.el.dom.src = '/images/no_image.png';
        } else {
            this.el.dom.src = '/images/thumbnail/' + new_value;
        }
    }
    ,initValue : function(){
          this.setValue(this.value);
    }

    ,initComponent: function() {
        Ext.apply(this, {

        });

        Ext.form.ImageField.superclass.initComponent.apply(this);
    }
});

Ext.reg('image_field', Ext.form.ImageField);

This is a bit simplified version as I removed the rails specific code, but anyway the main points are here:
line 2 creates image tag (rather than the default text field tag)
line 5 defaults the image src property to a placeholder (in case there's no image uploaded)
line 7 - in case there is a valid image it will change the src path to that image - in my case it's routed to Images controller and action thumbnail which returns a downscaled version of the image.

Other than this don't forget to set the FormPanel's fileUpload: true and refresh the form on submit success. I usually send the updated object JSON in the response to update action.:

  def update
    contact = Contact.find(params[:id])
    on_bind(contact)
    success = contact.save
    contact = Contact.find(contact.id) if success
    
    logger.info("saved")
    
    respond_to do |format|
      format.ext_json do
        json = contact.to_ext_json(:success => success,
                                   :methods => [:image_id],
                                   :includes => [:address],
                                   :messages => {:notice => l(:contact_updated), :warning => l(:contact_update_error)})
        render :json => json #:json => {:success => success}
        headers["Content-Type" ] = "text/html"
      end
    end
  end

!!!Notice line 16 - you have to manually set the content type to "text/html" because rails will default to "text/javascript" which will not work with file uploads!!! It works with fileUpload: false but it doesn't with fileUpload: true. I don't really know if Rails or EXT is right here but did take me some time to figure this out.

this will produce the following JSON (I removed the unnecessary fields):

{"success": true,
    "messages": {"notice": "Contact was successfully updated.", "warning": "contact_update_error"},
    "data": {"contact[image_id]": 15, "contact[active]": false, "contact[client_account_id]": 2, "contact[client_account_no]": "100",
        "contact[created_at]": "2009-07-07T10:31:54+08:00", "contact[credit_term_id]": 1}}

Now in the forms success method I can use the result.data to update the form:

success: function(form, action) {
                      form.setValues(action.result.data);
                  },

Losing Faith

Lately, I've been going through a lot of frustrations with Ruby (by ruby I don't meant just the language but the whole ecosystem and infrastructure based on Ruby). I am in a finishing phase on several projects. It's a time when the finer details get addressed. When there's no more time to do it later, differently or drop it all together. It's when everything that is supposed to work should actually work. The problem with Ruby is that it's been in this kind of finishing phase pretty much ever since I started using it (2004). If you look at it from far enough it seems like there is everything in it but when you come closer and actually try using it you'll find out that a lot of it is in less then pre-alpha stage (of course labeled as production). No language is an island unto itself. No matter how great the language you probably don't want to create all the basic libraries from scratch.

When I made a switch in 2004 Ruby was fairly new, Rails was fairly new so I expected this kind of problem, but you could feel the change in the air. Pretty much all the industry heavy weights were talking about it - Rails conf had a better key note line up than any other conference around, number of books was written and the whole Ruby ecosystem was created over almost overnight. Sad truth is that many of the heavy weights are still talking about it today but they're yet to do their first project in Ruby / Rails. Despite the explosion like surge in popularity little has changed in terms of actually usable Ruby libraries (actually I have the same feeling about 1.9 as well). This is a stark contrast to some of the uncelebrated languages out there. For example I find myself writing increasingly more and more code in Python. I don't think Python is the next Ruby, but I was quite surprised how easily it addressed things I struggled with for 5 years in Ruby.

An example could be excel libraries. I still use the same excel libraries I did 5 years ago (for export I switched recently to xlwt, for import because of the changes in roo I've been using JXL for years now switching to xlrd). The export is great only it doesn't support much formatting and breaks on spreadsheets over 10MB (actually somewhere over 6MB) and the files sometimes don't work on later versions of MS Office. There's been a rewrite that is able to go over 10MB but still without formatting I would need and with a license that doesn't allow commercial use. Python's xlwt - despite a very crude and unpolished appearance is able to produce a complete excel spreadsheet with all the formatting (numbers, dates) (including page setup) and all that in much shorter time and with less resources (Ruby 1.8 4m 28s, Ruby 1.9 2m 36s and Python 33s). I have quite a few more examples but I really don't mean this as a Ruby - Python comparison. This experience, however, got me thinking if Ruby is right for the job I've been trying to do (ie uncool enterprise systems). There are pretty much 2 things for me in Ruby (okay - 2.5 the 0.5 being RubyMine - I'll write about it in some other post). One of them is expressiveness of the language and the other is Rails. Ruby as a medium is incredibly expressive, reads almost like English and allows to do pretty much anything (iterating/mapping an array in 1 line, etc). The down side - as Robert C. Martin put it - is that it's really easy to make a mess. After several months of going through somebody's else code I couldn't agree more. A nice entanglement of block and yields and procs can challenge most of the PHP spagetti code any day. While this expressiveness is incredibly powerful, many times I opt for more verbose solution - Java programmer's code on Monday morning as Gregg Pollack would say - that I will be able to understand 2 years down the road - or that my junior developer will understand.

The other thing in Ruby for me is Rails. The only thing with Rails is that after the years it's becoming a bit too entangled, while it made a lot of sense initially to bundle everything together (ORM, routing, templating, etc.) initially now it starting to be a reason why Rails slowly falls behind in certain areas (either that or Rails team has simply decided to focus on Web 2.0 start up market and let others handle enterprise problems). As surprising as it may seems the world around has not been snoozing after Rails was created and many people took the best ideas (just like Rails did from J2EE world) replaced what didn't work and the whole industry moved forward. While few years back Ruby and Rails were pretty much necessities now there's a lot of alternatives and in many different language. To channel out my current frustrations I started experimenting with quite a number of frameworks and platforms. After so many years I really know what I want and how it fits with my existing projects and infrastructure. Very briefly -

• I am looking for a modular framework where I can choose different ORM, different templating engine and so on where each component is developed independently by people who understand that specific area. As each of my project is very different sometimes I need the flexibility to change components. It was a great thing back in my Java times (Spring + Hibernate + Webwork). 
• I really need an enterprise friendly dynamic language. Life's just too short to be wasting time trying to force a language to do things it's inventors don't consider important. 
• I need a basic ecosystem of libraries (from XML, JSON, through excel, PDF, OO to image manipulation). 
• I need a framework that supports TDD out of the box. 
• I need a platform with (at least some) formal education and certification available. I need it because I employ people and certification really helped me back in Java times. To pass even the first Java certificate required people to study and to really understand the language and they simply took it as an independent measure of their skills. 
• I need a language with a proper IDE. I use Vim every day - BUT not for programming. I've been using RubyMine for more than a month now and I feel like a blind man that got his eye sight back.
• I need reasonable deployment option - both scalable and resource efficient. 
• Increasingly more important - I don't want the language/platform community to offend others

It's almost uncanny how many items point back to Java :-). Actually, it's kind of sad that things I took for granted 5 years ago in Java are still not available in the most celebrated and most wildly endorsed language of today. If you look at it - nothing really substantial changed in the way we do the backend (MC) part of the systems. A lot has changed on the View side, but the models and controllers use the same principles as years back. I've used Python/Pylons quite a lot lately and it scores quite high on a number of areas (only so-so on the IDE, though). It seems much more mature compared to Ruby with more matured libraries supporting the enterprisey stuff (so ostracized in Rails community). On the other hand it's marketing is nowhere near Ruby's - you don't really hear people talking about wishing to work in Python. I am a bit worried about the performance - but that's something I just have to try on a real project. A big issue still remains the education and certification. I went through this with Ruby already - no matter how easy and great and superior the language you still need to spend some time learning and then even more time working in it before you really understand it. While there's quite a number of books on Python not much of training courses (at least not in Singapore) and there doesn't seem to be any certification around.

Extracting Data From MS Sql Server on MacOS

I've had really a rough time lately getting data from legacy systems based on MS Sql Server. In each case I got only the .bak backup file and had to do the rest. The whole thing is a bit easier if you run everything on windows, but as I did the development on Mac I'll cover how to connect to MS Sql from MacOS as well.

Restore database from backup

You will need a windows machine with MS Sql Server running (of course :-) Luckily, you can download it for free from here. Next connect to the database server using the command line utility osql (you will need to have it in your PATH)

cd C:\Program Files\Microsoft SQL Server\MSSQL\Binn
osql -E

You may need to create a new user:

use master
go
EXEC sp_addlogin 'peter', 'pass45'
go

Now let's have a look at the backup file (put it in some easy location so you don't have to type out the path):

Restore FILELISTONLY FROM DISK='c:\AMS.bak'

This query allows us to find out the logical name of the database and log file which is needed to appropriately restore a database to a new path. What you get is basically an original location of the data and the log file. In my case I got:

AMS_Data      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Data.MDF
AMS_Log      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Log.LDF

Which means that the original installation was on drive D:\. As my temp server is on C: I will have to recover with changing the locations of the files

RESTORE DATABASE ams
FROM DISK='c:\AMS.bak'
WITH
MOVE 'AMS_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams.mdf',
MOVE 'AMS_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams_log.ldf'
go

The only important thing here is that 'AMS_Data' and 'AMS_Log' names match the ones from the previous query. Now you should hopefully see that your database has been restored. Now, will just get access to the database:

use ams
go
EXEC sp_grantdbaccess 'peter'
go
sp_addrolemember db_owner,peter
go

There's more info on how to use osql here. Now we should be set to connect to the database from development machine. This was quite a number of steps just to restore the database - compared to createdb tmp; psql tmp <>Connecting to MS Sql from MacOS We will need to install and set up a couple of things:

sudo port install rb-dbi +dbd_odbc
sudo port install freetds

set up connection to database

/opt/local/etc/freetds/freetds.conf

Add your server to the end of the file:

[Ams]
 host = 192.168.11.106
 port = 1433
 tds version = 8.0

where the host is IP of your windows machine. After this step you should be already able to connect to MS Sql Server:

tsql -S Ams -U peter -P pass45

You should get the server database prompt (just like the osql on windows machine). You can try some commands like

use ams
go
select * from contacts
go

Now set up ODBC connection. Go to Applications -> Utils -> ODBC Administrator 1) add driver with following descriptions:

Description: TDS
Driver File: /opt/local/lib/libtdsobdc.so
Setup FIle: /opt/local/lib/libtdsobdc.so
Define as:   System

2) add User DNS DSN: Ams Desc: old AMS database server add 2 keyword/value placeholders. To update them click on it and press TAB. Set it to following values:

ServerName: Ams   (or whatever you set in freetds.conf)
Database: ams   (or whatever your database name)

now you should be able to test iODBC. Note that I am using sudo for this as it doesn't seem to work without sudo complaining Data source name not found and no default driver specified. Driver could not be loaded (0) SQLSTATE=IM002.

sudo iodbctest "dsn=Ams;uid=USERNAME;pwd=PASSWORD"

You should be now in interactive terminal again. Once all this works connecting from Ruby is really easy:

require 'rubygems'
require 'dbi'

DBI.connect('DBI:ODBC:ams', 'USERNAME', 'PASSWORD')

Rescuing the data

This is just a short ruby script I use to extract all the data from MS Sql and import it to Postgresql. It's not any functional database conversion - it's just to get the data out to something that's easier to work with and doesn't require windows machine running. It may have an issue with binary fields - it worked on most of them but it did choke on a couple of fields. DBI actually provides more metadata like NOT NULL attribute, primary key, etc. so the script could generate a more precise copy of the original database but this was enough for what I needed. You may run into unknown data type - especially if you try to import it to a different database engine - all you need to do is just update the method update_type to return correct mappings of the data types.

require 'rubygems'
require 'dbi'
require_library_or_gem 'pg'

def escape(text)
 return "NULL" if text.nil?

 text = PGconn.escape("#{text}")

 return "'#{text}'"
end

  def self.update_type(col)
    type = col["type_name"]
    type ||= 'timestamptz'
    type = type.downcase
    
    case type
    when 'clob'
      return 'varchar'
    when 'varbinary'
      return "oid"
    when 'long varbinary'
      return "oid"
    when 'double'
      return 'double precision'
    when 'tinyint'
      return 'smallint'
    when 'char'
      return "char(#{col['precision']})"
    else
      return type
    end
  end



 dbh = DBI.connect("DBI:ODBC:ams", "peter", "pass45")

 sth = dbh.prepare('select name from sysobjects where type = \'U\' ORDER BY NAME;')
 sth.execute
 tables = sth.collect{|row| row[0]}

 tables.each do |table|
   sth = dbh.prepare("select * from #{table} ")
   sth.execute

      create = "CREATE TABLE #{table}(\n"
      create << sth.column_info.collect{|col| "\"#{col['name'].downcase}\" #{update_type(col)}"}.join(",\n")
      create << ");\n\n"

      puts create

      sth.each do |row|
        create << "INSERT INTO #{table} (#{sth.column_info.collect{|column| "\"#{column['name'].downcase}\""}.join(', ')}) VALUES (#{sth.column_info.collect{|col| escape(row[col['name']])}.join(', ')});\n"
      end
      
      create << "\n\n"
      
      output = File.new("data_ams.sql", "ab+")
      output.puts create
      output.close
      
      #puts create
    end

To import the data to Postgresql is as simple as:

createdb ams_backup
psql ams_backup <> noise.txt

Software Development as a Profession?

A few years ago I used to be quite active in Rails community and then I stopped. I haven't been to a Singapore ruby/rails community meet up for 2 years and many of my friends ask me why - if I completely moved from Ruby to Python or simply what happened. Even though, I spend more and more time working with Python, I am still spend 13 - 14 hours a day doing Ruby programming. The actual reason has much more to do with Rails positioning. Back in April 2007, when the Singapore rails community was just starting to form and we had our first bigger meeting I wrote about it here. Most of it is now irrelevant but I was quite surprised when I read my 2nd point - purpose of the meetings - especially the part about the community divide to free hackers and enterprise developers. Back then I believed both were good and necessary for the community. I still believe it today only the situation has changed. While back then it was pretty much balanced, most of the community today consists of free hackers. After Dave Thomas moved from programming to music (okay - from rails to ruby 1.9) there hasn't been anybody to represent the enterprise interest and the rails community and rails image evolved to a cool web 2.0 free hacker paradise with little interest in enterprise.

This became very clear thanks to Bob Martin's key note on RailsConf '09 and very nice rebuttal by DHH.  Actually I've listened to that key note for several times now - Robert Martin is so expressive and (using the Smalltalk parallel) so diplomatic delivering something so critical to the community. For the past year and a half, I've been fighting with so many of the things he mentions - so many things I couldn't understand but make so much more sense now. 

It makes so much sense what he says about arrogance - I usually call it the 'cool' factor. Not doing the dirty job and only doing the clean things. Belief that our tools are somehow better, that our language is so good that we don't have to follow the rules, we don't have to do the regular (enterprise) things. It may be one of the reasons why Ruby still doesn't have a proper PDF or Excel handling libraries. The problem here is that this kind of arrogance is usually very subtle - but hits really hard and on places you would never expect. 

Rails is too easy to make a mess. This single statement pretty much sums up everything I said last week about software development being hard. Yes Rails is easy - in fact so easy that it makes you believe there is nothing you couldn't fix overnight. It's so easy that it makes you believe you don't need to know anything else. The problem is that all that ignorance will come back 3 months down the road (hopefully you'll be able to quit and let somebody else handle the mess).

Clean code. You look at the method it's pretty much what you'd expect. 

TDD. As a programmer I pretty much grew up on books and blogs of people like Martin Fowler, Robert Martin, Dave Thomas, Kent Beck and others from the group. TDD was the only way I was able to work. TDD is a No 1 book on my list of books for beginners. Yet I've been having the hardest time selling TDD - even to my own employees. They all know they HAVE to do tests - but only very few of them do it voluntarily or really understand why. TDD is simply not a culture around here (SG) - in fact, many find TDD culturally insensitive - at best a nice theory but totally useless in practice - just go through the Singapore ruby mailing list. Sensitive or not - it made it to our contracts for software developers.

Lastly, the discussion goes to software development as a profession and to what constitutes the professional portion of our job. Surprisingly enough, Robert Martin seems to think it's the discipline, rules and principles, it's the tough things we have to do and it's keeping them up when the pressure of a deadline mounts.  While there are people that are against being professional (see the DHH's rebuttal), for many people around me it makes sense. It seems so obvious that no body is really doing it.

Software Development is Hard

This is something that's been bothering me for the longest time. And NOT because I want to feel good about myself telling others how hard my job/work is. The reason is quite opposite. Our company has gone through expansion and shrinking cycle for several times. Each time we change a few things, achieve better results but frankly nowhere near the ideal situation. The last time round we came up with some important structural changes, but it doesn't really stop there. Given an appropriate and workable structure, the next question is: what kind of people are we looking for? What are the hard skills and what are the soft skills they should have? What should their motivation be? 

Long time ago I wrote about 10 books to start learning software development. It was an attempt to address the fundamental areas of software development - to take a side in plethora of different possible approaches to them and say this is how we do it here. I really didn't mean to say read those 10 books and you're a top class software developer. I used to start my Ruby on Rails trainings with the following picture:

I found the picture somewhere on the net - but as an ex-java developer I owned and read every single book from the stack (even though I ditched Struts for Spring in 2003 so my stack would have Spring books instead of struts). The ruby stack is pretty much all that was available back then. It would be very nice to think that after reading 2 books you'll know everything you need to know :-). Just for the record, my ruby and rails collection today has exactly 25 books plus a Ruby on Rails Essential Training from Linda.com. Regardless of which stack you're looking at, however, the fundamental problem with the picture is that those books will teach you only very small portion of what you need to know - they only cover 1 area of software development (to be fair the java stack has books on some of the other aspects). They only teach you the language but they are not making you a programmer - just like knowing all 12 notes doesn't make you a musician and being able to write doesn't make you a writer. You still have to add a stack on patterns and principles, databases, testing, basic design, HTML+CSS, object oriented analysis and design, agile techniques and practices, meta-programming, deployment and other aspects. It's extremely important to say that those books are not novels you don't just read them - you really have to work with them, do the examples, play with it try it in different scenarios - basically deep practice with them. Reading in a few hours is totally useless and waste of time. 

This is what you have to spend your first 10,000 hours on.

The problem is that after those 10,000 hours you only know HOW ... you still don't know WHAT - a.k.a. the domain. Depending on your job this can very narrow or very broad or your every project may be in a different domain. That's not to say that you have to become a domain expert in every domain - but you can hardly write an accounting software if you refuse to learn about invoices, receivables, GL postings, financial statements and so on. While in traditional model there's system architect (or designer) that is expected to define everything to the last step with screen layouts and function of the last button and programmer simply translates the UML diagrams to code agile models require a very opposite approach. Software developer is no longer just shoveling coal from one pile to another. Instead, he/she is an active part of the development process - sitting with customers, communicating and shaping the final product. 

For example, when I was working on my first inventory control system I spent 7 months in the warehouse doing every role from unloading the deliveries from suppliers, preparing deliveries to customers, picking the stock for walk-in customers, through weekly stock checks representing the logistics department on management meetings. I was even a delivery boy for several days to understand what the drivers go through. Doing accounting systems I was a part of accounting department from data entry clerk, going through the end of the month salary preparation stress, working on financial statements for management and going through audits - working with auditors to understand what they need and what's the best way to present it to them. You may say it's totally crazy - and to be perfectly frank many of my (ex)employees do - you did not finish university to be some store clerk and if you wanted to be an auditor you would have studied finance - not programing. As one of my ex-employee (he used to liken himself to Douglas Crockford) put it - I am a luminary programmer - I cannot care about invoices and whatever other domain nonsense. 

The problem is that our customers don't give a damn about how luminary programmers we are. They don't know the difference between Java or Ruby, Rails or Spring, Apache or Tomcat. In fact, a few years back when I said I was programming in Java one of my customer asked if it's not too expensive to fly to Java every week and asked if it's because of the environment or cheap massages. They don't need a drill - they need holes. I have long 3 - 5 hours meetings with customers every day but we don't talk about programming - we talk about their issues and their problems. It's about understanding those problems and finding a solution - yet I could hardly find a solution if I had no idea what they're talking about. Problem is that many programmers I know don't even have any interest to understand.

To say it differently - not so long ago I spent 9 months in a restaurant - doing everything from serving, cashier, bar-tending to helping in the kitchen. Kitchen is a single THE most stressful place in the restaurant. Surprisingly the difference between great chefs, mediocre chefs and fired-in-3-days chefs wasn't how well they could fry the fish or roast the beef or how they're goulash tasted - in one word how well they could COOK. They all had to be able and were able to cook. The difference was in preparation. Great chefs would come in as early as 7am and started by cleaning the kitchen, they personally received all the deliveries checking the quality of everything - pretty much smelling every tomato, fish or pork chop. They would check they tools, go through the dishes, cutlery, prepare the condiments and just before opening for lunch sweep the restaurant floor one more time and check every single table one more time just to make sure that it's all perfect and ready. In the afternoon break they would talk to suppliers prepare samples and work on improvements. Mediocre chefs would turn up around 11:00 just in time  to defrost the fish heat up the soup and have a cigarette. In the afternoon break they'd be sleeping in the office - because legally, it was they're break.

The software development is very hard - but not because of programming. Being able to program doesn't make you a great programmer just like being able to cook doesn't make you a great chef. It's a given that programmer should be able to write a clean code with tests. What sets them apart is how they approach the other aspects - especially the low glam aspects. I realize that all that I just said goes right against the core values of Rails framework and general perception of Rails. After four years in rails I am very sure I am not a COOL developer working on the next, much COOLER Google or Facebook or Web 3.75 webapp. We do clinically uncool systems - dare I say the word enterprise systems and only a part of the work is actual programming.

I started by asking about motivation - now looking back at what I said - one would have to be crazy just to do all that stuff once - what kind of an insane person would suffer through all this learning and then suffer again on every project (not even mentioning finish stress and starting stress and stress during every iteration - considering a good project is never finished it's like a living organism). Now imagine you find this kind of person, after all there all sorts of crazy people around (like climbers, actors, musicians, etc.) that go through a similar ordeal - how would you pay them? If a fresh grad without any experience asks for $6,000 because he has an option of stress free MNC job (at least here they do) - what can you offer? Martin Fowler once said that they start charging clients on 1 million - when it runs out they charge again. We're almost there - but not quite yet :-) 

Last week I read Talent is Overrated by Geoff Colvin. There is a section dedicated to the the motivation of top performers which is very interesting - so I'll come back with more on this topic (hopefully) soon.

The Talent Code

I read an interesting book couple of weeks ago. The Talent Code by Daniel Coyle. While the whole book is really good I particularly liked the section explaining the "sudden" appearance of extraordinary talents - like Brazilian football players, Bronte's sisters, Mozart, venetian sculptors and painters and so on. How it's believed that all those people were simply born with great talent and their great work just happened. And how the reality was almost a total opposite. The main idea behind the book is that no matter what you do, in order to achieve a world class skill you need to spend 10,000 hours in practice. The skill is basically "neural connections" in the brain. Those connections are created during practice but what determines the level of the skill is the "quality" of those connections. What increases the quality of the connection is myelin by insulating nerve fibers. The 10,000 hours is necessary to create sufficient myelin. What's interesting is that it's not just any practice - the best results are achieved in "deep practice".

The author goes on to explain how any of the great talents already had their 10,000 hours clocked in by the time they were "discovered". How bronte's sisters had written tens of training books before Jane Eyre or Wuthering Heights, how Mozart had his 10,000 hours of practice in very early age and how venetian painters and sculptors got their hours of training in apprenticeship. They were not born great but started from scratch and achieved their greatness by increasingly improving their skills. Interesting thing is that the apprenticeship didn't mean that they would simply paint the whole day and after 10 years became masters. No - they had to do all sorts of things - especially all sorts of "low level things" - like setting the canvas, preparing chisels and so on for their masters. Only after they learnt the basics could they move on to more difficult things. And this is what basically constitutes the "deep training" - choose a goal just outside your comfort zone and keep on failing until you achieve it. Then repeat the process. 

To sidetrack a bit to IT - this is where I think today's system breaks down. Software development is hard. And it only starts with the technical skills - you still need to understand the domain (when working on accounting system you have to have a very solid understanding of accounting), you have to be able to work with people and translate their ideas and feelings about something they've never seen to reality. And there is no magic shortcut - you have to have your 10,000 hours clocked in before you can really do something.  Programming is like writing - just like bronte's sisters - you have to write a lot a lot a lot and about everything you can find - only then your writing will start making sense. Unfortunately, I wouldn't really count any hours spent at school. Maybe it's an unfortunate situation around here - but the IT schools here are set to producing CIO's making decisions on a golf course rather than doing programing or any other actual IT work. To their defense that's pretty much what market here wants - most of the fresh grads from IT will find a good shake-leg work in a bank or MNC paying at least $10,000 per month. I meet quite a few of those people in my trainings or recently, as the banks scale down certain areas, in interviews. 

Anyway, my point here was that only very very few people in IT are willing (or forced) to go through the whole journey - from setting up networks, installing computers, moving to servers, maintaining different operating systems, doing backup / recovery, working with databases, optimizing databases, writing SQL, programming in several languages for several years, understanding patterns going through several releases and so on. And none of that means "I used MS Access in 3rd semester".

Another interesting thing was the difference between masters and beginners. The difference was explained on chess players. The difference between chess masters and beginners is that masters can remember the whole board set up on one look. There is a twist, though. They could only remember the set up from an actual game. If the figures were in random order the chess masters photographic memory vanished. The reason why chess masters could remember the board setup from actual games is because they were not seeing the individual pieces - they were seeing the patterns. 

I picked up the book because I wanted to know more about how people learn - I wanted to understand why after a year of training and working some of my employees are not able to do even a simple task, why is it that they spent a week installing a server without any success while somebody else can do the same work in 1 hour. Why some spend months and months programming a piece of functionality creating a total mess and somehow not "seeing" the simple solution that can be finished in a few hours or days. I used to be especially puzzled about the not seeing part. Even after showing them the simple solution they just couldn't understand it and had to go a big round of all possible wrong ways to arrive at the same solution. I guess some of them see patterns while others see thousands of lines of commands. As a result, in the light of the book's 10,000 hours I start to look very differently at this kind of things. The good news is: it's learnable. The bad news is: it may take 10 years for someone to learn it.

Rails 2.3 with Ruby 1.9

After the longest time we decided to upgrade the rails version in our applications. The last upgrade was to 1.2.6 year and a half ago. As we have quite a number of applications I am quite reluctant to upgrade unless it's really really worth it. There was quite a number of positive changes in Rails 2.2 (like the database connection pooling, thread safety and built in internationalization) but it wasn't compatible with Ruby 1.9 and I really didn't feel like going through all the apps just to go through them again to upgrade it to Ruby 1.9.

After release of Rails 2.3 compatible with Ruby 1.9 I could finally go ahead with the upgrade. And it turned out to be quite easy - but unfortunately some things haven't been ported to 1.9 yet. One of the main advantages of 1.9 is supposed to be increase in performance. It can be easily demonstrated by running recursive Fibonacci test. Unfortunately not even one 

of my applications needs faster Fibonacci :-). Having run most of our apps on 1.9 for around 2 weeks now I don't really see any tremendous performance increase or lower memory requirements. Several of our systems require extensive calculations (especially bigger accounting systems) but the performance there is dependent on many other factors like database speed, array sorting and grouping and so on and I am still working out the optimizations here.

What really really really helped me during the migration was David Black's book The Well-Grounded Rubyist. I heard him speaking several times but I was quite surprised by how well written the book was. It's really sharp

 and to the point without useless 'technicalities'. At the same time it's very readable and not dry like API reference.

Okay back to Rails. As I said most of the things went smooth BUT here's the list of some that didn't :-)

1) Mongrel doesn't work - not so much of a problem for me as I don't use it in production and in development I switched to Thin. 

2) File upload on nginx doesn't work - it seems to be an issue with Rack that is bundled with Rails 2.3. This is a bit of an issue as I was just starting to like nginx and I used it for some my lighter applications. 

3) Problems with encoding - this a single issue that I didn't manage to resolve or find a workaround. We've done some projects in Europe and they have to support slavic languages with all their funny characters. As 1.9 supports so many encodings I really didn't expect any problems there but I didn't manage to get pass the

incompatible character encodings: UTF-8 and ASCII-8BIT

when data from database contain special characters. There are several patches available in lighthouse here and here but non of them really worked for me - safe for

contact.name.force_enconding('utf-8')

4) send_data doesn't work on binary files with message

invalid byte sequence in UTF-8

. I used to use it for on the fly generated images but had to replace it with creating a temp file and then use

send_file

5) ruby zip hasn't been updated yet and still references ftools that have been replaced by fileutils. I had to change it manually along with File.move reference replaced with FileUtils.move

6) quite a number of plugins stopped working and as many are no longer supported I decided to ditch a few of them. I have never been a fan of the small rails plugins that only adding some syntactic sugar not just because they break on every update of Rails (even worse with Ruby upgrade) but mostly because of their obtrusiveness. Even though they make the code shorter it's usually much less readable as you have to learn and understand the plugin.

File Downloads in Rails on Apache

Over the past half a year we worked on several systems for logistics/courier/delivery companies. It was quite fun as a big logistics and inventory control system was my very first project when I arrived to Asia 9 years ago. Last week we launched one of them. It was slightly different as most of our systems are used mainly internally but this one is also used by clients of the courier company to book deliveries. Each delivery requires printing of confirmation documents, internal routing documents, then signed and stamped documents are uploaded back to the system as a proof of delivery for clients. It has quite a number of other (technically) interesting features like integration with google maps to geo-tag the orders and notifying drivers in the area - I'll write about it soon.

As this courier company handles several hundreds of deliveries every day it requires quite a number of uploads/downloads of documents and images. The issue with uploads and downloads is that they will tie up rails processes for much longer than usual requests. The problem is that rails processes are quite expensive in terms of memory (around 70 - 100MB for simple operations). Do 20 concurrent uploads/downloads and all your rails processes and memory will get tied up - and quite wastefully as there is no 'dynamic' processing during upload or download. The process just sits there and waits for the file to get delivered. Luckily there's a simple way how to move this burden to Apache. This kind of upload/download processes will take only around 2MB on average and your rails processes are free to handle the dynamic requests.

To let apache handle file downloads use x_sendfile. All you need to do is install x_senfile module on apache and adjust you config a little. Here's the installation instructions for Ubuntu 8.04:

cd /usr/local/src
wget http://tn123.ath.cx/mod_xsendfile/mod_xsendfile.c
apxs2 -cia mod_xsendfile.c

Open apache config file and add the module line:

sudo vim /etc/apache2/apache2.conf

LoadModule xsendfile_module /usr/lib/apache2/modules/mod_xsendfile.so

Add the following 2 lines to your virtual host file

sudo vim /etc/apache2/sites-available/my_app

XSendfile on
XSendFileAllowAbove on

You will have to restart the apache and you should be done:

sudo /etc/init.d/apache2 restart

Now in your controller handling download add x_sendfile attribute and set it to true

 send_file("/home/test/big_file.zip",
           :filename => "big_file.zip",
           :type => "application/zip,
           :x_sendfile=> true)

The uploads are equally easy. You just need to apache modules libapreq2 and modporter.

wget http://mirror.nus.edu.sg/apache/httpd/libapreq/libapreq2-2.12.tar.gz
tar -xzf libapreq2-2.12.tar.gz
cd libapreq2-2.12
./configure --with-apache2-apxs=/usr/bin/apxs2
make
sudo make install

Now download and install modporter. Get it from the github. Either use the download link and unzip or just clone the repository

git clone git://github.com/actionrails/modporter.git
cd modporter

On Ubuntu I had to change reference to apxs in Rakefile as the in ubuntu it's apxs2.

vim Rakefile

Change the APXS line to the following:

APXS           = "apxs2"

No just run rake:

sudo rake

add the following to your apache2.conf

sudo vim /etc/apache2/apache2.conf

LoadModule apreq_module /usr/lib/apache2/modules/mod_apreq2.so
LoadModule porter_module /usr/lib/apache2/modules/mod_porter.so

To actually use it in your rails application you will need modporter plugin.

script/plugin install git://github.com/actionrails/modporter-plugin.git

Add the following to the application_controller.rb (application.rb before Rails 2.3)

class ApplicationController < ActionController::Base 
mod_porter_secret = "secret"
.
.

and just enable porter in you vhost file:

sudo vim /etc/apache2/sites-available/my_app

Porter On
PorterSharedSecret secret

And that's all now if you check the uploaded files they will be a ModPorter::UploadedFile Credit for this goes to Koz from theRailsWay.com - you can find more information here.

Another Training

Another training's over. Very different but equally challenging.

Seems that Ruby's been making headway to corporate world as most of the participants this time were there on their employers' budget. When I compare it with our first RoR trainings around 2 years ago with Rails 1.0 just out and more than 30% of the things still on a wish list and the only way to learn RoR was Agile Web Development With Rails...this time it was much more business-like. There's like 20 books out, including a (bit obsolete) video from Linda.Com and pretty much everybody knows about rails.

Another aspect that has changed is Rails versions. With Rails 2.3 and Ruby 1.9 out there's a lot of changes and incompatibilities that complicate the learning. Most of the books and reference materials use Rails 2.0 or early versions of 2.2. This makes it little bit difficult to have training on 2.3 as most of the "further" reading is incompatible. While the incompatibilities may seem easy to solve for experienced developers they can be quite exasperating for a newbies. On the other hand, people are quite reluctant to start learning something that is already obsolete.

Similar is true for Ruby 1.9, even though after burning they're fingers several times until it got finally compatible with Rails most people would still prefer to use 1.8.6 or 1.8.7.