Ruby on Rails Hosting

This is a problem I've been (surprisingly) fighting with ever since I've been working with Ruby on Rails. Even though the situation has improved tremendously over the past 2 years it's still far from ideal. I am not sure if it's because insufficient demand or the increased complexity compared to hosting HTML or PHP the big hosting companies don't show much interest in Ruby hosting. Sure they list it as one of the options but the offer and support is flaky at best, many times offering only 1 rails application per account or missing ways to restart FCGI processes. Not surprisingly rails developers had to take the matter in their own hands and as a result almost any RoR hosting worth considering was started like that. I am not sure how much long term market sense it makes but looking at waiting time at slice host it really seems to be working for now.

When choosing RoR hosting the main decision you have to make is shared hosting vs. VPS. Shared hosting is usually cheaper and easier to set up but you're sharing the server with many other users and it takes only one bad neighbor to make the whole server unusable. VPS, on the other hand, is slightly more expensive and you have to set up everything your self. That gives you a lot of power and flexibility but you have to be able to configure Linux server (it's really no rocket science – there's plenty of How-To-s on the net – including this blog).

Shared Hosting

A few things to watch out for when choosing shared hosting:

1. FCGI support – most of the hosts offer FCGI (but there are still many that offer only CGI) including an easy way of restarting FCGI processes. You really don't want to open up a support ticket any time you update you application and especially not when you fix a very urgent bug.
2. 24/7 Support – while 24/7 support is claimed to be a standard in hosting industry I am yet to really find the host with this kind of support. That is not to say that the support doesn't matter – it really does and even if not 24/7 you should really test them out first – open up tickets outside of business hours, on public holidays, etc. and see the response time as well as helpfulness and professionalism. Many big hosting companies outsource their support offshore and all you'll get outside the working hours is “I will get a senior technician to look into the problem”. This is especially important if you're not in different timezone.
3. Number of applications you can host under one account. This includes number of domains, number of subdomains as well as number of databases that you can create. You don't have to worry that much about the actual limits of the server here because if you're not going to utilize the resources it doesn't make them available during peak, it just means that somebody else will.
4. DNS management tools / support. This is not so important if you have 1 application but as the number increases you will have to take care of multiple domains and multiple subdomains under each domain. With the standard shared hosting you will usually only get cPanel that lets you maintain only domains/subdomains hosting on the same server. Reseller account should come with WHM that has proper DNS management tool. Of course you have always option to host your DNS elsewhere.
5. Disk space / bandwidth limits. Usually not a problem with U.S. hosting but most of the hosting companies in Asia still offer 100MB accounts.
6. SSH. It's very hard if not impossible to set up your rails application without SSH access and yet I've seen several hosts (mostly in Asia) to offer ruby hosting without SSH. When I asked how to install the application they asked me for step by step instructions :-). I really don't think you want to do that.
7. SVN hosting – not crucial but a very nice bonus. It's beneficial even if you're a sole developer as it makes your repository available online. RailsPlayground.Com even bundles this with Trac.

Looking at the list seems like there's quite a few things to watch out for. From what I've used the best seems to be the RailsPlayground.Com. They have a very reasonable reasonable support, very few limits and offer quite interesting packages. The downside is that the servers do get overloaded sometimes and then they kill off your processes. This is really bad as the user will get a Rails Application Error but the Exception Notifier will not generate anything and there won't be anything in the logs. I've had several other issues there – longer HTTP POST will generate application error – again without any trace in logs and I had some intermittent problems with file upload / download. Other then that I would recommend them as most probably the best RoR shared hosting out there.

VPS Hosting

VPS experience a stellar launch to popularity over the past year solving most of the problems of shared hosting for only slightly higher price. It had an easy job replacing dedicated hosting offering roughly the same but for 10 times higher price.

Here are some things to watch out for:

1. 24/7 Support – even though it's all maintained by you and you're much less dependent on support, there still will be times when your VPS doesn't come back after restart or doesn't respond due to some runaway processes.
2. Choice of Linux distributions – many hosts offer a selection of distributions like Ubuntu, Debian, RedHat, etc. You just choose your desired flavor from the menu and it's automatically installed for you. It's really helpful when you have experience with only one Linux flavor or to synchronize your installations when you have multiple servers at several hosting companies.
3. Memory / Price ratio. The only important resource when choosing VPS is memory. Most of them come with sufficient disk space and run on multiprocessor machines but provide only limited memory. Anything below 256MB is not worth considering. You shouldn't pay more then US$ 29 for 256MB and usual price is around US$ 20. Some hosts provide burstable memory which means you can go over your memory limit if nobody else is using it. This can be very helpful during random hit surges or when processing memory extensive tasks. Be very careful as some of the hosts will mercilessly cut off any process that goes about the memory limit causing rails application error without any trace in logs (otherwise great VPSLink.com does that). Another thing to watch out for here is SWAP. While not ideal, it can save your ass during peak requests. Some hosts don't allow any SWAP which will cause your application to crawl when the memory limit is reached.
4. Scalability – check how easy it is to upgrade your account – either to increase memory or to add on another server. When number of users increases, adding another server is many times the only option to scale your application.

One of the best VPS providers is Slice Host providing all of the above for the lowest price on the market. So far, I've experienced only one short downtime. You can choose your Linux distribution, reinstall everything within a few minutes they have no nonsense policies, upgrades in both directions are painless. The only downside is a long waiting list if you're a new user. Another great host is Rose Hosting. They offer burstable memory and used to have very competitive pricing. I couldn't find any way to add another option to add servers to my account.

Some general things to watch out for:

1. cancellation policy. It's very important to read and understand it as most of the hosts have ridiculous cancellation policies – like you have to cancel at least 3 months ahead, or only 10 days before the end of the month or only on Monday, Wednesday and Sunday 3 – 4 am. Also, money back guarantee is much more a dream then reality.
2. Server location. Many people believe believe that the closer the server is to them the faster it is. I've heard many scientific explanations to this, but based on my experience there is usually no difference in access time unless it's on the same subnet (i.e. same provider) which is hardly a case. There is so many other factors (like aggregation, PC speed, last mile connection) that will affect the actual speed that for me it doesn't make any difference in speed for my servers located in the U.S. and servers located in Singapore. The problem with local (Singaporean, Malaysian :-) hosting is that it's several times more expensive, provides several times less resources (like space, bandwidth, memory, etc.) and only 9 – 5 support. I believe this is due to lack of local competition, lack of market awareness and undue local patriotism. Anyways, ...
3. Backup – some hosts provide automatic backup for very reasonable price (e.g. Slice Host offers images for US$ 5). While most of the hosts claim to have auto backup it happened to me several times that it took them 2 days to retrieve this backup. As such you should think of your own back up strategy – one of the ways is to use Amazons AWS. Most of our customers require direct access to back up files and recovery within 30 minutes (this means that no matter what happens we have to be able get back online within 30 minutes). Another issue is frequency of back up – most of the auto back up is daily, which is far too little for any production application. Our standard is hourly backup with possibility to increase this during peak hours.

Corporate Trainings

I finished my first commercial Ruby on Rails application in October 2005 and I've been using it ever since. For most of this time, we've been one of the very very few offering Ruby on Rails in Singapore. Most of our customers are not exactly technical so they wouldn't really care even if we used Cobol :-). What they did care about was the price and delivery time. I don't know why, but writing about advantages of Ruby (and Ruby on Rails) comes very unnatural to me. It feels like saying what everybody knows. At the very minimum, most of the technical people have heard about it – whether they like it or hate it at least the don't go like “You're programming in WHAT???”

The situation seems to be turning even in this part of the world. Thanks to Ruby brigade I met a handful of Ruby on Rails programmers and a few handfuls of wannabes. After almost 2 years in isolation it was really great to see and meet Ruby programmers (other then my own trainees and employees :-). While all this was nice, what really made an impact was ever increasing demand for corporate training. More and more big companies approach me with corporate trainings. The reason is simple...as many are parts of global corporations, many of their internal systems were created elsewhere and now they need a local support. It may come as a shock that MNCs use other systems than just SAP, but it seems to be easier and way cheaper to use a few smaller, independent systems for handling a portion of daily operation.

I started my first Ruby course in April 2005 and for the past 6 months I spent roughly 40% of my time doing training. This has been a major shift for me as only as recently as 1 year back I spent around 90% programming and only 10% on other stuff. I started training in Changi prison and it's almost unbelievable how rewarding this experience has been. I cannot even describe how much the guys have changed. When I walked there the first time I sure saw the potential – around 30 guys doing (or more fighting) with every possible limitation you can think of but still determined to succeed. It may be an old truth that you will appreciate the value of something only after you've lost it but it's really valid. All of the ubiquitous things like Internet connection, access to new books, access to community support (reading the posts or posting questions) – all this is off limits in prison. And yet I didn't find any surviving broken existence in there. So many times I have wished my own employees showed half of the determination, zeal and persistence I could see in there. The more you have, the more you start taking things for granted and the more comfortable you become.

What I felt the guys needed was just better resources and some direction (IT wise). We went through some really tough time – I still remember explaining recursion for the whole Sunday, OOP, TDD and many other things. Well, none of them had any prior formal IT education. But then, maybe it was more to their advantage. Most of the time I find it much harder to work with computer science graduates than with people who never studied IT but have some logical and mathematical thinking. It's surprising how difficult it is to unlearn all the coding cowboy habits as opposed to just starting afresh. This is not to say that OOP or Ruby or design patterns or TDD or agile methodologies or anything else I do is the only way (or that it's correct at all). It's just one of the ways and the one I happened (in reality it took much more then just happen :-) to believe in – but more about that in my other blog posts. Anyway, it's really great to walk in there now. It's completely a different crowd – you see people being able to think, to analyze and to even transform all that into a working product. Sure it's not all wine and roses...but I would really like to see this kind of commitment in many other organizations I worked with.

I realized much of this when talking to people at one of the Ruby Brigade meetings. That's when I realized that I may be running quite a tight ship in regards to development principles – but so far I've really appreciated that nobody on my staff would say things like: “I do tests only when I feel like”, “I've never been into testing much”, “We do bug driven development”, “What's that 'assert' thing mean?” and many others.

To conclude with, I must say that I have always been an extremely fast learner – I've never had problem to learn anything but it's a completely different ball game to teach what you know. One thing is to look at something and understand it and a completely different one is to explain it to somebody else. And I really love it :-). And it finally gives me a reason to start working on my e-learning platform I've been planning since high school.

Ruby on Rails in Singapore (finally :-)

Around a year ago I posted a long article on my blog about no Ruby programmers in Singapore. Back then I didn't manage find any programmer leave alone any company using Ruby (with or without Rails) in production. I started playing with Ruby a little less than 2 years ago and by that time I had several applications in production, and RoR had become our obvious choice for pretty much all of our new projects.

Year on, there are still not many companies using RoR here but the situation seems to be rapidly changing. Last Thursday National Library hosted a meeting of 20+ Ruby and RoR developers. Sure, 20+ may seem pathetic, considering the hopelessly sold out RailsConf and huge popularity that Ruby has gained around the world, but for the local “environment” this is really an achievement and great success for organizers (namely Doug, Sausheong and Choon Keat). It was really great to see that there are like-minded people around.

Let's cut to the chase - there were a few questions/issues raised during the meeting that I'd like address:

1) relationship with library – this question was raised by “the Rambling Librarian” Ivan, thanks to whom we were able to meet in a proper environment rather then a coffee house. I must say that I would really love if we could find ways to not only cooperate with library but find ways beneficial to more general public. We claim that we use the most advanced and cutting edge technology today and how great it is and how much fun it is, but based on the number of participants it seems that Singapore really couldn't care less. Whether it's true or not I see this as the biggest opportunity for cooperation. Library is (and should be) a perfect venue (or avenue?) for spreading this latest know-how. It will be years before it reaches formal educational system – if ever as there's no strong vendor to support the “buy-in”. In fact, I cannot think of a better opportunity to contribute to the goal set in the motto I found in library's corporate brochure: “We aim to provide Knowledge that ignites Imagination and transforms Possibility into reality”. Isn't this exactly what RoR strives to stand for? Just look at the the highlighted words – isn't this exactly how the creators and early adopters (DHH, Dave Thomas, etc.) used to sell RoR? Now unlike many other libraries in countries I lived in, NLB is highly respected among Singaporeans – and I must say rightly so (it could sure use some update of IT section – but overall it's the best library I've been a member of). As such it is in a perfect position to spread this latest knowledge through informal communities that gather around it. It brings due credibility to the scene and of course, it's up to us to show and prove that we are really worth our salt and we are worth of being NLB's partner. After all, I started RoR course in Changi Prison and I believe there's something we could do with library as well :-)

I'd suggest to meet up and discuss the possibilities and ideas – possibly together with Ivan so that we can better understand library's views and goals.

2) What's the purpose of those meetings – this question was raised by me as well as by Ivan. We discussed it quite a bit with Doug but I don't think we managed to close it on anything. Doug's idea is to simply build and grow RoR community in Singapore. I would be interested in slightly more than that. I think this roughly follows the informal divide in RoR community around the world – a divide between the free hackers inclining more to DHH and more corporate (or enterprise) developers inclining towards Dave Thomas. Doing what I do, I think it's quite clear to which category I belong. This is really not to say that one or the other is better – definitely no – they just have slightly different needs and as such they look for slightly different things. Ever since I came to Singapore (actually ever since I came to Asia 7 years ago) I've been trying to find a professional community of actual developers, people who work with the technology (whatever the technology I was using at the time). This seems to be the only one worth mentioning (so far) – I am not picking on the “sales” communities around the biggest vendors – they just weren't my cup of coffee. Thursday's meeting was a very interesting mix of people with representation from actual developers, potential developers as well as potential customers and partners. Each and every of those groups has its place and purpose in the community but all of them need and expect very different things. As such I hope we will be able to structure the meetings so that everybody can benefit and that nobody will have to sit through 2 hours of something that sounds like total gibberish to them. Again, I think this is for a longer discussion and I am very open to meet up and discuss the possibilities.

Lastly, we are a high-tech community – aren't we? All of us create huge, complicated systems every day – don't we? And yet – let's look at our presentation – just count how many times Ivan uses a word “guy” as opposed to the number of names. I think we could do much better in profiling ourselves – even the most techless business / marketing communities have websites nowadays :-)

Yes – it's a PDI (to myself as well).

Restless

Today's again one of those days that I feel restless. It may be difficult to imagine if you've never experienced it but most of the people I know do have this feeling from time to time. I feel like my head is going to explode. I am dying to do something – do anything – my hands are burning to start working.

It's usually a sign that I need a break or that something is bothering me (more like eating me alive) or that my mind is simply running out of operating memory and I need some “things” to be closed to free up the processing power. You know like a runaway process on the server that is eating up all of the resources and grinds all the other services to halt.

This whole week seems like a runaway process for me. It started at hectic pace with back to back meetings right until my flight to Hong Kong. Then my whole China trip was really a crazy ride. I didn't let my customer down and managed to do everything I was supposed to do – in fact, I don't know if it was more the stressful work or the new (unbelievably eye opening) environment but the whole situation caused my brain to soak with so much of information that there was hardly any space left and made me miss my flight back which caused even more stress as I had to rush to the Ruby on Rails meeting straight from the airport my clothes still smelling of beer that the drunk manageress of a night club spilled on me the night before. All that with not more than 2 hours of sleep. As if it was not enough what followed was a hectic Friday with a really intense meeting the whole morning and at least a bit more relaxed in the afternoon.

Now, this is exactly what makes me tick – all this is exactly why I love being in this business and what gives me enough adrenaline to wake up every morning and keeps me up until late night. Unfortunately, as much as all our muscles need stress to become stronger they need a time to relax at least as much. You flex and then release. Normally, my body knows how to control this and usually after my last meeting on Friday it turns off and I am able to get the rest I need. Sometimes, it's not as easy. Sometimes I need a reset – you know, vacuum the brain a bit and then fill that space with fresh things. Sometimes I just need a mental challenge – to watch a good movie or to read a good book, meet someone interesting, or a challenging discussion to get my mind somewhere else. At times (like today) none of those is nowhere around. I'd do anything but just cannot find the right thing. I see the clock ticking and the more they move the less time to relax and the more I feel like I am going to explode.

Apache 2.2.4 and Mongrel on Ubuntu

This second installment about installation of the new Apache 2.2.4 and it's configuration for production use with Ruby on Rails and Java. This installment discusses installation of Mongrel and it builds on installation of apache.

Mongrel Installation

Install mongrel and its supporting software. Make sure when prompted you select the most recent, non mswin32 option.

sudo gem install daemons gem_plugin mongrel mongrel_cluster --include-dependencies

Set up mongrel cluster

When deploying a mongrel cluster, none of the mongrel servers will be listening on a privileged port. This means that you don't have to run mongrel as root. I usually use just my normal non-root user to run the mongrel cluster. It also helps to avoid problems with file permissions.

For the purpose of this example we will just use a freshly minted rails app. You can adjust these instructions to work with your app, wherever you may have placed it. I usually put all the applications in /home/peter/rails/. So lets go set up our app and test that mongrel will serve it:

mkdir -p /home/peter/rails/mediacom/releases/
cd /home/peter/rails/mediacom/releases/
rails 1
cd ../
ln -s current releases/1
cd current
mongrel_rails start

You should now be able to see your application at http://host:3000/. If you can, you are good to go. Hit CTRL+C to stop the mongrel server. At a minimum the log directory of your app has to be writable by the peter user:

sudo chown -R peter:peter /home/peter/rails/mediacom/

With mongrel working and our webapp directory prepared we can proceed with the mongrel_cluster configuration step:

sudo mongrel_rails cluster::configure -e production \
   -p 8000 -N 3 -c /home/peter/rails/mediacom/current -a 127.0.0.1 \
   --user peter --group peter

This will write a configuration file in config/mongrel_cluster.yml. We have setup to run our cluster in production mode as the user mongrel and will start 3 mongrel servers listening on ports 8000, 8001, and 8002. Now, lets do a quick test of what we have setup so far:

sudo mongrel_rails cluster::start

Checking our host on ports 8000, 8001, and 8002 we should now be able to see our test application. We can stop all of those mongrels with

sudo mongrel_rails cluster::stop.

On Boot Initialization Setup

At this point, mongrel and mongrel_cluster are setup and working with our sample webapp. Ultimately, we want this cluster to start on boot. Fortunately, mongrel_cluster comes with an init script that we can just drop into place. All we need to do is put the configuration files in /etc/mongrel_cluster and take care of a few system tasks:

sudo mkdir /etc/mongrel_cluster
sudo ln -s /home/peter/rails/mediacom/current/config/mongrel_cluster.yml \
   /etc/mongrel_cluster/testapp.yml

The following step needs to be done only once and ofcourse you mongrel_cluster version may vary:

sudo cp \
   /usr/lib/ruby/gems/1.8/gems/mongrel_cluster-0.2.1/resources/mongrel_cluster \
   /etc/init.d/
sudo chmod +x /etc/init.d/mongrel_cluster

Now we have a typical System V init script that will launch our mongrel cluster. Actually, this script will launch any cluster that has a configuration file in /etc/mongrel_cluster/. So when we run /etc/init.d/mongrel_cluster start it will start all clusters. Likewise for stop and restart. To install the script use:

sudo /usr/sbin/update-rc.d -f mongrel_cluster defaults

Change the shebang line (the first line) of the mongrel_cluster_ctl to:

#!/usr/local/bin ruby

Apache configuration

Probably the most difficult part. As the apache was compiled from source the configuration files are located quite nonstandardly under /usr/local/apache/conf. As I use the apache for hosting multiple applications (I really wonder how many mongrel instances can such slice host sustain :-) I create a separate folder for configuration files for each of them. This example uses mediacom application and will be accessible from virtual domain mediacom.nextlogic.net. Of course when setting up another application instead of mediacom in files and folder names use the appropriate name :-)

1) Create folder for configuration files

mkdir /usr/local/apache/conf/mediacom.conf
cd /usr/local/apache/conf/mediacom.conf

2) Create file mediacom.common

  ServerName mediacom.nextlogic.net
 DocumentRoot /home/peter/rails/mediacom/current/public

 
   Options FollowSymLinks
   AllowOverride None
   Order allow,deny
   Allow from all
 

 RewriteEngine On

 # Uncomment for rewrite debugging
 #RewriteLog logs/myapp_rewrite_log
 #RewriteLogLevel 9

 # Check for maintenance file and redirect all requests
 #  ( this is for use with Capistrano's disable_web task )
 RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
 RewriteCond %{SCRIPT_FILENAME} !maintenance.html
 RewriteRule ^.*$ /system/maintenance.html [L]

 # Rewrite index to check for static
 RewriteRule ^/$ /index.html [QSA]

 # Rewrite to check for Rails cached page
 RewriteRule ^([^.]+)$ $1.html [QSA]

 # Redirect all non-static requests to cluster
 RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
 RewriteRule ^/(.*)$ balancer://mongrel_cluster_mediacom%{REQUEST_URI} [P,QSA,L]

 # Deflate
 AddOutputFilterByType DEFLATE text/html text/plain text/css
 # ... text/xml application/xml application/xhtml+xml text/javascript
 BrowserMatch ^Mozilla/4 gzip-only-text/html
 BrowserMatch ^Mozilla/4.0[678] no-gzip
 BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

 # Uncomment for deflate debugging
 #DeflateFilterNote Input input_info
 #DeflateFilterNote Output output_info
 #DeflateFilterNote Ratio ratio_info
 #LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
 #CustomLog logs/myapp_deflate_log deflate

3) Create mediacom.conf

    Include conf/mediacom.conf/mediacom.common

 # This is required to convince Rails (via mod_proxy_balancer) that we're
 # actually using HTTPS.
 RequestHeader set X_FORWARDED_PROTO 'https'

   ErrorLog logs/mediacom_errors_log
   CustomLog logs/mediacom_log combined
 

4) Create mediacom.proxy_cluster.conf

    BalancerMember http://127.0.0.1:8000
   BalancerMember http://127.0.0.1:8001
   BalancerMember http://127.0.0.1:8002
 

Note; Of course the port numbers will differ (based on whatever you configured when setting up mongrel cluster.

5) Create mediacom.proxy_frontend.conf

Listen 8100

 
   SetHandler balancer-manager
   Deny from all
   Allow from peter.dyndns.org
 

Note
Of course the port number will have to be unique on the server. Allow access from your current location (or whatever the location you will need the access from)

6) Inform httpd.conf about your configuration files At the following to the very bottom of /usr/local/apache/conf/httpd.conf

Include conf/mediacom.conf/*

7) Restart apache and you should be good to go. Of course, you will have to make sure that the virtual host subdomain actually points to the server (i.e. that mediacom.nextlogic.net will actually point to your server - for testing just make an entry in /etc/hosts)

/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl start

8) Go to http://mediacom.nextlogic.net and you should be able to see your app :-)

Top 10 Software Development Books for Beginners

This topic has been boiling inside of me for quite some time now. In fact, ever since I hired my first employee in my first job in Asia. I've been always reading a lot of books – even when I had no money I would spend Sundays in MPH or Kinokuniya :-). Even though I've had Safari Books Online subscription like forever (actually since my master's degree thesis) hardly a month goes by that I wouldn't buy a new book.

Anyway, my problem is to select a “reasonable” number of books for my new employees / trainees to read so as to give them essential (really basic and necessary) information and still not overload them to the point when the loose the sight of why they're doing it. Here I must point out that 99% of all people I've interviewed and around the same percentage of people I eventually hired had never read (and pretty much never heard of) any of my top 10 essential books. Most of those were computer science graduates from Singapore, Malaysia, Germany, India, Slovakia, Myanmar, etc.

The books to start with are especially tricky because of a different learning style for different people. Another issue is to avoid (or undo) possible bad habits. I may not be able to stress this enough...but this is realy not about book "reading". Our orientation process means working with those books daily - working out the examples, trying them in different languages and putting the concepts to work first on testing projects and later on real projects.

Put together, Top 10 that my employees / trainees have to bite through looks like this:

1) Test Driven Development by Kent Beck – this is definitely the first and the last book of software development for me. I think without this book made all the agile methodology possible and it pretty much defined way of programming today.

2) Learn To Program by Chris Pine - this is quite a recent addition – in May 2006 when it was clear that most of our development had moved from Java to Ruby, this book replaced Head First Java by Kathy Sierra and Bert Bates. It may sound a bit strange to include an actual programming book so early in the process but it sort of provides a tool or media for further learning.

3) Head First Object-Oriented Analysis and Design by Brett McLaughlin, Gary Pollice, David West - what the hack took you so long guys? This book finally filled in the spot shared by several books on OOP. Even though OOP has been a standard for more than 20 years I still find it one of the most difficult part for people to understand. I even like that the book uses Java for all the examples, because doing those examples in Ruby gives you plenty of opportunity to see Java and Ruby code side by side and to really see the differences between dynamic and static languages.

4) Head First HTML with CSS & XHTML by Elisabeth Freeman, Eric Freeman - what to say...after real hard work with book no 3 this one reads like a comic book. It's really a pleasure to read and I would make it required reading for all web designers.

5) Head First Databases – by I wish Unfortunately, this spot is still open. At this point I would need some down to earth, fun to read, easy to understand non-reference database book. Please let me know if you know of anything

6) Agile Web Development with Rails by Dave Thomas and David Heinemeier Hansson We're getting serious :-). For some of you it may seem like a long time to wait but only around this point I usually feel that my new hires are ready to not only understand but to really appreciate the beauty of rails. With rails simplicity and generators (we use custom made generators that generate pretty much everything besides the business logic) it's just way too easy to slip to monkey coding without actually understanding it.

7) Pragmatic Version Control: Using Subversion (SE) by Mike Mason Again something that universities (and many small companies) should look into. Nothing else to say...required reading.

8) Extreme Programming Explained (SE) by Kent Beck It's really hard to understand and appreciate this book without at some working experience. Everything seems so obvious and natural and yet it's so rarely done that way. At least around this part of the world. Most of the time I hear people say that Asian boss wouldn't approve of agile principle, that Asian boss wouldn't stand two people sharing a computer, but in my experience that's rarely the case. More about it in some other blog...

9. Refactoring by Martin Fowler - after having written some code, you're ready to change it. This book provides not only solutions to common code smells but it also helps you to train your nose for those smells.

10. Beyond Software Architecture: Creating and Sustaining Winning Solutions by Luke Hohmann - this may be a little different from the rest, but it very nicely sums up all the 'other' questions regarding software development and actual deployment. Even though it's not platform specific some sections would deserve a small update...but overall a very complete reference.

okay...so that's my top 10 learn to program (business applications) list. If you didn't find your favourite in there don't worry - it's most probably because this list is meant for beginers. In fact, I cannot believe how much this list changed since only year and a half ago.

what's next? My intermediate top 10 will follow soon :-)

Deployment Using Apache 2.2.4

Couple of weeks ago I found a new respect for Apache. I was installing a new VPS and I needed to deploy ruby on rails applications, Java applications and PHP side by side. As the site is for mostly internal use, the traffic was originally quite low, but due to some problems with one of my other hosting providers I was forced to move there some of our external applications with considerably higher traffic. As soon as the traffic grew beyond the resources of this server I realized how easy it was to scale out and instead of moving some of the applications away I was able to retain this server as the main balancer that is sending traffic to servers around.

Here's a simple walk through for installing Apache 2.2.4 on Ubuntu. It's compiled from various sources – they are listed below – of course, they did the tough job :-). I just put everything in one place to provide a complete example.

Prerequisites

As the Apache 2.2.4 is not yet in Ubuntu repositories you will need to do manual installation (i.e. compile from source). Before that you have to make sure that whatever apache installed using apt-get is removed from the system:

sudo dpkg --purge apache apache2

Install the GCC compilers and developer tools:

sudo apt-get install build-essential

Some extra libraries needed for Ruby/Apache to work:

Install zlib:

wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar xvfz zlib-1.2.3.tar.gz
cd zlib-1.2.3/
./configure
make
sudo make install

alternatively you can use:

sudo apt-get install zlib1g zlib1g-dev

Install openssl:

sudo apt-get install openssl libssl-dev

Install Readline:

apt-get install libreadline5 libreadline5-dev

If you haven't done so already install ruby:

sudo apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 libreadline-ruby1.8 libruby1.8

ONLY FOR 64-bit processors you will need to use Ruby 1.8.5 or higher to make postresql gem work, which means, you will need to compile it from source. In this case download the latest Ruby source:

wget http://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6.tar.gz
tar -xzf ruby-1.8.6.tar.gz
cd ruby-1.8.6
./configure
make
sudo make install

Now create a few links to ruby that will be used later (this may or may not be necessary - but no hurt doing it :-):

sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

Install readline support for ruby:

cd ext/readline
ruby extconf.rb
make
sudo make install
cd ../../../

Apache Installation

Download and compile Apache 2.2:

wget http://apache.rmplc.co.uk/httpd/httpd-2.2.4.tar.gz
tar -xvf httpd-2.2.4.tar.gz
cd httpd-2.2.4
./configure --prefix=/usr/local/apache --enable-proxy   --enable-proxy-http \
--enable-proxy-balancer --enable-dav --enable-rewrite    --enable-so \
--enable-http   --enable-ssl    --enable-expires  --enable-headers  \
--enable-mods=deflate_module --with-php \
--with-mysql --with-susexec --disable-info  \
--without-berkeley-db --enable-dav=shared \
--enable-dav-lock=shared --with-included-apr
make
sudo make install

ln -s /usr/local/apache/bin/apachectl /usr/sbin/

Apache at start-up

It's a good idea to have Apache start at boot time automatically:

sudo cp /usr/local/apache/bin/apachectl /etc/init.d/apachectl
sudo chmod +x /etc/init.d/apachectl
sudo vim /etc/init.d/apachectl

Add the followinig, so the top of the file looks like:

#!/bin/sh
#
# chkconfig: - 85 15
# description: Apache is a web server.

Now we need to register it with the start-up manager:

sudo /usr/sbin/update-rc.d apachectl defaults

Securing Apache

It's also a good idea to create a dedicate Apache system user account. It'll make your install more secure.

sudo adduser --system apache

To make apache actually use it edit the configuration file:

sudo vim /usr/local/apache/conf/httpd.conf

You need to find the lines that say:

User daemon
Group daemon

And change them so they look like:

User apache
Group nogroup

Install PHP

While you're at it install the PHP as well... You may need to install bison and flex first:

sudo apt-get install bison
sudo apt-get install flex

Download the php from www.php.net. In my case it was PHP4:

wget http://sg.php.net/distributions/php-4.4.6.tar.gz
tar -xzf php-4.4.6.tar.gz
cd php-4.4.6
./configure --with-apxs2=/usr/local/apache/bin/apxs --with-mysql

make
sudo make install

sudo cp php.ini-dist /usr/local/lib/php.ini

Add the following to httpd.conf (sudo vim /usr/local/apache/conf/httpd.conf) - put it somewhere around the other AddType definitions:

AddType application/x-httpd-php .php .phtml

Update line

DirectoryIndex index.html

to

DirectoryIndex index.html index.php

Restart apache and you're done

sudo apachectl restart

Wow! We're done with the first part. Once you're done here you're able to add the actual deployment servers. Here are some of my favorite:

• Mongrel Cluster to deploy Ruby on Rails applications
• Tomcat to deploy Java applications

References

• http://www.simplisticcomplexity.com/2006/8/13/apache-2-2-mod_proxy_balancer-mongrel-on-ubuntu-6-06
• http://blog.codahale.com/2006/06/19/time-for-a-grown-up-server-rails-mongrel-apache-capistrano-and-you/
• http://mongrel.rubyforge.org/docs/apache.html
• http://mongrel.rubyforge.org/docs/mongrel_cluster.html
• http://www.kodefoo.com/2007/2/18/deploying-rails-on-ubuntu-dapper/
• http://davidwinter.me.uk/articles/2006/10/17/building-apache-22-from-source-for-ubuntu-dapper/

Fixed Price Contracts in Agile Projects

Today I got an interesting question about how our company deals with fixed scope/price contracts. Here's my slightly enhanced reply :-)

To say right from the beginning - we have never done a fixed scope contract. We have done fixed price contracts, but none of the systems we've ever done (from small CRM for 2 people company to quite large systems spanning geographical locations, departments, etc.) has ever turned up the way it was originally "envisioned" by the customer. And that applies even to systems that were replacing some existing systems and we could work with the original developers/domain experts who knew precisely what they wanted. In fact, it applies even to systems we built for ourselves :-). Our first release comes usually after 1 week and by 3rd week the customers usually have a completely different idea about what they really need and, especially, what is possible. I would even go as far as to say that if the system is delivered the same as originally requested, there's something wrong.

As such, our contracts with customers usually define the system only on modular level with “hints” of the most important specific functionality requests. It may sound funny, but the details are going to change anyways. (more about it can be found here Fixed Scope Mirage) I usually get asked how we decide what to work on if the scope is so vague. This is very simple – it's always decided by users' priorities. And at the end, it's the business priorities of users that will define the scope of the system.

Is it always possible? Definitely not. It really depends on the personality and experience of your users. It's going to be really hard for someone who's never done anything with accounting to define what his accounting system should do. Most probably the worst category is users (or customers) that don't care. In those cases I would rather suggest an off-the-shelf system. Another situation when it's really difficult is when there are no real users yet. The crucial point here is instant communication and working directly with the actual users (for us at least once a week). Of course, to do that you need to be "releasing" daily (depending on the phase). Our customers also have access to our issue tracking system so at any time they know what we're working on and how long it takes.

Now to the more difficult question...fixed price contracts. How do you put a price on something that nobody knows what it is? It's possible to have a fixed price even in agile project ( Fixed Price) and I must say that most of our projects start with the fixed price. We do it either by specifying a subset of functionality that the customer believes is well defined and we do that first. When/if we deliver or when the customer gets the touch and feel of what we do and how we do it they decide to go on with the scope at which point we provide another quotation (with usually much better conditions). The second way - in case the customer simply wants the whole system we simply agree on the buffer that will have to cover the difference. Once the buffer is over, we wrap up and the rest is negotiated/quoted and invoiced separately. Regardless of that, however, our payments are always progressive - by iterations. At the end of the iteration the payment is made and we continue only after we've received it. Of course, doing the above will effectively ruin the time plan. There are countries, however, where you need to be really careful and we require deposit. Of course, that is not to say that we've only had perfect contracts - regardless of the contract, there are always challenges. The rule of thumb is get something done and get the money.

Having done projects in several countries in Asia and Europe I must say that contracts vary really a lot depending on the given country/culture/legal system and so on. Whereas in Europe everybody requires 20 pages legal document that is handled by lawyers, in most of Asia the contracts are worthless and the only thing that matters is money in the hand.

Programming and Music

This article was inspired (if not provoked) by Dave Thomas. It was several of his latest blogs that really made me stop, take a few steps back (actually 10000 or so km dividing Singapore and Slovakia) and take a break from the constant work cycle. Our circumstances couldn't be more different – he is an accomplished writer / programmer and founder of the pragmatic cult. Yet our daily issues, things we notice couldn't be more similar.

It started with Dave's blog drawing parallels between music and programming. Having finished 12 years of musical school (and frankly not having touched a musical instrument since). I've always wondered if we could get any inspiration for teaching / learning programming from teaching music. Maybe the most appropriate would be a jazz example, as jazz is probably the most flexible yet having a strong internal structure, patters and the same tune played by different musicians or played for different audiences may turn up completely differently. Just think of different renditions of Wonderful World. The parallel here is with pretty much any “standard software solution”. I worked on more than 20 inventory control systems and all of them were different – they used the same structure and patterns but the result was completely different for national distributor carrying 120 thousands different products and for a farm producing and selling bean sprouts.

Anyway, this is not so much about the end result as it is about the process. When you start playing a musical instruments you have to practice scales, etudes and and arpeggios. Sure you'll be playing some simple tunes as you go along but your daily bread for many years will be scales, etudes and then some more scales, and yes, etudes. From time to time, to boost up your motivation and to show the parents where their money is going, you play some concert gigs. From two note polka, when you're six through some Mozart when you're twelve to some Iron Maiden covers on your graduation party. Okay, maybe not that but when we played 6 saxophones + piano rendition of Chattanooga Choo Choo graduating from “school of classical music” everyone was equally shocked.

And this is exactly what I miss in my programming trainings. The first thing you learn on a new musical instrument is playing the scales, as they give you basic touch and feel of the instrument. As the notes are always the same you can fully focus on exploring each individual note, possibly different ways of playing it and slowly you learn to transition from note to note and you're getting comfortable with the full scale of the instrument. In software development I imagine this as a collection of simple challenges covering the basics of a language. Something like the really really great book Learn to Program, just a bit more structured and with many many more practice challenges. If you programmed in other languages and you're familiar with the concepts you will need just a few hours to go through and after you're done you will know the basic building stones of the language.

Later you move to etudes. Each etude is focused around a specific riff specific transition or specific problematic move but still having some kind of melody to make the learning process less boring. The idea is to practice this one thing over and over and over until you completely internalize it. This is another thing I really miss when teaching programming. To explain the collections in ruby is very easy – if you talk really a lot then you need maybe 10 minutes. Their so easy that their usually forgotten right away. Even if my students go through several “challenges” they're usually way too few and way too simplistic to really understand and internalize the collections so that next time they need a collection they will choose the right one without even realizing it and they will write iteration loop without even knowing it. And this is true for every part of the language (any language). When learning to play saxophone I went through around 4 etudes books every year. When teaching ruby programming I have nothing. The programming etudes would do the same as the musical ones – introduce various concepts, first in isolation later with more and more context and provide a wealth of challenges and questions to solidify the knowledge.

The last part of learning is working with the real life concepts – tinkering and playing with them and as there usually isn't just one way seeing it from different perspective and different angle. This part seems to be more addressed in programming as there are several good resources – for example Dave Thomas' Code Kata or book Ruby Quiz.

This is of course not to say that there are no resources for learning at all. Of course, there's plenty of books for every possible aspect of programming. Most of them, however, are written in more exploratory/reference style that is suitable for self study and exploration of the language, than for internalizing the concepts. An exception here is Java as there is plenty of preparation books for the first certificate and they provide sufficient number of questions and challenges. The truth is that ruby may not have reached the critical mass yet and it's used by people who “know how to use it”.