Tuesday, April 14, 2009

Another Training

Another training's over. Very different but equally challenging.
Seems that Ruby's been making headway to corporate world as most of the participants this time were there on their employers' budget. When I compare it with our first RoR trainings around 2 years ago with Rails 1.0 just out and more than 30% of the things still on a wish list and the only way to learn RoR was Agile Web Development With Rails...this time it was much more business-like. There's like 20 books out, including a (bit obsolete) video from Linda.Com and pretty much everybody knows about rails.
Another aspect that has changed is Rails versions. With Rails 2.3 and Ruby 1.9 out there's a lot of changes and incompatibilities that complicate the learning. Most of the books and reference materials use Rails 2.0 or early versions of 2.2. This makes it little bit difficult to have training on 2.3 as most of the "further" reading is incompatible. While the incompatibilities may seem easy to solve for experienced developers they can be quite exasperating for a newbies. On the other hand, people are quite reluctant to start learning something that is already obsolete.
Similar is true for Ruby 1.9, even though after burning they're fingers several times until it got finally compatible with Rails most people would still prefer to use 1.8.6 or 1.8.7.

Monday, April 13, 2009

Dynamic Columns In Ext JS Grid

I just had another unproductive afternoon. Another thing I would expect to work out of the box that turned out to be not that easy. It was not the first time I needed dynamic number of columns in the grid but until now I was always able to find a workaround. Fortunately, today I couldn't go around and had to solve the problem once and for all.

What I needed was a column model dynamically adjusting depending on the data received from data store (i.e. JSON provided by server). Something like the following:



Where each site has a different set of dimensions (rows) and different set of auditors (columns).

It turned out that the problem was actually two-fold - first dynamically change the columns and then display the editable checkboxes instead of true/false values.

To change the columns I used metaData attribute in my JSON. Including metaData attribute causes Ext.data.JsonReader to be adjusted to the new set of fields and other config parameters and fire metachange event. Here's an example of the JSON with metaData:
{
 "results": 3,
 "auditors": [
  {"dimension_name": "Audits and Reviews", "auditor_3": false, "dimension_id"
: 10, "id": 10, "auditor_4": false, "auditor_18": false},
  {"dimension_name": "Emergency Management & Fire Safety", "auditor_3": false, "dimension_id": 8, "id": 8, "auditor_4": false, "auditor_18": false}
  .
  .
  .
 ],
 "metaData": {
  "fields": [
   {"name": "id"},
   {"type": "string", "mapping": "dimension_name", "name": "auditor[dimension]"},
   {"type": "int", "mapping": "dimension_id", "name": "auditor[dimension_id]"},
   {"type": "bool", "name": "auditor_3", "header": "Dasha Alexo"},
   {"type": "bool", "name": "auditor_18", "header": "Tester 3"},
   {"type": "bool", "name": "auditor_4", "header": "Zobak Zobakovic"}],
  "root": "auditors",
  "totalProperty": "results",
  "id": "id"
 }
}
In this case it's the number of auditors (and their names) that's changing. My data types are booleans but displaying any other data type would look pretty much the same.

Next I needed to update the column model based on the new meta data. I did that by adding metachange listener to my store
auditors_store.addListener("metachange", function(store, meta){
 var grid = Ext.getCmp('auditors-grid');

 var columns = [
  {header: 'Dimension', dataIndex: 'auditor[dimension_id]', hidden: true},
  {header: 'Dimension', dataIndex: 'auditor[dimension]', width: 200}
 ];
 
 for (var i = 3; i < meta.fields.length; i++ ) {
  var plugin = new Ext.grid.CheckColumn({
    header: meta.fields[i].header,
    dataIndex: meta.fields[i].name,
    grid: grid,
    width: 120
   });
  
  columns.push(plugin);
  plugin.init(grid);

  //use columns.push( { header: meta.fields[i].header, dataIndex: meta.fields[i].name, type: meta.fields[i].type }); for fields that don't require plugins
 }
 grid.reconfigure(auditors_store, new Ext.grid.ColumnModel(columns));
});

I had to create and initialize the plugin for each new column as I needed editable checkboxes. For normal fields just use the usual column config object { header: meta.fields[i].header, dataIndex: meta.fields[i].name, type: meta.fields[i].type }.

The actual magic happens is in the last line. Grid has an obscure method reconfigure( Ext.data.Store store, Ext.grid.ColumnModel colModel ) that allows to switch the store and column model (the columns displayed) on the fly.

My next problem was to make the checkboxes editable. To do that I had to put a small fix to the CheckColumn plugin (see the plugin) as the original plugin assumes that the grid is not rendered when initializing, however, when dynamically adding columns the grid is already rendered so I had to change the following:
 init : function(grid){
  this.grid = grid;
   this.grid.on('render', function(){
    var view = this.grid.getView();
    view.mainBody.on('mousedown', this.onMouseDown, this);
   }, this);   
 },
to the following:
 init : function(grid){
  this.grid = grid;
  if (this.grid.rendered) {
   var view = this.grid.getView();
   view.mainBody.on('mousedown', this.onMouseDown, this);
  } else {
   this.grid.on('render', function(){
    var view = this.grid.getView();
    view.mainBody.on('mousedown', this.onMouseDown, this);
   }, this);   
  }
 },
And that's all - nice and simple. If only there were more books on Ext :-(

Wednesday, April 8, 2009

Agile Employment Contracts (part 1)

A lot has been said said about agile development and agile principles like content delivery over contract negotiation, flexibility of contracts and no upfront specification. We've been loving and doing that for some years now. One thing I was not able to wrap my head around for the longest time, though, was how to manage (traditional) employees in this kind of a non-traditional arrangement. Most of the agile books spend only very limited time on this topic pretty much always retorting to anecdotal evidence of "self motivated teams rising to the occasion and living happily ever after"

When dealing with clients, both parties are bound by the contract (however agile and flexible) and there are clear and simple business rules to be followed. In our case it comes down to if client doesn't like what they're getting they don't pay. The payment happens after every iteration so both parties have a tangible motivation/interest to get things done. Both parties enter the contract without previous baggage and maintain pretty much equal status.

Not at all as simple with (traditional) employees. They usually start with limited knowledge and require extensive training. In our company it takes several months and even after they've mastered the technical skills they still need months and months of experience to grasp various aspects of different domains and business processes we deal with every day. We use commission from closed iterations and collected payments as a motivation. Main portion of the money, however, still comes in the form of monthly salary. Given high level of stress, steep learning curve of ever changing technology, demanding customers, even more demanding bosses it's only matter of time when the initial motivation wears out and everyday routine kicks in. In many of my ex-employees, this was accompanied with a feeling that they've learned more than they'd ever thought they would and their price on the market has soared indefinitely. Sarcasm aside, after reaching this level, people start to be more relaxed and try to find motivation/amusement in experimentation with latest gadgets, obscure technologies or start working on they're own version of Rails or other frameworks. What's worse, it slowly starts to affect the delivery of project(s).

Now let's switch to our shoes. We spent the money on training and non-productive period of the employee. We see that the results are diminishing and we see that the employee is even starting to abuse others requiring help and claiming that he is a slow programmer and that it's not his domain knowledge, etc. Problems in delivery cause delayed payments and smaller and less frequent commission for the employee. No matter how little employee produces (and this can quickly become negative value when the quality reaches throw away and rework levels) his salary is still guaranteed to him by employment act.  If we let him go we lost all our investment in training and we will incur extra costs replacing him mid-project. During all this time there is absolutely nothing we can do to protect our side. Yes, we can motivate, but if that doesn't work we've got nothing.

Agile development contracts seem to be calling for agile employment contracts.


Tuesday, April 7, 2009

Ubuntu 8.04 mod_rails

Around a half a year ago I started slowly switching my servers to Apache mod_rails. I wasn't very keen after my first test around September on one of my Slicehost servers. Without any application running apache, ruby and postgresql processes tookup around 500MB of memory - around 100MB per processes. When I experimented with my other servers I found out this happens on 64bit architecture - 32bit servers took up less than half of that memory. I am still running performance tests to compare the performance and behavior of mod_rails vs Lighttpd - I'll be writing about it soon. For now here's the installation for Hardy Heron (I assume you have the build-essential, ruby and so on already installed):
sudo apt-get install apache2 apache2.2-common apache2-mpm-prefork apache2-utils libexpat1 ssl-cert apache2-prefork-dev
sudo gem install passenger
sudo passenger-install-apache2-module
Now add the 3 generated lines (almost last on the screen) to config file:
sudo vim /etc/apache2/apache2.conf
Right now it looks something like the following - BUT THE VERSION NUMBERS WILL CHANGE SO COPY THE LINES FROM THE PREVIOUS PROCESS
LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/passenger-2.1.2/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.1.2
PassengerRuby /usr/local/bin/ruby
In the same config file add the server name and NameVirtualHost otherwise the server will complain at restart:
ServerName XYZ
NameVirtualHost *:80
save the config file and continue - we'll need to enable modrewrite and disable default site:
sudo a2enmod rewrite
sudo a2dissite default
now just configure your application by adding your_app_name to /etc/apache2/sites-available/ with the following content:
sudo vim /etc/apache2/sites-available/your_app_name
<VirtualHost *:80>
ServerName  abc.def.com
ServerAlias xyz.com

DocumentRoot /home/bohm/rails/access/current/public

</VirtualHost>
where abc.def.com and xyz.com are the URLs of your application and DocumentRoot points to the public folder of your application. To enable the application run
sudo a2ensite your_app_name
sudo /etc/init.d/apache2 reload
To disable it run
sudo a2dissite your_app_name
sudo /etc/init.d/apache2 reload
And you're done :-)

Monday, April 6, 2009

Multiline Tree nodes in ExtJS

While I really like Ext and the component model behind it, some simple things seems to be missing - an example would be multiline nodes in TreePanel. After some searching I managed to find a  MultilineTreeNodeUI plugin.
Here's how to use it:
new Ext.tree.TreePanel({
 //renderTo:'treecontainer',
 id: 'upcoming_events',

 loader: new Ext.tree.TreeLoader({
  dataUrl:'/datasource/upcoming_events_json',
  baseParams: {},
  baseAttrs: {uiProvider: Ext.ux.tree.MultilineTreeNodeUI}
 }),
 root: new Ext.tree.AsyncTreeNode({
  expanded: true
 }),
 rootVisible: false,
 collapsed: false
})

Just include the MultilineTreeNodeUI.js in the page and create a usual TreePanel. The only difference is to include baseAttrs: {uiProvider: Ext.ux.tree.MultilineTreeNodeUI} in your loader. Multilne tree requires the data in the following format:
{
 text: '09 April 2009',
 iconCls: 'calendar',
 details: ['Program for the day...'],
 uiProvider: Ext.ux.tree.MultilineTreeNodeUI,
 children:[
  {
   text: '9:30',
   iconCls: 'time',
   details: ['Win $1M', 'Sure win...'],
   uiProvider: Ext.ux.tree.MultilineTreeNodeUI,
   leaf: true
  },
  {
   text: '11:30',
   iconCls: 'time',
   details: ['Buy private jet', 'Or something'],
   uiProvider: Ext.ux.tree.MultilineTreeNodeUI,
   leaf: true
  }
 ]
}
Where the additional lines are provided by details attribute. It's an array - one item per line - maybe not an ideal solution but still workable - worst case you just split the longer line by number of characters. The uiProvider will generate the multiple lines. Your controller doesn't have to do anything about it as it will be added automatically by the baseAttrs: {uiProvider: Ext.ux.tree.MultilineTreeNodeUI} we added in the loader.
And here's the outcome:

Saturday, April 4, 2009

SMS with GSM Modem

Last time I wrote about sending SMS using online tools. The main problem there remains 2 way communication. Luckily, there's a simple solution to receiving problem. Since the start of 3Gmobile internet, GSM modems became quite ubiqitous and reasonably cheap. Just like the SMS software. I used Smstools that proved to be a snap to install and use.
wget http://smstools3.kekekasvi.com/packages/smstools3-3.1.3.tar.gz
tar -xzf smstools3-3.1.3.tar.gz
cd smstools3
sudo ./install.sh
I am using Huawei Mobile Modem E270 and E170 from Starhub. The only change I had to make now is update the config file. Here's my config file /etc/smstools.conf
devices = GSM1
logfile = /var/log/smsd.log
loglevel = 7

[GSM1]
device = /dev/tty.HUAWEIMobile-Modem
incoming = yes
#pin = 1111
init = AT+CNMI=2,0,0,2,1
After that just start/restart the smsd daemon
sudo killall smsd; sudo smsd;
Sending of SMS is now as simple as
sudo sendsms +6584199983 'Your SMS message....'
or just put SMS File to /var/spool/sms/outgoing/
To: 491721234567

Hello, this is the sms.
More info about the structure of the file is here.
As for receiving, all the received messages will be downloaded to /var/spool/sms/incoming/ so all you have to do is create a cronjob to check this folder for new messages and then forward the message to your application.

SMS Integration Without Hardware

When I tried to integrate one of our applications with SMS messages (both incoming and outgoing) several years ago we didn't manage to find not even one SMS gateway provider that would be able to pass our simple criteria. Allow sending (da :-) and receiving of messages with some reasonable notification of received messages (simple webservice / forward to email, anything). The replies should preferably be handled by a local number and sending/forwarding should be within reasonable price range :-) Is it too much to ask?

There is a number of online portals for sending (some support also receiving) SMSes like Clickatel, SMSZilla, SMSCountry. Their pricing per message varies from SGD0.05 - 0.1 which is quite reasonable and some provide nice and easy API for sending (when you think about it how complicated can it get to write a form handler with 3 or 4 fields) - some ask you to pay for API extra. For better or worse they would accomplish the sending but the receiving was either very painful or expensive or both. On top of that sending a hundred messages to and Indian or US number shows on your bill so we had to find something better.

For systems that require only sending we mostly used Singtel Ideas or Starhub Gee. Both are just a little short of and actual webservice, nevertheless worked quite well. Here's an example how to send message using StarHub Gee (12345678 is your phone number)

require 'rubygems'
require 'mechanize'

mech = WWW::Mechanize.new
page = mech.get('http://groupsms.starhubgee.com.sg/VCRLS/login.cgi?cb=http%3A%2F%2Fgroupsms.starhubgee.com.sg%2Fcgi-bin%2Fs.pl')

#login first
form = page.form('loginform')
form.action = "https://login.starhubgee.com.sg/eam/login"
form['cb'] = 'http://groupsms.starhubgee.com.sg/cgi-bin/s.pl'
form['vcid'] = 'groupsms'
form['pudn'] = 'starhubgee'
form['domain'] = 'starhubgee'

form['fake_uid'] = '12345678'
form['uid'] = '12345678@starhubgee'
form['password'] = 'password'

page = mech.submit(form)  

page = mech.get("http://groupsms.starhubgee.com.sg/cgi-bin/s.pl")
form = page.form('compose')
form['to']='comma separated recipients numbers'

form['message']= "Your message...."

page = mech.submit(form)

This would take care of sending with much lower cost as it utilizes bundled SMSes in the package (400 - 1000 messages) with nothing else required. The problem, however, was still receiving. A little glimpse of hope was SMS+ introduced by SingTel last year, which allows you to auto forward your received messages to email address. This would have been ideal - if not for a small glitch - it only works with messages sent from SingTel numbers. OH MY GOOOOOD!!!!!!!! I almost cried when I found out that.

Lately I found a nice and simple Singaporean provider www.gsmexchange.com able to handle 2 way communication. Nothing to say about sending - just send a http request with username, password, phone numbers and message and you will receive message ID that you can use for tracking. Replies are slightly more complicated as I didn't find any API for reading replies - nevertheless - there is a list page that you can parse with mechanize to get the responses. There is an option to forward messages to email but for 5c per message it's a bit too expensive. Actually, also pricing for sending of the messages is on the higher end 10c per message which makes it much less attractive for our customers.

Luckily I managed to find a much cheaper solution - running my own SMS server :-) See the next post...

Friday, April 3, 2009

In the Cloud

On of the biggest pains in the ass with web apps is not really the development, changing technologies nor extremely demanding customers. It's actually hosting - seems like the easiest thing yet it's something that, most of the times, you have only a very little control over. Of course you still can put the server in the office but if you have multiple locations you need pretty good connection to support it. I've written countless times about problems with hosting in Singapore - from crazy prices to the total lack of support and overselling of capacities. With pretty much the only "reasonable" hosting provider here (Frro) going down I found myself in the middle of yet another server migration. It seems easy but installation of around 10 servers can be quite daunting :-).

Luckily (I so hope this luckily is going to last at least some time), there's a new kid in town - Rack Space Cloud. There's been so much buzz around the clouds over the past year or so (Amazon, Google, Microsoft), but none of them was really useable for hosting of rails applications. Rackspace Cloud offers Cloud Servers - they're quite similar to Slice Host's slices, however, come with some "cloud" extras. I don't really understand the technicalities of the processor computing power sharing on the cloud and as (I would assume) they don't have that many customers yet I cannot say how it's going to affect the performance.

What's great about the cloud servers, however, is the flexibility of other resources together with hourly billing. And the prices are currently more than reasonable. For the cheapest set up - 256 RAM and 10GB will cost you 1.5c per hour + the band with 8c per incoming GB and 22c per outgoing GB. Most of my apps will rarely hit transfer of more than around 5GB so my total costs works out to around $12 - 13 per month. The same set up on Slice Host is even $20. But it's really not just a price comparison. One of the main strengths of the cloud is flexibility to resize the servers as you need - increase the resources during peak hours/days and shrink it back off peak. This could be nicely automated with the API for Cloud Server that is being developed. Personally, I have mostly used this flexibility to test and compare different configurations - mod_rails vs. Lighttpd, Postgresql per instance vs. shared, various Postgresql settings and all this with different memory sizes. This experimentation actually led to some surprising (for me) results.

Another great feature is back ups. They allow you to create new servers from the back up images. What it means is that I have images with my typical configurations (mod_rails, Lighttpd, Tomcat + Apache, database server, etc.) and when I need to add a new server I just choose the image I need and within a few minutes the server is ready with everything I need, and all I have to do is add the new IP to hosts and configure the DNS. As we manage over 50 different applications, this comes extremely handy - not just for adding new ones but also to keep the existing ones in sync.

No matter how great the initial set up, things will always go wrong so I was quite curious about the support. Rackspace has been know for their great customer service ("Fanatical Support"). As the control panel is very new (about 2 weeks) there are still some glitches, however, I found the live chat always available and ready to help. For the "bugs" in the control panel - the support seems to be automatically notified and fixes everything within minutes. How I wish this was the case with Singaporean hosting where my server can be down for hours without any response from support. Enough bitching. Over the years I've grown to be very skeptical about hosting (and internet) providers. They usually start great but as soon as I prepay for several months everything goes down. Let's hope this one's going to be different :-)