Sunday, July 19, 2009

Ext JS Image Field

Yesterday I spent way too much time on another simple thing in EXT that I would have expected to work out of the box. What I needed was to upload a company logo to a profile and, of course, display that logo in the form. While it's very simple to display an image in a panel or box, it's a totally different thing to display the in image in the form and update it together with other form fields. What I needed is something like this:
and after form submit it should automatically load the new image:
For image upload I used the image upload plugin from EXT examples. To display the image I needed to extend the field component like this:
Ext.form.ImageField = Ext.extend(Ext.form.Field, {
    autoCreate: {tag: 'img'}
    ,setValue: function(new_value){
        if (new_value == undefined || new_value == null) {
            this.el.dom.src = '/images/no_image.png';
        } else {
            this.el.dom.src = '/images/thumbnail/' + new_value;
        }
    }
    ,initValue : function(){
          this.setValue(this.value);
    }

    ,initComponent: function() {
        Ext.apply(this, {

        });

        Ext.form.ImageField.superclass.initComponent.apply(this);
    }
});

Ext.reg('image_field', Ext.form.ImageField);

This is a bit simplified version as I removed the rails specific code, but anyway the main points are here:
line 2 creates image tag (rather than the default text field tag)
line 5 defaults the image src property to a placeholder (in case there's no image uploaded)
line 7 - in case there is a valid image it will change the src path to that image - in my case it's routed to Images controller and action thumbnail which returns a downscaled version of the image.
Other than this don't forget to set the FormPanel's fileUpload: true and refresh the form on submit success. I usually send the updated object JSON in the response to update action.:
  def update
    contact = Contact.find(params[:id])
    on_bind(contact)
    success = contact.save
    contact = Contact.find(contact.id) if success
    
    logger.info("saved")
    
    respond_to do |format|
      format.ext_json do
        json = contact.to_ext_json(:success => success,
                                   :methods => [:image_id],
                                   :includes => [:address],
                                   :messages => {:notice => l(:contact_updated), :warning => l(:contact_update_error)})
        render :json => json #:json => {:success => success}
        headers["Content-Type" ] = "text/html"
      end
    end
  end

!!!Notice line 16 - you have to manually set the content type to "text/html" because rails will default to "text/javascript" which will not work with file uploads!!! It works with fileUpload: false but it doesn't with fileUpload: true. I don't really know if Rails or EXT is right here but did take me some time to figure this out.
this will produce the following JSON (I removed the unnecessary fields):
{"success": true,
    "messages": {"notice": "Contact was successfully updated.", "warning": "contact_update_error"},
    "data": {"contact[image_id]": 15, "contact[active]": false, "contact[client_account_id]": 2, "contact[client_account_no]": "100",
        "contact[created_at]": "2009-07-07T10:31:54+08:00", "contact[credit_term_id]": 1}}
Now in the forms success method I can use the result.data to update the form:
success: function(form, action) {
                      form.setValues(action.result.data);
                  },

Friday, July 17, 2009

Losing Faith

Lately, I've been going through a lot of frustrations with Ruby (by ruby I don't meant just the language but the whole ecosystem and infrastructure based on Ruby). I am in a finishing phase on several projects. It's a time when the finer details get addressed. When there's no more time to do it later, differently or drop it all together. It's when everything that is supposed to work should actually work. The problem with Ruby is that it's been in this kind of finishing phase pretty much ever since I started using it (2004). If you look at it from far enough it seems like there is everything in it but when you come closer and actually try using it you'll find out that a lot of it is in less then pre-alpha stage (of course labeled as production). No language is an island unto itself. No matter how great the language you probably don't want to create all the basic libraries from scratch.

When I made a switch in 2004 Ruby was fairly new, Rails was fairly new so I expected this kind of problem, but you could feel the change in the air. Pretty much all the industry heavy weights were talking about it - Rails conf had a better key note line up than any other conference around, number of books was written and the whole Ruby ecosystem was created over almost overnight. Sad truth is that many of the heavy weights are still talking about it today but they're yet to do their first project in Ruby / Rails. Despite the explosion like surge in popularity little has changed in terms of actually usable Ruby libraries (actually I have the same feeling about 1.9 as well). This is a stark contrast to some of the uncelebrated languages out there. For example I find myself writing increasingly more and more code in Python. I don't think Python is the next Ruby, but I was quite surprised how easily it addressed things I struggled with for 5 years in Ruby.

An example could be excel libraries. I still use the same excel libraries I did 5 years ago (for export I switched recently to xlwt, for import because of the changes in roo I've been using JXL for years now switching to xlrd). The export is great only it doesn't support much formatting and breaks on spreadsheets over 10MB (actually somewhere over 6MB) and the files sometimes don't work on later versions of MS Office. There's been a rewrite that is able to go over 10MB but still without formatting I would need and with a license that doesn't allow commercial use. Python's xlwt - despite a very crude and unpolished appearance is able to produce a complete excel spreadsheet with all the formatting (numbers, dates) (including page setup) and all that in much shorter time and with less resources (Ruby 1.8 4m 28s, Ruby 1.9 2m 36s and Python 33s). I have quite a few more examples but I really don't mean this as a Ruby - Python comparison. This experience, however, got me thinking if Ruby is right for the job I've been trying to do (ie uncool enterprise systems). There are pretty much 2 things for me in Ruby (okay - 2.5 the 0.5 being RubyMine - I'll write about it in some other post). One of them is expressiveness of the language and the other is Rails. Ruby as a medium is incredibly expressive, reads almost like English and allows to do pretty much anything (iterating/mapping an array in 1 line, etc). The down side - as Robert C. Martin put it - is that it's really easy to make a mess. After several months of going through somebody's else code I couldn't agree more. A nice entanglement of block and yields and procs can challenge most of the PHP spagetti code any day. While this expressiveness is incredibly powerful, many times I opt for more verbose solution - Java programmer's code on Monday morning as Gregg Pollack would say - that I will be able to understand 2 years down the road - or that my junior developer will understand.

The other thing in Ruby for me is Rails. The only thing with Rails is that after the years it's becoming a bit too entangled, while it made a lot of sense initially to bundle everything together (ORM, routing, templating, etc.) initially now it starting to be a reason why Rails slowly falls behind in certain areas (either that or Rails team has simply decided to focus on Web 2.0 start up market and let others handle enterprise problems). As surprising as it may seems the world around has not been snoozing after Rails was created and many people took the best ideas (just like Rails did from J2EE world) replaced what didn't work and the whole industry moved forward. While few years back Ruby and Rails were pretty much necessities now there's a lot of alternatives and in many different language. To channel out my current frustrations I started experimenting with quite a number of frameworks and platforms. After so many years I really know what I want and how it fits with my existing projects and infrastructure. Very briefly -
  • I am looking for a modular framework where I can choose different ORM, different templating engine and so on where each component is developed independently by people who understand that specific area. As each of my project is very different sometimes I need the flexibility to change components. It was a great thing back in my Java times (Spring + Hibernate + Webwork). 
  • I really need an enterprise friendly dynamic language. Life's just too short to be wasting time trying to force a language to do things it's inventors don't consider important. 
  • I need a basic ecosystem of libraries (from XML, JSON, through excel, PDF, OO to image manipulation). 
  • I need a framework that supports TDD out of the box. 
  • I need a platform with (at least some) formal education and certification available. I need it because I employ people and certification really helped me back in Java times. To pass even the first Java certificate required people to study and to really understand the language and they simply took it as an independent measure of their skills. 
  • I need a language with a proper IDE. I use Vim every day - BUT not for programming. I've been using RubyMine for more than a month now and I feel like a blind man that got his eye sight back.
  • I need reasonable deployment option - both scalable and resource efficient. 
  • Increasingly more important - I don't want the language/platform community to offend others
It's almost uncanny how many items point back to Java :-). Actually, it's kind of sad that things I took for granted 5 years ago in Java are still not available in the most celebrated and most wildly endorsed language of today. If you look at it - nothing really substantial changed in the way we do the backend (MC) part of the systems. A lot has changed on the View side, but the models and controllers use the same principles as years back. I've used Python/Pylons quite a lot lately and it scores quite high on a number of areas (only so-so on the IDE, though). It seems much more mature compared to Ruby with more matured libraries supporting the enterprisey stuff (so ostracized in Rails community). On the other hand it's marketing is nowhere near Ruby's - you don't really hear people talking about wishing to work in Python. I am a bit worried about the performance - but that's something I just have to try on a real project. A big issue still remains the education and certification. I went through this with Ruby already - no matter how easy and great and superior the language you still need to spend some time learning and then even more time working in it before you really understand it. While there's quite a number of books on Python not much of training courses (at least not in Singapore) and there doesn't seem to be any certification around.

Monday, July 13, 2009

Extracting Data From MS Sql Server on MacOS

I've had really a rough time lately getting data from legacy systems based on MS Sql Server. In each case I got only the .bak backup file and had to do the rest. The whole thing is a bit easier if you run everything on windows, but as I did the development on Mac I'll cover how to connect to MS Sql from MacOS as well.

Restore database from backup

You will need a windows machine with MS Sql Server running (of course :-) Luckily, you can download it for free from here. Next connect to the database server using the command line utility osql (you will need to have it in your PATH)
cd C:\Program Files\Microsoft SQL Server\MSSQL\Binn
osql -E
You may need to create a new user:
use master
go
EXEC sp_addlogin 'peter', 'pass45'
go
Now let's have a look at the backup file (put it in some easy location so you don't have to type out the path):
Restore FILELISTONLY FROM DISK='c:\AMS.bak'
This query allows us to find out the logical name of the database and log file which is needed to appropriately restore a database to a new path. What you get is basically an original location of the data and the log file. In my case I got:
AMS_Data      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Data.MDF
AMS_Log      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Log.LDF
Which means that the original installation was on drive D:\. As my temp server is on C: I will have to recover with changing the locations of the files
RESTORE DATABASE ams
FROM DISK='c:\AMS.bak'
WITH
MOVE 'AMS_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams.mdf',
MOVE 'AMS_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams_log.ldf'
go
The only important thing here is that 'AMS_Data' and 'AMS_Log' names match the ones from the previous query. Now you should hopefully see that your database has been restored. Now, will just get access to the database:
use ams
go
EXEC sp_grantdbaccess 'peter'
go
sp_addrolemember db_owner,peter
go
There's more info on how to use osql here. Now we should be set to connect to the database from development machine. This was quite a number of steps just to restore the database - compared to createdb tmp; psql tmp <>Connecting to MS Sql from MacOS We will need to install and set up a couple of things:
sudo port install rb-dbi +dbd_odbc
sudo port install freetds
set up connection to database
/opt/local/etc/freetds/freetds.conf
Add your server to the end of the file:
[Ams]
 host = 192.168.11.106
 port = 1433
 tds version = 8.0
where the host is IP of your windows machine. After this step you should be already able to connect to MS Sql Server:
tsql -S Ams -U peter -P pass45
You should get the server database prompt (just like the osql on windows machine). You can try some commands like
use ams
go
select * from contacts
go
Now set up ODBC connection. Go to Applications -> Utils -> ODBC Administrator 1) add driver with following descriptions:
Description: TDS
Driver File: /opt/local/lib/libtdsobdc.so
Setup FIle: /opt/local/lib/libtdsobdc.so
Define as:   System
2) add User DNS DSN: Ams Desc: old AMS database server add 2 keyword/value placeholders. To update them click on it and press TAB. Set it to following values:
ServerName: Ams   (or whatever you set in freetds.conf)
Database: ams   (or whatever your database name)
now you should be able to test iODBC. Note that I am using sudo for this as it doesn't seem to work without sudo complaining Data source name not found and no default driver specified. Driver could not be loaded (0) SQLSTATE=IM002.
sudo iodbctest "dsn=Ams;uid=USERNAME;pwd=PASSWORD"
You should be now in interactive terminal again. Once all this works connecting from Ruby is really easy:
require 'rubygems'
require 'dbi'

DBI.connect('DBI:ODBC:ams', 'USERNAME', 'PASSWORD')

Rescuing the data

This is just a short ruby script I use to extract all the data from MS Sql and import it to Postgresql. It's not any functional database conversion - it's just to get the data out to something that's easier to work with and doesn't require windows machine running. It may have an issue with binary fields - it worked on most of them but it did choke on a couple of fields. DBI actually provides more metadata like NOT NULL attribute, primary key, etc. so the script could generate a more precise copy of the original database but this was enough for what I needed. You may run into unknown data type - especially if you try to import it to a different database engine - all you need to do is just update the method update_type to return correct mappings of the data types.
require 'rubygems'
require 'dbi'
require_library_or_gem 'pg'

def escape(text)
 return "NULL" if text.nil?

 text = PGconn.escape("#{text}")

 return "'#{text}'"
end

  def self.update_type(col)
    type = col["type_name"]
    type ||= 'timestamptz'
    type = type.downcase
    
    case type
    when 'clob'
      return 'varchar'
    when 'varbinary'
      return "oid"
    when 'long varbinary'
      return "oid"
    when 'double'
      return 'double precision'
    when 'tinyint'
      return 'smallint'
    when 'char'
      return "char(#{col['precision']})"
    else
      return type
    end
  end



 dbh = DBI.connect("DBI:ODBC:ams", "peter", "pass45")

 sth = dbh.prepare('select name from sysobjects where type = \'U\' ORDER BY NAME;')
 sth.execute
 tables = sth.collect{|row| row[0]}

 tables.each do |table|
   sth = dbh.prepare("select * from #{table} ")
   sth.execute

      create = "CREATE TABLE #{table}(\n"
      create << sth.column_info.collect{|col| "\"#{col['name'].downcase}\" #{update_type(col)}"}.join(",\n")
      create << ");\n\n"

      puts create

      sth.each do |row|
        create << "INSERT INTO #{table} (#{sth.column_info.collect{|column| "\"#{column['name'].downcase}\""}.join(', ')}) VALUES (#{sth.column_info.collect{|col| escape(row[col['name']])}.join(', ')});\n"
      end
      
      create << "\n\n"
      
      output = File.new("data_ams.sql", "ab+")
      output.puts create
      output.close
      
      #puts create
    end
To import the data to Postgresql is as simple as:
createdb ams_backup
psql ams_backup <> noise.txt

Friday, July 3, 2009

Software Development as a Profession?

A few years ago I used to be quite active in Rails community and then I stopped. I haven't been to a Singapore ruby/rails community meet up for 2 years and many of my friends ask me why - if I completely moved from Ruby to Python or simply what happened. Even though, I spend more and more time working with Python, I am still spend 13 - 14 hours a day doing Ruby programming. The actual reason has much more to do with Rails positioning. Back in April 2007, when the Singapore rails community was just starting to form and we had our first bigger meeting I wrote about it here. Most of it is now irrelevant but I was quite surprised when I read my 2nd point - purpose of the meetings - especially the part about the community divide to free hackers and enterprise developers. Back then I believed both were good and necessary for the community. I still believe it today only the situation has changed. While back then it was pretty much balanced, most of the community today consists of free hackers. After Dave Thomas moved from programming to music (okay - from rails to ruby 1.9) there hasn't been anybody to represent the enterprise interest and the rails community and rails image evolved to a cool web 2.0 free hacker paradise with little interest in enterprise.

This became very clear thanks to Bob Martin's key note on RailsConf '09 and very nice rebuttal by DHH.  Actually I've listened to that key note for several times now - Robert Martin is so expressive and (using the Smalltalk parallel) so diplomatic delivering something so critical to the community. For the past year and a half, I've been fighting with so many of the things he mentions - so many things I couldn't understand but make so much more sense now. 

It makes so much sense what he says about arrogance - I usually call it the 'cool' factor. Not doing the dirty job and only doing the clean things. Belief that our tools are somehow better, that our language is so good that we don't have to follow the rules, we don't have to do the regular (enterprise) things. It may be one of the reasons why Ruby still doesn't have a proper PDF or Excel handling libraries. The problem here is that this kind of arrogance is usually very subtle - but hits really hard and on places you would never expect. 

Rails is too easy to make a mess. This single statement pretty much sums up everything I said last week about software development being hard. Yes Rails is easy - in fact so easy that it makes you believe there is nothing you couldn't fix overnight. It's so easy that it makes you believe you don't need to know anything else. The problem is that all that ignorance will come back 3 months down the road (hopefully you'll be able to quit and let somebody else handle the mess).

Clean code. You look at the method it's pretty much what you'd expect. 

TDD. As a programmer I pretty much grew up on books and blogs of people like Martin Fowler, Robert Martin, Dave Thomas, Kent Beck and others from the group. TDD was the only way I was able to work. TDD is a No 1 book on my list of books for beginners. Yet I've been having the hardest time selling TDD - even to my own employees. They all know they HAVE to do tests - but only very few of them do it voluntarily or really understand why. TDD is simply not a culture around here (SG) - in fact, many find TDD culturally insensitive - at best a nice theory but totally useless in practice - just go through the Singapore ruby mailing list. Sensitive or not - it made it to our contracts for software developers.

Lastly, the discussion goes to software development as a profession and to what constitutes the professional portion of our job. Surprisingly enough, Robert Martin seems to think it's the discipline, rules and principles, it's the tough things we have to do and it's keeping them up when the pressure of a deadline mounts.  While there are people that are against being professional (see the DHH's rebuttal), for many people around me it makes sense. It seems so obvious that no body is really doing it.