Desperate times calls for desperate measures

These are desperate times and they surely call for some desperate measures. However when I look around I find people doing not-so-desperate things. The only mantra for cost cutting seems to be employee cutting! The truth is that most of the corporations are so bloated and top-heavy that they cannot think about anything else. It is like having a tummy so big that you are unable to see the feet, the truth!, the foundation.

Simple things, like Lehman brothers new york office not switching off external lights/display in months of acute crisis, strike me really really odd. I am not sure I do understand it correctly but isn't it very counter intuitive that when you are undergoing such a bad time you should spend thousands of dollars on some lousy display that nobody cares about ? Have corporations completely forgotten "saving" a primary and basic principle of survival ?

How can we save at least some money in these times ?

1. Start spending as less as possible on stationary. Ask employees to bring pens and diaries.
2. Keep that office printer starved of paper. Better still decommission those which are not really needed or are back up. Put per person quota on printers.
3. Do the same with phones. Ask employees to not use phones for personal purposes. Monitor the bills and slap it on people if it exceeds a certain limit.
4. Sublet office space if you can.
5. Sell that office car if it is not really used
6. Stops those team meets which require a long distance call or travel
7. Rather, stop meetings as much as possible
8. Start switching off everybody's computers at night.
9. Restrict internet usage to bare minimum
10. Stop that friday free lunch, ask people to pay up.
11. Make the coffee dispensing machine pay-on-use
12. Reduce the office cooling up to 25 deg C
13. Make the office gym pay-per-use
14. Switch off all the servers that you don't really need
15. Remove those tissue papers from cafeteria and bathroom. Ask people to carry handkerchiefs
16. Allow people to work from home
17. If possible move the office to a place where rent is lower (suburbs)
18. Completely stop those corporate gifts.
19. Try to move your call center to a web chat based model and save on phone and employee costs. (This may be a good idea in all times)
20. Motivate people to work more, put in more efforts by assuring them that no firing is going to take place.

More to follow as I observe more useless expenses happening around me.
In my view this is much better than firing people left, right and center.

Fastest and easiest way to book train tickets

After a 2 month long grind, we have managed to launch train bookings on Cleartrip. Under the hood we have implemented a really fast search and fare engine to accomplish the feat of serving search results in sub 100 milliseconds. This means one can search, check availability and book train tickets in less than one minute. This is a remarkable change over the long queues that everybody had to stand on reservation counters.

Business today

"When the productive need to ask permission from the unproductive in order to produce, then you may know that your culture is doomed."-Ayn Rand

Specially true for IT business :)

Use mongrel handler for slow requests

I work in a industry where almost everything is a API call to a remote and slow system usually written in java. Rails is not at all the way to implement such things. I think the easiest way to implement this from ruby world is to write a mongrel handler.

class SleepHandler < Mongrel::HttpHandler
  def process(request, response)
    response.start do |head,out|
      head["Content-Type"] = "text/html"
      sleep 300 # Long running task
      out.write("test") # Push out the result
    end
  end
end

#Handler configs
config = Mongrel::Configurator.new :host => ARGV[0], :port => ARGV[1] do
  listener(:num_processors => 80, :timeout => 400) do
    uri "/sleep", :handler => SleepHandler.new
    uri "/nosleep", :handler => NoSleepHandler.new
  end

  trap("INT") { stop }
  run
end

puts "Mongrel running on #{ARGV[0]}:#{ARGV[1]} with docroot #{ARGV[2]}"
config.join

I stressed this mongrel handler with 50 concurrent requests and everything came back in 5 min and 45 secs and maximum memory comsumption was 32MB so in my view decent performance by mongrel.

Why are my flights always so late ?

My morning flights are always very late and there is a lot of rush on the airports. It got me thinking. How many planes are actually in air at a given moment? I got hold of some data of airline schedules and tried to plot it on a map(Google). This is a snapshot of the same.


Red bubbles show approx. location of a flight in air. Red lines show the path.
It depicts Indian sky at 8 in the morning. Some 414 flights in Air!

Cleartrip trains now has more features

• Dateless search - Now you can search without specifying and date of travel and the result will be presented to you in a form of table which will show you a list of trains between the specified pair of stations/cities, days it runs on etc. One can further specify the date from "Modify your search " link and check for availability.

• Google maps of trains - We launched this feature quiet sometime back. However I still like to brag about it as I had to spend a lot of time into it. A few maps just look so beautiful. For instance

http://www.cleartrip.com/trains/map/68

• Route Images - After lot of hard work from Anurag we were finally able to launch a really cool feature : Route images like the ones we find on IR timetable. Inspired by London underground maps these look amazing

http://www.cleartrip.com/trains/show/1047

The best thing about these maps is that they highlight the important stations and that was achieved after a little voodoo :)

• Flight and hotel cross sell - Last but not the least...definitely not the least! Now we show you cheapest flight available (if any) for the date and sector you are searching for. We also suggest you a good hotel at the destination city. For example http://www.cleartrip.com/trains/results?from_city=Mumbai&to_city=New%20Delhi&class=1&date=26-4-2008
  

• Search by station code - You may now search by station code but I would discourage that as we have comprehensive look ups/autocomplete for this purpose. For instance http://www.cleartrip.com/trains/results?from_station=pnbe&to_station=ndls&class=1&date=26-04-2008

RDBMS vs Object/document DB

I am now sick and tired of people telling/writing blogs about how Object DB will rule the world and the RDBMS is crap. Well in my view both they are both right and wrong. It is true that Document DBs will rule the roost in years to come. This is primarily due to the fact that web applications are mostly normal CRUD (Create, Read, Update, Delete) applications where object/document/flat DBs work well. Due to sharp decline in storage media prices normalizing a datastore is not that useful/cost-saving anymore. Even if we have millions of records, keeping them in a denormalized form for easy fetching makes sense.
That said, this does not mean that RDBMS are things of past. They are in fact quiet important for a lot of things which either cannot be done on Object DBs or will be much slower than RDMSs. For instance, suppose you have to make a small data analytics system for a log file which has 2 million entires everyday which DB will you use ? (Hint: The one that allows you to normalize, join , index, sort and group data in any manner you like!)
I think all these RDBMS bashing crowd is
1. Have never used databases at all and read blogs and created a very strong opnion against RDBMSs
2. Have used databases but don't know jack shit about database design and query optimization
3. Are lispers and think all data is code or the other way round!

I did do this small experiment involving a log file having about 2 million entries. We pushed it into a lisp based object DB called acache and generated some statistics. We also pushed the same log file in a MySQL DB. Although data entry time was almost the same (for MySQl about 24-25mins and for acache about 30 mins) there was a massive difference when we tried the analytics part. Acache took around 2-3 minutes to generate only one statistics from the data. MySQL one generated 10 in 52 seconds!!!
Beat that!

Closed source hurts your bottomline twice

Oracle is decent database. It has good performance, rich features and is reasonably fast. However this doesn't mean, if you have spent $100K on licensing it and $20K on the machine it runs on, it will never fail. In fact I happen to be in a situation where I have seen Oracle crash and burn very regularly.
Being closed source and a paid application oracle hurts your bottomline more than once. Not only do you spend money buying the expensive license+machine but you end up paying more for maintenance and administration.

Lets compare the cost of a MySQL/Postgres database to an Oracle.

	Oracle	MySQL/Postgres
DBA cost	High ($30K/year)	Medium ($20K/year)
Machine/resource cost	High	Medium
License cost	High ($600K + $100K/year ~ approx)	Nil ($0k)
Downtime cost	High (Revenue)	High Revenue)

And after all this money spent when it crashes it hurts much more as the cost of the downtime is the same i.e. loss of revenue. It you much more as it has already eaten into your revenue. Speed wise there is hardly anything that can match MySQL. However it is not as reliable as Oracle/Postgres. But then if I maintain say a 3 node cluster of MySQL it is much more reliable and cost effective than Oracle.

Oracle has always been touted as the most feature rich database. However I don't think most of the websites out there use any of those advanced features.

Journey is MUCH more important

I always had this belief that journey is much more important than the destination. And now I have no doubt that it is true. There is really no freaking destination out there! It is all about the journey and that what really matters

Watch this talk by DHH. Nobody can explain it better than him
http://www.justin.tv/hackertv/97862/DHH_Talk__Startup_School_2008

Write your own Orkut in Rails - Part 1

So you want to start a new social networking portal? (well almost every third guy I meet is trying to do so!). Lets start writing one in Rails. It may not scale to the level of original orkut but it will surely be able to handle a couple of thousand odd users who login 4-5 times a day(you may be able to easily optimize to handle upto 10K probably on a decent server grade machine or two). Few things I can guarantee you that it will be a lot of fun (since it is in rails ) and will be faster than the original one (just a little bit).

First thing is to create a database table structure for it. I use mysql and i pretty good. Did you know that initially orkut was written on mysql and aspx ? or was it mssql...anyway it now runs on bigtable.

First thing to create is a person table like this:
CREATE TABLE `people` (
`id` int(11) NOT NULL auto_increment,
`emailid` varchar(35) NOT NULL default '',
`password` varchar(100) NOT NULL default '',
`display_name` varchar(100) NOT NULL default '',
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
PRIMARY KEY (`id`),
KEY `display_name` (`display_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=6 ;

This table will have data of people logging to your orkut like email id, password (in md5 hash) and display name.

Lets create friends table now
CREATE TABLE `friends` (
`person_id` int(11) NOT NULL,
`friend_id` int(11) NOT NULL,
KEY `person_id` (`person_id`,`friend_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
That was easy! Well I have not added a friend, good friend, best friend etc column. You add it later if you need it. For now lets keep things simple.

CREATE TABLE `scraps` (
`id` int(11) NOT NULL auto_increment,
`body` varchar(1024) NOT NULL default '',
`created_at` timestamp NOT NULL default CURRENT_TIMESTAMP,
`person_id` int(11) NOT NULL,
`owner_id` int(11) NOT NULL,
PRIMARY KEY (`id`),
KEY `owner_id` (`owner_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=3 ;

Thats pretty trivial and mundane..isn't it ?

Lets create the tricky one. Profiles

CREATE TABLE `profiles` (
`id` int(11) NOT NULL auto_increment,
`type` smallint(6) NOT NULL,
`body` text,
`person_id` int(11) default NULL,
`created_at` datetime default NULL,
`updated_at` datetime default NULL,
PRIMARY KEY (`id`),
KEY `person_id` (`person_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2 ;

Well this table only has a type and a body. Type will contain the information about the type of the profile(like personal, business etc) and body will contain the whole profile in YAML format.

Similarly create pictures
CREATE TABLE `pictures` (
`int` int(11) NOT NULL,
`file_name` varchar(255) NOT NULL default '',
`caption` varchar(255) default NULL,
`person_id` int(11) NOT NULL,
KEY `file_name` (`file_name`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

We will give a miss to communities and likes for right now(who needs them anyway!)

After database schema is ready lets create a rails application. To be honest we should have created the rails app first and then create this schema from generators. But I prefer to have more control on the schema.

Lets dip our hands into models now.

class Person < ActiveRecord::Base
  has_and_belongs_to_many :friends, :class_name => "Person", :join_table => "friends", :foreign_key => "person_id", :association_foreign_key => "friend_id"
  has_many :pictures
  has_many :profiles
  has_many :scraps
end

This will create the friends relationships for us. Every person can now have many friends in a many to many way.

Picture, scrap, profile etc all are similar

class Profile < ActiveRecord::Base
  belongs_to :person, :foreign_key => :person_id
end

Nothing to see here. Move on.


Now we open a console and create a few people here

p1=Person.create(:emailid => 'piyush.@foobar1.com', :password => 'piyush', :display_name => 'pr1')
p2=Person.create(:emailid => 'piyush@foobar2.com', :password => 'piyush', :display_name => 'pr2')

Now lets make them friends

p1.friends << p2
p2.friends << p1

Notice here I make two calls to friends from each person object because if p1 added p2 as friend the relationship is not immediately established as p2 has to acknowledge. Also p1 may add p1 as 'best friend' but p2 may add him/her is 'don't know this person'. For a friend relationship to established two way connection is a must. This may however be done by adding a few more columns to the friends table but I want to keep it lean.

This is it for now. Lets do the views and controllers of it in part 2

Faster and easier train search

at http://www.cleartrip.com/trains

and more features to come