One line window pagination in Ruby

def getPages(current_page, minimum=1, maximum=20, window=2)
return((minimum+window<current_page ? minimum.upto(window).collect : minimum.upto(current_page+window).collect) + (current_page-window > minimum+window ? [".."] : []) + (current_page>minimum+window ? (current_page-window > minimum+window ? current_page-window : minimum+window).upto(current_page+window > maximum ? maximum : current_page+window).collect : [])+(current_page+window+1<maximum-window ? [".."] : [])+(current_page<maximum-2*window ? maximum-window : current_page+window+1).upto(maximum).collect)
end

Output:
>> 1.upto(20){|x| print x; print " : "; p getPages(x,1,20,1)}

1 : [1, 2, "..", 19, 20]
2 : [1, 2, 3, "..", 19, 20]
3 : [1, 2, 3, 4, "..", 19, 20]
4 : [1, "..", 3, 4, 5, "..", 19, 20]
5 : [1, "..", 4, 5, 6, "..", 19, 20]
6 : [1, "..", 5, 6, 7, "..", 19, 20]
7 : [1, "..", 6, 7, 8, "..", 19, 20]
8 : [1, "..", 7, 8, 9, "..", 19, 20]
9 : [1, "..", 8, 9, 10, "..", 19, 20]
10 : [1, "..", 9, 10, 11, "..", 19, 20]
11 : [1, "..", 10, 11, 12, "..", 19, 20]
12 : [1, "..", 11, 12, 13, "..", 19, 20]
13 : [1, "..", 12, 13, 14, "..", 19, 20]
14 : [1, "..", 13, 14, 15, "..", 19, 20]
15 : [1, "..", 14, 15, 16, "..", 19, 20]
16 : [1, "..", 15, 16, 17, "..", 19, 20]
17 : [1, "..", 16, 17, 18, 19, 20]
18 : [1, "..", 17, 18, 19, 20]
19 : [1, "..", 18, 19, 20]
20 : [1, "..", 19, 20]

Dunno if I should be proud of this or worried!

Dump those fixtures

I hate testing/debugging a rails application on "unreal" data. At the same time I find writing fixtures by hand is a big pain. I wrote this rake task to dump all the data in fixtures folder by (rake test:fixtures:dump) on a production environment, bring it to development environment and do a rake test:fixtures:load

desc "Dump all the data from DB to fixtures in YML format"
namespace :db do
  namespace :fixtures do
    task :dump => :environment do
      require 'active_record/fixtures'
      ActiveRecord::Base.establish_connection(Rails.env)
      base_dir = ENV['FIXTURES_PATH'] ? File.join(Rails.root, ENV['FIXTURES_PATH']) : File.join(Rails.root, 'test', 'fixtures')
      fixtures_dir = ENV['FIXTURES_DIR'] ? File.join(base_dir, ENV['FIXTURES_DIR']) : base_dir

      Dir.entries(File.join(RAILS_ROOT, "app", "models")).each{|filename|
        if /\.rb$/.match(filename)
          model = Kernel.const_get(filename.gsub(".rb", "").camelcase)
          if model.ancestors.include?(ActiveRecord::Base) and model.table_exists?
            puts "Dumping #{model}"
            file = File.open(File.join("test", "fixtures", model.table_name+".yml"), "w")
            counter = 1
            model.find_in_batches(:batch_size => 5000){|rows|
              rows.each{|row|
                file.puts "#{counter}:"
                file.puts row.attributes.to_yaml.split("\n")[1..-1].collect{|x| "\s\s"+x}.join("\n")
                counter+=1
              }
            }
            file.close
            puts " -- Created a file with #{counter} rows"
          end
        end
      }
    end
  end
end

Shootout: Jruby1.3.1 vs Ruby1.8.7 vs Ruby1.9.2 vs Python2.6 vs Java6

Click on the image to view it clearly

Jruby is the fastest among Jruby, Ruby and Python. Benchmark program was a fractal geometry program posted here.

Language	Time for 100 iterations	times slower than java with -server
java –server	0.18	1
Ruby1.8	7.78	44.07
Ruby1.9.2	4.2	23.78
Jruby	2.5	14.16
Jruby1.3.1—sever	2.31	13.1
java	0.18	1.01
python	3.04	17.21

cricketr

A few weeks ago I decided to quit my job and do something on my own. An idea, to let cricket crazy fans do commentary about anything and everything related to cricket, has been lingering on in my head for very long. I have been thinking on and off about this idea for about an year now. It is finally launched and the URL is cricketr. As such the would cricketr is a typo for cricketer but the "r" actually signifies that cricketr is a web2.0 site.
The idea itself is not very unique. It is basically a twitter clone for cricket. There are, however, a few very subtle but a few important differences.

1. Unlike twitter, it does cater only one domain and hence domain ontology can be used to make sense of a commentary. For instance, if I type "Sachin just hit a massive six", it can deduce that sachin is a player who belongs to Mumbai Indians(IPL context). Not only that, if a match is going on, or has just finished/about to start it will also identify the match and tag this commentary to that match tag. This way it is possible to generate crowdsourced commentary which is extremely fast. On a demo run today I found this to be about 2 minutes faster than cricinfo score and commentary updates and about 1 minute faster than cricbuzz's score and commentary updates.

2. Personal interaction is important but not the goal. Idea is to let users talk about cricket and derive knowledge out of it.

It is still in a very fluid state(it is basically a product of 4 days of almost 24hr marathon hacking sessions) and a lot of things need to be decided upon. Some big features planned pretty soon. Hope people will like it and do a lot of commentary on it :)

Fingers crossed!

Mongrel as a stand alone server

I have been using rails for quiet sometime now. Rails is easy but I feel stupid using it. It is bulky and slow for some of my use cases. I wanted something lean. I wanted something more geeky and powerful to handle lots of long running requests(more than 100K/day with avg. 30sec processing time).
I have been playing with mongrel handlers for about an year now and they work like a charm. In this post I will give a sneak peek into how to write a highly scalable back-end for doing real stuff. Mongrel is a very fast server and can take heavy load. So here we go writing our own mongrel server for heavy processing

require 'rubygems'
require 'mongrel'
require 'mysql'

PORT=4444
class LogHandler < Mongrel::HttpHandler
  def initialize
    @@mysql=Mysql.connect("host", "username", "password", "databse")
  end

   def process(request, response)
     response.start(200) do |head, out|
       logs = @@mysql.query("select * from huge")
       # Do some heavy processing on this data
       sleep 10
       # done
       logs.each{|row|
         out.write(row)
       end
     end
   end
end

config = Mongrel::Configurator.new :host => "0.0.0.0", :port => PORT do
  daemonize(:cwd => Dir.pwd, :log_file => "server.log")
  listener(:num_processors => 150, :timeout => 300) do
    uri "/", :handler => LogHandler.new
  end
   trap("INT") { stop }
end

config.run.join

This piece of code registers a url "/" on the machine on port provided (in this case 4444) and serves a log huge file. Not very developer friendly is it ?


Well we can use ERB along with it and that will make things look a little easier.
Lets add some more code to class LogHandler for ERB stuff.

class LogHandler < Mongrel::HttpHandler
   def initialize
      @mutex = Mutex.new
   end
# This is for making instance variables of this class available at the template
  def get_binding
    binding
  end

# We change the process function to render a rhtml file called view.rhtml
  def process(request, response)
     response.start(200) do |head, out|
       head["Content-Type"] = "text/html"
       logs = @@mysql.query("select * from huge order by id DESC limit 20")
       sleep 20 # Some heavy processing on logs      
       rhtml = ERB.new(File.read("view.rhtml"))
       @mutex.synchronize{
          @logs = logs
          out.write rhtml.result(self.get_binding) 
      }
     end
   end

Mutex lock is important to implement here as mongrel reuses instance variables of this class for subsequent requests and may lead to a race condition.

View.rhtml looks something like this

<html><body><title>Log console</title>
<h1>Last 20 hits on our page</h3>
<table>
        <th>IP</th>
        <th>URL fetched</th>
        <th>Came from</th>
        <th>time taken</th>
     <%@logs.each{|td|%>
<tr>
        <td><%=row[0]%></td>
        <td><%=row[1]%></td>
        <td><%=row[2]%></td>
        <td><%=row[3]%></td>
</tr>
<%}%>
</table>
</div>

Not bad right ? I can now render rhtml as I 'd do from a rails application.

But hey what about the Routing, MVC stuff, activerecord, logging, form helpers, javascript helpers, view side helpers, callbacks, migrations etc etc ? Well well! This is NOT a full scale framework or a Rails substitute. If you want to do view-side-heavy things use Rails or Merb. If you want to do processing heavy jobs which results in simple-html/no-html then use this.

That said we can very easily sneak in a few of the Rails goodies.

1. Active record - That is easy. Just require 'active_record'; establish_connection; create models by doing this

def User < ActiveRecord::Base
end

2. Logging
Rails does logging in two parts. Request logging and response logging. You may add callbacks in the 'process' function to log a request and reponse at start and end of the function respectively.
Something like

def process
   requestLogger(request)
   # Do stuff
   responseLogger(response)
end

3. Routing
This one much more difficult/the most difficult to implement. However if you have only a few urls to match and most of them are not dynamic it is a easier to hard code them. However it is dirty to do so. To implement a light weight routing is not that difficult and not so dirty. We take this path.

First step to implement routing it to get all the parameters. Both post and get parameters. You may use something like this:

  def post_params(request)
    post_params = {}
    request.body.readlines.first.split('&').each{|x|
      k,v=x.split('=')
      post_params[k.to_sym] = CGI.unescape(v)
    }
    return post_params
  end

   def get_params(request)
    get_params = {}
    request.params["QUERY_STRING"].split("&").each{|x|
      k, v = x.split("=")
      get_params[k.to_sym] = CGI.unescape(v)
    }
    return get_params
  end

Actual implementation of routing is a little complex and is not easy to cover in one blog post. Also I may have done it wrong so I do not want to put it out there. I will cover this in detail when I am sure about it.

The more stuff you add to this thing the more it will start looking like Rails! IMHO it is not a bad idea to implement your own framework. I did implement a framework to run rails code as it is. But I never used it on my production servers. Apparently that is the why how it should be :)

Update 1: I have changed some code to highlight that mysql query is not the heavy call and processing is being on done elsewhere (sleep in this case). Default mysql libraries are not thread safe however one may use something like Neverblock

Desperate times calls for desperate measures

These are desperate times and they surely call for some desperate measures. However when I look around I find people doing not-so-desperate things. The only mantra for cost cutting seems to be employee cutting! The truth is that most of the corporations are so bloated and top-heavy that they cannot think about anything else. It is like having a tummy so big that you are unable to see the feet, the truth!, the foundation.

Simple things, like Lehman brothers new york office not switching off external lights/display in months of acute crisis, strike me really really odd. I am not sure I do understand it correctly but isn't it very counter intuitive that when you are undergoing such a bad time you should spend thousands of dollars on some lousy display that nobody cares about ? Have corporations completely forgotten "saving" a primary and basic principle of survival ?

How can we save at least some money in these times ?

1. Start spending as less as possible on stationary. Ask employees to bring pens and diaries.
2. Keep that office printer starved of paper. Better still decommission those which are not really needed or are back up. Put per person quota on printers.
3. Do the same with phones. Ask employees to not use phones for personal purposes. Monitor the bills and slap it on people if it exceeds a certain limit.
4. Sublet office space if you can.
5. Sell that office car if it is not really used
6. Stops those team meets which require a long distance call or travel
7. Rather, stop meetings as much as possible
8. Start switching off everybody's computers at night.
9. Restrict internet usage to bare minimum
10. Stop that friday free lunch, ask people to pay up.
11. Make the coffee dispensing machine pay-on-use
12. Reduce the office cooling up to 25 deg C
13. Make the office gym pay-per-use
14. Switch off all the servers that you don't really need
15. Remove those tissue papers from cafeteria and bathroom. Ask people to carry handkerchiefs
16. Allow people to work from home
17. If possible move the office to a place where rent is lower (suburbs)
18. Completely stop those corporate gifts.
19. Try to move your call center to a web chat based model and save on phone and employee costs. (This may be a good idea in all times)
20. Motivate people to work more, put in more efforts by assuring them that no firing is going to take place.

More to follow as I observe more useless expenses happening around me.
In my view this is much better than firing people left, right and center.

Fastest and easiest way to book train tickets

After a 2 month long grind, we have managed to launch train bookings on Cleartrip. Under the hood we have implemented a really fast search and fare engine to accomplish the feat of serving search results in sub 100 milliseconds. This means one can search, check availability and book train tickets in less than one minute. This is a remarkable change over the long queues that everybody had to stand on reservation counters.

Business today

"When the productive need to ask permission from the unproductive in order to produce, then you may know that your culture is doomed."-Ayn Rand

Specially true for IT business :)

Use mongrel handler for slow requests

I work in a industry where almost everything is a API call to a remote and slow system usually written in java. Rails is not at all the way to implement such things. I think the easiest way to implement this from ruby world is to write a mongrel handler.

class SleepHandler < Mongrel::HttpHandler
  def process(request, response)
    response.start do |head,out|
      head["Content-Type"] = "text/html"
      sleep 300 # Long running task
      out.write("test") # Push out the result
    end
  end
end

#Handler configs
config = Mongrel::Configurator.new :host => ARGV[0], :port => ARGV[1] do
  listener(:num_processors => 80, :timeout => 400) do
    uri "/sleep", :handler => SleepHandler.new
    uri "/nosleep", :handler => NoSleepHandler.new
  end

  trap("INT") { stop }
  run
end

puts "Mongrel running on #{ARGV[0]}:#{ARGV[1]} with docroot #{ARGV[2]}"
config.join

I stressed this mongrel handler with 50 concurrent requests and everything came back in 5 min and 45 secs and maximum memory comsumption was 32MB so in my view decent performance by mongrel.

Why are my flights always so late ?

My morning flights are always very late and there is a lot of rush on the airports. It got me thinking. How many planes are actually in air at a given moment? I got hold of some data of airline schedules and tried to plot it on a map(Google). This is a snapshot of the same.


Red bubbles show approx. location of a flight in air. Red lines show the path.
It depicts Indian sky at 8 in the morning. Some 414 flights in Air!