easyXDM made easy. Really!

Today’s world is connected. In the early 1950s, the degrees of separation was 6. Today, with all the Facebooking and Twittering it is just 3. Can you imagine? Just 3. Times have changed. Phew!

With this connectedness comes the additional technical challenge of talking to other web services to make yourself relevant. It isn’t straight forward to call another service as it includes lots of security considerations such as CSRF, failing which you can kiss you ass goodbye!

Enter easyXDM, a JS library that enables developers to easily work around the limitation set in place by the Same Origin Policy. easyXDM is really high-profile with customers such as Twitter and LinkedIn. When my work required me to talk to other web services, I jumped into easyXDM.

And that’s when the terror started. Documentation is bad. There are no easy-to-follow tutorials. Everything was as cryptic as easyXDM itself. But with lots of help from my team member, we were able to get it working like a charm. I know the pain that we faced and I don’t want you to go through the same.

In any easyXDM integration, there are always two components – Provider and Consumer. Provider provides the easyXDM services whereas Consumer consumes those services provided by the Provider.

So without any further delay, I will list down the steps required to integrate easyXDM into your application. Follow these steps and you will not have any issues.

1. easyXDM ships with name.html and easyxdm.swf. Upload these files to a CDN of your choice.

2. easyXDM also ships with index.html under the cors folder. This file HAS to be served from the Provider server if you need to work with cookies.

3. Either serve easyXDM.min.js and json2.js from CDN or include it inline.

4. Don’t know why, but set useAccessControl = false in index.html and save yourself a couple of days of headache.

5. Initialize the RPC object as shown –

     var REMOTE = "CDN_URL";
     function createRPCObject(providerUrl) {
          var rpc = new easyXDM.Rpc({
               local: REMOTE + "/name.html",
               swf: REMOTE + "/easyxdm.swf",
               remote: providerUrl + "/cors/index.html",
               remoteHelper: REMOTE + "/name.html"
          }, {
                  remote: {
                       request: {}
          return rpc;

6. Now that RPC object is initialized and ready, you can start making cross origin requests.

          url: "path_to_cross_origin_request",
    	  method: "GET or POST",
    	  data: "optional_data_to_be_sent"
     }, function(data) {
             // success callback
             var jsonData = $.parseJSON(data.data); // access your response data
        function (data) {
             // error callback

It’s that simple! Go ahead, give it a try.

Points to remember

1. It can be used for compatibility with older versions of browsers too. No special changes required.

2. In index.html, the following code is present –

     if (errorMessage) {
     else {
               data: req.responseText,
               status: req.status,
               headers: headers

success and error functions are the callbacks. As you can see, the response that you send is present in the data: req.responseText. And in the success callback of the rpc.request(), data contains all of the success parameters. To access your response, you have to call $.parseJSON(data.data) as shown earlier.

3. When the form method GET was used, I observed that cookies were not being sent in IE. I did not investigate it further but I think it is best to use POST.


1. http://easyxdm.net/

Posted in Technical | Tagged , , | Leave a comment

South Bollywood

You exactly know what this blog post is all about if you have read my tweet.

The growing trend in Bollywood is to “go down”. (pun intended but geographically ;-)) The stories are going down, the performances are going down and the stunts are going down. No really. Bollywood movies are becoming crap. There is absolutely no grace in the movies anymore. Well I do know that there is “poetic license” for the movies too and there should be but how far can it be stretched is something to ponder about. Case in point: Bhaag Milkha Bhaag.

Why did this trend start in the first place? “North” people were bored with the “same old same old”. They wanted something different. Not as different as that of Anurag Kashyap and Ram Gopal Verma because they couldn’t understand it. Something more masala-ish… that’s all they asked. And where can you find the true masala – South India.

South Indian movies were, I repeat were, predominantly love stories. Rich girl poor guy, Rich guy poor girl, Family opposition to their love and so on. But always with a twist. And that twist was “larger than life” picturizations. See this video to understand what I mean. South Indian heroes make Rambo seem like a kindergarten kid! Their movies are simple and commercial. And people love it. It’s a classic case of – “Patient also wanted the same thing. Doctor also prescribed the same thing”. (It packs more punch when I say it in Kannada but you get the drift)

For Bollywood, it was as simple as – There is a proven formula down South. Why not “take it and go”! AK, SK and the one with an ‘R’ in between “taked it and goed” to glorious heights. AK remade a South movie but I don’t remember which one (wink wink…). If I talk about SK, then I would immediately be put on the Wanted list and I would need a Bodyguard to protect me. The one with an ‘R’ glorifies stereotypes to a whole new level. Enna rascala, mind it! Needless to day, they all found their mojo and are happy with their 100 crores. And Bollywood is basking in the glory again being the richest film industry. Credit needs to be given to the South but I think South-stereotype jokes are running around 😦 “Biting the hand that feeds u” – not good macha.

The world knows that India is the answer to the “Life, Universe and Everything”. And within India, the answer is down South! I am a proud South Indian. And I am not the way you depict us.

We are like that only, ok va!

Posted in Personal | Leave a comment

Bhaag Milkha Bhaag – The dialogue, not the movie

The highly anticipated, much awaited Bhaag Milkha Bhaag had a grand opening. (pun intended – read the post completely, u will understand :-)) People thronged to the theaters during office hours to see the legend come alive. I too saw it first day first show at Cauvery Theater (technically first day first show but not technically technically, if you know what I mean!) The main drawing point was Farhan Akhtar’s 6 pack abs and his athletic look. Man, was I impressed!

After I saw the movie, I did some reading on Wikipedia and really hats-off to the great man! Everyone does mistakes. To err is human.. remember. Lets leave it at that. He has moved on and so should you, rather than discussing about his ‘turning back leading to his failure’. (On a side note, I saw a video which shows that he was not in the fray from the start.)

Another point that caught my attention was – his affair with a foreign girl. I could not find any mention of any affair of his on the internet. Is it so secretive that it’s not on the internet or is it that he told only the director about it or is it just bollywoodized? Can the director show “Inspired by a true story” at the start and take liberty at will? I don’t know.

This blog post is related to his affair.

But before that, a small context check – Cauvery theater is a local theater. I usually go to malls to watch movies (sigh… first world problems). This time, I wanted to watch it with the ‘not-so-sophisticated’ crowd. Watching the movie in Cauvery… man, had I missed something for a long time. The environment was awesome, lively, energetic & lovely. People shouting, whistling, screaming… phew.. it was good. That’s when I got into the mood! I waited for the right moment to join in with the crowd. And that right moment was when he goes into the room with that foreign girl. And that’s when everyone knows there will be a kissing scene. And that’s when the theater became silent. And that’s when he started kissing the foreign girl. And that’s when I shouted “Bhaag Milkha Bhaag”. And that’s when the theater came to whistles again!

And that’s when I realized I am local too! And that’s when you realize how yours truly rolls!

Posted in Personal | Leave a comment

Play Framework – A simple cookbook

Play Framework is creating a lot of hype but it is still way behind that generated by Bitcoin. Just saying.

Anyways, let’s get to the point without beating around the bush.

The documentation of Play sucks big (no pun intended :-)). The differences between Play 1.x and 2.x are so profound that it requires a complete code change. This can be a huge thumbs-down in deciding to use Play in the first place. But, you know, LinkedIn is using it. So there must be something about Play. Innit?

Bearing this in mind, I decided to go ahead with Play come what may. There were lots of places where I got stuck for days with no help whatsoever except from yours truly :-). This (running) blog post aims to make the lives of some developers easy by giving detailed instructions on some of the issues I faced and how to overcome it.

Async Programming

Play is built on Akka which bills itself as “… toolkit and runtime for building highly concurrent, distributed, and fault tolerant event-driven applications on the JVM”. Hence, async must come natively. It does. But it isn’t very easy to understand how to implement it.

Any initial documentation search tells us to use –

F.Promise<WS.HttpResponse> remoteCall = WS.url("http://www.pristinecrap.com").getAsync();

This works perfectly as long as we use URLs directly. What if we use third party libraries? This is where the documentation tends to get murky.

Do not worry. The below code snippet worked for me –

import play.libs.F.Promise;
import play.libs.Akka;
import play.mvc.Result;
import play.mvc.Controller;

public class ApplicationController extends Controller {
     public static Result asyncProcess() {
          Promise<BusinessObject> businessObject = Akka.future(new Callable() {
               public BusinessObject call() {
                    return newBusinessObject(originRequestParameters);
          return async(
                    new Function () {
                         public Result apply(BusinessObject businessObject) {
                              return (ok(page.render()));

I am yet to figure out how to implement async operations outside of the Controller 😦

Jobs & Global Settings

Play 1.x has support for Jobs. It was very easy to understand the concepts with annotations such as @OnApplicationStart for running bootstrap jobs, @Every and @On for running scheduled and cron jobs.

Play 2.x completely removed this concept of Jobs. Why? Don’t ask me. Nevertheless, I spent a few hours to figure out how to simulate a job. The @OnApplicationStart functionality is obtained by extending GlobalSettings and writing your logic inside onStart(Application app) function.

import play.GlobalSettings;

public class ApplicationGlobal extends GlobalSettings {
     public void onStart(Application app) {


Ok you explained how to implement a bootstrap job in Play 2.x. Nice! But how do we implement a scheduled job? This question really frustated me as none of the available documentation worked.

The first implementation that you will come across is for Play 2.0 –

import play.libs.Akka;
import akka.util.Duration;

     Duration.create(0, TimeUnit.MILLISECONDS),
     Duration.create(30, TimeUnit.MINUTES)

I am not sure if the above code snippet works in Play 2.0. But in Play 2.1, it doesn’t compile. The package akka.util.Duration doesn’t even exist and it is not a one-off case. Why have they made so many changes in minor version upgrades too? Blows my mind! I spent one day just to find out how to implement a scheduled job in Play 2.1. You don’t have to go through the same mental torture –

import play.libs.Akka;
import scala.concurrent.duration.FiniteDuration;
import java.util.concurrent.TimeUnit;

Akka.system().scheduler().schedule(new FiniteDuration(0, TimeUnit.SECONDS), new FiniteDuration(1, TimeUnit.MINUTES), new MyCustomJob(), Akka.system().dispatcher());

The above job runs every 1 minute and executes MyCustomJob() which is a class that implements Runnable.

To run a scheduled job just once at a pre-determined time, use the following –

import play.libs.Akka;
import scala.concurrent.duration.FiniteDuration;
import java.util.concurrent.TimeUnit;

Akka.system().scheduler().scheduleOnce(new FiniteDuration(1, TimeUnit.MINUTES), new MyCustomJob(), Akka.system().dispatcher());

As you can see, these are just some of the issues that I have faced and their solutions. Just pray that I run into more issues 🙂

Posted in Technical | Tagged , , , , , , , | Leave a comment

Play Framework – Good or Bad?

The latest buzz word doing the rounds is Play Framework. It is deployed at high-profile companies such as LinkedIn. Chances are that you would not have heard of Play or Akka (because I hadn’t :-))

When we started work on a new product, we were asked to evaluate Play and its feasibility for our requirements. I, being over enthusiastic, took up this task and, lo & behold, what did I find out?

But before the specifics, let’s take a look at the test environment – 1 m1.large Amazon Linux EC2 instance (AppServer), 1 m1.large Amazon Linux EC2 instance (RabbitMQ Server), 1 m1.large JMeter Master Windows EC2 instance and 1 m1.large JMeter Slave Amazon Linux EC2 instance, all in the same AZ. The test environment also contains 2 AWS DynamoDB tables with an appropriately provisioned throughput. The load for the experiments mentioned below was 500 threads running in parallel with each thread looping 100 times.

Now for the details for the various test cases –

1. A sample Play project
AppServer was able to handle 4000+ requests per second at a max of 40% CPU utilization. Initial impression – Not bad!

2. A sample Play project + 1 RabbitMQ call
AppServer was able to handle 500+ requests per second. Impression – Dramatic reduction in concurrency. Is RabbitMQ Server to blame?

3. A sample Play project + 1 AWS DynamoDB call (Callback Async)
AppServer was able to handle 300 requests per second at 100% CPU utilization.

4. A sample Play project + 2 AWS DynamoDB calls (Futures Async – 100 milliseconds sleep per thread)
AppServer was able to handle 50 requests per second at 60% CPU utilization.

5. A sample Play project + 2 AWS DynamoDB calls (Futures Async – 10 milliseconds sleep per thread)
AppServer was able to handle 75 requests per second at 100% CPU utilization.

6. A sample Play project + 2 AWS DynamoDB calls (Sync) + 1 RabbitMQ call
AppServer was able to handle 90+ requests per second at 100% CPU utilization.

Test case 6 gives a throughput of 500+ requests per second on a c1.xlarge EC2 instance.

Why is Play not living up to its hype? Is m1.large not powerful enough? (Earlier experiments suggest that m1.large is a pretty mean machine!) Are my coding skills bad? 😦 Is Scala better than Java when it comes to performance? Am I missing something very trivial in Play? Lots of questions remain. But for the time being, we are going ahead with Play and using both scale-up and scale-out to meet our requirements.

Has anyone played with Play? Was it your ideal play ground? What were your experiences?

Posted in Technical | Tagged , , , , , , | Leave a comment

To all my ex-crushes, ex-GFs and one not-so-special-anymore…

I write this post from the bottom of my heart.

I have had my share of crushes and GFs over the years. It is natural to love someone and there is no reason to hide it. But at the same time, you have to understand that there is a time and a place for the world to know about it. For me, the time and the place is now!

Like every student, I had a crush on my teacher. And another teacher. The best part is that both of their names start with S. So let’s call them Mrs. S from now on. (Fortunately, it leads to confusion to my beloved readers :-))

Mrs. S used to come in my dreams and I don’t know why I still remember that dream! It seems like it happened yesterday but the dream is still so fresh in my mind. I don’t know where she is and how she is.

Mrs. S was very fond of me. I was usually in the top 5. But somehow she liked me more than the student who used to get first rank. In fourth standard, I got second rank in the finals. I had gone to collect my report card with my parents and she kissed me on my left cheek. Yippie! I know where she lives and she is doing fine.

No more one-way love stories from school days apart from a couple of F.L.A.M.E.S! #youremember

We will now proceed to the most happening phase in any student life – College. Just to add more confusion to my loved readers, I shall refer to College to mean PUC and B.E.

In college for the first time in my life, I had 3 one-way GFs. But here’s the twist: the one-way was not from my side! My friend suggested to keep nicknames – Pink, White & Diagonal. I could understand Pink and White. But what’s with Diagonal? My friend explained that as mine was a rectangular love story, Diagonal was the farthest from me. Truer words were never explained in such a simple way!

I have already shared with you this incident – Why I Love YouTube!

All of these were mostly puppy love. Once I finished my golden phase of life, I had one true GF. One serious GF. It all started when I was in my early second decade of my life. (sigh… I am in third decade now!) She (Thank God… it was a she… yaaay!) was this gorgeous, sexy-looking girl. It started as an infatuation, remained as an infatuation for a long time and then some. Slowly it graduated from that and I mustered courage again and asked her out for which she readily accepted. The love phase was awesome and like any other BF-GF, we roamed around Bangalore trying our best not to meet people we knew. I have to give credit to myself… I didn’t meet anyone I knew when I was with her. Pat on the back for yours truly! But all this was short-lived and then started the on-off-on-off thingy. It was depressing, really. It was the saddest phase in my life and I consider myself lucky as I didn’t end up in NIMHANS. There were just 2 memorable moments with her – both happened to be on the same day. (wink wink :-)) I don’t where in which part of the world she is now and I don’t intend to know too.

To all my ex-crushes, ex-GFs and one not-so-special-anymore, I really thank you for breaking up with me. My life is perfect without you!

Happy ValENDtines Day!

Posted in Personal | Tagged , , | Leave a comment

Solr – Pre & Post 4.0

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world’s largest internet sites.

The major feature introductions in Solr 4.0 are SolrCloud, Near Real Time Search, Atomic Updates and Optimistic Concurrency. Before getting into the details, first let’s understand how distributed indexing & querying and master-slave index replication worked in Solr 3.6 or earlier.

Solr in AWS: Shards & Distributed Search provides a detailed explanation of what a shard is, initializing shards, indexing documents into shards and performing a distributed search. Learnings from this article are as follows –

  • Solr doesn’t have any logic for distributing indexed data over shards. It is up to you to ensure that the documents are evenly distributed across shards.
  • Performing a distributed search requires you to have knowledge of all the existing shards so that the entire data set can be queried.
  • Both distributed indexing and querying is NOT a transparent process
  • What happens when one of the shards goes down? Is the data incomplete now?

Distributed indexing & searching is a very significant feature, no doubt. But can it be handled better and in a much more transparent way?

Solr in AWS: Master Slave Replication & Mitigation Strategies offers a detailed explanation of what an index replication means, setting it up in master-slave mode and mitigation steps needed when the master fails. Learnings from this article are as follows –

  • Master is the single point of failure (SPOF). Once the master is down, no more writes to the index are possible.
  • Master has no clue of the number of slaves that are present.
  • Absence of a centralized coordination service for maintaining configuration information and providing distributed synchronization leading to inevitable race conditions.
  • Replication is done pull-style which leads to index inconsistencies due to replication lag
  • Nomination of one of the slaves as the new master requires implementation of custom scripts and / or deployment of NFS.
  • What happens when the master goes down and you do not receive any notification?

It is obvious that setting up distributed indexing & searching and master-slave replication is a complex process and requires the presence of a Solr expert.

Do not fret! Enter Solr 4.0 and its state-of-the-art capabilities.


SolrCloud is the name of a set of new distributed capabilities in Solr. Passing parameters to enable these capabilities will enable you to set up a highly available, fault tolerant cluster of Solr servers. Use SolrCloud when you want high scale, fault tolerant, distributed indexing and search capabilities.

Truth be told – Setting up a sharded, replicated, highly available, fault tolerant SolrCloud cluster takes less than 15 minutes! Ok, enough of the theatrics. Let’s get our hands dirty.

A SolrCloud cluster consists of one or more shards, each having one or more replicas and coordinated by ZooKeeper to facilitate high availability and fault tolerance.


Figure 1: 2 Shard 2 Replica Architecture


ZooKeeper is used as a repository for cluster configuration and coordination – think of it as a distributed file system that contains information about all of the Solr servers. A single ZooKeeper service can handle the cluster but then it becomes SPOF. In order to avoid such a scenario, it is recommended that a ZooKeeper Ensemble, i.e. running multiple ZooKeeper servers in concert, is in action. Every ZooKeeper server needs to know about every other ZooKeeper server in the ensemble, and a majority of servers (called a Quorum) are needed to provide service. For example, a ZooKeeper ensemble of 3 servers allows anyone to fail with the remaining 2 constituting a majority to continue providing service. 5 ZooKeeper servers are needed to allow for the failure of up to 2 servers at a time.

Installing & running ZooKeeper Ensemble

cd /opt
wget http://apache.techartifact.com/mirror/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz
tar -xzvf zookeeper-3.4.5.tar.gz
cd zookeeper-3.4.5
cp conf/zoo_sample.cfg conf/zoo.cfg
vim conf/zoo.cfg
cd /opt/zookeeper/data
vim myid
	1 (or 2 or 3… depending on the server)
/opt/zookeeper-3.4.5/bin/zkServer.sh start
/opt/zookeeper-3.4.5/bin/zkServer.sh stop

Follow the above steps in all the ZooKeeper servers. Refer Clustered (Multi-Server) Setup and Configuration Parameters for understanding quorum_port, leader_election_port and the file myid. We now have a 3-node ZooKeeper Ensemble running!

Installing & running Solr 4.0

cd /opt
wget http://apache.techartifact.com/mirror/lucene/solr/4.0.0/apache-solr-4.0.0.tgz
tar -xzvf apache-solr-4.0.0.tgz 
rm -f apache-solr-4.0.0.tgz
cd /opt/apache-solr-4.0.0/example/
java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -DnumShards=2 -DzkHost=<server1_ip>:<client_port>,<server2_ip>:<client_port>,<server3_ip>:<client_port> -jar start.jar
java -DzkHost= DzkHost=<server1_ip>:<client_port>,<server2_ip>:<client_port>,<server3_ip>:<client_port> -jar start.jar

A couple of points worth mentioning –

  • -DnumShards: the number of shards that will be present. Note that once set, this number cannot be increased or decreased without re-indexing the entire data set. (Dynamically changing the number of shards is part of the Solr roadmap!)
  • -DzkHost: a comma-separated list of ZooKeeper servers.
  • -Dbootstrap_confdir, -Dcollection.configName: these parameters are specified only when starting up the first Solr instance. This will enable the transfer of configuration files to ZooKeeper. Subsequent Solr instances need to just point to the ZooKeeper ensemble.

The above command with –DnumShards=2 specifies that it is a 2-shard cluster. The first Solr server automatically becomes shard1 and the second Solr server automatically becomes shard2. What happens when we run a third & fourth Solr instances? Since it’s a 2-shard cluster, the third Solr server automatically becomes a replica of shard1 and the fourth Solr server becomes a replica of shard2. We now have 4-node (2-shard 2-replica) Solr cluster running!

An important point to remember: The first Solr server that joins shard1 is assigned the role of a Leader of the shard1 sub-cluster and subsequent servers that join shard1 are the replicas. All writes happen on the Leader and the same are pushed to the replicas. If the Leader goes down, automatic fail over kicks in and a replica will automatically be promoted to the role of the Leader.

Indexing documents into the cluster

Unlike pre Solr 4.0, the entire indexing process is transparent. Any document write operation can be sent to any shard leader / replica and the indexing module ensures that the document is sent to the correct shard.

The index process is as follows –

  • If an index document request is sent to a shard replica, the same is forwarded to the leader of that shard.
  • If an index document request is sent to a leader of a shard, then the leader first checks if the document belongs to the same shard using MurmurHash3 algorithm.
    • If yes, then the document is written to its transaction log.
    • If no, then the document is forwarded to the leader of the correct shard which in turn writes to its transaction log.

It then transfers the delta of the transaction log to all its replicas. The replicas in turn  update their respective transaction logs and send an acknowledgement to the leader.  Once the leader receives acknowledgements that all the slaves have received the updates, the original index request is responded to.

Performing a distributed search

Unlike pre Solr 4.0, the entire distributed search is completely transparent. You need not know the number of shards nor the number of replicas. All this information is maintained in ZooKeeper.

When a query request is sent to one of the instances, either a leader or a replica, Solr load balances the requests across replicas, using different servers to satisfy each request. SolrCloud can continue to serve results without interruption as long as at least one server hosts every shard. To return just the documents that are available in the shards that are still alive (and avoid a 503 error), add the following query parameter: shards.tolerant=true.

Soft Commit

A hard commit calls fsync on the index files to ensure that they have been flushed to stable storage and no data loss will result from a power failure. It is an expensive operation as it involves disk IO.

A soft commit, introduced in Solr 4.0 to handle NRT search, is much faster as it only makes the index change visible to the searchers but does not call fsync. This is accomplished by writing the changes to the transaction log. Think of it as adding documents to an in-memory writeable segment. As the size of the transaction log increases over time, it is imperative that these are written to disk using hard commit periodically.

Near Real Time Search

The /get NRT handler ensures that the latest version of a document is retrieved. It is NOT designed to be used as full-text search. Instead, its responsibility is the guaranteed return of the latest version of a particular document.

When a /get request is raised by providing an id or ids (a set of id), the search process is as follows –

  • It checks if the document is present in the transaction log
    • If yes, then the document is returned from the transaction log
    • If no, then the latest opened searcher is used to retrieve the document

Atomic Updates

Atomic Updates is a new feature in Solr 4.0 that allows you to update on a field level rather than on a document level (pre Solr 4.0). This means that you can update individual fields without having to send the entire document to Solr with the update fields’ values. Internally Solr re-adds the document to the index with the updated fields. Please note that fields in your schema.xml must be stored=true to enable atomic updates.

Optimistic Concurrency

Optimistic concurrency control is a Solr NoSQL feature that allows a conditional update based on the document _version_. Using optimistic concurrency normally involves a READ-MODIFY-WRITE process. Solr automatically adds a _version_ field to all documents. A client can specify a value for _version_ on any update operation to invoke optimistic concurrency control.

  • If the value of _version_ matches what currently exists in Solr, then the update operation will be successful and a new _version_ value will be set to that document.
  • If the _version_ does not match what currently exists in Solr, then a HTTP error with code 409 (Conflict) will be returned.


  1. How are the replicas assigned to shards?
    1. When a new Solr server is added, it is assigned to the shard with the fewest replicas, tie-breaking on the lowest shard number.
  2. Why should I specify all the ZooKeepers for the parameter –DzkHost?
    1. TCP connections are established between the Solr instance and the ZooKeeper servers. When a ZooKeeper server goes down, the corresponding TCP connection is lost. However, other existing TCP connections are still functional and hence this ensures fault tolerance of the SolrCloud cluster even when one or more ZooKeeper servers are down.
  3. What is a transaction log?
    1. It is an append-only log of write operations maintained by each node. It records all write operations performed on an index between two commits. Anytime the indexing process is interrupted, any uncommitted updates can be replayed from the transaction log.
  4. When does the old-style replication kick in?
    1. When a Solr machine is added to the cluster as a replica, it needs to get itself synchronized with the concerned shard. If more than 100 updates are present, then an old-style master-slave replication kicks off. Otherwise, transaction log is replayed to synchronize the replica.
  5. How is load balancing performed on the client side?
    1. Solr client uses LBHttpSolrServer. It is a dumb round-robin implementation. Please note that this should NOT be used for indexing.
  6. What will happen if the entire ZooKeeper ensemble goes down or quorum is not maintained?
    1. ZooKeeper periodically sends the current cluster configuration information to all the SolrCloud instances. When a search request needs to be performed, the Solr instance reads the current cluster information from its local cache and executes the query. Hence, search requests need not have the ZooKeeper ensemble running. Please bear in mind that any new instances that are added to the cluster will not be visible to the other instances.
    2. However, a write index request is a bit more complicated. An index write operation results in a new Lucene segment getting added or existing Lucene segments getting merged. This information has to be sent to ZooKeeper. Each Solr server must report to ZooKeeper which cores it has installed. Each host file is of the form host_version. It is the responsibility of each Solr host/server to match the state of the cores_N file. Meaning, each Solr server must install the cores defined for it and after successful install, write the hosts file out to ZooKeeper. Hence, an index write operation always needs ZooKeeper ensemble to be running.
  7. Can the ZooKeeper cluster be dynamically changed?
    1. ZooKeeper cluster is not easily changed dynamically but is part of their roadmap. A workaround is to do a Rolling Restart.
  8. Is there a way to find out which Solr instance is a leader and which one is a replica?
    1. Solr 4.0 Admin console shows these roles in a nice graphical manner.
  9. How much time does it take to add a new replica to a shard?
    1. For a shard leader index size of 500MB, an m1.medium EC2 cold-start replica instance takes about 30 seconds to synchronize its index with the leader and become part of the cluster.
  10. Is it possible to ensure that the SolrCloud cluster is HA even when an entire AZ goes out of action?
    1. When an entire AZ is down and the cluster is expected to be running successfully, the simplest and recommended approach is to have all leaders in one AZ and replicas in other AZs with the ZooKeeper ensemble spanning across AZs.


  1. http://wiki.apache.org/solr/SolrCloud
  2. http://zookeeper.apache.org/
  3. http://wiki.apache.org/solr/SolrReplication
  4. https://pristinecrap.com/2013/01/12/solr-in-aws-shards-distributed-search/
  5. http://wiki.apache.org/solr/UpdateXmlMessages
  6. http://wiki.apache.org/solr/Atomic_Updates
  7. http://yonik.com/solr/realtime-get/
  8. http://yonik.com/solr/optimistic-concurrency/
Posted in Technical | Tagged , , , , , , , , | 5 Comments

Solr in AWS: Master Slave Replication & Mitigation Strategies

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, near real-time indexing, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world’s largest internet sites.

A single Solr server can handle only a certain number of requests without affecting performance. In such, not so uncommon, scenarios it is best to set up a Solr master-slave cluster so that the load can be balanced effectively. Master usually takes up the task of index updates whereas the slaves’ responsibilities are to poll the master for updates and handle the ever-increasing search requests.

This article explains the index replication that works over HTTP and how to set it up using Solr 3.6. Let’s get started!

Index Replication

A master-slave replication includes both index replication and (an optional) configuration files replication. Index replication, as the phrase indicates, is the replication of Lucene index from the master to the slaves. The slaves poll the master for any updates and the master sends a delta of the index so that everyone can be in sync.

Setting up master-slave replication

Open the file solrconfig.xml and add the following –

<requestHandler name="/replication" class="solr.ReplicationHandler">
   <lst name="master">
      <!--Replicate on 'startup' and 'commit'. 'optimize' is also a valid value for replicateAfter. -->
      <str name="enable">${enable.master:false}</str>
      <str name="replicateAfter">startup</str>
      <str name="replicateAfter">commit</str>
      <str name="commitReserveDuration">00:00:10</str>
   <lst name="slave">
      <str name="enable">${enable.slave:false}</str>
      <str name="masterUrl">http://master_server_ip:solr_port/solr/replication</str>
      <str name="pollInterval">00:00:20</str>

Note that ${enable.master:false} and ${enable.slave:false} are false indicating that currently this machine is neither set up as a master nor a slave. These settings HAVE to be overridden by specifying the values in the file solrcore.properties which is located under the conf directory of each core’s instance directory.

On the master server, open the file solrcore.properties and add the following –


On the slave server, open the file solrcore.properties and add the following –


Fire up these machines and you have a master-slave Solr cluster ready!


A master may be able to serve only so many slaves without adversely affecting performance. Some organizations have deployed slave servers across multiple data centers. If each slave downloads the index from a remote data center, the resulting download may consume too much network bandwidth. To avoid performance degradation in cases like this, you can configure one or more slaves as repeaters. A repeater is simply a node that acts as both a master and a slave. (enable.master=true

Note: Be sure to have replicateAfter ‘commit’ setup on repeater even if replicateAfter is set to optimize on the main master. This is because on a repeater (or any slave), only a commit is called after index is downloaded. Optimize is never called on slaves.

As with our lives, nothing is certain in the lives of machines too! Any machine can go down at any time and there is nothing we can do about it except to plan for such inevitable cases and have a mitigation strategy in place.

Mitigation Strategies when master is down

Since master-slave replication is done pull-style, there are always inconsistencies with the indices of the master and the slaves. When some loss of updates are acceptable –

Mitigation Plan 1: Every machine is either a master or a slave and not BOTH

1. Nominate one of the slaves as master
2. Stop the Solr server on the new master
3. Change the solrcore.properties to promote it as master.
4. Start the Solr server of new master
5. Detach the EIP from the failed master and associate with the new nominated master.
6. That’s it!

Mitigation Plan 2: Every machine is both master and slave (Concept of Repeater)

1. Nominate one of the instances as master
2. Detach the EIP from the failed master and associate with the new nominated master.
3. That’s it!

In each of the mitigation plans, the first step is to nominate a slave. The obvious question arises – How do we decide which slave is the best-fit?

We have to choose that slave whose index is closest to master. To carry out this operation, use the LukeRequestHandler (enabled by default) and query the version parameter. This parameter shows the timestamp in milliseconds of the last index operation. Pick the slave which satisfies the following conditions –

1. Retrieve the version attribute on the master from S3. (Aside: Since the master is down currently, there is no way you can get the version of the master. Hence, you have to query and store the master version in S3 periodically when the master was running!)
2. Query the version on all solr slaves.
3. Among the slaves, pick the slave that has the highest version. That is the best nomination.
4. As a double check, check that the nominated slave’s version is closest or equal to that of the master (Replicating a master index means copying index as-is from the master to slaves. That’s why lastModified and version are the same on a slave once replication is successful. This is the reason why slave version can never be greater than that of the master)

However in production environments, any loss of updates is not acceptable and hence more robust mitigation plans need to be in place.

Mitigation Plan 1

1. Detach EBS from master and mount to any slave in the same AZ. (This is because of EBS restriction)
2. Reattach EIP from master to the slave
3. That’s it!

Mitigation Plan 2

1. Use GlusterFS as network file system. Index is automatically replicated across AZ and regions.
2. Reattach EIP to the secondary master.
3. That’s it!

Mitigation Plan 3

1. Use SolrCloud feature of Solr 4.0!


1. http://wiki.apache.org/solr/SolrReplication

Posted in Technical | Tagged , , , , , , | 1 Comment

Amazon CloudSearch & Apache Solr 3.6

With the advent of the “Information Age”, massive amounts of data & information is being added into our lives every second. Gone are the days of MB & GB. Today everything is in the order of TB and PB. Raw data is worthless unless meaningful information can be extracted out of it & made searchable so that end users derive value.

Search plays the most integral role in today’s websites and online applications. Search has the power to make or break a business. Hence, sufficient time has to be spent to make search a meaningful experience to the users.

We will look at David & Goliath of search technologies – Amazon CloudSearch & Apache Solr.

Amazon CloudSearch is a “fully-managed search service in the cloud that allows customers to easily integrate fast and highly scalable search functionality into their applications. It seamlessly scales as the amount of searchable data increases or as the query rate changes, and developers can change search parameters, fine tune search relevance, and apply new settings at any time without having to upload the data again. It supports a rich set of features including free text search, faceted search, customizable relevance ranking, configurable search fields, text processing options, and near real-time indexing.

Apache Solr is a “popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly scalable, providing distributed search and index replication.”

First, let’s compare these two search technologies with respect to their feature set –


Apache Solr is completely open source. It is written entirely in Java and uses Lucene under the hood. However, Amazon CloudSearch is a proprietary creation and is based on Amazon’s A9 technology.

Setup Effort

Considerable time & effort is needed to configure Apache Solr and get it up & running. It includes tasks such as Solr download, knowledge of Java, configuration of environment variables, deploying it in a server, understanding the commands needed to start / stop / index the Solr server, applying patches and upgrading to newer versions.

In contrast, Amazon CloudSearch is a fully managed search service in the cloud. You can set it up and start processing queries in less than an hour with a few clicks in the AWS Management Console. You don’t need to worry about running out of disk space or processing power.

Multilingual Support

Apache Solr has multilingual support. Custom analysers and tokenizers have to be written and plugged in. Also, the recommended approach is to have a multi-core architecture with each core addressing one language. But Amazon CloudSearch currently supports only English for tokenizing words in the index.

This is a good-to-have feature but cannot be seen as a critical one.


Scaling is an important design consideration for high volume / high growth architectures. Scaling can be done in two ways: scale-up & scale-out. Scale-up is the process of migrating from a small instance to a larger instance whereas scale-out is the process of spawning multiple instances. Refer Figure 1 below.

Apache Solr has scaling support but it is a manual process. When the search traffic increases beyond the threshold of a particular server, we have to manually spawn new Solr servers, transfer the index, auto-warm the caches and re-route the search queries to point to the new Solr servers. It is an involved process and needs an expert to get it done right.

An expert Solr admin is needed to keep a close watch on the performance of the Solr servers. Solr provides an admin interface, which has information regarding documentCache, filterCache, resultCache and statistics such as cache hit rate, cache lookups, cache hit ratio and cache size. Observing these metrics, the Solr admin has to decide on scaling the Solr server. One of the signals that an admin uses to make this decision is to have a look at the cache hit ratio. If this metric is low, then it means that the cache is not able to serve a majority of Solr search requests.  The admin then proceeds to increase the cache size (i.e. scale-up) so that searches will be quicker. Similarly when a scale-up is not possible, scale-out comes in handy. But scale-out is not as easy an operation as scale-up. As you will see later in this article, scale-out involves partitioning the index and performing a distributed search. The admin needs to be careful when partitioning the index (as it usually leads to the re-index of the entire data set) and search queries have to be modified to support the presence of multiple indices across distributed Solr servers. Manual scaling is a strenuous task, no doubt.

As Amazon CloudSearch is a fully managed search service, it scales up and down seamlessly as the amount of data or query volume increases. Amazon CloudSearch determines the size and number of search instances required to deliver low latency, high throughput search performance. When a search instance reaches over 80% CPU utilization, CloudSearch scales up your search domain by adding a search instance to handle the increased traffic. Conversely, when a search instance reaches below 30% CPU utilization, CloudSearch scales down your search domain by removing the additional search instances in order to minimize costs. This is one of the most important points in favour of Amazon CloudSearch.

In today’s information age, scaling (along with index partitioning and replication) is considered as a must-have feature.

Partitioning the Index

When it is not possible to store the data in its entirety on a single server, we have to partition the index and store the part-indices in separate servers. This is also known as sharding.

Apache Solr supports partitioning the index but it is a manual process. When performed manually, it is not a completely transparent operation. Solr doesn’t have any logic for distributing indexed data over shards. Then when querying for data, you supply a shards parameter that lists which Solr shards to aggregate results from. Lastly, every document needs a unique key (ID), because you are breaking up the index based on rows, and these rows are distinguished from each other by their document ID.

uniqueId.hashCode() % numServers determines which server a document should be indexed at. The ability to search across shards is built into the query request handlers. You do not need to do any special configuration to activate it. In order to search across shards, you would issue a search request to Solr, and specify in a shards URL parameter a comma delimited list of all of the shards to distribute the search across. You can issue the search request to any Solr instance, and the server will in turn delegate the same request to each of the Solr servers identified in the shards parameter. The server will aggregate the results and return the standard response format. A sample distributed search URL will be of the form –


Amazon CloudSearch, being a fully managed search service, automatically partitions the index as your data volume grows. It will partition the index across multiple search instances. Conversely, when your data volume shrinks, it will fit your data in one index.

Figure 1: CloudSearch Scaling showing index partition & replication

Replication of Index

Replication of index is used to handle high volumes of search traffic.

Apache Solr has the support to replicate the index. But it is a manual process and includes spawning new instances and configuring them to enable replication between the servers. A replication handler has to be configured on both master and slave machines. On the master, you have to specify the replicateAfter values and on the slave you have to set the fully qualified URL of the master replication handler for the attribute masterUrl. If at any time the URL of the master changes, then all the slaves have to be stopped to make the necessary changes and restarted again.

Amazon CloudSearch automatically scales your search domain to meet your traffic demands. As your traffic increases beyond the processing capacity of each search instance, the partition is replicated to provide additional capacity.

Faceted Search

Faceting allows you to categorize your results into sub-groups, which can be used as the basis for another search. In recent times, faceting has gained popularity by allowing users to narrow down search results in an easy-to-use and efficient way.

Faceting is best explained with the help of a picture (See Figure 2 below). As you can see, a search for “java programming” results in a lot of hits. Observe on the left side of the figure. You can clearly see that the search resulted in 3 facets (or sub-groups) using which you can narrow down your search. For example: if you click on “PDF” in the “Format” facet (see “Facet 2” in the figure), the search query now essentially means “java programming AND only pdf format”, thereby narrowing down the search space eventually leading you to better and convenient results. You can also observe that each member of a facet is accompanied by a number called Facet Count. In the “Format” facet, you can see “PDF (14)” which means that there are 14 “java programming” results in PDF format. The important aspect of facets is that as you go deeper using facets, the resultant search space is vastly reduced and hence the search will be considerably faster.

Both Apache Solr and Amazon CloudSearch allows the user to perform faceting with minimal effort.

Figure 2: Faceted Search

Field Weighting / Boosting

Field Weighting is a process of assigning different prominences to the same word when present in different places in a document. For example when the phrase “Harry Potter” is present in the title of a document, it is ranked higher than when the same phrase is present in the References section of a document.

Both Apache Solr and Amazon CloudSearch allows field boosting with minimal effort.

“Did you mean…” feature

One of the better ways to enhance the search experience is offering spelling corrections. This is sometimes presented at the top of search results with such text as “Did you mean …”. Many a times, a user might not know the correct spelling thus leading him to undesired results. Such a feature would vastly reduce users’ time and effort. It is sad when businesses deliberately skip this feature in order to show increased search traffic in their monthly / annual term sheets.

Apache Solr supports this with the Spellcheck search component. The recommended approach is to build a word corpus based on the index principally because your data will contain proper nouns and other words not present in a general-purpose dictionary.

Unfortunately, Amazon CloudSearch has no support for “Did you mean…” feature.


A popular feature of most search applications is the auto-suggest feature where, as a user types their query into a text box, suggestions of popular queries are presented. The suggestions list is refined as additional characters are typed in. If you think about it for a minute, you will realize that this feature is even better than a “Did you mean…” feature.

Apache Solr has the support for autosuggest. It can be facilitated in many ways – NGramFilterFactory, EdgeNGramFilterFactory or TermsComponent. When used in conjunction with jQuery, it becomes a very powerful autosuggest experience for the user.

Amazon CloudSearch has no support for autosuggest feature.

Geospatial Search

Consider the following example – when a user performs a search for “Starbucks”, the search engine must show the nearest outlet based on the user’s current location. Location-aware search will always produce significantly better results and helps the user in finding the information more effectively and efficiently. This use-case signifies the importance of Geospatial search. In today’s on-the-go world, it is a must-have feature as it is a win-win situation for both users and businesses.

Apache Solr supports geospatial search through the implementation class solr.LatLonType. Actions such as sorting the results by distance and boosting documents by distance can be performed.

Amazon CloudSearch has a very limited geospatial search feature set. As of now, CloudSearch has the capability to return documents within a specific area. Missing features include sorting by geographical distance and faceting by distance.

“Find Similar” feature

This suggests similar records based on a particular record. It is similar to the “Find Similar” feature used by popular search engines. It is more common for applications to request it to be performed on just one document when a user requests it, which occurs at some point after a search. E-commerce sites benefit from this feature as research suggests that users typically compare products before making a transaction and are likely to buy a product which is better & hence slightly expensive than what they had initially intended to buy.

Apache Solr implements this using MoreLikeThisHandler or MoreLikeThisComponent.

Amazon CloudSearch currently does not support this feature.

Rich Documents

Support for rich documents (HTML, PDF, Word, etc.) is an essential feature of a search server. Data & information will be in different formats and it would be foolish to expect it only in certain formats. It is best if search servers provide an intuitive interface for ETL operations and natively support this feature.

Apache Solr has support for rich document parsing & indexing using Apache Tika.

Amazon CloudSearch expects data to be in Search Data Format (JSON & XML) and hence we need to use AWS Console or command line tools (cs-generate-sdf)  to convert the rich text to sdf format.


Specific features need not be supported natively because there might not be sufficient demand for them. In such cases, customizations play an important role. It is really a good-to-have feature as different businesses have different requirements and they would need the capability to customize appropriately to suit their needs.

Amazon CloudSearch, being a proprietary creation, does not allow for any customization either through plugin integration or via extending functionalities.

Apache Solr, being open source, allows of customizations of analysers, tokenizers, indexers, query analysis and the like through plugins and via extending the code base.

Having seen the differences between these two search technologies in terms of their feature set, let’s us now compare them with respect to the production environments and fitment –

Increasing / Decreasing / Spike Traffic

Scaling Apache Solr is a manual process and an admin has to spawn new instances, having a close look on the traffic patterns, well in advance so that traffic requests are not dropped.

However, Amazon CloudSearch is a fully managed service and hence no human intervention is needed for scaling. As mentioned earlier, when a search instance reaches over 80% CPU utilization, CloudSearch scales up your search domain by adding a search instance to handle the increased traffic. Conversely, when a search instance reaches below 30% CPU utilization, CloudSearch scales down your search domain by removing the additional search instances in order to minimize costs.

Support for protocols

Both Amazon CloudSearch and Apache Solr support HTTP & HTTPS. Amazon CloudSearch supports HTTPS and includes web service interfaces to configure firewall settings that control network access to your domain. The same but manual process is recommended for Solr too.


Apache Solr has many algorithms including cache implementations such as LRUCache and FastLRUCache. Solr, being open source, can be extended by adding our own algorithms. Since Amazon CloudSearch is proprietary, neither is there information on the algorithms being used nor can they be extended. But please bear in mind that the default CloudSearch algorithms will suffice for most applications.

High Availability

Amazon CloudSearch, like other AWS components, is a highly-available service. It completely automates the time consuming tasks of managing and scaling it. But in the case of Apache Solr, the high-availability has to be built manually which is a strenuous task.


Consider a sample e-commerce business: 100 MB of search data is present in their servers. Search traffic is likely to be 100,000 requests per day. 50 batch uploads per day, where each batch contains 1000 1 KB documents. Their entire index has to be re-built 4 times every month.

Based on the above requirements, cost of running Amazon CloudSearch 24 / 7 will be close to $87. The costs will definitely rise as the data and search traffic increases since larger and more search instances will be spawned to meet the growing needs. The plus side, however, is that all the managing and scaling tasks are automated.

Running Apache Solr on existing servers will have no additional cost when the deployments are in the small-to-medium range. Since there is no such thing called a free lunch, additional efforts & time will be spent in setting up high availability and scaling.

Few Limitations of CloudSearch

  • Available only in US East region
  • It can scale up to a maximum of 50 instances and partition up to 10 instances. Further scaling is likely on prior approval from AWS team
  • The maximum batch size is 5 MB and the maximum document size is 1 MB
  • Stopwords, synonyms dictionary size is limited

Apache Solr is a highly stable product with a rich feature set and high profile deployments but requires significant human effort to scale and manage it. Amazon CloudSearch is still in its infancy and has a lot of catching-up to do. But the most important benefit is that managing and scaling is completely automatic. It has a lot of promise and we are eagerly awaiting future enhancements!

Posted in Technical | Tagged , , , , | 2 Comments

Hive – A Petabyte Scale Data Warehouse using Hadoop

Web has been growing rapidly in size as well as scale during the last 10 years and shows no signs of slowing down. Statistics shows us that every year more data is produced than all of the previous years combined. Apache Hadoop, inspired by Google’s MapReduce and Google File System (GFS) papers, is a framework that allows for the distributed processing of such large data sets across clusters of machines. Hadoop Distributed File System (HDFS) is the primary storage system used by Hadoop applications. HDFS creates multiple replicas of data blocks & distributes them on compute nodes throughout a cluster to enable reliable, extremely rapid computations.

Using Hadoop was not easy for end users, especially for the ones who were not familiar with MapReduce framework. End users had to write map/reduce programs for simple tasks like getting raw counts or averages. Hadoop lacked the expressibility of popular query languages like SQL and as a result users ended up spending hours (if not days) to write programs for typical analysis.

Enter Hive!

Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query and analysis. While initially developed by Facebook, Apache Hive is now used and developed by other companies.

Hive was created to make it possible for analysts with strong SQL skills (but meager Java programming skills) to run queries on the huge volumes of data to extract patterns and meaningful information. It provides an SQL-like language called HiveQL while maintaining full support for map/reduce.

A HiveQL statement is submitted via the Command Line Interface (CLI), the web UI or an external client using the Apache Thrift, ODBC or JDBC interfaces. The Driver first passes the query to the Compiler where it goes through the typical parse, type check and semantic analysis phases, using the metadata stored in the Metastore. An optimized plan in the form of a directed acyclic graph of map-reduce tasks and HDFS tasks is generated. The Execution Engine then executes these tasks in the order of their dependencies, using Hadoop. In short, any HiveQL query is divided into MapReduce tasks which run on the robust Hadoop framework.

Hive has access to files stored either directly in HDFS or in other data storage systems such as HBase. HBase tables can be accessed as if they were native Hive tables.  As a result, a single Hive query can perform complex operations such as join, union, and aggregation across combinations of HBase and native Hive tables. But the recommended approach is to pump the data from HBase (or any other storage) into Hive and then analyze it there. The benefit is 2-fold: (a) once the data is in Hive, queries against it will typically run faster (since Hive is optimized for warehousing) and (b) no risk of excessive load originating from cluster nodes onto the external storage system

Loading bulk data into Hadoop from production systems or accessing it from map-reduce applications running on large clusters can be a challenging task. Transferring data using scripts is inefficient and time consuming. How do we efficiently move data from an external storage into Hive? Meet Apache Sqoop. Sqoop allows easy import and export of data from structured data stores such as relational databases, enterprise data warehouses, and NoSQL systems. Using Sqoop, you can provision the data from an external system on to HDFS, and populate tables in Hive and HBase. The dataset being transferred is sliced up into different partitions and a map-only job is launched with individual mappers responsible for transferring a slice of this dataset.

Posted in Technical | Tagged , , , , | Leave a comment