Article published in Integration Consortium

My article on "More the data, less the business intelligence" is published here.
On my blog here

Respect vs Authority - IT view

In one of the training sessions I attended recently, I was involved in a fierce debate about the 'need to reign in IT programmers'. A lady said that on many occasions, the programmers go on and do things as they want and there was a need to check this. ANother gentlemen piped in saying that,"this is the only profession, where you don't need to have any qualification to be called an expert in". That's where he thought, "IT ALL WENT WRONG".

The disdain was rankling. For one, I think the advances that have happened in/using IT has far surpassed anything else in the last two decades. Think of home pcs, mobiles, gaming, internet for God's sake (the lady chose to differ!!). One of the primary reasons It has happened because of the immense freedom that the IT developers get. These programmers has been either working independently or have got enough freedom in their job to go away and do something that they thought was path-breaking, interesting and fun. Because the resources available to a programmer are essentially available free, he can go about this task all alone, with minimal supervision, and funding.

IT is a field that allows for the greatest exchange of ideas, simply because you can share the essential building block - the code - with anyone in the world. You can copy already available code and create something completely new. If what you come up with is good, people will use, approve, and promote it. If not, you get feedback to improve or bin it.

To say that management needs to always reign in these guys is not understand how fundamentally different things operate in the computing world. Creativity in any field is fostered through breaking down barriers and allowing maximum freedom. It's especially so in IT. Of course there needs to be checks on what the final output is, but even here, the old way of using authority to do that is obsolete and obstructive. Rather, we now have what is called respect. If you create something great - an article, paper, code, or design - you publish it to the world. The world studies it, reviews it, and scores you on your artifact. This is much more democratic, personality independent method of weighing the value of a creation.

You see it in blogs, movie rating,articles and code. Many times the rating is accompanied with enlightened comments from the reviewers, clearly articulating what aspects they liked or disliked. This feedback is much more useful than quality checks done through a single dedicated team of people.

Of course, sometimes the artifact you create, cannot be published to the whole world. But then, you can use private cloud / intranet to access vast pool of people in your organization to review your product.

But the moot point is, in World 2.0, RESPECT WORKS BETTER THAN AUTHORITY. Its high time we recognize and use it fact to our advantage.

CBIP recognition in UK

From my experience, it seems that there isn't enough recognition of the CBIP certification in the UK. The best measure of this is to compare the number of data warehouse / BI jobs advertised here in the UK and US.

In the US
Lets try Monster - It throws 3 jobs with CBIP as a requirement or prefered

At Jobster - it has 7


In the UK
Monster - 0

At IT Job Board - 0

Any given day, you will find similar trend - definitely recognized in the US, but not in UK. Wonder why that is? Not enough marketing/awareness building by TDWI

More the data, less the business intelligence

SUMMARY

Often, there is a quest to do more with the data we have. The notion is - "We have this piece of data, probably, it means something important. Lets find out!" This has led to creation of mammoth corporate data warehouses which are poor in imparting knowledge and big on maintenance. Most of the data that they contain is fractured, incorrect, and redundant. Good decision making doesn't need ALL the data that is available. In fact, good decision making demands limited data, i.e. KPIs. This means that data warehouses which are our decision support engines need to built as per specs (not as per what source systems contain). Moreover, we need to educate ourselves to look beyond data - recent global events have highlighted acutely the inherent limitations of data and people who use it.

DATA WAREHOUSES AS DUMPING GROUNDS
As organizations become more tech-savvy, they want to do more with the data they generate. Letting the data be just created and consumed for its fundamental need – facilitating a core business transaction - is unacceptable. Organizations want to store, examine, tweak and mix data, and hope that it will be useful in some way.

More and more buzzwords – Business performance management, digital dashboards, scorecards, web analytics, on-demand BI – have added to the confusion in the information management space, all the while, making the CIO worried that he is not doing enough with the data he owns. Organizations are trying to extract the last ounce of meaning from the smallest data element. The notion is - "We have this piece of data, probably, it means something important. Lets find out!" I wonder if it is the right approach.

Data stores are typically built for analysis and provide "business intelligence". Large ones - data warehouses - contain aggregate of all the organizational data. Data marts focus on specific subject areas or departments. Operational data stores cater to immediate needs - minute-by-minute or daily updates, and are smaller in size. The thing to note is that these stores are not core transactional systems themselves. They are decision support instruments that help the management in their decision making process and come up with business plans based on them.
Capturing all the available data in these stores is fraught with numerous problems.

1. More data we bring into them, the more noise we add. Some of this noise can be attributed to incorrect system entry or untidy manual processing. However, sometimes, there is a genuine anomaly e.g. sudden freak rains on a given day mean that footfall on the high street was reduced by 40%. But does this information help us in business planning? Not really.

2. It is difficult to sieve through loads and loads of data. It obscures rather than informs. You might organize all the data in neat multi-dimensional fancy reports. However, someone has to trawl through them to understand what the hell all those arrays, tags, and numbers are. By the time you do that, the time to make a critical decision may have already passed.

3. Capturing minutiae into a data warehouse requires huge spending. The infrastructure to clean the data, pump it into the warehouse, and store it adds massively to the costs. For companies already struggling to build a plain vanilla data warehouse, it's a plain vanilla foolishness to waste money on a real-time all singing, all dancing BI- behemoth.

4. Be prepared for poor performance from large data warehouse – normal databases cannot match the performance of data warehouse appliances, which are specifically targeted for large data volumes, and sophisticated data analysis. However such appliances are extremely expensive. Regular RDBMS databases struggle to process vast quantity of data, the load times and SQL queries are much slower, whatever the optimization strategy used. A lot of the user queries result in full table scans and fetching of millions of records at a time. This is thanks to the approach "Let's see what I can get out of the system" rather than "What do I really want?" It puts undue pressure on the system leading to poor performance. No wonder, it leaves users dissatisfied, who in the end, turn away from the system.

5. With the oceans of data available, the decision makers are free to glean information that fits their views or needs. This phenomenon is called confirmation bias. Managers can use it to justify the spending last year or budget for the next, demonstrate exceptional performance, or apportion blame for poor performance on some external factors. It doesn't matter whether people do this selective data filtering consciously or sub-consciously. The fact is we do it.

WHY DOES ALL THE DATA GET ASKED FOR?
Most of the projects are still run as IT change initiatives. This means that business is considered ancillary to the whole activity. The IT project managers don’t put sufficient demand for business resources. Unfortunately, the team managers do not want to take out key resources from their day-to-day activities to engage in something that provides no immediate value-add and may harm their team's performance. In the absence of clear business requirements, the IT team is forced to design a system to cater to any eventuality. Alas, that is not possible.

Some blame rests on the IT team as well. The business representatives don't know what features or functionality will be available in the new world. Business can do with some handholding about the capabilities of the new architecture – screen layouts, filters, collaboration, scheduling, security, flexibility etc. so business knows what to expect and provide their requirements accordingly. A few mock-ups go a long way in flicking the light bulb on, giving the business users a number of ideas about ways to use the new architecture effectively and also demonstrate its limitations. In the absence of any business-IT workshops early on, business will keep drifting and won’t knowing what to expect. They will keep ask for everything.

DATA AND DECISION MAKING
Decision support stores provide a chance to see through the convoluted, shifting data spaghetti and glean out the most pertinent information from it. The starting point always has to be 'what do we need to find out' backed by a strong 'why' and then move to 'How can we achieve this?'

We fail to understand the intrinsic difference between data, information, and knowledge. Data is unprocessed content. Information is a higher abstraction which provides additional meaning to the data. Knowledge is application of the underlying information or data, it's about true understanding of 'what lies beneath'. The important thing to note is that more data doesn’t mean more knowledge. And, more information doesn’t equal more knowledge. In fact, just the opposite may be true.

Good decision making doesn't need ALL the data that is available. In fact, good decision making demands limited data, what is known as KPIs - key performance indicators. E.g. KPI for recruitment team is recruitment increase percentage. It can be derived by number of new recruits this year over previous year. You might capture a whole lot of information about the new recruits, but most of it is peripheral to the success of the recruitment team. A doctor doesn't need to record all your physiological variables to correctly diagnose a problem. Just 3-4 key indicators are enough to tell him if you are having a heart attack. However, as we humans believe that more information one can acquire to make a decision, the better. This is known as information bias. However, extra information cannot affect our decision – what is not worth knowing is not worth knowing. Based on this, it is critical to limit the content of any design support system - be it a warehouse, or operational data store.

Some of the really good decision making is intuitive. With just handful of information and unconscious rapid cognition, we can arrive at an accurate judgement very quickly. It is what Malcolm Gladwell calls thin-slicing in his fascinating book, Blink. It is a powerful, sophisticated tool for taking quick decisions with minimum information. In fact, in the face of mountains of data, this ability no longer functions. Intuition, by definition is fragile and short-lived, and too much information can often paralyze it.

WHAT TRUE ANALYSIS DEMANDS
Real, meaningful analysis needs time and distance. There is no benefit in seeing all the variables right this second to be able to make a decision about the ‘next XYZ strategy’. Over a period of time, the data will be sufficiently stable to be able to give a reliable picture of current business trends and decide on how to tackle the future ones. Distance refers to the fact that one doesn’t need granular, detailed data for analysis. One needs to zoom out slightly to see it in a broader context. Summarized data rolled up at reasonable levels spliced across relevant dimensions will provide a more substantive, vivid depiction of the overall performance and help sharper, quicker decision making.

It’s a fallacy that ALL the answers are there in the data we capture. A school might be deemed to have exceptional teaching model based on the performance of the students at Grade 10 tests. However, this conclusion doesn’t take into consideration the fact that the school has a rigorous test based admission process, ensuring only the best students get admitted to the school. No school maintains un-admitted students on their rolls. This is called survivorship bias or problem of silent evidence.

The other trouble is inability to predict cataclysmic events like earthquakes, terrorist attacks, fires or for that matter, a recession. Such is the disruptive force of these events that the forecasting and predictive models just don’t stand up to scrutiny. This, at a time when you expect such tools to guide you the most. The need is to look beyond the obvious.

Massive data stores give us a sense of being all-knowing. We think we can just immerse into them and we will come out enlightened. The truth, as explained, is not that simple. In reality, the tools are as good as the data they contain and the people that use them. Unfortunately, both are inherently flawed. We need to remember two things - Data doesn’t tell everything, and you need to know what you are looking for.

What beautiful scroll

Just extraordinarily peculiar.

Whole story here

Tiger numbers diminishing rapidly

Tendulkar dedicates ton to tiger conservation

Hamilton (PTI): Master batsman Sachin Tendulkar today dedicated his 42nd Test hundred, which put India in command against New Zealand, to the cause of tiger conservation.

"I would very much like to like to dedicate this hundred to tiger conservation because that is what right from the start of this tour the whole team had decided. I have given couple of messages as well in this regard. So I would dedicate this one to tiger conservation," said Tendulkar, who smashed a sublime 160 at the Seddon Park.

On the eve of the first Test, Tendulkar had supported the cause, expressing his concerns at the dwindling numbers of tigers in India.

"At the start of the century, there were nearly 40,000 tigers in India. Today, that number has shrunk to 1,700 and we are losing at least one tiger a month. The rate at which the tiger is being hunted down is alarming," Tendulkar had said.

Tendulkar has got the rate of 1 tiger loss per month wrong. There were 40000 tigers at the start of his century and by the end of it, there were only 1700. So, that would be a loss of 13.82 tigers every 1 second.

That's a truly staggering rate of decline.

P.S. I am just kidding of course, Tendulkar's concerns are well-placed.

Practical advice

Well, the governments, banks, social services are all giving us practical advice to cope with the recession.

No, I don't want practical advice. How about just some theoritical advice?

Listen till the very end - Jon Stewart, the bond!

The Daily Show With Jon StewartM - Th 11p / 10c

CNBC Gives Financial Advice

Daily Show Full Episodes
Important Things With Demetri Martin

Political Humor
Joke of the Day

The servility towards "criminally rich" CEOS is vomit inducing.

Curious case of Benjamin Button

Terrible waste of my time. A movie without a soul, purpose, or any need to exist.

Predictioner

I think I can be a good jyotishi.
Here is what I said about the Oscar for Supporting Role

And the result - Heath Ledger wins

Anurag Kashyap for Prime Minister.

I am pleased. Very very pleased. It's been a heady night getting high on some fantastic music. Heard Delhi 6 first and now bloody this. India, I am so fucking proud.

Hallucinating about mosquitoes

I took my daughter - aged 1 year and some - to India to meet my family and friends. I didn't consider mosquitoes in that list. Evidently the pests didn't take well to that and ravaged my daughter all over. She had chicken pox just a while ago. She recovered but her face now looks worse. Also, being fair (both her chromosomes seem to come from her mom), her skin turned crimson red.

On the plane to London, people were asking me if I was a millionaire, because I (naturally) and my daughter (thanks to the scabs) were looking like we lived in the slums. I said no, we don't live in slums, thank you! But the mosquitoes don't know that and keep bugging us.

While people were asking inane questions, I also noticed two mosquitoes on the wall opposite (we get front seats thanks to the little one). They had boarded the flight without passport, plane tickets, security checks, and worse, were not wearing seat belts. I couldn't debase myself by reaching for them and trying to squash them. Instead, lost about 0.01 pints of blood stoically getting bitten. Finally, killed one of them while it was in the act.

Back in London, I still couldn't get over those rascals in Mumbai and the ones with unknown address. While cradling my daughter to sleep, I was hallucinating that mosquitoes were prancing around us. I could hear the buzzing in my head, just like you hear it in cell-phones in vibrating mode. I was reaching for the bat (its actually a racquet, except you don't play with it, but zap mosquitoes. Which means it may not be a racquet. Hmmm!). Except, there was no racquet. And there were no mosquitoes either. Except those in my head.

At night, I was covering up my daughter in quilt, not because it was cold, but because I was afraid more mosquitoes would bite and her face might start looking like a monkey's bum (and people would then think that both her chromosomes came from me). Realised through the jet lagged sleep that there are no mosquitoes here.

Yanked the quilt from her and wrapped me warm!

Rahman should popularise ghazals

Rahman should popularise ghazals, says Jagjit Singh

And Jagjit Singh should populatise EVERY other music style. If thats possible.

Actually the idiot said, that in the earlier decades Hindi film music was ghazal based. Hello! Apart from a few exceptions like Umrao Jaan and the nasal Salma Agha's movies, ghazal has never been central to Hindi films. But the moron is too insular and petty to recognize the place of ghazals (important)and that of Rahman (very important)

Slumdog Millionaire

For a Mumbaikar turned radical Mumbai-baiter, Slumdog Millionaire should be such a great mouth piece. Showing Bombay in all its gory, disgusting, repulsive, and obscene glory, it should have made me say to the city loyalist, "Look, is this why you really like the city. Look at this ugly mess of the city you got!".

But somehow, it made it long for the place. Because for some reason, the ugly mess is mine (may be I even contributed to it before shamefully copping out ).

I am glad I didn't have a life anywhere close to Jamal's. I did start out in one of those millions shacks near Golibar road, Santacruz, but thanks to the resilience and grit of my grandmother and my parents, I came out of that pretty quickly. At age 5, I was living in a 1 BHK flat - a place a million worlds apart.

Slumdog Millionaire makes you believe in your heart that even fantastically great things do happen to good people. Most of the poorer folks work incredibly hard to earn a living, without purposely trying to subvert the system or break the rules. I think if most of these guys who keep trying and fighting come out winning. Bombay offers them the opportunities to work. They toil terribly hard, subside on vada-pav, send their kids to school, put money away for the rainy day, and find themselves in substantially better place than where they started. I truly think there are a million Slumdog Millionaire stories out there. And I feel proud that a city that is filthy, cramped, and insensitive is still a place where such inspiring stories bloom.

Rabbi Shergill - Jinhe Naaz Hai Hind Par Wo Kahaa Hai

For all that is wrong with our country, for all those things that rile and rankle, for all that disgusts us, for all those things that make you go, "When will we ever learn?", there is a truly the most amazing song.



If there needs to be a patriotic song in these times, then this is it. For in the last few days, it is not the external factors that have threatened India. It's the ugliness within. The insurmountable agonies, the unspeakable debasement, the sheer insensitivity that ordinary Indians have to face. 26/11 is NOT an external attack. It is direct result of the callouness of our government and bureaucracy.

The song so beautifully summarizes our inadequacies in that regard. After all, Bilqis Yakoob Rasul, Manjunath Shanmugham, Satydendra Dubey and Navleen Kumar were not victims of the criminals who killed them. That were victims of the system that never enforced and ensured decent protection for those who stood up or swift and suitable justice to those who fell.

Criminals in this country (especially politicians, police and bureaucrats) and outside (terrorits) know this and play to its weaknesses.

Slumdog Millionaire - Indian Film?

It is galling, the ignorance of some of our Bollywood actors. Anil Kapoor says in this interview that Slumdog Millionaire, a film in which he plays a pivotal role (and not just a walk-in part) is a Hollywood film. Now it was ok for our entertainment reporters with their low IQs and poor GK to say something like that. But AK has worked on the film, got paid by the producers (Celador Films and Film4), taken instructions from the director, said the lines that the writer, Simon Beaufoy wrote.

Can you tell me AK, where does Hollywood come into the picture.

CDMP using CBIP

The credits you get on CBIP can be transferred to get CDMP - Certified Data Management Professional. Most speciality exams on CBIP are valid to get a CDMP. All you have to do is to contact ICCP or DAMA, send an application to transfer the credit (admin cost $50).

CDMP is conducted by ICCP and DAMA (Data Management Association)
Previous posts on CBIP 1, 2 and 3

Lloyds TSB is terrible

Following is the official complaint I made with Lloyds TSB. I am definitely moving my bank to protest against the ill-treatment and brazen rip-off I was subjected to. Please don't fall prey to their cross-selling tactics.

*********************************************************************

I am furious at the way I have been literally duped by Lloyds TSB with my credit card account. I have faced recurring problems with Privacy Guard and then again with Payment protection and have lost almost £50 which I have been surreptitiously and incorrectly charged.

Privacy Guard - I took up the Privacy Guard facility based on a cross-sell that it was going to cost me £1 for the first month and that I would get my credit report from Privacy Guard. I was unable to get my credit report in spite of trying thrice. Subsequently I called up before the 1 month period to cancel my account. However, due to appalling processes, the cancellation was not registered in their systems. I was subsequently charged for the next month and had to call up again to cancel the service (not surprisingly, they didn't have any records of my previous call).

Payment Protection - I signed up to this when I visited York branch (Pavement) for some other banking needs. I clearly remember being rushed into this, and more importantly not being told anything about any monthly charge (my wife, who is an Accountant, was with me at the time and for a change, agrees with me on this one). I was not explained at all that this is actually an insurance for which I need to pay a monthly premium.

When I called up Lloyds TSB, it was made as my fault that I did not read the documentation that was sent later. That may be the case, but the advisor in the bank should have explained the main terms and conditions clearly to me, which he didn’t. The documentation is written obviously not to be read; else it would be simple, summarized and intelligibly show the charges with examples. It may be my fault that I did not read the documentation, but I have no doubt in my mind that I was misguided when I was sold the scheme and that the cost implications were not explained at all.

I think on both cases Lloyds TSB has used unscrupulous means to sell new schemes to an unsuspecting and gullible customer like me. I am hugely disappointed and angry at having being cheated by your bank.

I will be moving my credit card account soon to somewhere else and subsequently do the same with all my accounts with Lloyds TSB – current as well as online saver.

*********************************************************************

article from outlookindia.com

---

From Outlook

---

magazine | Dec 08, 2008

Link to this Story : Ah, Bombay...
Falling to the mother of all security lapses

---

Subscribe Online: Outlook Magazine

Seige of Bombay - The ugliness within

In the aftermath of the seige of Bombay, there is a lot of praise and pride of how the security agencies performed. I want to only focus on the ugly.

I think thanks to the utter failure of the top brass in the police to manage and control the situation, exacerbated no doubt by complete lunacy and depravity of the government, as also lack of training and common sense, the security forces deployed - police, commandos, army - have been exposed badly.

Exhibit A - Inadequately armed policeman runs away from the crime scene.
Exhibit A - Armed policemen unwilling to fire a single shot.
Exhibit C - Killing the terrorits who could have been captured
Exhibit D - Still some terrorists are at large.

Heads need to roll everywhere. Good that Home Minister has FINALLY resigned. Hope he dies of guilt tomorrow. CM, Deputy CM should go, too. For too long, they have sat there impotent, giving undue attention to dance bars and family weddings. But more than that, the heads of most intelligence agencies need to go. The work of these agencies is shrouded in secrecy - not only for outsiders but also for the guys who work there. They have no clue what they are supposed to do. If you don't know, I will tell you.

Protect Indians, you motherfuckers.

The whole response to the attack, particularly at Taj hotel was nothing short of a farce. Police turned out in shirt and jeans, many of them still without bullet proof vests. Some of just chatting away to each other, in the cover of a car or a wall.

Failure to control the flow of information was also stupefying. Why should anyone on the ground be ALLOWED to speak directly to the media, when the top officials themselves are clueless. No clampdown on media itself to stop images of a covert operation from being trasmitted live. Similarly mobile jammers are now used in schools, colleges and conference rooms. Have you even heard of the technology. Why were people in the Taj and Oberio able to communicate to the outside.

This is most retarded I have seen the establishment to be. But politicians and all officials come from the amongst us. I wonder if there is something fundamentally wrong with us. We are after all fine within the confines of our homes, but outside, we are a callous lot. We dont care of others, we disrepect our poor, and our general involvement for larger public causes is abysmal. Our organizational and planning skills in social environments (where the incentives are not directly aligned to your abilities) are inadequate, which is why no one got a measure of what the terrorists were upto.

Yes, I think the attack came from within India.