A Need for change in your IT infrastructure | Big Data

A Need for change in your IT infrastructure

Companies earnings outstrip forecasts, consumer confidence is retuning and city bonuses are back. What does this mean for business? Growth! After the recent years of cost cutting in IT budgets, there is the sudden fear induced from increased demand. Pre-existing trouble points in IT infrastructures that have lain dormant will suddenly be exposed. Monthly reporting and real time analytics will suffer as data grows. IT departments across the land will be crying out “The engine canna take no more captain”. What can be done?

What we need is a scalable system that grows with the business. A system that can handle sudden increases in data growth without falling over. There are two core principles to a scalable system (1) Users experience constant QoS as demand grows (2) System Architects can grow system capacity proportionally with the available resources. In other words, if demand increases twofold, it is “enough” to purchase twice the hardware.

This is linear growth. Is it enough to satisfy CEOs IT demands? Generally, the achievement of linear growth is favorable. In computational science, software problems are characterized by their complexity denoted as O(n). Trivial problems to solve are O(n). An example, is the simple summing of employee salaries. With increasing problem complexity we can observe O(n2), O(n3), or O(2(n)). How does this relate to ensuring I get the same performance as my data grows? Well, an O(n3) problem means that if the number of database records grows 3x, the amount of hardware needs to grow 27x to maintain QoS.

Computational complexity can be minimized using code level techniques, especially if the code is badly written.  However, the fact is that with certain problems, their complexity will never be lowered. For instance, NP-complete problems require non polynomial computation time. In other words, only small datasets can be processed due to their massive computation times - optimization of cargo storage is one such example.

However, if badly implemented, the  simple summing of employee salaries can become NP-complete! What can be learnt? Consider your-self lucky if the problem grows linearly.

So, are we cursed to buying X times hardware each time our domain grows X items? No. Polynomials have many parameters (y=a+ b*x + c * x*x…). “a” indicates the starting barrier. “b, c” indicate the steepness of the curve. For small sets, it is possible that the problem of a cubic complexity grows slower than linear. For large sets, it is impossible.

Need for change
The spiraling growth in data can be anticipated and planned for. The need can be predicted based on (1) past experience, (2) known amount of data, eg by market impact, (3) complexity calculations. Take Facebook, which has 850 million photos and 7 million videos that need storing per month. To cope with exponential data increase, expenditure on its data centers has risen from $30 million, 2007 to $100 billion, 2009.

Facebook's rapid expansion was only achieved with a clear and forward thinking IT strategy. Successful IT planning needs (1) responsible people (2) brave decision makers. A culture of responsibility needs to be instilled within businesses. Underlying problems of the IT infrastructure must not be hidden from upper management. Decision makers need to know about problems. Only then, can pro-active decisions be made that guarantee an agile response to growth. IT cannot be allowed to block business.

Often, warning signs are ignored and the need for change comes too late: “We need our new system tomorrow”. Tomorrows systems only add up to more steel. This is better known as “save your ass tactics” - a quick fix with poor ROI.

In most business, the need for change can be predicted. We have instruments for it and the data is available. Yet, many business only take action when it is too late. There is no blanket solution, but there are methods.