The Big Data Refinery: distilling intelligence from Big Data.: In a world where data volumes are literally exploding in size, the winners will not necessarily be those who can 'outlift' others in the data processing arena: it will be those who can use information intelligence to manoeuvre nimbly around their competitors.

PositionA BAE Systems Detica Whitepaper - Company overview
  1. Gaining an 'information edge'

    Organisations of all kinds face two uncompromising challenges to their traditional business models and long-held assumptions. On the one hand, economic pressures are forcing the public and private sectors alike to do more for less. These pressures include both the recent economic downturns in Europe and the US, and also the slower--but relentless--tectonic shift in economic power towards lower-cost competitors as a consequence of Globalisation.

    On the other hand, the information revolution continues unabated, creating threats as well as opportunities. The proliferation of Big Data--from diverse sources including sensors, web logs, multimedia files and transaction records--is spawning whole new industries whilst threatening to undermine others. The volume of data created worldwide is growing at 54% per year: 90% of the data in the world today has been created in the last two years alone (1).

    In this context, gaining an 'information edge' using data is not only a cost effective way of making efficiency savings, but can also be a way of generating new revenue streams and adapting to new business models.

  2. Exploiting Big Data

    Many businesses are waking up to the possibility of exploiting Big Data to generate new insights to build competitive advantage. The gains for both individual organisations and the wider economy could be huge. The Centre for Economics and Business Research recently estimated that harnessing Big Data could yield a cumulative benefit of [pounds sterling] 215bn to the UK economy between 2012 and 2017, including [pounds sterling] 74bn realised from better customer intelligence, [pounds sterling] 46bn from supply chain improvements, and [pounds sterling] 6bn in the public sector from fraud detection and more accurate performance management (2).

    Gleaning valuable intelligence from Big Data requires new tools and techniques. UK companies with over 1,000 employees have each amassed an average 870TB of data, with many financial institutions and telecommunications providers storing over a petabyte. Size is not the only challenge. This data also has ever-more diverse structures and formats, and often is generated in continuous streams--leading to data with levels of scale, speed and complexity which traditional business intelligence tools are not designed to cope with.

    As a result, we are witnessing an exciting chapter in the development of data analytics, with a plethora of new tools and innovative start ups to address the storage, processing and analysis challenges presented by Big Data.

    However, whilst many major online brands are famous for pioneering the approaches and technologies behind Big Data--such as Hadoop--organisations in other sectors often face significant challenges when attempting to leverage the same technology and have been slower to adapt.

    Exploding data volumes are already driving up spend on data storage and consuming more and more of the IT budget. Integrating a Big Data solution into the existing IT estate may add further cost and complexity. Whilst many new Big Data solutions are generating excitement and interest, there are often immature compared to other enterprise platforms, and depend on specialists with deep technical skills to make them work.

    Without a clear understanding of what Big Data solution will best fit their needs, and how it will work within their organisation, enterprises run the risk of turning a Big Data opportunity into a costly mistake.

    In light of these challenges, how can organisations identify what Big Data solution will generate the actionable intelligence the business requires, and help bring data storage costs under control?

  3. The Big Data Refinery

    BAE Systems Detica has developed what we call the Big Data Refinery: an architecture and a toolkit that can define a solution which meets your business needs, complements your existing business intelligence platform and helps manage down escalating data storage costs.

    There is no one-size-fits-all approach to Big Data, and often one tool will not fully cover all Big Data processing needs. Detica's Big Data Refinery (BDR) is comprised of an end state Big Data reference architecture and decision guide, to enable organisations to determine what tools they need to harness the value from Big Data, and...

To continue reading

Request your trial