Bevil Wood­ing

Gov­ern­ments and or­gan­i­sa­tions have been stor­ing da­ta about in­di­vid­u­als for gen­er­a­tions. Whether through ed­u­ca­tion, em­ploy­ment, bank­ing, tax or med­ical records, we have been shar­ing per­son­al da­ta with trust­ed in­sti­tu­tions as part of the rou­tine of life. How­ev­er, the trend in shar­ing and stor­ing per­son­al da­ta elec­tron­i­cal­ly has ex­plod­ed in re­cent years. This is be­ing fu­elled by the spec­tac­u­lar suc­cess of In­ter­net-based ser­vices pro­vid­ed by on­line gi­ants such as Face­book, Twit­ter, YouTube, Ama­zon, LinkedIn and Google. In amaz­ing­ly short time­frames, these ser­vices gen­er­ate ex­po­nen­tial rates of growth of da­ta about peo­ple and their pref­er­ences. Trans­lat­ing these mas­sive da­ta stores in­to ac­tion­able de­ci­sion- mak­ing as­sets is cre­at­ing new val­ue for busi­ness­es. It al­so promis­es new pow­er and op­por­tu­ni­ty to those savvy enough to har­ness this so-called Big Da­ta rev­o­lu­tion.

Big Da­ta: A grow­ing busi­ness

Big Da­ta is the term in­creas­ing­ly be­ing used to de­scribe the mas­sive amounts of elec­tron­ic da­ta gen­er­at­ed from tra­di­tion­al busi­ness ac­tiv­i­ty and from new, es­pe­cial­ly-In­ter­net-based, sources such as so­cial me­dia. Ac­cord­ing to re­search by MGI and McK­in­sey's Busi­ness Tech­nol­o­gy Of­fice, analysing these large da­ta sets, "will be­come a key ba­sis of com­pe­ti­tion, un­der­pin­ning new waves of pro­duc­tiv­i­ty growth, in­no­va­tion, and con­sumer sur­plus." In our in­creas­ing­ly digi­tised world, every­thing it seems now cre­ates an elec­tron­ic record: every pur­chase, bank trans­ac­tion, doc­tor's vis­it, or on­line "like" or new "friend". As or­gan­i­sa­tions amass this in­for­ma­tion, they are con­stant­ly look­ing for more so­phis­ti­cat­ed means of analysing it.

This process of min­ing Big Da­ta, helps or­gan­i­sa­tions bet­ter un­der­stand their mar­kets and cus­tomers. It even helps pre­dict what's next or what should be next. Big Da­ta an­a­lyt­ics has been cru­cial to the glob­al suc­cess of In­ter­net com­pa­nies like Face­book, Google, Net­flix and Ama­zon. But even tra­di­tion­al re­tail, man­u­fac­tur­ing and ser­vi­cein­dus­try firms are em­ploy­ing big-da­ta an­a­lyt­ics to grow mar­ket share and im­prove their com­pet­i­tive ad­van­tage. And why not? This bur­geon­ing mar­ket to ex­tract in­tel­li­gence from the ter­abytes of da­ta be­ing gen­er­at­ed each day, cross­es all in­dus­tries and sec­tors. As I wrote in a re­cent piece on the Open Da­ta move­ment, "Da­ta is the new cap­i­tal of the glob­al econ­o­my, and as busi­ness­es and na­tions seek in­creased com­pet­i­tive­ness, and sus­tain­able growth, the pres­sure to in­tel­li­gent­ly ex­ploit da­ta re­sources will on­ly in­crease."

Analyse this

Big Da­ta is al­ready big busi­ness for the mak­ers of the sys­tems and soft­ware that do the heavy-lift­ing an­a­lyt­i­cal work. Ac­cord­ing to the mar­ket re­search firm IDC, "big da­ta" com­pa­nies that ad­dress these needs had sales of US$3.2 bil­lion in 2010. By 2015, to­tal in­dus­try rev­enue is ex­pect­ed to reach near­ly US$17 bil­lion. This is a growth rate about sev­en times faster than the over­all IT mar­ket. The usu­al sus­pects like IBM, Or­a­cle and SAP are al­ready in the game. But crash­ing the par­ty, are a raft of star­tups, in­clud­ing Odi­a­go, Plat­fo­ra, Splunk, and Sumo­Log­ic.

Oth­er ben­e­fi­cia­ries are com­pa­nies that col­lect and sup­ply da­ta and pro­vide da­ta bro­ker­age ser­vices. The rapid growth rate of the sec­tor means there is room for more play­ers. Soft­ware de­vel­op­ment shops and an­a­lyt­i­cal star­tups in emerg­ing mar­kets can com­bine soft­ware en­gi­neer­ing, sta­tis­ti­cal analy­sis and lo­cal knowl­edge skill sets to craft their own so­lu­tions. Pri­vate sec­tor and lo­cal uni­ver­si­ties should see the ben­e­fit in in­vest­ing in build­ing home­grown ca­pac­i­ty and repos­i­to­ries. This is strate­gi­cal­ly im­por­tant to sup­port on­go­ing re­search and to safe-guard long term in­for­ma­tion se­cu­ri­ty.

Big Da­ta, small busi­ness­es,bet­ter gov­ern­ments

Big Da­ta is not just for big busi­ness­es. The con­flu­ence of sev­er­al streams is de­moc­ra­tis­ing ac­cess to Big Da­ta and its ben­e­fits. These streams in­clude open source soft­ware plat­forms for pro­cess­ing large vol­umes of da­ta; ac­ces­si­ble and in­creas­ing­ly af­ford­able cloud com­put­ing stor­age, and Big Da­ta sets be­ing made avail­able in the pub­lic do­main as part of open da­ta ini­tia­tives. These de­vel­op­ments set the stage for en­tre­pre­neurs and small-busi­ness­es to re­alise sig­nif­i­cant re­turns from Big Da­ta. Op­por­tu­ni­ties abound to cre­ate new busi­ness­es and find ways to ac­cel­er­ate the growth and im­prove the ef­fi­cien­cies in ex­ist­ing ones. Gov­ern­ments too stand to ben­e­fit from lever­ag­ing Big Da­ta.

Of­ten the largest hold­er of in­for­ma­tion on in­di­vid­u­als, gov­ern­ments of­ten strug­gle to lever­age its da­ta stores to im­prove the de­liv­ery of pub­lic ser­vices and per­son­alise its in­ter­face with the pub­lic. In­vest­ments in digi­tis­ing pub­lic records can be made to bear more rel­e­vant fruit if gov­ern­ments move to more mod­ern ap­proach­es to da­ta man­age­ment and analy­sis. The MGI study pro­ject­ed that in the de­vel­oped economies of Eu­rope, gov­ern­ment ad­min­is­tra­tors could save more than &eu­ro;100 bil­lion (US$149 bil­lion) in op­er­a­tional ef­fi­cien­cy im­prove­ments alone by us­ing big da­ta, not in­clud­ing us­ing big da­ta to re­duce fraud and er­rors and boost the col­lec­tion of tax rev­enues. And users of ser­vices en­abled by per­son­al­lo­ca­tion da­ta could cap­ture US$600 bil­lion in con­sumer sur­plus. The po­ten­tial ben­e­fits and cost sav­ing can be even more sig­nif­i­cant and far-reach­ing in de­vel­op­ing economies, more so in small-is­land states.

Pri­va­cy and Promise

Much of the in­ter­est in Big Da­ta comes from the val­ue stored in the rich trail of dig­i­tal bread­crumbs we all leave when we use on­line ser­vices. Big Da­ta an­a­lyt­ics helps or­gan­i­sa­tions make con­nec­tions be­tween dis­parate snip­pets of per­son­al in­for­ma­tion; a shop­ping pref­er­ence here, a news in­ter­est there, a per­son­al de­ci­sion some­where else. It all adds up, al­low­ing Big Da­ta min­ers to re­veal far more about us than we may in­tend­ed, or that we may even know about our­selves! Con­se­quent­ly, the col­lec­tion and use of Big Da­ta is in­ex­tri­ca­bly linked with the grow­ing glob­al de­bate over pri­va­cy and the use of per­son­al da­ta.

Still, in an age where in­for­ma­tion is pow­er and knowl­edge gives con­trol, Big Da­ta rep­re­sents a pow­er­ful new re­source for or­gan­i­sa­tions. The in­creas­ing vol­ume and de­tail of in­for­ma­tion cap­tured by or­gan­i­sa­tions, the rise of the In­ter­net, so­cial me­dia, mo­bile ac­cess and elec­tron­ic com­merce will con­tin­ue fu­elling ex­po­nen­tial growth in da­ta for the fore­see­able fu­ture. For pri­vate and pub­lic in­sti­tu­tions, re­al­is­ing the full po­ten­tial of Big Da­ta will re­quire more than just savvy techies work­ing in a back of­fice. Lead­ers and pol­i­cy­mak­ers in every sec­tor will have to co-or­di­nate their ap­proach to man­ag­ing the busi­ness of Big Da­ta in or­der to re­alise its promise.

Bevil Wood­ing is the Founder and Ex­ec­u­tive Di­rec­tor of Bright­Path Foun­da­tion, an ed­u­ca­tion-fo­cused not-for-prof­it de­liv­er­ing val­ues-based tech­nol­o­gy train­ing pro­grams in­clud­ing dig­i­tal pub­lish­ing and eBook cre­ation work­shops. He is al­so Chief Knowl­edge Of­fi­cer of Con­gress WBN. Fol­low on Twit­ter: @bevil­wood­ing and Face­book: face­book.com/bevil­wood­ing