The way ahead: Non-intrusive big data

The coming years will see an increased debate on data privacy and secrecy, regulations that are structural as well as knee jerk reactions and most importantly, approaches that are ‘non-intrusive’ 
Image: Shutterstock

The three buzz words - big data, artificial intelligence and machine learning are getting even more prominent and at the cost of getting most of us jittery. If there was any unfound fear about AI replacing human intelligence initially and then dominating it, the recent annihilation of the chess engine, Stockfish, by Google’s AI-based Alpha Zero only adds credence to it. Stockfish, a chess engine that Grand Masters hone their skill against, is programmed with a collection of all the games played by the very best over centuries, was handed out a drubbing by an AI-based engine that only practised with itself for 4 hours an just knew the rules of chess! The implications are tremendous, the best of collective human intelligence stands little chance against the machine! AI would slice, analyse and process data which human beings cannot fathom! This is already happening. Where does it leave us? Most importantly, where do we see the next big areas of concern and the next innovations therefore?

We must embrace ourselves to see a greater shift from targeting the average individual to a more specifically designed targeting of individuals. Two things, will enable this - one, new analytical techniques aided by AI and two, collecting as much data as possible of individuals’ decision making. With the impending, all pervasive data analysis techniques, there is now a growing fear whether too many lives will be controlled by too few. The coming years will see an increased debate on data privacy and secrecy, regulations that are structural as well as knee jerk reactions and most importantly, approaches that are ‘non-intrusive’! The uneasiness on the data privacy and secrecy is understandable. Data secrecy is not merely a matter of consent, it has far reaching consequences on individuals who didn’t even consent. I am not referring to snooping here as the decision to consent by one may affect others irrespective of what they choose to do. A relevant example is the on-going imbroglio on how much did data analytics influence election outcomes? With election outcomes, the eventual choice affects us all. It does not matter, whether I had consented that an analytics firm can use my data or not. Going forward, there will be more cheques and balances as expected, backed by revised privacy laws. However, what is more interesting is the endogenous transformation data analytics will undertake and this will be primarily driven by unstructured data. Let me elaborate with an example. 

Ads by ZINC

Consider the case of farmer distress. Currently large areas in India are under severe agrarian stress while there are many more farmers, not hailing from distressed areas but nonetheless face perhaps greater crisis. Currently, our approach to solving these problems are time consuming, intrusive as well as reactionary. We spend an enormous amount of resource to identify the distressed farmers and in most cases even the distressed areas. As a result, any response by the Government is reactionary, made ineffective by leakages.  With available resources, the big data analytics has already started to give us a fresh way of approaching the issue. One uses Satellite image data, applies AI on it to be able to generate real time productivity losses at individual plot levels! Increasingly, the benefits of using unstructured data to solve traditional problems are dominating the landscape. Techniques like these are endogenous response to the issue of data privacy and secrecy!

To leverage this what we need to see now, is the future of data analytics in conjunction with appreciating the business problem and understanding the data extraction and structuring process. One can expect a greater synergy of the three going forward, reflected both in terms of how we train the professionals as well as how they solve the problems. That is the future. Leverage the human intelligence with AI coupled with computational storage!