011 – Top 10 Data Predictions for 2017

Happy New Year!
Thank you to all listeners and subscribers for your support this past year.

10 – Data borders will break down – logical data lakes and logical data warehouses will grow as companies embrace data virtualization products like Denodo. Data preparation tools, like the new Project Maestro from Tableau, will allow people to seamlessly pull from a) on-premise databases and excel files; b) cloud repositories like Redshift and BigTable; and c) hosted products like Workday and Salesforce.

9 – Data Quality and “Refined” data sets will become more important – with the uptick in BigData, sensor data, and data lakes, users will have a glut of information at their disposal (some have this already). Automated solutions that assess data quality or specially created intermediate data sets will become more and more important6. In many Data Lake architectures and Hadoop based ecosystems, curated or moderately processed datasets are becoming the norm for widespread usage by the enterprise. Data scientists and power users will continue to harness raw data sets for their explorations, but these refined data sets will be used to reduce heavy lifting and “recreating the wheel” for many analysts.

8 – Collaborative BI and analytics will become more mainstream – Sites like data.world and collaborative features in products such as Tableau will be embraced by more users than ever before in 2017. Taking cues from social media, these tools and techniques will produce more living datasets and visualizations with near real-time data as static reporting continues to decline as a percentage of overall reporting. Users will interact with each other and gain economies of scale by not reinventing the wheel when someone else has already done the heavy lifting.

7 – Internet of Things (IoT) will continue to expand – Currently, most firms use an age or time-based approach to maintain and replace equipment. Up to 50% of spend using this approach may be wasted, according to ARC Advisory Group3. This study also found that 82% of failures occur randomly. New sensors will be deployed and real-time data will continue to swing upward across many industries. Businesses will be able to use this data to respond to events like power outages as they occur and use predictive analytics and historical information for preventative maintenance. Using this data will allow companies to move from a time-based or cyclical check schedules to an event-based ones that can detect even small changes in performance that may spell trouble.

6 – Converged Intelligence will improve our lives – the trend for companies to share datasets and provide APIs to their services will enable more collaborative experiences to help customers and differentiate companies from their competitors. Services like IFTT (If this, then that) will offer more and more connections, largely driven by community contributions. Partnerships like SolarCity, Nest, and the Tesla Powerwall will share data to produce synergies that can save money and reduce energy dependence. People will leverage internet of things (IoT) [see #1 above] devices and home automation like SmartThings to make us more comfortable. Whether it is automatically adjusting your lights, TV, and devices when you want to watch a movie or automatically adjusting your Thermostat when you leave and arm your alarm, connected living will grow.

A word of caution: data sharing may be open and driven by users opting-in, but in some instances it will be hidden and used to exploit customers without their knowledge.

5 – Data breaches will continue – Stakes are getting higher as hackers attempt to sway political campaigns, ransomware is on the rise, and data breaches are increasing. As data becomes more open and shareable, attack vectors are much greater and opportunities are higher. Enterprises need to make sure they are vetting cloud and hosted solutions properly to make sure they are secure, but they also need to realize that cloud providers may be able to provide economies of scale and make data safer than individual organizations can on their own.

4 – So…Security will have to get more proactive – As hackers start to use IoT and continue DDOS, companies need to work together to defend against threats. Tools like Watson for Cyber Security will user in this new era. We will move from predictive analytics into cognitive to discover threats, identify all assets exposed, and then perform a second-order threat analysis to see what other services may suffer or what may be targeted next. These tasks can be performed by machine clusters faster and more completely than an army of analysts.

3 – You’ll continue to hear about blockchain initiatives, but it will be mostly hype in 2017 – According to Gartner, Blockchain is nearing the peak of the hype cycle4. However, I think other items close to the peak, like home automation and IoT will see more adoption than blockchain. IMHO, these others can be adopted on a smaller scale and are more readily available to the general public than blockchain related deployments. Many people are forecasting that blockchain related tech won’t hit mainstream for another 5-10 years5. Nevertheless, the concept and some early uses of it are pretty interesting, such as Smart Contracts. Also, friendly FYI, something that uses a blockchain is not automatically anonymous, as in the case of bitcoin.

2 – The line between Data Scientist and analyst/programmer will blur even more – analysts and programmers will take special courses here and there to beef up their statistics and data science chops. I think the demand for data scientists will bifurcate in 2017: a subset of organizations will spring for data scientists and the high salaries they command; however, the majority of firms will push for their analysts or tools to do low level data science work. Tools like Tableau and R Studio are making it easier for analysts to dabble in statistical and predictive analytics. Firms, such as New Knowledge, are offering “Data Scientist as a Service”, and tons of online courses, e-books, and knowledge bases have sprung up to spread data science fundamentals to the masses.

1 – BYOT, Bring Your Own Tool, will continue to gain momentum – Enterprises can no longer place all their eggs in one basket when it comes to a BI or reporting tool. Tools such as Tableau have proven their ability to uproot entrenched stalwarts like IBM Cognos, and traditional BI tools appear stale and financially infeasible compared to a plethora of specialized, cheaper alternatives. Traditional BI tools will still have their place in firms that have enterprise-level agreements and are slow to change, but as more and more users demand features that these tools can’t support, or go out and acquire alternatives through “shadow procurement”, the traditional tools and expertise in firms will erode. It is now more important than ever for IT organizations to focus on architectures that make a wide array of data available to the entire organization regardless of device or access tool of choice. Good governance policies and data czars needs to focus on data quality, establishment and maintenance of metadata, and publishing best practices around the types of tools and reports/visualizations that are best for specific scenarios. Firms need to evaluate the benefits of having multiple tools and the flexibility and productivity it gives their employees vs. the supportability and procurement benefits of working with a smaller number of providers.

Music: Auld Lang Syne by Fresh Nelly, from Free Music Archive.

Sources:

Podcast: Play in new window | Download

Happy New Year! Thank you to all listeners and subscribers for your support this past year.

Happy New Year!
Thank you to all listeners and subscribers for your support this past year.