Displaying items by tag: big data - AIRS
Website Search
Research Papers
Articles
FAQs
Easy Profile - Search plugin
Courses & Exams
Pages
Specialized Search Engines
Events Calender
Upcoming Events

Searching video surveillance streaming for relevant information is a time-consuming mission that does not always convey accurate results. A new cloud-based deep-learning search engine augments surveillance systems with natural language search capabilities across recorded video footage.

The Ella search engine, developed by IC Realtime, uses both algorithmic and deep learning tools to give any surveillance or security camera the ability to recognize objects, colors, people, vehicles, animals and more.

It was designed with the technology backbone of Camio, a startup founded by ex-Googlers who realized there could be a way to apply search to streaming video feeds. Ella makes every nanosecond of video searchable instantly, letting users type in queries like “white truck” to find every relevant clip instead of searching through hours of footage. Ella quite simply creates a Google for video.

Traditional systems only allow the user to search for events by date, time, and camera type and to return very broad results that still require sifting, according to businesswire.com. The average surveillance camera sees less than two minutes of interesting video each day despite streaming and recording 24/7.

Ella instead does the work for users to highlight the interesting events and to enable fast searches of their surveillance and security footage. From the moment Ella comes online and is connected, it begins learning and tagging objects the cameras see.

The deep learning engine lives in the cloud and comes preloaded with recognition of thousands of objects like makes and models of cars; within the first minute of being online, users can start to search their footage.

Hardware agnostic, the technology also solves the issue of limited bandwidth for any HD streaming camera or NVR. Rather than push every second of recorded video to the cloud, Ella features interest-based video compression. Based on machine learning algorithms that recognize patterns of motion in each camera scene to recognize what is interesting within each scene, Ella will only record in HD when it recognizes something important. The uninteresting events are still stored in a low-resolution time-lapse format, so they provide 24×7 continuous security coverage without using up valuable bandwidth.

Ella works with both existing DIY and professionally installed surveillance and security cameras and is comprised of an on-premise video gateway device and the cloud platform subscription.

Source: This article was published i-hls.com

Published in Search Engine

Big data is an incredibly useful platform for any business, big or small. It allows brands to delve deeper into the information and insights that fuel their products, services and processes.

For example, you can use data collected on past product performance to make a more informed decision about a future launch or development cycle.

That said, big data as a whole isn’t exactly what you’d call accessible. For starters, you need to deploy the systems and processes to collect useful data.

Then, you need to have a team of data analysts and scientists to sort through it all and find actionable intel.

Finally, you need someone to take that practical data and put it to good use. A company executive just might not have a clear plan for, or understand the applications of, a niche data set.

This doesn’t mean applying big data is impossible. It just means it’s a potentially involved and time-consuming process.

Naturally, this can give organizations and decision makers enough doubt to avoid big data systems. The number of companies using predictive analytics to drive processes and make decisions, for instance, remains low at 29 percent, according to a 2016 PwC press release.

Adoption for these technologies and systems is rising, but not at the rate it could be.

So if you’re reluctant to get involved with big data, you’re not alone. Luckily, there are tools and resources to help you manage the transition.

1. Marketing ROI: Kissmetrics

Big data isn’t just about the data itself. This means that even if you collect customer or visitor data and use it to your advantage, you’re not necessarily using big data fully.

In today’s highly digital landscape, collecting and analyzing data is par for the course. Tracking the number of visitors or traffic referred to your website isn’t necessarily “big data.” An alarming 61 percent of employees say their company is not using big data solutions despite collecting data regularly.

This all relates to marketing, because you’re constantly reviewing data to inform your decisions and actions. The performance of a new product launch will tell you whether or not the resources that went into it were worthwhile.

If it fails to catch on, you know not to waste more resources on similar products or services. But again, this is just surface information — not true insights.

Kissmetrics is a data-powered tool that can help you boost your marketing ROI and processes. It does more than just track information like pageviews, heatmaps, demographics and more.

It actually churns that data and spits out usable intel. You can use the platform to create triggers that resonate with your audience and further boost customer engagement and behaviors.

2. Sales Calling: PhoneBurner

Big data has become a key player in the evolution of modern sales and marketing departments. Few examples illustrate this point better than the ways in which big data has been incorporated into modern sales calling software.

PhoneBurner is a power dialing company that automates the sales calling process so that, when a potential customer answers the phone, they are connected to a live sales agent (with no annoying pause in between).

If the call goes to voicemail, the power dialers leaves your pre-recorded voicemail and integrates the call status directly into a software dashboard so you can easily follow up via email at a later date.

3. Third-Party Integration: InsightSquared

One key obstacle in big data is fragmentation. There are so many tools, platforms, third-party portals and information streams that combining and parsing everything can be incredibly daunting.

InsightSquared is a data-driven tool that solves this with solid integration with third-party platforms and services. It can connect to popular enterprise solutions you’re likely already familiar with, such as Google Analytics, ZenDesk, QuickBooks, Salesforce and more.

The information is then mined and analyzed, making it more accessible to you, your teams and even parties who don’t work with data regularly. If you connect a customer relationship tool or CRM, it syncs up the data to offer efficient lead generation, customer tracking, pipeline forecasting and even profitability predictions.

4. Machine Learning and Predictive Analytics: IBM’s Watson Analytics

Big data solutions can offer some pretty amazing insights into your business, customers and strategies. Machine learning and predictive analytics are the way to achieve this.

IBM’s Watson Analytics relies on the IBM Watson machine learning API to deliver remarkable analysis of your data. More importantly, it automates the entire process intelligently to leave you more time to focus elsewhere.

The best feature of Watson is that it unifies all data analysis projects into a single channel or source. It can be connected to marketing and sales tools, finance and human resources, customer data and performance and much more.

Watson also employs a unique AI system to deliver “natural language” insights, which is a fancy way of saying the data it returns is easy to understand.

5. Credit and Payment Analytics: TranzLogic

Love it or hate it, credit card transactions and related payment systems can deliver boatloads of invaluable and necessary data. Even so, associated data streams are not always accessible — especially to smaller businesses or teams — and they can be super complex and confusing.

TranzLogic is designed to process this information and extract actionable intel. Want to measure sales performance and customer patterns or improve promotions? What about using payment data to improve loyalty programs and boost engagement across your customer base?

Also, it’s a turnkey tool that doesn’t require additional knowledge or experience. Even if your specialty lies beyond IT or data, you can still make sense of everything reported through TranzLogic.

6. Customer Feedback: Qualtrics

Research can be invaluable to any business or brand. In fact, by conducting studies, surveys and simple questionnaires, you can extract highly useful insights about your audience and products. Why do you think polls and surveys are so popular on social media networks? People love to share their opinions. This is even a great way to discover and hear about new ideas or concepts.

Qualtrics is a customer feedback solution for related big data sources. With the tool, you gain access to three different types of real-time insights: market, customer and employee trends.

This includes things like customer satisfaction, exit interviews and market research. You can even unlock academic research and mobile studies, too. There’s a lot of data here you could put to use.

7. Long-Term Data: Google Analytics

Google Analytics probably needs no introduction. What’s special about Google’s toolset is that it can help you extract long-term information and stats. For instance, you get to see where traffic is coming from and how that fluctuates over time.

With information like this, you can fill in any gaps. You could, for instance, use this to your advantage during the holidays to target common referrals through marketing and promotions.

Social media traffic is another great source of data, which is also tied into Google’s platform.

It’s more of a robust, multi-platform toolset that can be used to track the kind of information you’d want to track anyway.

More importantly, it’s instantly accessible through your browser.

Source: This article was published bigdata-madesimple.com By Kayla Matthews

Published in Business Research

 

Are you a liar? You bet you are but the real you is emerging through your online activities. What Big Data knows about the real you.

There are things about which we all lie. We lie about our innermost hopes, fears and desires. We lie to our friends, spouses, doctors, pollsters, even to ourselves. But our truth is being discovered because we willingly reveal it every day through our activities online.

It’s all being tracked and through big data a new picture about us is emerging which contradicts much of what we previously believed about each other.

Seth Stephens-Davidowitz, author of Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are

New York Times op-ed writer Seth Stephens-Davidowitz has studied this and reveals it all in his book, Everybody Lies. He says that data from the internet is like "digital truth serum," revealing how we really behave when no one's watching. Mike Collins talks with him about some of his findings.

Some highlights from the show:

On the events in Charlottesville

“There was a period when we thought we lived in a post-racism society. You could see online, even when people were telling pollsters politically correct truths, like they weren’t racists and didn’t care that Barack Obama was black, online they were telling a different story. They were making racists searches, usually for jokes mocking African-Americans with a shocking frequency. Stormfront is the biggest, most popular hate site in the United States. The demographics are young people, which you also saw at the events in Charlottesville. Neo-Nazis exist and I’ve known this for years because of internet research. Young people were becoming obsessed with neo-Nazis and the clear cause of it was Barack Obama.” 

On young adults age 19 to 21

“It’s a very impressionable group. It’s not a stupid or uneducated group. The most popular interest for Stormfront members is reading. They’re obsessed with philosophers, evolution and they’re political junkies. Many people say they join Stormfront because of a dating experience. Perhaps an African-American dated someone they wanted to date and it created this rage that led them to this material.”

Google reveals the most about the human psyche

“We’re in a habit of lying to make ourselves look better. That carries over to surveys, there’s no incentive to tell the truth. With Google there’s an incentive, you tell the truth, you get information you need. People are lying to surveys saying they aren’t racists. Compare that to the Google searches. It’s so clear there’s a very different truth about society that was being missed by the traditional way of understanding people. ”

“Google trends compare the rates of searches to different parts of the United States or world and when these searches are highest. You can learn interesting patterns. Anxiety has doubled in the last five years. It’s highest in Kentucky, Maine and rural areas. The recent rise in anxiety and panic attacks almost perfectly track rises in searches related to opioids.”

Guest

Seth Stephens-Davidowitz - New York Times op-ed contributor, visiting lecturer at The Wharton School, and a former Google data scientist. He is the author ofEverybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are

Read an excerpt from Everybody Lies. 
"The power in Google data is that people tell the giant search engine things they might not tell anyone else. Google was invented so that people could learn about the world, not so researchers could learn about people, but it turns out the trails we leave as we seek knowledge on the internet are tremendously revealing."

Seth on NPR's Hidden Brain podcast: What Our Google Searches Reveal About Who We Really Are
"I think there's something very comforting about that little white box that people feel very comfortable telling things that they may not tell anybody else about their sexual interests, their health problems, their insecurities. And using this anonymous aggregate data, we can learn a lot more about people than we've really ever known."

Source: This article was published wfae.org By ERIN KEEVER

 

Published in Science & Tech

Big data has and will change how advertisers work and businesses market.

There are plenty of words online about how big data will change every facet of our lives, and a substantial chunk of those words are devoted towards how big data will affect advertising. But instead of haphazardly leaping on the change bandwagon, advertisers need to sit down and understand what big data has changed and yet what still remains the same.

At its core, advertising is about communication as it seeks to inform consumers about a business’s product and services. But different consumers want to hear different messages, which becomes all the more important as new customers join the internet thanks to the growing popularity of mobile.

Big data can refine those messages, predict what customers want to hear with predictive analytics, and yield new insights in what customers want to hear. All of this is certainly revolutionary and will change how consumers and marketers approach advertising. But it will still be up to advertisers to create messages in the name of their clients.

Algorithms and targeting

Some things which many people do not think about as advertising are in fact a conflation of big data and marketing. Netflix is a terrific example of this. Netflix obviously does not have advertisements, but it heavily relies on algorithms to recommend shows to its viewers. These algorithms save Netflix $1 billion per year by reducing the churn rate and marketing the right shows to the right customers.

Netflix’s efforts to target consumers with the right shows is hardly unusual, as websites and online stores like YouTube, Amazon, or Steam do this all the time these days. But the key here is the reliance on algorithms to make targeting more accurate.

These algorithms require a constant stream of data to stay up to date. But now that data is everywhere. Internet users leave a constant stream of data not just on social media websites, but anywhere they go in the form of digital footprints.

This represents new opportunities and challenges for advertisers. On one hand, the digital footprints which everyone creates offers new insights to advertisers into what we truly want which can be more accurate than what we say on social media. But at the same time, advertisers do have to worry about protecting consumer privacy and security. This is not just a moral thing; advertisers or websites that are flagrantly cavalier with their user data will spark a backlash that will hurt business.

Advertising targeting has already been in place for some time now. But as advertisers collect more data, targeting will become more personalized and thus effective. Advertisers will fight not just to collect as much data as possible, but to collect data which accurately represents individual customers to market to their individual tastes.

Changing forms of advertising

Big data can uncover new information about each individual customer, but the advertiser must craft a message to appeal to said customer. But with these new insights, advertisers can entirely change how they approach marketing as they craft entirely new strategies.

This is not completely new. The rise in content marketing is often cited as a major beneficiary of big data, but content marketing as a concept is older than the Internet. Nevertheless, the rise in content marketing as well as other strategies like native advertising or the endless dance around search engine optimization.

These rising advertising strategies are fascinating because just as advertisers rely on data to craft new strategies, they give data right back to the consumer. Content marketing is all about giving consumers details about a business such as how they make food, what it is like to work there, and so on. By sharing this data, the company makes the customers feel like they are part of a group which knows common information. And in turn the customer ends up giving up his data to the company which lets it construct new advertising strategies.

This symbiosis between consumer and company shows that data is not just about cold analytics, but is about creating a bond between the two groups like all advertising sets out to do. Similarly, businesses must take the complexity of big data, analyze trends, and then create simple guidelines which their customer staff can use. All the advertising in the world will not make as big of an impression on a customer as one surly or confused customer representative.

Big data has and will change how advertisers work and businesses market to consumers through more personalized and targeted advertisement as well as creating new forms of advertising. But big data is less important than smart data and strategy. Business leaders who can break big data down into small chunks, come up with a smart strategy, and formulate an effective message will still thrive just as much as they would have in the past. In this way, big data is not quite the revolutionary change that many think.

This article is published as part of the IDG Contributor Network. Want to Join?

Published in Search Engine

The world runs on data. Businesses are inundated by it. From information received from mobile phones, sensor networks and Internet of Things-enabled devices, industries have a world of information at their fingertips. But making sense of all this data is no easy endeavour.

According to research firm IDC, companies spent over $20 billion in big data technology and services in 2015. Those same companies are also expanding the number of data types and sources they analyze. But what is big data anyway? We break down the concept and answer some questions you may be afraid to ask, in partnership with Cisco.

data

What is it?

In simple terms, “big data” refers to large or complex data sets. This includes unstructured data (typically text-heavy database and IoT information including dates, numbers, and facts) and structured data (easily organized and searchable content such as relational database information).

Regardless of industry sector or vertical, the average organization stores data from a wide range of sources: regular business transactions, IoT devices, email, videos, Internet traffic and even social media. Take manufacturing, for example. Every machine on the factory floor is constantly generating vital data such as production output, equipment health, and inventory levels. All of this makes managing real-time information a challenge.

Big data is a big deal. And while the concept of big data may seem simple, it’s actually a technology phenomenon that has the potential to unlock business value and innovation on a global scale.

And make no mistake, the companies that are able to harness the power of big data via data analytics, preparation, and management tools will be the ones that innovate, thrive, and survive in an increasingly competitive marketplace. But the typical organization still relies on aging and/or outdated virtualization tools and data management systems — making the need for a new way of managing data a strategic concern.

Indeed, as the world of data changes before our eyes, today’s organizations need real-time technology tools that can quickly identify the value in the data to solve critical business problems.

data

Why does it matter? 


The rise of big data means that the organizations that embark on a process of digital transformation — adopting digital technologies such as analytics tools and services to help manage, streamline and unify data sets into business operations— will be the ones that will be better prepared to make faster business decisions.

This includes the use of advanced tools such as user behaviour or predictive analytics. Employing either of these will help you make data connections more quickly, help you identify business trends, and conduct complex calculations, transactions or simulations to better respond to customer needs and demands.

Understanding big data — and its intrinsic value and potential for future business growth — will help businesses get more data-driven insights, foster better customer experiences and establish stronger internal and external communications and interactions.

Working with an industry leader such as Cisco can help you develop intelligent, integrated, and agile technology environments that help swiftly curate, automate, and manage digital data in real-time. Simply put, taking advantage of big data to analyze both structured and unstructured information, enables companies to extract deeper market value and develop tomorrow’s business insights, today.

Cisco works with leading companies to make sense of big data, helping them prioritize the information that matters to make better business decisions. They help businesses develop analytics strategies to maintain a competitive edge in an increasingly connected, global world. To learn more about how your company can benefit from big data, download this free report.

Source : huffingtonpost.ca

Published in Others

Search and big data analytics have evolved significantly over the last few years, and organizations are increasingly using these technologies to meet their mission-critical needs.

At the beginning of 2016, we were talking a lot about machine learning and semantic search and how they would be key developments in this space. 

Those have certainly been hot topics and continue to be areas that companies are seeking to exploit for their data-driven applications. 

But what will we be talking about in 2017 as it pertains to this space?

Here’s a look at five areas you can expect to hear more of this year and beyond.

Open Source Rises to the Top

Open source technologies are becoming more prominent in a wide range of use cases, from traditional enterprise search to log analytics, e-commerce search, and even government document search.

In fact, current data shows open source search engines have gained significant popularity because of flexibility, cost and features. According to DB-Engines, Elasticsearch and Solr — two open source search engines based on Lucene — top the list of leading commercial and open source search engines.

Just last month I discussed the features and limitations of Elasticsearch and Solr in this article.

With its growing use in the commercial and government space, the move to open source will continue to be a hot topic as organizations seek greater features, costs savings, and more flexibility with their search and big data analytics solutions.

Life Without Google Search Appliance

In early 2016, Google announced its end of support for the Google Search Appliance (by March 2019) as part of its strategic move to a cloud-based platform. Since then, many have begged the question: What now?

In my article last summer, I provided some tips on moving from the GSA and looked at some of the replacement options available at the time.

As we enter 2017, there are still no concrete details from Google about a new cloud-based search solution, so users are forging ahead with seeking out and comparing their existing alternatives.  With the March 2019 deadline getting closer, we’ll be hearing a lot more about the alternatives, and maybe even get word on Google’s cloud-based plans. 

Analytics Powered by Enterprise Data Lakes

Enterprises have a lot of data but how well they use it to derive insights is key to success. Over the last year, we’ve been hearing a lot of hype around enterprise data lakes (or enterprise data hubs) to bring together data silos and make the right data available to the right users at the right time. 

There is a wide variety of structured and unstructured data in enterprise data lakes. That said, search engines are the ideal tool for storing, processing, accessing, and presenting this data because they are schema-free and can scale to billions of records.

Data lakes’ search and analytics capabilities are nearly endless when we combine search engines, big data techniques, and visualization dashboards in groundbreaking use cases such as bioinformatics, precision agriculture, and precision medicine.

As data lakes continue to gain popularity as a way to store massive amounts of data and analytics, we’ll see organizations continuing to have the conversation in 2017 about how to best exploit this.

Search Engines Become 'Insight Engines1'

Just like Google, Cortana and Siri, search is becoming much more than just keyword matching.

We’re now heading into the age of search results personalization. Search engines are becoming personal digital assistants or as Gartner calls it, Insight Engines. This was made possible with big data analytics techniques like machine learning and predictive analytics.

It’s pervading the modern business world in a multitude of use cases from intranet search, to e-commerce search, recruiting, medical research, media and publishing, and many others. Organizations are just beginning to skim the surface on how real-time personalization is significantly enhancing their operations so we expect a lot of talk about how to implement this in the coming year.

Search Engine Scoring

We know that analyzing statistically valid scores helps increase search engine relevancy over time. But, while this method can significantly increase business value and the bottom line, not many organizations have started engine scoring or are implementing it effectively.

With that said, we are observing proven success with newer, better algorithms used in the scoring process, some of which are discussed in this article.  This is, without a doubt, a solid technique that will continue to grow and be discussed in the coming years as organizations seek to improve their user’s search experience.

In conclusion, with the rise of open source, massive volumes of structured and unstructured data, and the need to do complex analytics, we will continue to hear a lot on these topics throughout 2017 as search and big data continue to converge. 

Source : cmswire.com

Published in Search Engine

Evolution of digital marketing

As a digital marketing agency with 10 years’ experience, we have witnessed the increasing adoption of digital marketing among advertisers in Hong Kong. Annual internet advertising spending in Hong Kong grew from HK$1.5 billion in 2011 to HK$5.9 billion in 2016, a 293% growth in five years. Our interaction with customers also evolved over the years
from our previously educating and persuading advertisers to place internet ads, to currently advertisers proactively asking us what’s new on the internet whilst we keep on sending the latest trends to customers, to help them keep in pace with the market.

The pull strategy in search engine marketing (SEM) contributes 56% (HK$3.3 billion) to total internet spending in 2016. Complemented with push advertising like display ad, social media ad and soft selling approach from social media content and KOL, all these demonstrate a holistic digital marketing landscape.

Targeting and goal setting

In the early days of digital advertising, advertisers used to review the ad campaign performance based on the number of clicks and impressions. Nowadays, big data is crucial for advertisers to focus and effectively spend advertising dollars on the right target customers, thus achieving better value for money in their advertising spending. In response to market needs, we, equipped with the latest technology and knowledge, utilise tracking and web analytics tools to enhance advertisers’ digital ads and help them reach the target customers by devices, by geographics, by behavior targeting, by demographics, by search terms, by specific dates and time… As a local agency, we have built a knowledge base with data related to keywords in the web, such historical data which is gathered daily throughout the year and from different industries will be transformed into valuable market insights for advertisers.

With so much data “bombarding” the advertisers, it is important for them to set up marketing goals for their campaign before determining the type of digital marketing channels to be adopted. These goals can be leads generation, online sales, website visit, phone calls, apps downloads, brand awareness and returning customers etc. With clear goal setting, the planning and execution of advertising campaigns can be better measured, reviewed and optimised.

There are no limits to being new for digital media which is ever-evolving. New iMedia strives to be the trusted digital marketing partner of our customers with a view to building and sustaining mutually beneficial collaborations. Through conceptualising and executing differentiated one-stop digital marketing solutions for our customers, riding on our experience, creativity and technological expertise, we work side-by-side with our customers to achieve their business goals while living our core values.

Source : http://www.marketing-interactive.com/features/big-data-key-understanding-customers-better/

Published in Others

Search and big data analytics have evolved significantly over the last few years, and organizations are increasingly using these technologies to meet their mission-critical needs.

At the beginning of 2016, we were talking a lot about machine learning and semantic search and how they would be key developments in this space. 

Those have certainly been hot topics and continue to be areas that companies are seeking to exploit for their data-driven applications. 

But what will we be talking about in 2017 as it pertains to this space?

Here’s a look at five areas you can expect to hear more of this year and beyond.

Open Source Rises to the Top

Open source technologies are becoming more prominent in a wide range of use cases, from traditional enterprise search to log analytics, e-commerce search, and even government document search.

In fact, current data shows open source search engines have gained significant popularity because of flexibility, cost and features. According to DB-Engines, Elasticsearch and Solr — two open source search engines based on Lucene — top the list of leading commercial and open source search engines.

Just last month I discussed the features and limitations of Elasticsearch and Solr in this article.

With its growing use in the commercial and government space, the move to open source will continue to be a hot topic as organizations seek greater features, costs savings, and more flexibility with their search and big data analytics solutions.

Life Without Google Search Appliance

In early 2016, Google announced its end of support for the Google Search Appliance (by March 2019) as part of its strategic move to a cloud-based platform. Since then, many have begged the question: What now?

In my article last summer, I provided some tips on moving from the GSA and looked at some of the replacement options available at the time.

As we enter 2017, there are still no concrete details from Google about a new cloud-based search solution, so users are forging ahead with seeking out and comparing their existing alternatives.  With the March 2019 deadline getting closer, we’ll be hearing a lot more about the alternatives, and maybe even get word on Google’s cloud-based plans. 

Analytics Powered by Enterprise Data Lakes

Enterprises have a lot of data but how well they use it to derive insights is key to success. Over the last year, we’ve been hearing a lot of hype around enterprise data lakes (or enterprise data hubs) to bring together data silos and make the right data available to the right users at the right time. 

There is a wide variety of structured and unstructured data in enterprise data lakes. That said, search engines are the ideal tool for storing, processing, accessing, and presenting this data because they are schema-free and can scale to billions of records.

Data lakes’ search and analytics capabilities are nearly endless when we combine search engines, big data techniques, and visualization dashboards in groundbreaking use cases such as bioinformatics, precision agriculture, and precision medicine.

As data lakes continue to gain popularity as a way to store massive amounts of data and analytics, we’ll see organizations continuing to have the conversation in 2017 about how to best exploit this.

Search Engines Become 'Insight Engines'

Just like Google, Cortana and Siri, search is becoming much more than just keyword matching.

We’re now heading into the age of search results personalization. Search engines are becoming personal digital assistants or as Gartner calls it, Insight Engines. This was made possible with big data analytics techniques like machine learning and predictive analytics.

It’s pervading the modern business world in a multitude of use cases from intranet search, to e-commerce search, recruiting, medical research, media and publishing, and many others. Organizations are just beginning to skim the surface on how real-time personalization is significantly enhancing their operations so we expect a lot of talk about how to implement this in the coming year.

Search Engine Scoring

We know that analyzing statistically valid scores helps increase search engine relevancy over time. But, while this method can significantly increase business value and the bottom line, not many organizations have started engine scoring or are implementing it effectively.

With that said, we are observing proven success with newer, better algorithms used in the scoring process, some of which are discussed in this article.  This is, without a doubt, a solid technique that will continue to grow and be discussed in the coming years as organizations seek to improve their user’s search experience.

In conclusion, with the rise of open source, massive volumes of structured and unstructured data, and the need to do complex analytics, we will continue to hear a lot on these topics throughout 2017 as search and big data continue to converge. 

About the Author

Kamran Khan is co-founder, president and CEO of Search Technologies, a company dedicated to developing, implementing and supporting search and big data solutions. He has been developing, supporting, selling and managing in the computer software/services industry for 25 years with a focus on search engine technology.

Author : Kamran Khan

Source : http://www.cmswire.com/big-data/search-big-data-analytics-in-2017-5-hot-topics/

Published in Science & Tech

As more personal information is collected up by ever-more-powerful computers, giant sets of data – big data – have become available for not only legitimate uses but also abuses.

Big data has an enormous potential to revolutionize our lives with its predictive power. Imagine a future in which you know what your weather will be like with 95 percent accuracy 48 hours ahead of time. But due to the possibility of malicious use, there are both security and privacy threats of big data you should be concerned about, especially as you spend more time on the Internet.

What threats are emerging? How should we address these growing concerns without denying society the benefits big data can bring?

The size of the potential problem

First of all, due to the sheer scale of people involved in big data security incidents, the stakes are higher than ever. When the professional development system at Arkansas University was breached in 2014, just 50,000 people were affected. That’s a large number, but compare it with 145 million people whose birth dates, home and email addresses, and other information were stolen in a data breach at eBay that same year.

From the perspective of a security professional, protecting big data sets is also more daunting. This is partly due to the nature of the underlying technologies used to store and process the information.

Big data companies like Amazon heavily rely on distributed computing, which typically involves data centers geographically dispersed across the whole world. Amazon divides its global operations into 12 regions each containing multiple data centers and being potentially subject to both physical attacks and persistent cyberattacks against the tens of thousands of individual servers housed inside.

Difficulties with access control

One of the best strategies for controlling access to information or physical space is having a single access point, which is much easier to secure than hundreds of them. The fact that big data is stored in such widely spread places runs against this principle. Its vulnerability is far higher because of its size, distribution and broad range of access.

In addition, many sophisticated software components do not take security seriously enough, including parts of companies’ big data infrastructure. This opens a further avenue of potential attack.

For instance, Hadoop is a collection of software components that allows programmers to process a large amount of data in a distributed computing infrastructure. When first introduced, Hadoop had very basic security features suitable for a system used by only a few users. Many big companies have adopted Hadoop as their corporate data platform, despite the fact that its access control mechanism wasn’t designed for large-scale adoption.

Consumer demand drives security and privacy

For consumers, then, it is critical to demand a heightened level of security through vehicles such as terms and conditions, service level agreements, and security trust seals from organizations collecting and using big data.

What can companies do to protect personal information? Countermeasures such as encryption, access control, intrusion detection, backups, auditing and corporate procedures can prevent data from being breached and falling into the wrong hands. As such, security can promote your privacy.

At the same time, heightened security can also hurt your privacy: it can provide legitimate excuses to collect more private information such as employees’ web surfing history on work computers.

When law enforcement agencies collect information in the name of improved security, everyone is treated as a potential criminal or terrorist, whose information may eventually be used against them. The authorities already know a lot about us but could ask companies such as Apple, Google and Amazon to provide more intelligence such as a decrypted version of our data, what search terms we are using and what we are buying online.

The fundamental security principle used to justify this type of blanket surveillance (which is now more affordable and feasible due to the use of big data technologies) is “nobody can be trusted.” Once collected, those data join the rest of the information in being susceptible to abuse and breaches, as demonstrated in snooping incidents involving National Security Agency employees.

And yet when used properly, big data can help enhance your privacy by allowing more information to be leveraged and eventually improve the quality (especially, the accuracy) of intelligence on potential attacks and attackers in cyberspace.

For example, in an ideal world we don’t have to worry about fraudulent emails (also called phishing) because a big data analytics engine would be able to pick out malicious emails with pinpoint accuracy.

How big data is used – for you or against you

There are also other privacy concerns about big data. Companies are eager to deliver targeted advertising to you and tracking your every online move. Big data makes this tracking easier to do, less expensive and more easily analyzed.

A service like IBM’s Personality Insights can build a detailed profile of you, moving well beyond basic demographics or location information. Your online habits can reveal aspects of your personality, such as whether you are outgoing, environmentally conscious, politically conservative or enjoy travel in Africa.

Industry representatives make benign claims about this capability, saying it improves users’ online experiences. But it is not hard to imagine that the same information could be very easily used against us.

For example, insurance companies could start questioning coverage to consumers based on these sorts of big-data profiles, which has already begun to happen.

Banning large-scale data collection is unlikely to be a realistic option to solve the problem. Whether we like it or not, the age of big data has already arrived. We should find the best way of protecting our privacy while allowing legitimate uses of big data, which can make our lives much safer, richer and more productive.

For example, when used legitimately and securely, big data technology can drastically improve the effectiveness of fraud detection, which, in turn, frees us from worrying about stolen identities and potential monetary loss.

Transparency is the key to letting us harness the power of big data while addressing its security and privacy challenges. Handlers of big data should disclose information on what they gather and for what purposes.

In addition, consumers must know how the data is stored, who has access to it and how that access is granted. Finally, big data companies can earn public trust by giving specific explanations about the security controls they use to protect the data they manage.

Author : Jungwoo Ryoo

Source : http://theconversation.com/big-data-security-problems-threaten-consumers-privacy-54798

Published in Internet Privacy

Wow! What a year 2016 has been. The big data industry has significant inertia moving into 2017. In order to give our valued readers a pulse on important new trends leading into next year, we here at insideBIGDATA heard from all our friends across the vendor ecosystem to get their insights, reflections and predictions for what may be coming. We were very encouraged to hear such exciting perspectives. Even if only half actually come true, Big Data in the next year is destined to be quite an exciting ride. Enjoy!

IT becomes the data hero. It’s finally IT’s time to break the cycle and evolve from producer to enabler. IT is at the helm of the transformation to self-service analytics at scale. IT is providing the flexibility and agility the business needs to innovate all while balancing governance, data security, and compliance. And by empowering the organization to make data-driven decisions at the speed of business, IT will emerge as the data hero who helps shape the future of the business. – Francois Ajenstat, Chief Product Officer at Tableau

In 2017, we’re going to see analytics do more than ever to drive customer satisfaction. As the world of big data exploded, business leaders had a false comfort in having these mammoth data lakes which brought no value on their own when they were sitting unanalyzed. Plain and simple, data tells us about our customers — it’s how we learn more about customers and how to better serve them. As today’s customers expect a personalized experience when interacting with a business, we’re going to see customer analytics become the spinal cord of the customer journey, creating touch points at every level of the funnel and at every moment of interaction. – Ketan Karkhanis, SVP and GM of the Salesforce Analytics Cloud

Knowing the Unknown Unknowns – Enterprises that apply Big Data analytics across their entire organizations, versus those that simply implement point solutions to solve one specific challenge, will benefit greatly by uncovering business or market anomalies or other risks that they never knew existed. For example, an airline using Big Data to improve customer satisfaction might uncover hiccups in its new aircraft maintenance scheduling that could impact equipment availability. Or, a mobile carrier looking to grow its customer base might discover ways to improve call center efficiency. Discovering these unknown unknowns can enable organizations to make changes or fix issues before they become a problem, and empower them to make more strategic business decisions and retain competitive agility. – Laks Srinivasan, Co-COO, Opera Solutions

Democratization of Data Analysis – In 2017 I believe that C-suite executives will begin to understand that there is a real gap between their data visions and the ability of their enterprise to move data horizontally throughout the organization. In the past, big data analysis has lagged in implementation compared to other parts of the business being transformed by advanced technology such as supply chains. I believe companies will begin to place different data storage systems into the hands of end users in a fast and efficient manner that has user self-direction and flexibility, democratizing data analysis. –  Chuck Pieper, CEO, Cambridge Semantics

The battleground for data-enriched CRM will only continue to heat up in 2017. Data is a great way to extend the value proposition of CRM to businesses of all sizes, especially those in the small-to mid-size range. By providing pre-populated data sets, the amount of “busy work” done by sales and other CRM users is reduced, and the better the data, the more effective individuals can be every moment of the day. A lot of M&A as well as in-house development and partnerships will fuel more data-powered CRM announcements in 2017. The key, of course, is seeing which providers provide the most seamless and most sensible use cases out of the box for their customers.” – Martin Schneider, Vice President of Corporate Communications, SugarCRM

In 2017 (and 2018), streaming analytics will become a default enterprise capability, and we’re going to see widespread enterprise adoption and implementation of this technology as the next big step to help companies gain a competitive advantage from their data. The rate of adoption will be a hockey stick model and ultimately take half the time it has taken Hadoop to rise as the default big data platform over the past six years. Streaming analytics will enable the real-time enterprise, serving as a transformational workload over their data platforms that will effectively move enterprises from analyzing data in batch-mode once or twice a day to the order of seconds to gain real-time insights and taking opportunistic actions. Overall, enterprises leveraging the power of real-time streaming analytics will become more sensitive, agile and gain a better understanding of their customers’ needs and habits to provide an overall better experience. In terms of the technology stack to achieve this, there will be an acceleration in the rise and spread of the usage of open source streaming engines, such as Spark Streaming and Flink, in tight integration with the enterprise Hadoop data lake, and that will increase the demand for tools and easier approaches to leverage open source in the enterprise. – Anand Venugopal, Head of Product, StreamAnalytix, Impetus Technologies

The unique value creation for businesses comes not just from processing and understanding transactions as they happen and then applying models, but by actually doing it before the consumer, or the sensor, logs in to do something. I predict we will quickly move from post-event and even real-time to preemptive analytics that can drive transactions instead of just modifying or optimizing them. This will have a transformative impact on the ability of a data-centric business to identify new revenue streams, save costs and improve their customer intimacy. – Scott Gnau, Chief Technology Officer, Hortonworks

Text analytics will be subsumed by ML/AI in 2017. The terms Text Mining and Text Analytics never really gained the kind of cachet and power in the marketplace that most of us hoped they would. This year will see the terms be subsumed by ML/AI and they’ll become component pieces of AI. – Jeff Catlin, CEO, Lexalytics

IT will start automating the choices for data management and analysis, leading to standardized data prep, quality, and governance. BI tools have been making more decisions for people and automating more processes. The knowledge for doing this — e.g., choosing one chart type over another — was embedded into the tools themselves. Data prep and management tends to be different, because the required rules are specific to the business requirements rather than being inherent in the data. Rule-based data management will enable IT to define rules that the business uses in its analytics processes, making business analysts more productive while still ensuring reliability and reproducibility. For a use case, consider a data scientist who sources data externally, and lets the data tools automatically choose which enterprise data prep and cleansing processes need to be applied. – Jake Freivald, Vice President, Information Builders

Managing the sprawl: Self-service analytics technologies have put analysis into the hands of more users and as a byproduct, led to the creation of derivative artifacts: additional datasets and reports, think Tableau workbooks and Excel spreadsheets. These artifacts have taken on a life of their own. In 2017, we will see a set of technologies begin to emerge to help organize these self-service data sets and manage data sprawl. These technologies will combine automation and encourage organic understanding, guided by well thought-out, but broadly applicable policies. – Venky Ganti, CTO, Alation

We will move from “only visual analysis” to include the whole supply chain of data. We will eventually see visualizations in unified hubs that show us more data, including asset management, catalogs, and portals, as well as visual self-service data preparation. Further, visualizations will become a more common means of communicating insights. The result of this is that more users will have a deeper understanding of the data supply chain, and the use of visual analysis will increase. – Dan Sommer, Senior Director and Market Intelligence Lead, Qlik

Artificial Intelligence

AI, ML, and NLP innovations have really exploded this past year but despite a lot of hype, most of the tangible applications are still based on specialized AI and not general AI. We will continue to see new use-cases of such specialized AI across verticals and key business processes. These use-cases would primarily be focused on the evolutionary process improvement side of the digital transformation. Since the efficiency of ML is based on constant improvement through better and wider training data, this would only add to the already expanding size of the data enterprise needs to manage. Good data management policies would be key to achieving a scalable and sustainable AI vision. For the business users this would mean better access to actionable intelligence, and elimination of routine tasks that can be delegated to the bots. For users who want to stay relevant in the new economy, this would allow them transform their roles in to knowledge workers that focus on tasks that can still only be done based on the general intelligence. Business users that can train the AI models would also be very hot commodity in the economy of future. – Vishal Awasthi, Chief Technology Officer, Dolphin Enterprise Solutions Corporation

Why machine-led, human-augmented intelligence is the next tech revolution – In 2017, more C-suite executives are going to prioritize data-driven business outcomes. As C-level executives see the potential for analytics, they’ve begun to show greater participation in getting analytics off the ground in their organizations, and I expect they’ll be leading the charge this year to ensure insights permeate every level and department of the business. All of the true technological revolutions have happened when people at a mass scale are empowered. So, shifting data science from an ivory tower function to giving everyone in an organization access to advanced, interactive AI will help each employee become smarter and more productive. It’s becoming clearer that when data can inform each and every decision a business user is making, businesses are going to see a real a competitive advantage and business outcome. – Ketan Karkhanis, SVP and GM of the Salesforce Analytics Cloud

Graph-Based Databases for Emerging Tech – The key applications companies are exploring — IoT, machine learning and AI – will be constrained by relational database technology. These areas will move towards sitting on top of graph-based architecture, which by definition, expands much more quickly in response to the output of those learnings. If you think of AI, it cycles back on data many, many times, and once it has a conclusion, it asks for more information. If that information in a relational format is not already there, all those AI, IoT and machine learning programs stop. But if it’s on a graph-based arch it automatically allows itself those multiple levels of joins to bring in more information. That will help unleash the real potential of some of those new technologies. – –  Chuck Pieper, CEO, Cambridge Semantics

The symbiotic relationship between man and machine will enable better decisions. Machines will never replace man, but they will empower and complement the data-driven efforts of workers in the coming years, especially as data becomes more accessible across departments and organizations. The democratization of data, the self-service movement and data’s continued simplicity means more people will be leveraging it in more applications – paving the way for a better man vs. machine relationship. For example, IBM Watson can go through medical papers, research and journals and then present top choices, but only a trained doctor can make the right decision for a specific patient. Adding to that, the reskilling of the workforce through nanodegrees will simplify data even further. Technology is sharpening the workforce and putting the power of data into the hands of business users – AI and machine-learning will only help them achieve more.” – Laura Sellers, VP of Product Management, Alteryx

My prediction about Big Data is that it will be subsumed into the topic of AI, as big data is an enabler of AI not an end in itself. The lack of focus on big data will actually let the field mature with only the serious players and result in much better business results. – Anil Kaul, Co-Founder and CEO of Absolutdata

Companies will stop reinventing the AI wheel. More and more companies are applying artificial intelligence and deep learning into their applications, but a unified, standardized engine to facilitate this process has lagged behind. Today, to insert AI into robots, drones, self-driving cars, and other devices, each company needs to reinvent the wheel. In 2017, we will see the emergence of unified AI engines that will eliminate or greatly mitigate these inefficiencies and propel the formation of a mature AI tech supplier industry.” – Massimiliano Versace, cofounder and CEO, Neurala

AI will (still) be the new black. One topic that was covered ad nauseam in 2016 was AI. While it’s important to be cautious about all of the AI hype (especially when it comes to use cases that sound like science fiction), the reality is that this technology is going to evolve even faster from here on out. It’s just in the past few years that innovative business-to-business companies have started using AI to achieve specific business outcomes. Keynoters at this year’s IBM World of Watson conference highlighted ways in which it is already delivering impressive business value, as well as examples of how it might help a CEO decide whether to buy a competitor, or help a doctor diagnose a patient’s symptoms in just the next three to five years. – Sean Zinsmeister, Senior Director of Product Marketing, Infer

Artificial intelligence (AI) initiatives will continue, but in the vein of commoditisation – AI is garnering interest in the legal sector, but a closer inspection of the tools and apps being made available reveal that they are presently more similar to commoditised legal services in the form of packaged, low cost modules for areas such as wills, contracts, pre-nuptials and non-disclosure agreements for the benefit of consumers. Undoubtedly, AI offers tremendous potential and some large law firms have launched initiatives to leverage the technology. However, there’s a significant amount of work to be done in defining the ethical and legal boundaries for AI, before the technology can truly be utilised for delivering legal services to clients with minimal human involvement. Until then, in 2017 and perhaps for a few more years yet, we will continue to see incremental innovative efforts to leverage the technology, but in the vein of commoditisation – similar to what we have seen in the last 12 months. – Roy Russell, CEO of Ascertus Limited

AI and analytics vendor M&A activity will accelerate — There’s no doubt that there’s a massive land grab for anything AI, machine learning or deep learning. Major players as diverse as Google, Apple, Salesforce and Microsoft to AOL, Twitter and Amazon drove the acquisition trend this year. Due to the short operating history of most of the startups being acquired, these moves are as much about acquiring the limited number of AI experts on the planet as the value of what each company has produced to date. The battle for AI enterprise mindshare has clearly been drawn between IBM Watson, Salesforce Einstein, and Oracle’s Adaptive Intelligent Applications. What’s well understood is that AI needs a consistent foundation of reliable data upon which to operate. With a limited number of startups offering these integrated capabilities, the quest for relevant insights and ultimately recommended actions that can help with predictive and more efficient forecasting and decision-making will lead to even more aggressive M&A activity in 2017. – Ramon Chen, CMO, Reltio

AI and machine learning are already infiltrating the workforce across a multitude of industries. In fact, when it comes to HR and people management, more and more companies are starting to deploy technologies that bring transparency to data around the work employees do. This is creating huge opportunities for businesses to leverage frequent touch points, check-ins and opportunities to provide feedback to employees and get a holistic picture of what’s driving work. In 2017 we can expect to see data and analytics used more in HR and management to help visualize behaviors of employees, from the time they were hired to their success down the road, and understand why they have been so successful. By using machine learning companies can focus on building teams to support long-term goal achievement, instead of frantically hiring to fill immediate needs. – Kris Duggan, CEO of BetterWorks

Artificial intelligence (AI) is rapidly becoming more accessible. Previously, you needed a lot of training to implement AI, but this is becoming less and less true as technology becomes more intelligent. Over the next several years, we can expect AI to become more of a commodity and companies like Google and Microsoft will make it extremely easy for developers to analyze large amounts of data on their platform. Once that data analysis is done, developers will be able to implement processes based on those results, which is essentially AI. In the next year we can expect that AI will become much easier to implement for developers via API calls into their applications. – Kurt Collins, Director of Technology Evangelism & Partnerships, Built.io

This year we saw customer interactions evolve from traditional question and answer dialogues, to intelligent machines now enhancing the process and experience. Machines are learning patterns and providing answers to customers to help eliminate some of the mundane tasks that customer service agents used to handle; and intelligent machine personas like the Alexa in the Amazon Echo and Siri in various Apple devices, are paving the way. In 2017, we’ll see more capabilities when it comes to artificial intelligence and customer service like Alexa triggering a call from contact center based on a question about online order status, thermostats submitting a trouble ticket after noticing a problem with the heater, or Siri searching through a cable company’s FAQ to answer to a commonly asked question about internet service troubleshooting. However, one thing will always remain true – human interactions will still be critical when dealing with complex situations or to provide the empathy that is needed in customer service. – Mayur Anadkat, VP of Product Marketing, Five9

For some, the mere mention of artificial intelligence (AI) corresponds to a fashion return from decades ago. So yes, those wide ties are back, and in 2017 we’ll see the rapid adoption of AI in the form of relatively straightforward algorithms deployed on large data sets to address repetitive automated tasks. First a brief history of AI. In the 1960s, Ray Solomonoff laid the foundations of a mathematical theory of AI, introducing universal Bayesian methods for inductive inference and prediction. In 1980 the First National Conference of the American Association for Artificial Intelligence (AAAI) was held at Stanford and marked the application of theories in software. AI is now back in mainstream discussions and the umbrella buzzword for machine intelligence, machine learning, neural networks, and cognitive computing. Why is AI a rejuvenated trend? The three V’s come to mind: Velocity, Variety and Volume. Platforms that can process the three V’s with modern and traditional processing models that scale horizontally providing 10-20X cost efficiency over traditional platforms. Google has documented how simple algorithms executed frequently against large datasets yield better results than other approaches using smaller sets. We’ll see the highest value from applying AI to high volume repetitive tasks where consistency is more effective than gaining human intuitive oversight at the expense of human error and cost. – John Schroeder, Chairman and Founder, MapR

The Cognitive Era of computing will make it possible to converge artificial intelligence, business intelligence, machine learning and real-time analytics in various ways that will make real-time intelligence a reality. Such “speed of thought” analyses would not be possible were it not for the unprecedented performance afforded by hardware acceleration of in-memory data stores. By delivering extraordinary performance without the need to define a schema or index in advance, GPU acceleration provides the ability to perform exploratory analytics that will be required for cognitive computing. – Eric Mizell, Vice President, Global Solutions Engineering, Kinetica

We expect three of the well-funded ML/AI companies to go out of business, while a number of the lesser funded companies will not get off the ground. In addition, we’ll lose more than a few pure-play text analytics companies as ML/AI subsumes more and more of the functionality. The influx of cash isn’t infinite, and companies will need to learn the importance of ROI/TCO analysis. Do you really need a slide or firepole between floors? No. Do you need to have budget for things like, say, salary and advertising, yes. Another common failure will be over-investing in the engineering aspect of the business. While it’s critical to have a great product, people also need to hear about it. If you can’t clearly articulate your business necessity, then it doesn’t matter how cool the product is. – Jeff Catlin, CEO, Lexalytics

Deep Learning will move out of the hype zone and into reality. Deep learning is getting massive buzz recently. Unfortunately, many people are once again making the mistake of thinking that deep learning is a magic, cure-all bullet for all things analytics. The fact is that deep learning is amazingly powerful for some areas such as image recognition. However, that doesn’t mean it can apply everywhere. While deep learning will be in place at a large number of companies in the coming year, the market will start to recognize where it really makes sense and where it does not. By better defining where deep learning plays, it will increase focus on the right areas and speed the delivery of value. – Bill Franks, Chief Analytics Officer, Teradata

By the end of 2017, the idea of deep learning will have matured and true use cases will emerge. For example, Google uses it to look at faces and then determine if the face is happy, sad, etc. There are also existing use cases in which the police is using it to compare the “baseline” facial structure to “real time” facial expressions to determine intoxication, duress or other potentially adverse activities. – Joanna Schloss, Director of Product Marketing, Datameer

The future of all enterprise processes will be driven by Artificial Intelligence, which requires the highest quality of data to be successful. AI is where all business processes are headed; however, with the recent push of AI technology advancements for businesses – many companies have not addressed how they will ensure that the data their AI models are built on is high quality. Data quality is key to pulling accurate insights and actions and in 2017, we will see more companies focus on solving the challenge of maintaining accurate, valuable data, so that AI technology lives up to its promise of driving change and improvement for businesses. – Darian Shirazi, CEO and Co-Founder, Radius

Prediction: Artificial Intelligence will Create New Marketing Categories, Like the B2B Business Concierge. In 2017, AI will allow marketers to create highly personalized ads tailored to buyer’s specific interests in real-time through superior and infinite knowledge. AI will also make mass email marketing tools obsolete (and the resulting spam email), automatically scanning out the “bad” leads and creating custom, personalized communication instead. As AI continues to advance, we can expect to see the recommendation engines that power companies like Netflix and Amazon develop specifically for the B2B market. This will start to pave the way for a B2B business concierge – a completely automated and customized buyer’s journey throughout the funnel that is driven by AI. – Chris Golec, Founder & CEO, Demandbase

AI-as-a-Service will take off: In 2016 AI was applied to solve known problems. And as we move forward, we will start leveraging AI to gain greater insights into ongoing problems that we didn’t even know existed. Using AI to uncover these “unknown unknowns” will free us to collaborate more and tackle new, interesting and life-changing challenges … AI will amplify humans: We have made enormous leaps forward to build machines capable of understanding and simulating human tasks, even mimicking our thought process. 2017 will be the year of knowledge-based AI, as we develop systems based on knowledge, which learn and retain knowledge of prior tasks, rather than pure automation of tasks we want performed. This will completely disrupt the way we work as human capabilities are amplified by machines that learn, remember and inform … AI will be seen as solving the workforce crisis, not creating it: As the baby boomer generation retires, enterprises are on the brink of losing significant institutional mindshare and knowledge. With the astronomical price tag of losing these workers, enterprises are turning to knowledge management and machine learning to train AI to capture institutional knowledge and act on our behalf. In the coming year and beyond, we will see AI adoption not only come from technological need, but also from the need to capture current employee insights and know-how. – Abdul Razack, SVP & Head of Platforms, Big Data and Analytics, Infosys

How Does AI Fit in an Enterprise? Whatever the industry, we can take better advantage of AI by making our current work tools — apps, medical devices, supply chain systems — much better through machine learning. The key is in the delivery — in other words, the “operationalization” of the analytics. I like to use the analogy of the self-driving car. The best autonomous vehicle systems will surely be able to handle the driving task in typical conditions; there are lots of little decisions to be made, but they are straightforward and easy to make. It’s when conditions become more challenging that the magic happens; the car will not only know when a human should intervene but also will smoothly transfer control to the driver and then back again to the machine. We’re on the cusp of where our everyday work apps and devices shift from repositories to assistants — and we need to start planning for it. Today, employees — or their boss — determine the next set of tasks to focus on. They log into an app, go through a checklist, generate a BI report, etc. In contrast, AI could automatically serve up 50% (or more) of what a specific employee needs to focus on that day, and deliver those tasks via a Slack app or Salesforce Chatter. Success will be found in making AI pervasive across apps and operations and in its ability to affect people’s work behavior to achieve larger business objectives. – Dan Udoutch, CEO, Alpine Data

Many Fortune 500 brands are already using chatbots, and many more are developing them as we speak. What’s ahead for the industry? Though it may not seem sexy, the next year will be a foundational one when it comes to applying AI. Chatbots are only as valuable as the relationships they build and the scenarios they can support, so their level of sophistication will make or break them. Investing in AI is only one piece of the puzzle, and 2017 will be the year that companies need to expand their AI initiatives while also doubling down on investing to improve them with new data streams and integration across channels. – Dave O’Flanagan, CEO, Boxever

The AI Hypecycle and Trough of Disillusionment, 2017: IDC predicts that by 2018, 75 percent of enterprise and ISV development will include cognitive/AI or machine learning functionality in at least one application. While dazzling POCs will continue to capture our imaginations, companies will quickly realize that AI is a lot harder than it appears at first blush and a more measured, long-term approach to AI is needed. AI is only as intelligent as the data behind it, and we are not yet at a point where enough organizations can harvest their data well enough to fulfill their AI dreams. – Ashley Stirrup, CMO, Talend

Hybrid Deep Learning systems. In 2017 we’ll see the rise of embedded analytics, optimized by cloud-based learning. The hybrid architectures used by autonomous vehicles – systems embedded within the vehicle to make numerous decisions per second, augmented by cloud-based learning platforms capable of optimizing decisions across the fleet – will serve as the foundation for the next generation of IoT machines. – Snehal Antani, CTO, Splunk

The focus will shift from “advanced analytics” to “advancing analytics.” Advanced analytics will continue to grow, and eventually be brought into self-service tools. With more users advancing their analytics, Artificial Intelligence (AI) might play a bigger role in organizations. But that means AI will also need to have high levels of usability as well, since users will need it to augment their analyses and business decisions. – Dan Sommer, Senior Director and Market Intelligence Lead, Qlik

Big Data

Many companies have ideas and initiatives around big data, but not a solid understanding of how it, along with the subsequent insights, will help them better the business or develop new solutions. Technology suddenly gave organizations the ability to process large amounts of data at a high frequency. That together with the move to mobile (as every consumer has one or more devices that they are constantly online with) drives a lot of data – whether through social networks, search engines or more. You have the information but it needs to be taken one step further – you need to analyze it. The question for big data is “what can I learn from it? Where can I make meaningful insights? – Dr. Werner Hopf, CEO and Archiving Principal, Dolphin Enterprise Solutions Corporation

Big data becomes fast and approachable. Sure, you can perform machine learning and conduct sentiment analysis on Hadoop, but the first question people often ask is: “How fast is the interactive SQL?” SQL, after all, is the conduit to business users who want to use Hadoop data for faster, more repeatable KPI dashboards as well as exploratory analysis. In 2017, options will expand to speed up Hadoop. This shift has already started, as evidenced by the adoption of faster databases like Exasol and MemSQL, Hadoop-based stores like Kudu, and technologies that enable faster queries. – Dan Kogan, director of product marketing at Tableau

Big Data, More Data, Fragmented Data – As we amass more enterprise data and blend third-party data, we create greater opportunity for insight and impact. However, let’s be honest. All companies are not created equal when it comes to their Big Data learning curves and sophistication. We will continue to see companies investing in, yet struggling with building their data layers.  Opera Solutions expects to see more attention and focused on data flow, data layers, and the emergence of the insights layer. – Georges Smine, VP Project Marketing, Opera Solutions

Moving into SMB – I see the advent of the big data analytics and discovery for SMB to start taking root in 2017. Big, rich, data environments such as pharma, healthcare, life sciences, financial services, insurance are the current industries leading big data analytics but graph-based databases can also be used by small companies, where you don’t want to spend your time coding and recoding every time you change your mind about what it is you want to look for. –  Chuck Pieper, CEO, Cambridge Semantics

Despite the hype and promise of big data and AI, few clear examples exist today where these technologies impact our lives on a daily basis. Serving relevant ads to website visitors and detecting fraud in credit card transactions come to mind. These companies have invested in big data and machine learning for years, which has allowed them to develop solid data architectures. Companies that have lived with NoSQL databases for more than a year know that ignoring data model design and instead leaning too heavily on the flexible, schema-free capabilities of these databases leads to poorly performing applications, difficult maintainability, and ultimately rework. In 2017, I predict the discipline of data modeling will gain strength as a sought-after skill set and project activity, particularly for companies dedicated to building impactful data strategies. Tools, such as well-designed industry clouds provide the professional data model design necessary for long-term success.” – J.J. Jakubik, Chief Architect, Vlocity

The sheer volume of data generated by applications and infrastructure will only increase, resulting in data overload. For the first time, IT Operations teams will embrace an algorithmic approach – also known as Algorithmic IT Operations, or AIOps – to detect signal from noise to ensure successful service delivery. AIOps platforms will provide IT Operations teams with situational awareness and diagnostic capabilities that were not previously possible using manual, non-algorithmic techniques.” – Michael Butt, Senior Product Marketing Manager at BigPanda

We’re living in a big data glut. But in 2017, we’ll see data become more intelligent, more useable, and more relevant than ever. The cloud has opened the doors to more affordable, smart data solutions that make it possible for non-technical users to explore, through visualization tools, the power of predictive analytics. We’re also seeing the increasing democratization of artificial intelligence which is driving more sophisticated consumer insights and decision-making. Forward-thinking organizations need to approach predictive analytics with the future and extensibility in mind. Today’s tools may not be the best for tomorrow’s needs. Cloud solutions are still evolving and haven’t reached functionality maturity yet, but by merging cloud, open source, and agile development methodologies into their predictive analytics stack, organizations will be able to easily adopt as technology advances.  – Slava Koltovich, CEO, EastBanc Technologies

One Team, One Platform – Data is the common thread within the enterprise, regardless of where the source might be. In the past data handlers have relied on disparate systems for data needs. Next year, the goal will be to move data into the future by providing a one-stop shop to access, develop and explore data. Companies will now look to one data platform for integrated cloud services with easy access and consistent behavior that is equipped to satisfy the needs of diverse data-hungry professionals across the organization. Just as you can easily access a variety of apps on your smartphone, business users and data professionals will look to deploy one platform that allows their organization to tap into the rich capabilities of data. – Derek Schoettle, General Manager, Cloud Data Services, IBM Watson and Cloud Platform

Next year will bring about another deluge of data brought on by advancements in the way we capture it. As more hardware and software is instrumented especially for this purpose, such as IoT devices, it will become easier and cheaper to capture data. Organizations will continue to feed on the increased data volume while the big data industry struggles through a shortage of data scientists and the boundaries imposed by non-scalable legacy software that can’t perform analytics at a granular level on big data data. Healthcare will especially be hard hit in this regard. Sources of huge healthcare data sets are becoming more abundant, ranging from macro-level sources like surveys by the World Health Organization, to micro-level sources like next-generation Genomics technologies. Healthcare professionals are leveraging these data to improve the quality and speed of their services. Even traditional technology companies are venturing into this field. For example, Google is ploughing money into its healthcare initiatives like Calico, its “life-expansion” project, and Verily, which is aimed at disease prevention. We expect the demand for innovative technical solutions in all industries, particularly healthcare to explode in popularity next year. – Michael Upchurch, COO, Fuzzy Logix

Data lakes will finally become useful — Many companies who took the data lake plunge in the early days have spent a significant amount of money not only buying into the promise of low cost storage and process, but a plethora of services in order to aggregate and make available significant pools of big data to be correlated and uncovered for better insights. The challenge has been finding skilled data scientists that are able to make sense of the information, while also guaranteeing the reliability of data upon which data is being aligned and correlated to (although noted expert Tom Davenport recently claimed it’s a myth that data scientists are hard to find). Data lakes have also fallen short in providing input into and receiving real-time updates from operational applications. Fortunately, the gap is narrowing between what has traditionally been the discipline and set of technologies known as master data management (MDM), and the world of operational applications, analytical data warehouses and data lakes. With existing big data projects recognizing the need for a reliable data foundation, and new projects being combined into a holistic data management strategy, data lakes may finally fulfill their promise in 2017. – Ramon Chen, CMO, Reltio

I believe customers will choose solutions in Big Data that deliver faster time to value, simple deployment with ease of management, interoperability with open source tools and solutions that help bridge the skills gap. I predict that Big Data technologies like Hadoop will be adopted at an accelerated rate because customers must get smarter about data. Based on customer conversations, they understand they could be disrupted by a new competitor with a data driven business model. Hadoop will be at the core of a data driven business allowing organizations to be more agile, know more about their customers, and offer new services ahead of the competition. I believe the strength of the community, the work of Cloudera and Hortonworks along with maturing ecosystem tools, as well as interoperability with analytical tools, will provide a secure, enterprise ready data platform. – Armando Acosta, Hadoop Product Manager and Data Analytics SME, Dell EMC

Open source and faux-pen source data technology choices will continue to proliferate, but the new model will redistribute rather than purely reduce costs for enterprises. Vendors are walking away from traditional database and data warehouse business models. Prime examples of this are Pivotal open sourcing Greenplum, Hewlett Packard Enterprise (HPE) spinning off Vertica and other assets, and Actian stopping support for Matrix (formerly ParAccel). Open source projects – or in many cases, vendor sponsored faux-pen sources – are becoming the new model for data processing technology. But while open source reduces the costs of vendor licensing, it also shifts responsibility to the enterprise to sort through the options, assemble stacks and productionize open source projects. This increase in complexity and consumption challenges requires new hiring and/or partnering with as-a-Service cloud vendors. – Prat Moghe, Founder and CEO, Cazena

In 2017 organizations will shift from the “build it and they will come” data lake approach to a business-driven data approach. Use case orientation drives the combination of analytics and operations. Approaching a data lake as “Imagine what your business could do if all your data were collected in one centralized, secure, fully-governed place that any department can access anytime, anywhere.” could sound attractive at a high level, but too frequently results in a data swamp that looks like a data warehouse rebuild and can’t address real-time and operational use case requirements. Once in place the concept is to “ask questions”. In reality, the world moves faster today. Today’s world requires analytics and operational capabilities to address customers, process claims and interface to devices in real time at an individual level. For example any ecommerce site must provide individualized recommendations and price checks in real time. Healthcare organizations must process valid claims and block fraudulent claims by combining analytics with operational systems. Media companies are now personalizing content served though set top boxes. Auto manufacturers and ride sharing companies are interoperating at scale with cars and the drivers. Delivering these use cases requires an agile platform that can provide both analytical and operational processing to increase value from additional use cases that span from back office analytics to front office operations. In 2017, organizations will push aggressively beyond an “asking questions” approach and architect to drive initial and long term business value. – John Schroeder, Chairman and Founder, MapR

Big data goes self-service. Organizations that have realized the value of big data now face a new problem: IT and data teams are being flooded with requests from users to pull data. To address this, we’ll see more organizations opt for a self-service data model so that anyone in the company can easily pull data to uncover new insights to make business decisions. A self-service infrastructure allows any employee to easily access and analyze data, saving IT and data teams precious time and resources. To make this a reality, all types of data in every department will need to be published so that users can self-serve. – Ashish Thusoo, CEO, Qubole

2017 will be the year organizations begin to rekindle trust in their data lakes. The “dump it in the data lake” mentality compromises analysis and sows distrust in the data. With so many new and evolving data sources like sensors and connected devices, organizations must be vigilant about the integrity of their data and expect and plan for regular, unanticipated changes to the format of their incoming data. Next year, organizations will begin to change their mindset and look for ways to constantly monitor and sanitize data as it arrives, before it reaches its destination. – Girish Pancha, CEO and Founder, StreamSets

Companies have been collecting data for awhile, so the data lake is well-stocked with fish. But the people who needed data most couldn’t generally find the right fish. I support the notion of a data lake, dumping all your raw data into one data warehouse. But it doesn’t work if you don’t have a way to make it cohesive when you query it. There have been great innovations by companies like Segment, Fivetran and Stitch, which make moving data into the lake easier. Modeling data is the final step that brings it all together and helps some of the best companies in the world see through data.
Companies like Docker, Amazon Prime Now and BuzzFeed are using all their data to create comprehensive views of their customers and of their businesses. When these final two steps are added, the data lake can finally be a powerful way to get all your data into the hands of every decision-maker to make companies more successful. – Lloyd Tabb, Founder, Chairman & CTO, Looker

In 2017, organizations will stop letting data lakes be their proverbial ball and chain. Centralized data stores still have a place in initiatives of the future: How else can you compare current data with historical data to identify trends and patterns? Yet, relying solely on a centralized data strategy will ensure data weighs you down. Rather than a data lake-focused approach, organizations will begin to shift the bulk of their investments to implementing solutions that enable data to be utilized where it’s generated and where business process occur – at the edge. In years to come, this shift will be understood as especially prescient, now that edge analytics and distributed strategies are becoming increasingly important parts of deriving value from data. – Adam Wray, CEO, Basho Technologies

In 2017, the reports of Big Data’s death will be greatly exaggerated, as will the hype around IoT and AI. In reality, all of these disciplines focus on data capture, curation, analysis and modeling. The importance of that suite of activities won’t go away unless all businesses cease operation. – Andrew Brust, Senior Director, Market Strategy and Intelligence, Datameer

Big data or bust in 2017? Big data is an example of something that didn’t get as far along as people predicted. Of course, it wasn’t stagnant. But nearly everyone involved in the enterprise sector would like it to move faster. The problem is that companies struggle, in general, to make sense of big data because of its sheer volume, the speed in which it is collected and the great variety of content it encompasses. Looking ahead, we can expect to see newer tools and procedures that will help companies house and examine these massive amounts of data and help them move toward truly making data-driven decisions. – Bob DeSantis, COO, Conga

In the new world of data, DBMS is really the management of a collection of data systems. This deserves a new thinking or approach to how we manage these systems and the applications that leverage them. The enterprise has long relied on raw logs and systems monitoring solutions to optimize their Big Data applications—and as companies continue to adopt numerous disparate Big Data technologies to help meet their business needs, complexity is only increasing while the time required to diagnose and resolve issues grows exponentially, all of which is underlined by an acute shortage of talent capable of effectively running and maintaining these intricate Big Data systems. The primary challenge faced by the enterprise is finding a single full-stack platform capable of analyzing, optimizing and resolving any issues that exists with Big Data applications and the infrastructure supporting them. In the year ahead, the enterprise will search for a solution that addresses the unmet challenges of data teams that find themselves spending much of their day digging through machine logs in order to identify the root cause of problems on a Big Data stack. These problems, if not eradicated, will continue to reduce application performance and divert teams from their real mission of deriving the full value from their Big Data. Ideal solutions will be ones that resolve problems automatically, detecting and pinpointing performance and reliability issues with Big Data applications running on clusters; solutions that open up the doors to data equality across the enterprise, that with just the click of a button, drastically accelerate the time-to-value of Big Data investments. – Unravel Data

Big data wanes – Big data will continue to wane as a term. The focus now turns from infrastructure to applications with specific purposes. Companies will look to applications and new business models for concrete value, rather than the more general idea that data can be useful at scale. – Satyen Sangani, CEO, Alation

Business Intelligence

Self-service extends to data prep. While self-service data discovery has become the standard, data prep has remained in the realm of IT and data experts. This will change in 2017. Common data-prep tasks like data parsing, JSON and HTML imports, and data wrangling will no longer be delegated to specialists. With new innovations in this transforming space, everyone will be able to tackle these tasks as part of their analytics flow. – Francois Ajenstat, Chief Product Officer at Tableau

Many Big Data systems are lacking simple UI’s for data input and classification. This usually requires highly technical staff and costs for the configuration, ongoing use, and for the interpretation of Big Data. This produces a high cost of entry and ongoing expenses. To add insult to injury, even once deployed, if the tool cannot be completely adopted by all necessary end users due to complexity, all BI efforts may be for naught. Successful User Interfaces (UI’s) are simple and flexible and modify to the needs of a variety of users and any changes to fluid data sets. This is the future of Big Data. Making Big Data even more accessible accurate, and therefore indispensable. Just as other technologies have evolved, BI is evolving to be more accessible than ever to today’s business. This will only continue in the future. – Dave Bethers, Chief Operations Officer, TCN

Digital transformation will be a CIO imperative for greater than 50% of all institutions. As such, IT will no longer be pushing Big Data technologies to the business owners. Instead, IT will need to respond to the demands for faster and more predicative analytics. Data scientists will be embedded into the business units in larger companies and in the smaller firms, everyone will be considered a citizen data scientist. Regardless, business intelligence will no longer be considered a department but an attitude. A way of life. At least for those who plan to be in business by 2019. – Anthony Dina, Director Data Analytics, Dell EMC

In 2017, business people will become ‘data mixologists’, capable of blending data from any combination of systems – centralized and decentralized – to produce new insights on their own, share them with others, and make better, more trusted business decisions. Historically, mixing together data from spreadsheets, databases, or applications like Marketo, Salesforce and Google Analytics has been an inaccessible capability for business people, as well as a data governance nightmare. Until now, self-service data prep tools have been designed for data scientists who work in silos of disconnected data – a phenomenon known as “data discovery sprawl”. These silos produce inaccurate and unreliable insights, and they don’t put those insights in the hands of business decision-makers. In the coming year, we will see business users choose modern tools that help them become data mixologists, making empowered decisions from trustworthy data sets. – Pedro Arellano, VP of Product Strategy, Birst

Cloud

The move to serverless architectures will become more widespread in the coming years, and will impact how applications are deployed and managed. Serverless architectures allow users to deploy code and run applications without managing the supporting infrastructure. Instead, the supporting infrastructure is managed by a third party. AWS’ cloud service Amazon Lambda is an example, and we anticipate growth in the number of providers and the breadth of enterprise-ready applications. As use of serverless architectures begin to rise, the overall application development and deployment strategy will begin to shift away from operations and more towards business logic. More cloud providers will also begin migrating to this form of architecture, allowing for a more competitive market with more expansive application support. As such, it will be important for database solution providers to be ‘cloud-ready.’ – Patrick McFadin, Chief Evangelist for Apache Cassandra, DataStax

The conversation around vendor lock-in is becoming much more prominent in senior level meetings, spurred on by many enterprises’ decision to move to the public cloud. To this point, the issue of vendor lock-in was initially discussed as a black or white situation. However, in 2017 we are going to see this conversation shift to acknowledge the many shades of gray, as executives realize and consider the varying degrees of lock-in and how it impacts various departments and levels of management. Examining the potential consequences of using proprietary technology on the different levels of the hardware and software stack will be an important issue within companies this year as more enterprises implement digital transformation initiatives. – Bob Wiederhold, CEO, Couchbase

Big data and the cloud will go hand-in-hand. Five years ago concerns over security and compliance kept enterprises from embracing big data in the cloud. Now, best practices and advancements in technology have allayed those concerns while the cloud’s agility and ease of use are becoming must-have’s for processing big data. As big data moves from an experiment to an organization-wide endeavor, the cost, time and resources needed to manage a massive data center don’t make sense. As a result, more and more companies will look to the cloud to help with the costs of data management. In 2017, expect enterprises to move their big data projects to the cloud in droves. – Ashish Thusoo, CEO, Qubole

2017 will be the year big data platforms go operational with the rise of hybrid clouds. We will see more customer cloud apps, such as Salesforce CRM and Oracle CX, accessing big data insights directly from on-premises big data platforms, which are the foundations of enterprises’ digital transformation and omni-channel marketing strategies. Examples of big data insights that support additional functional areas, such as sales and marketing, include predictive models, lead scoring or personalization. This typically starts with the ingestion of customer and marketing data into a data lake, where the source data is commonly stored in hybrid cloud and on-premises systems. And to operationalize those insights, we’ll see greater demand for standard REST interfaces to big data sets primarily accessible from SQL (such as Hive, Impala or Hawq) for hybrid connectivity from SaaS applications or cloud and mobile application development. For on-premises consumers of hybrid data, we expect hosted big data platforms such as IBM BigInsights on Cloud, Amazon EMR, Azure HDInsights or SAP Altiscale to run more big data workloads, not suitable for local data centers, in the cloud and sending only the insights to on-premises systems for core business operations. – Sumit Sarkar’s, Chief Data Evangelist, Progress

Big-Data-as-a-Service. Big Data continued to see rising adoption throughout 2016, and we’ve observed an increasing number of organizations that are transitioning from experimental projects to large-scale deployments in production. However, the complexity and cost associated with traditional Big Data infrastructure has also prevented a number of enterprises from moving forward. Until recently, most enterprise Hadoop deployments were implemented the traditional way: on bare-metal physical servers with direct attached storage. Big-Data-as-a-Service (BDaaS) has emerged as a simpler and more cost-effective option for deploying Hadoop as well as Spark, Kafka, Cassandra, and other Big Data frameworks. As the public cloud becomes a more common deployment model for Big Data, we anticipate many of these deployments shifting to BDaaS offerings in 2017. In addition to solutions offered by newer BDaaS vendors like BlueData and Qubole, we’ll see more initiatives from established public cloud players like AWS, Google, IBM, and Microsoft. We can also expect a range of other announcements that will further validate the trend toward BDaaS, including both major partnerships (such as VMware’s recent embrace of AWS) and acquisitions (SAP buying Altiscale). As the ecosystem expands, customers will have the flexibility to choose from a range of BDaaS solutions, including public cloud as well as on-premises and even hybrid options (e.g. compute in the cloud and data stored on-premises). – BlueData

Data Governance

The Chief Data Officer Moves to New Heights – In this past year, we’ve seen the Chief Data Officer emerge as an instrumental part of the organization’s plan to harness the full value of data for competitive advantage. In 2017 we will see this role evolve further with the acceleration of CDO hires across industries to help with competitive pressures, aggressive global regulations (things like GDPR and BCBS 239) and the general increasing speed of business. Gartner predicts that by 2019, 90% of large organizations will have a CDO. We see this happening much quicker with the CDO rising as data hero within the organization when faced with the new challenges of managing the big data overload dispersed in separate systems and data silos among specific groups and users enterprise-wide. Wearing a super cape, CDOs will figure out a way to break down the data unrest that likely exists today by implementing business-focused governance processes and platforms and enabling and empowering every user across the enterprise to use and capitalize on data for competitive advantage. – Stan Christiaens, co-founder and CTO of data governance leader Collibra

In 2017, the governance vs. data value tug of war will be front and center. Enterprises have a wealth of information about their customers and partners. Leaders are transforming their companies from industry sector leaders to data driven companies. Organizations are now facing an escalating tug of war between governance required for compliance, and the use of data to provide business value and implement security to avoid damaging data leaks and breeches. Financial services and heath care are the most obvious industries with customers counting in the millions with heavy governance requirements. Leading organizations will manage their data between regulated and non-regulated use cases. Regulated use cases data require governance; data quality and lineage so a regulatory body can report and track data through all transformations to originating source. This is mandatory and necessary but limiting for non-regulatory use cases like customer 360 or offer serving where higher cardinality, real-time and a mix of structured and unstructured yields more effective results. – John Schroeder, Chairman and Founder, MapR

Moore’s Law holds true for databases. Per Moore’s law, CPUs are always getting faster and cheaper. Of late, databases have been following the same pattern. In 2013, Amazon changed the game when they introduced Redshift, a massively parallel processing database that allowed companies to store and analyze all their data for a reasonable price. Since then however, companies who saw products like Redshift as datastores with effectively limitless capacity have hit a wall. They have hundreds of terabytes or even petabytes of data and are stuck between paying more for the speed they had become accustomed to, or waiting five minutes for a query to return. Enter (or reenter) Moore’s law. Redshift has become the industry standard for cloud MPP databases, and we don’t see that changing anytime soon. With that said, our prediction for 2017 is that on-demand MPP databases like Google BigQuery and Snowflake will see a huge uptick in popularity. On-demand databases charge pennies for storage, allowing companies to store data without worrying about cost. When users want to run queries or pull data, it spins up the hardware it needs and gets the job done in seconds. They’re fast, scalable, and we expect to see a lot of companies using them in 2017. – Lloyd Tabb, Founder, Chairman & CTO, Looker

The rise of “applied governance” to unstructured data. Earlier this year, more than 20,000 pages of top-secret Indian Navy data, including schematics on the their Scorpene-class submarines, were leaked. It’s been a huge setback for the Indian government. It’s also an unfortunate case study for what happens when you lack controls over unstructured information, such as blueprints that might be sitting in some legacy engineering software system. Now, replace the Indian Navy scenario with a situation involving the schematics for a Nuclear power plant or consumer IoT device, and the value of secure content curation becomes even more immeasurable. If unstructured blueprints and files are being physically printed or copied, or digitally transferred, how will you even know that content now exists? Tracking this ‘dark data’ – particularly in industrial environments – will be a top security priority in 2017. – Ankur Laroia, Leader – Solutions Strategy, Alfresco

Organizations have viewed data governance as a tax. It’s something you had to do for compliance or regulatory reasons, but it wasn’t adding value to the business. In reality, governance is crucial to driving business value. Think about the enormous amount of time and money being spent these days to harness the value of data – the whole Big Data movement. Organizations know there is tremendous value to be had, but many of them aren’t actually getting the value despite their investment. Gartner says: Through 2018, 80% of data lakes will not include effective metadata management capabilities, making them inefficient. Why? Two reasons: First, they don’t have the lineage and provenance of the data they’re analyzing. When they put bad or misleading data into their analysis, they’re going to get unreliable results back out. That’s a lack of data governance. Second, and perhaps even worse, organizations are afraid to share the data they’ve gone to great expense to create. They can’t answer questions such as: Under what agreements was the data collected? Which pieces are personal information? Who’s allowed to see it? In which geographies? With what redistribution rights? If you can’t answer these questions, you can’t share the data. Your data lake is fenced off. This is another failure of governance. Businesses will realize that governance gives them the highest quality results, that can be shared with the right audiences, and drive the greatest business value. – Joe Pasqua – EVP Products, MarkLogic

The Chief Data Officer position will pick up steam significantly. This is a sure sign of the pendulum swinging back: A company officer centrally managing the value of data. And a CDO’s job isn’t to empower analysts per se, although that will often be part of what they do. If that were all it was, companies could save a lot of money by handing out tools and not creating the CDO position. The CDO’s job is to extract maximum value from data. That can be done in many ways, including customer-facing portals, large-scale analytical apps, data feeds that stem from unified views of business entities, embedded BI inside other enterprise applications, and so on.So as the CDO position picks up steam, we can expect to see larger data-focused projects where information is managed and shared across divisional and even company boundaries, leading to better data monetization, lower per-user cost of data, and higher business value per unit of data. – Jake Freivald, Vice President, Information Builders

Data Science

In 2017 we will see an increased valuation of the critical thinking in the workplace, as people realize that there is not a deficit of data in the enterprise, but a deficit of insight. Companies will realize that data without additional tenets of knowledge or value, is both polarizing and damaging. The role of data scientist will evolve to become “the knowledge engineer.” We will see fewer “alchemists” – promising magic from data patterns alone, and more “chemists” — combining the elements of knowledge, data, context, and insight to deliver productivity enhancements that we have yet to imagine. – Donal Daly, CEO, Altify

We spend a lot of time thinking about what developers want & need in a tool, both right now and in the future. In software development, complexity is inevitable – tech stack, libraries, formats, protocols – and that complexity won’t be decreasing any time soon. The most successful tool is one that is simple, but not dumbed down or less powerful. I believe that tools will need to become even more powerful in 2017, and the successful tools will be ones that work for the developer rather than the other way around. Tools will need to be smarter to learn from the user automatically, proactive to inform the user automatically, collaborative to connect users with others, and visual and tangible to show and manipulate. This meta-increase in toolsets is possible now for a number of reasons. Memory, processing power, and connectivity speed continue to explode, while at the same time visual tools (like 4K screens) get better and better. Plus, the continued rise of social coding increases the need to powerful collaborative tools to support the developer. – Abhinav Asthana, CEO of Postman

2017 will be the “Year of the Data Scientist.” According to the McKinsey Global Institute, demand for data scientists is growing by as much as 12 percent a year and the US economy could be short by as many as 250,000 data scientists by 2024. Thanks to advances driven by AI companies in 2017, however, 2018 is when AI will become buildable – not just usable – but buildable by non-data scientists. This is not to say that data science will become less useful or in-demand post-2017, rather that some of the simpler problems will be solvable through a hyper-personalized AI built by someone who is not a data scientist. This will open up capabilities for coders and data scientists that will be mind-blowing. – Jeff Catlin, CEO, Lexalytics

SQL will have another extraordinary year. SQL has been around for decades, but from the late-1990s to mid 2000s, it went out of style as people started exploring NoSQL and Hadoop alternatives. SQL however, has come back with a vengeance. The renaissance of SQL has been beautiful to behold and I don’t even think it’s near it’s peak yet. The innovations we’re seeing are blowing our minds. BigQuery has created a product that is essentially infinitely scalable, the original goal of Hadoop, AND practical for analytics, the original goal of relational databases. Additionally, Google recently announced that the new version, BigQuery Standard SQL is fully ANSI compliant. Prior to this release, BigQuery’s Legacy SQL was peculiar and so presented a steep learning curve. BigQuery’s implementation of Standard SQL is amazing, with really advanced features like Arrays, Structures, and user-defined functions that can be written in both SQL and Javascript. SQL engines for Hadoop have continued to gain traction. Products like SparkSQL and Presto are popping up in enterprises and as cloud services because they allow companies to leverage their existing Hadoop clusters and cloud storage for speedy analytics. What’s not to love? To top it all off, companies like Snowflake, and now Amazon Athena, are building giant SQL data engines that query directly on S3 buckets, a source that was previously only accessible via command line. 2016 was the best year SQL has ever had — 2017 will be even better. – Lloyd Tabb, Founder, Chairman & CTO, Looker

The data skills gap widens. Problem: The demand for data scientists and data engineers continues to challenge enterprises who need to make the most of their data. And even when there are the right skillsets at play, the New York Times recently reported that these critical personnel are often spending more time cleaning the data than actually mining it. Prediction: Businesses will seek any tool that help to put more data in the hands of business analysts with the minimum data scientist intervention. In addition, new machine learning tools will emerge to help automate some of these data-focused tasks to scale the models that data scientists create. – SnapLogic

There will continue to be a shortage of qualified data scientists. I don’t expect the market to be in equilibrium until 2019 at the earliest. Every major university will have a data science program in place by 2017. – Michael Stonebraker, Ph.D., co-founder and CTO, Tamr

Data Scientists failed to predict the election—will they fail to predict your business? The other day I was giving a talk on ‘What is Machine Learning?’ and, barely two minutes in, someone said, ‘You’re saying we can do all these amazing things with big data and algorithms, but you had all the data in the world for the election, and you got it wrong. Why should we trust you?’ There are plenty of important takeaways from the election: First, Nate Silver and enterprise data scientists both try to learn from historical events to predict future events, and the margins of error can behigh in both. But in predicting an election you only get one chance. In business, you make predictions constantly, and the cost of error tends to be low. Also, there are fewer curve-balls in business. Customers and businesses tend to be pretty predictable. Voters and politicians are not. Second, the media committed the same sin we see business people make every day: falling too hard for the analytic ‘black box’ that does seemingly magical number crunching. Without a basic understanding of what types of analyses have been done on different types of data and why, the end users will never know the true value of the information they have at their disposal or how they should use it. There’s no better illustration of this than the little needle on The New York Times’ election ‘dial’ which bounced violently from Clinton to Trump in the middle of the evening and had me screaming at my phone. – Steven Hillion, Chief Product Officer, Alpine Data

GPUs and HPC

2017 will be the year when “accelerated compute” becomes known just simply as “compute”. This is a direct response to the use cases driving up utilization the most, and the explosion of accelerator availability in both the data center and the public cloud. As these use cases continue to ramp up in the Enterprise (particularly machine learning), we’ll see even more demand for computational accelerators. CPUs have been king for decades, and serve the general purpose quite well. But what we’re seeing now is an emphasis on deriving insight from data, versus just indexing it, and this requires orders of magnitude faster (and more specialized) resource in order to deliver feasible economics. It’s not that computational accelerators are necessarily “faster” than CPUs, but rather, they can be deployed as coprocessors and therefore take on very specialized identities. Because of this specialization, they can be programmed to do certain very discrete computations much quicker and at lower aggregate power consumption. Application developers and ISVs are pouncing on these capabilities (and their increasing availability) to create amazing new products and services. A good example of a red-hot technology in this space are GPU-accelerated databases, such as GPUdb from Kinetica (available as a turnkey workflow on the Nimbix Cloud). Rather than focusing on indexing massive amounts of information like a traditional RDBMS, it’s used to ingest fragments into memory for tremendously fast queries. In fact the queries are so fast that it blurs the line between analytics and machine learning (after all, machine learning involves processing massive data sets very quickly in order to create “models” that operate somewhat like human brains). Despite the advanced computing underneath, these tools serve traditional enterprise markets, not just “research labs”. Not only does its product name imply it, but the use case simply would be impossible without GPUs. This is a very real example of mainstream technology that demands computational accelerators. In talking with customers and business partners, the one common thread they all seek is more accelerated computational power (at reasonable economics) to do even more advanced things. I don’t see this trend slowing down anytime soon, which is why I’m predicting that we’ll drop the “accelerated” in front of “compute” as it will become a given. – Leo Reiter, CTO, Nimbix

Graphical Processing Units (GPUs) are capable of delivering up to 100-times better performance than even the most advanced in-memory databases that use CPUs alone. The reason is their massively parallel processing, with some GPUs containing over 4,000 cores, compared to the 16-32 cores typical in today’s most powerful CPUs. The small, efficient cores are also better suited to performing similar, repeated instructions in parallel, making GPUs ideal for accelerating the compute-intensive workloads required for analyzing large streaming data sets in real-time. – Eric Mizell, Vice President, Global Solutions Engineering, Kinetica

Amazon has already begun deploying GPUs, and Microsoft and Google have announced plans. These cloud service providers are all deploying GPUs for the same reason: to gain a competitive advantage. Given the dramatic improvements in performance offered by GPUs, other cloud service providers can also be expected to begin deploying GPUs in 2017. – Eric Mizell, Vice President, Global Solutions Engineering, Kinetica

Hadoop

As I predicted last year, 2016 was not a good year for Hadoop and specifically for Hadoop distribution vendors. Hortonworks is trading at one-third its IPO price and the open source projects are wandering off. IaaS cloud vendors are offering their own implementations of the open source compute engines – Hive, Presto, Impala and Spark. HDFS is legacy in the cloud and is rapidly being replaced by blob storage such as S3. Hadoop demonstrates the perils of being an open source vendor in a cloud-centric world. IaaS vendors incorporate the open source technology and leave the open source service vendor high and dry. Open source data analysis remains a complicated and confusing world. Wouldn’t it be nice if there were one database that could do it all? Wait, there is one, it’s called Snowflake. – Bob Muglia, CEO, Snowflake Computing Inc.

Don’t be a Ha-dope! For all those folks running around saying Hadoop is dead – they’re dead wrong. In 2017, we’re going to see an increased adoption of Hadoop. So far this year, I haven’t talked to a single organization with a digital data platform who doesn’t see Hadoop at the center of their infrastructure. Hadoop is an assumed part of every modern data architecture and nobody can question the value it brings with its flexibility of data ingestion and its scalable computational power. Hadoop is not going to replace other databases but it will be an essential part of data ingestion in the IoT/digital world. – George Corugedo, CTO, RedPoint Global

Hadoop distribution vendors will have crossed the chasm — unstructured data in Hadoop is a reality. But, since the open source problem has not been addressed, they aren’t making much money. As such, there will be an acquisition of many of these vendors by bigger players. As well as the idea that bigger ISV Hadoop vendors will band together and create larger entities in hopes of capitalizing on the economy of scale. – Joanna Schloss, Director of Product Marketing, Datameer

The Failure (and future) of Hadoop. Problem: Fifty percent of Hadoop deployments have failed. While it’s commanded the lion’s-share of attention, it’s suffered from product overload. Because new projects are added every month and the nature of the data in the Hadoop cluster is ever-growing, it’s created a complex, multidimensional environment that’s difficult to maintain in production. Prediction: To actually make Hadoop work beyond a test environment, enterprises will shift it to the cloud in 2017, and abstract storage from compute. This enables enterprises to select the tools they want to use (Spark, Flink or others) instead of being forced to carry excessive Hadoop baggage with them. – SnapLogic

In-Memory Computing

In 2017, in-memory computing will enter the mainstream as the enabling technology for adding operational intelligence to live systems, and it will supplant legacy streaming technologies. In 2017, the adoption of in-memory computing technologies, such as in-memory data grids (IMDGs), will provide the enabling technology to capture perishable opportunities and make mission-critical decisions on live data. Driven by the need for real-time analytics, the IMDG market alone – currently estimated at $600 million – will exceed $1 billion by 2018, according to Gartner. Unlike big data technologies, such as Spark, created for the data warehouse and legacy streaming technologies, in-memory computing enables the straightforward modeling and tracking of a live system by analyzing and correlating persistent data with live fast-changing data in real time, and it provides immediate feedback to that system for automated decision making. Gartner has recently elevated the term “digital twin” in its recent Top 10 strategic technology trends for 2017 to describe the shift in focus from data streams to the data sources which produce those streams. In-memory computing technology enables applications to easily create and manage digital representations of real-world devices, such as Industrial Internet of Things (IIoT) sensors and actuators, and this enables real-time introspection for operational intelligence. – Dr. William Bain, CEO and founder, ScaleOut Software

In-Memory and Temporary Storage become more important as new sources of data growth such as augmented and virtual reality, AI and machine learning become popular: While analyzing these new sources of data is becoming critical to long-term business goals, storing the data long term is both impractical and unnecessary when the results of analysis are more important than the data itself. Although 2017 will see plenty of data growth that will require permanent storage, most net new data generated next year will be ephemeral; it will quickly outlive its usefulness and be discarded. So despite exponential data growth, there won’t be as much storage growth as we might otherwise have expected. – Avinash Lakshman, CEO, Hedvig

IoT

The future of IoT will be focused on security. Recently, a major DDoS attack caused outages at major organizations. This is going to be a growing issue in the near future, and the concern at the forefront of IoT will be safeguarding networks and connected devices. – Dr. Werner Hopf, CEO and Archiving Principal, Dolphin Enterprise Solutions Corporation

IOT grows up – The enterprise has paid attention to IOT for some time, though this year will be the year we move past the “wow” phase and into the “how do we do we securely and effectively bring IOT to the enterprise, how do we handle the high speed data ingest, and how do we optimize analytics and decisions based on IOT data.” Those will be the questions enterprises will need to solve in 2017. – Leena Joshi, VP of Product Marketing, Redis Labs

IoT continues to pose a major threat. In late 2016, all eyes were on IoT-borne attacks. Threat actors were using Internet of Things devices to build botnets to launch massive distrubted denial of service (DDoS) attacks. In two instances, these botnets collected unsecured “smart” cameras. As IoT devices proliferate, and everything has a Web connection — refrigerators, medical devices, cameras, cars, tires, you name it — this problem will continue to grow unless proper precautions like two-factor authentication, strong password protection and others are taken. Device manufactures must also change behavior. They must scrap default passwords and either assign unique credentials to each device or apply modern password configuration techinques for the end user during setup. – A10 Networks

The Internet of Things (IoT) is widely acknowledged as a big growth area for 2017. More connected devices will create more data, which has to be securely shared, stored, managed and analyzed. As a result, databases will become more complex and the management burden will increase. Those organizations which can most effectively monitor their database layer to optimize peak performance and resolve bottlenecks will be more strongly placed in a better position to exploit the opportunities the IoT will bring. – Mike Kelly, CTO, Blue Medora

The future of retirement is gearing up for a major shift and Internet of Things (IoT) along with it. Baby boomers are retiring, and there are many economic and lifestyle reasons for them to live in their homes longer. This means changes for insurance companies, healthcare, medical devices, and appliance manufacturers. The proliferation of the IoT or “the connected life” allows for monitoring the elderly in their homes, from monitoring blood pressure to typical daily habits such as whether or not they turned on the TV or opened the refrigerator. Elderly parents want autonomy and their children want them to be safe – connected technology can bridge the gap between the two. Basic monitoring as well as more advanced medical monitoring is shifting the way we will live out our retirement. – Kevin Petrie, Attunity

The Internet of Things (IoT) is still a popular buzzword, but adoption will continue to be slow. Analyzing data from IoT and sensors clearly has the potential for massive impact, but most companies are far (FAR!) from ready. IoT will continue to get lots of lip service, but actual deployments will remain low. Complexity will continue to plague early adopters that find it a major challenge to integrate that many moving parts. Companies will instead focus resources on other low-hanging fruit data and analytics projects first. – Prat Moghe, Founder and CEO, Cazena

The Internet of Things is delivering on the promise of big data. IoT will deliver on the promise of big data. Increasingly, big data projects are going through multiple updates in a single year – and the Internet of Things (IoT) is largely the reason. That’s because IoT makes it possible to examine specific patterns that deliver specific business outcomes, and this has to increasingly be done in realtime. This will drive a healthier investment, and faster return in big data projects. – Ettienne Reinecke, Chief Technology Officer, Dimension Data

Next year, organizations will stop putting IoT data on a pedestal, or, if you like, in a silo. IoT data needs to be correlated with other data streams, tied to historical or master data or run through artificial intelligence algorithms in order to provide business-driving value. Despite the heralded arrival of shiny new tools that can handle IoT’s massive, moving workloads, organizations will realize they need to integrate these new data streams into their existing data management and governance disciplines to gain operational leverage and ensure application trust. – Girish Pancha, CEO and Founder, StreamSets

The Internet of Things Architect role will eclipse the data scientist as the most valuable unicorn for HR departments. The surge in IoT will produce a surge in edge computing and IoT operational design. 1000s of resumes will be updated overnight. Additionally, fewer than 10% of companies realize they need an IoT Analytics Architect, a distinct species from IoT System Architect. Software architects who can design both distributed and central analytics for IoT will soar in value. – Dan Graham, Internet of Things Technical Marketing Specialist, Teradata

At Least one Major Manufacturing Company will go belly up by not utilizing IoT/big data: The average lifespan of an S&P 500 company has dramatically decreased over the last century, from 67 years in the 1920s to just 15 years today. The average lifespan will continue to decrease as companies ignore or lag behind changing business models ushered in by technological evolutions. It is imperative that organizations find effective ways to harness big data to remain competitive. Those that have not already begun their digital transformations, or have no clear vision for how to do so, have likely already missed the boat—meaning they will soon be a footnote in a long line of once-great S&P 500 players. – Ashley Stirrup, CMO, Talend

Machine Learning

In-memory computing techniques will leverage the power of machine learning to enhance the value of operational intelligence. The year 2017 will see an accelerated adoption of scenarios that integrate machine learning with the power of in-memory computing, especially in e-commerce systems and the Internet of Things (IoT). E-commerce applications benefit by offering highly personalized experiences created by tracking and analyzing dynamic shopping behavior. IoT applications, such as those associated with windmills and solar arrays, benefit by delivering predictive feedback based on rapidly emerging patterns. In both of these applications, machine learning techniques can dramatically deepen the introspection and enhance operational intelligence. Once only practical only on supercomputers, machine learning techniques have evolved to become increasingly available on standard, commodity hardware. This enables IMDGs to apply them to the analysis of fast changing data and specifically to dynamic digital models of live systems. The ability of IMDGs to perform iterative computation in real-time and at extreme scale enables machine learning techniques to be easily integrated into stream processing which provides operational intelligence. – Chris Villinger, Vice President, Business Development and Marketing, ScaleOut Software

Machine learning will change the fabric of the enterprise – Machine learning will enable the adaptive enterprise, one that aligns business outcomes and customer needs in new and different ways. – Leena Joshi, VP of Product Marketing, Redis Labs

In 2017, I expect to see an increased emphasis on democratization of machine learning and artificial intelligence (AI). We’ve seen machine learning evolve from IBM Watson a few years ago to most recently with Salesforce and Oracle. While many think machine learning has gone mainstream, there is the potential for much more, such as performance monitoring and intelligent alerting. While companies might face false starts and initial mishaps while trying to crack the code, the increased number of organizations turning to AI and machine learning will lead to more successes next year. This increased adoption will help bring innovations faster to market, especially from a wide range of industries. – Mike Kelly, CTO, Blue Medora

There has been a lot of hype around machine learning for some time now, but in most cases it hasn’t been used very effectively. As we move forward, organizations are learning how to bring together all the ingredients needed to leverage machine learning – and I think that’s the story for 2017. We’ll see machine learning move from a mystical, over-hyped holy grail, to seeing more real-world, successful applications. Those who dismiss it as hocus-pocus will finally understand it’s real; those who distrust it will come to see its potential; and companies that are poised to leverage this capability for appropriate, practical applications will be able to ride the swell. It will still be a few years before machine learning becomes a tidal wave, but in 2017 it will be clear that it has a credible place in the business toolkit. – Jeff Evernham, Director of Consulting, North America, Sinequa

In 2017, ‘centralized-only’ monolithic software and silos of data disappear from the enterprise. Smart devices will collaborate and analyze what one another is saying. Real time machine-learning algorithms within modern distributed data applications will come into play – algorithms that are able to adjudicate ‘peer-to-peer’ decisions in real time. Data has gravity; it’s still expensive to move versus store in relative terms. This will spur the notion of processing analytics out at the edge, where the data was born and exists, and in real-time (versus moving everything into the cloud or back to a central location). – Scott Gnau, Chief Technology Officer, Hortonworks

Machine Learning will become de rigeur in the enterprise without many even noticing: What’s unique to today’s machine learning technology is that much of it originated and continues to be open source. This means that many different products and services are going to build machine learning into their platforms as a matter of course. As a result, more enterprises will be adopting machine learning in 2017 without even knowing they’re doing it because vendors are actively using ML to make their products smarter. Even existing products will soon use some variety of machine learning that will be delivered via an update or as an extra perk. – Avinash Lakshman, CEO, Hedvig

The Future of Machine Learning. We will finally deliver on the promise of machine learning: building models that can directly suggest or take action for large audiences. When we effectively scale machine learning, we can greatly increase the action-taking bandwidth of an enterprise. Instead of presenting a small number of business users in the enterprise with historical statistics à la business intelligence, companies can bring specific recommendations to thousands of front-line individuals responsible for taking action on behalf of the business. – Josh Lewis, VP of Product, Alpine Data

Machine learning-washing – Expect the market to be flooded with solutions that promise machine learning capabilities and grab headlines, but deliver no substance. – Toufic Boubez, VP Engineering, Machine Learning, Splunk

NoSQL

In 2017, NoSQL’s coming of age will be marked by a shift to workload-focused data strategies, meaning executives will answer questions about their business processes by examining the data workloads, use cases and end results they’re looking for. This mindset is in contrast to prior years when many decisions were driven from the bottom up by a technology-first approach, where executives would initiate projects by asking what types of tools best serve their purposes. This shift has been instigated by data technology, such as NoSQL databases, becoming increasingly accessible. – Adam Wray, CEO, Basho Technologies

Security

Cloud and data security agility will gain further importance — This is a rather obvious prediction, given the phobia of data breaches and the reticence of industries such as the financial sector to use public cloud technologies. Meanwhile, life sciences and retail, to name two industries, continue to forge ahead, realizing efficiencies while adhering to some of the strictest privacy and governance requirements set forth by regulators. With requirements such as the General Data Protection Regulation (GDPR) now in effect, companies not only have to ensure that their data is physically housed in the right geographic centers, but that the access complies with the most stringent regulations related to personal access and approvals for use of that data. Many vendors are now taking steps to provide the most secure, validated and agile infrastructure possible. Partnerships and use of Amazon Web Services, Google Cloud, and Microsoft Azure go a long way to providing the confidence and flexibility that many companies are looking for. In 2017, vendors offering Platform as a Service (PaaS) and tools themselves must also do their part in complying to Service Organization Control (SOC) types, as well as in the case of healthcare data, HITRUST (Health Information Trust Alliance), that provides an established security framework that can be used by all organizations that create, access, store or exchange sensitive and regulated data. – Ramon Chen, CMO, Reltio

Under the covers, machine learning is already becoming ubiquitous as it is embedded in many services that consumers take for granted. Increasingly, machine learning is becoming embedded in enterprise software and tooling for integrating and preparing data. Machine learning is placing a stress on enterprises to make data science a team sport; a big area for growth in 2017 will be solutions that spur collaboration, so the models and hypotheses that data scientists develop do not get bottled up on their desktops. – Ovum

Expect IoT to be even more vulnerable. Previous hacks into connected devices can be deemed as minor or inconvenient. But the recent DDoS attack involving Dyn shows IoT hacks are taking place on a larger and more disruptive scale. Hacking lightbulbs or setting off fire alarms is on the more mischievous side of the spectrum, but having the ability to override a car’s brake system or a “smart” pacemaker, for example, can turn connected devices into deadly weapons. Even worse, the lack of one standard for IoT (unlike Wi-Fi) will just make our devices more susceptible to large-scale breaches. Vendors have to recognize the parallels between security issues when Wi-Fi hit the mass market, and what’s happening with IoT. If they don’t move quickly to address the vulnerabilities, government regulations will need to come into play. Still, it would take something disastrous to galvanize government into action. – Richard Walters, SVP of Security Products, Intermedia

Over the past year there has been increased focused on data privacy, especially with the passing of the GDPR which represented one of the most comprehensive and refined set of standards put forth to date. In 2017, the trend line will to continue to move in the same direction and there will be a higher premium on data protection. With increased sensitivity around personal data, software vendors and enterprises will need to focus on what is being done to protect and manage personal data within the enterprise. To be successful companies must embrace privacy by design for themselves and the service providers they work with.” – Anthony West, CTO, Actiance

Spark

Spark and machine learning light up big data. In a survey of data architects, IT managers, and BI analysts, nearly 70% of the respondents favored Apache Spark over incumbent MapReduce, which is batch-oriented and doesn’t lend itself to interactive applications or real-time stream processing. These big-compute-on-big-data capabilities have elevated platforms featuring computation-intensive machine learning, AI, and graph algorithms. Microsoft Azure ML in particular has taken off thanks to its beginner-friendliness and easy integration with existing Microsoft platforms. Opening up ML to the masses will lead to the creation of more models and applications generating petabytes of data. In turn, all eyes will be on self-service software providers to see how they make this data approachable to the end user. – Dan Kogan, director of product marketing at Tableau

Analytics will experience a revolution in 2017. In the past, conversations about big data always included Hadoop (HDFS). But the industry today has hit a wall with its limitations to back up and preserve big data. As a result big data has become a black hole in the HDSFS cluster with no one managing it. In 2017, the Spark operating model – through ‘in memory analytics’ – will become a popular Big Data analytics option due to its ability to significantly reduce data movement and allow analytics to occur much earlier and faster in the process. – Vincent Hsu, VP, IBM Fellow, CTO for Storage and Software Defined Environment, IBM

Storage

People may think backup and recovery is dead, but they are sorely misunderstood and the move to the cloud actually makes backup and recovery more important than ever to safeguard data. Relying on the cloud won’t take care of everything! The need for backup and recovery will become very real as organizations continue betting on enterprise applications. Moreover, backup and recovery will take center stage as IT Ops and others in organizations have never stopped worrying about recovery, particularly as companies aggressively move toward modernized application and data delivery and consumption architectures. The likelihood of not knowing how to address or who to turn to in the event of an outage is just too great a risk. – Tarun Thakur, Co-founder and CEO at Datos IO

The Rise of the JBOD. In 2017, more users will come to understand that the storage for their scale-out nodes — whether you call it software-defined, “server SAN,” DAS, hyperconverged, whatever — can be attached externally to servers instead of buying servers with lots of disks and SSDs, without losing any of the performance or ease-of-use of internal DAS. Using simple, dumb, industry standard SAS JBODs (Just a Bunch Of Disks) means not having to throw away your storage when you upgrade your servers and vice-versa. It also gives you better flexibility and density in your deployments. – Tom Lyon, Chief Scientist, DriveScale

Verticals

One of the ongoing challenges in using big data to improve outcomes in healthcare has been its siloed natured. Healthcare providers have detailed clinical (patient) data within their organizations, while health insurers (payers) have more general claims data that goes across many providers. That is beginning to change, though, as the move to value-based care is encouraging providers and health payers to share their data to create a more complete picture of the patient. The latest trend is to bring in additional behavioral data, such as socio-economic and attitudinal data, to create more of a 360 degree view of not only what patients do but also what drives them to do it. Much as Facebook and Amazon.com use behavioral data to match users to relevant content. By applying next-generation analytics to this larger dataset, providers and payers can work together to help patients become healthier and stay healthy, reducing costs while helping them lead happier, more productive lives. – Rose Higgins, President, SCIO Health Analytics

We’ll usher in the next iteration of personalized care. Increased self-tracking, preventative care efforts, and advances in data science will give us more information on patients than ever before. We’ll use this data to create highly individual portraits of patients, that in turn, enable us to match physicians to patients in a very specific way. We can assign physicians based on their past success in treating similar patients and enable patients to have more informed and personal care. – Mark Scott, Chief Marketing Officer, Apixio

Data Analytics will go vertical (financial, medical, etc), and companies that build vertical solutions will dominate the market. General-purpose data analytics companies will start disappearing. Vertical data analytics startups will develop their own full-stack solutions to data collection, preparation and analytics. – Ihab Ilyas, co-founder of Tamr and Professor of Computer Science at the University of Waterloo

Big Data Will Transform Every Element of the Healthcare Supply Chain: The entire healthcare supply chain has been being digitized for the last several years. We’ve already witnessed the use of big data to improve not only patient care, but also payer-provider systems, reducing wasted overhead, predict epidemics, cure diseases, improve the quality of life and avoid preventable deaths. Combine this with the mass adoption of edge technologies to improve patient care and wellbeing such as wearables, mobile imaging devices, mobile health apps, etc. However, the use of data across the entire healthcare supply chain is about to reach a critical inflection point where the payoff from these initial big data investments will be bigger and come more quickly than ever before. As we move into 2017, healthcare leaders will find new ways to harness the power of big data to identify and uncover new areas for business process improvement, diagnose patients faster as well as drive better more personalized, preventative programs by integrating personally generated data with broader healthcare provider systems. – Ashley Stirrup, CMO, Talend

Author:  Daniel Gutierrez

Source:  http://insidebigdata.com/2016/12/21/big-data-industry-predictions-2017

Published in Science & Tech
Page 1 of 2

airs logo

AIRS is the world's leading community for the Internet Research Specialist and provide a Unified Platform that delivers, Education, Training and Certification for Online Research.

Subscribe to AIRS Newsletter

Receive Great tips via email, enter your email to Subscribe.
Please wait

Follow Us on Social Media

Read more content?
Register or Login as "AIRS Guest"
Enjoy Guest Account
or

x
Create an account
x

or