[youtube https://www.youtube.com/watch?v=n95hCVvuPKQ?rel=0]
Data Scientists frequently need to create applications that enable interactive data exploration, deliver predictive analytics APIs or simply publish results. Cloud Foundry provides an ideal platform for data scientists by making it easy to quickly deploy data driven apps backed by a variety of data stores.
Wikibon: For the calendar year 2014, the Big Data market – as measured by revenue associated with the sale of Big Data-related hardware, software and professional services – reached $27.36 billion, up from $19.6 billion in 2013. While growing significantly faster than other enterprise IT markets, the Big Data market’s overall growth rate slowed year-over-year from 60% in 2013 to 40% in 2014. This is to be expected in an emerging but quickly maturing market such as Big Data, and Wikibon does not believe this slightly slower growth rate indicates any structural market issues.
Now that most of the hype around big data has died down, overtaken by the buzz over the Internet of Things, we are sometimes treated to serious discussions of the state-of-the-art (or science, for that matter) in data analysis. If you are planning a career as a data scientist or you are a business executive trying to understand what the data scientists are telling you, three recent books provide excellent and accessible overviews:
Data Mining For Dummies by Meta S. Brown
Data Science For Dummies by Lillian Pierson
Bill Franks is the Chief Analytics Officer for Teradata, and his specialty is translating complex analytics into terms that business users can understand. The Analystics Revolution follows Franks’ Taming the Big Data Tidal Wave, which was listed on Tom Peters’ 2014 list of “Must Read” books.
“With all the hype around big data, it is easy to assume that nothing of interest was happening in the past if you don’t know better from experience” says Franks. The over-excitement about big data caused many organizations to re-create solutions that already exist and build new groups dedicated to big data analysis, separate from their traditional analytics functions. As a correction, Franks advocates “a new, integrated, and evolved analytics paradigm,” combining traditional analytics on traditional data with big data analytics on big data.
The focus of this new approach–and the book–is Operational Analytics. It takes us from the descriptive and predictive analytics of traditional and big data analytics to prescriptive analytics. It pays close attention to the numerous decisions and actions, mostly tactical, taking place every day in your business. Most important, it places great emphasis on the process of analytics, on embedding it everywhere, and on automating the required response to events and changing conditions.
“Of course,” says Franks, “it takes human intervention to decide that an operational analytics process is needed and to build the process.” But once the process is designed and turned on, the process accesses data, performs analysis, makes decisions, and then actually causes actions to occur. And humans are crucial to the success of this new brand of automated analytics, not only at the design phase, but also in the on-going monitoring and tweaking of the process.
An example of operational analytics is the development of an improved maintenance schedule using sensor data. There will be no value in the Internet of Things without an automated process for data analysis and action based on that analysis. “As traditional manufacturers suddenly find themselves embedding sensors, collecting data, and producing analytics for their customers, industry lines blur. Not only are new competencies needed, but the reason customers choose a product may have less to do with traditional selection criteria than with the data and analytics offered with the product,” says Franks.
The practical advice Franks provides in the book ranges from how to set up an analytics organization to developing and maintaining a corporate culture dedicated to discovery (finding new insights in the data and quickly acting on them) to implementing operational analytics. The Analytics Revolution is an excellent guide to the new business world of blurred industry lines and innovative data products.
If you are ready to move on from understanding the why of analytics today and how to think about it in a broad business and organizational context to a more specific understanding of the how of analyzing data, Data Mining for Dummies by Meta Brown should be your first step. The book was written for “average business people,” showing them that you don’t need to be a data scientist and “you don’t need to be an expert in statistics, a scientist, or a computer programmer to be a data miner.”
Brown is a consultant, speaker and writer with hands-on experience in business analytics. She’s the creator of the Storytelling for Data Analysts and Storytelling for Tech workshops. In Data Mining for Dummies, Brown tells the story of what data miners do.
It starts with a description of a day in the life of a data miner and goes on to discuss in clear, easy-to-understand prose all the key data mining concepts, how to plan and organize for data mining, getting data from internal, public and commercial sources, how to prepare data for exploration and predictive modeling, building predictive models, and selecting software and dealing with vendors. Data Mining for Dummies is an excellent step-by-step guide to understanding data mining and how to become a data miner.
If you are ready to move on from understanding data mining and being a data miner to more advanced tools and applications for data analysis, Data Science for Dummies by Lillian Pierson should be your first step. The book was written for readers with some technical and math skills and experience, but it aims to provide a general introduction to one and all: “Although data science may be a new topic for many, it’s a skill that any individual who wants to stay relevant in her career field and industry needs to know.”
Pierson is a data scientist and environmental engineer and the founder of Data-Mania, a start-up that focuses mainly on web analytics, data-driven growth services, data journalism, and data science training services. “Data scientists,” she explains, “use coding, quantitative methods (mathematical, statistical, and machine learning), and highly specialized [domain] expertise in their study area to derive solutions to complex business and scientific problems.
Data Science for Dummies is an excellent practical introduction to the fundamentals of data science. It provides a guided tour of the data science landscape today, from data engineering and processing tools such as Hadoop and MapReduce to supervised and unsupervised machine learning, statistics and mathematical modeling, using open-source applications such as Python and the R statistical programming language, finding resources for publicly-available data, and data visualization techniques for showcasing the results of your analysis. Stressing the importance of domain expertise for data scientists, Pierson provides detailed examples of applying data science in specific domains such as journalism, environmental intelligence, and e-commerce.
“A lot of times,” says Pierson, “data scientists get caught up analyzing the bark of the trees that they simply forget to look for their way out of the forest.” The three books reviewed here provide a handy map to the maze of data analysis and a safe conduct pass for business executives, IT staff, and students, ensuring that they successfully get in and out of the data forest. Remember, as ones and zeros eat the world, data is the new product and operational analytics, data mining, and data science is the new process of innovation.
You probably heard the news about the CFO of Morgan Stanley becoming the new Google CFO and the on-going migration of Wall Street bankers to Silicon Valley (here and here). But did you know that this is just a surface manifestation of an underlying secular trend, nothing less than a new restructuring of the IT industry and the emergence of completely new developers and sellers of information technology products and services?
You could argue that at any point in time in the last half a century you could say about the fast-changing IT industry that it is not your father’s IT industry. But remarkably, the structure of the IT industry has been very stable, with only one major redrawing of its landscape throughout its history. Today, the “IT industry” is being redefined with two major clusters of companies increasingly competing with the incumbents and reshaping the old IT world order.
First, what did your father’s IT industry look like? Until about 1990, the industry was dominated by IBM and a few smaller companies (e.g., DEC, HP, Wang, Prime). These companies were offering one-stop-shopping, providing to enterprises the entire IT stack, from hardware platforms (including proprietary chips) to peripherals to operating systems to networking to applications and to services.
The advent of the PC and, in particular, the networking of PCs in the 1980s, gave rise to a new tidal wave of digital data. As a result, between 1990 and 2000, the industry’s structure has expanded to include large vendors focused on one layer of the IT stack: Intel in semi-conductors, EMC in storage, Cisco in networking, Microsoft in operating systems, Oracle in databases. IBM saved itself from the fate of DEC, Wang, and Prime by focusing on services. DEC could have saved itself if it realized in time the value of focusing on its biggest competitive advantage—networking—instead of letting a number of focused players enter this market (which Cisco eventually dominated). At the same time, a number of focused PC vendors, primarily Compaq and Dell, carved a larger role for themselves by gradually expanding their reach into other layers of the IT stack, emulating the old model of a vertically-integrated IT vendor. By and large, they were less successful than the new, horizontally-focused IT vendors.
The restructured IT industry, and specifically, the focused, “best-in-class” vendors answered a pressing business need. Digitization and the rapid growth of data unleashed new business requirements and opportunities that called for new ways to sell and buy IT. The new competitive and business pressures to keep more and more data online and for longer duration, to mind and mine the data, to share and move it around, all contributed to the demand for a flexible IT infrastructure where buyers assemble together the pieces of their IT infrastructure from different vendors. Most important, in the late 1980s and early 1990s, businesses fundamentally changed their attitude towards and the scope of what they did with data: From primarily an internal, back-office bookkeeping, “how did we do last quarter?” focus on the past, to external, customer-, partner-, and supplier-oriented data creation, collection, and mining, with a focus on the present and “let’s understand how to serve our stakeholders better.”
Now, how is today’s or tomorrow’s IT industry different from what it was just fifteen years ago? We still have more or less the same dominant players—IBM, Cisco, EMC, Oracle, HP, a reinvigorated Dell, etc. If there has been any marked change, it has been a shift back to a vertically-integrated model, with all of these players providing the whole IT stack (or what they call, with their insatiable appetite for new buzzwords, “converged infrastructures”).
I would argue that this “back to the future” model is on its last legs and the real future belongs to two clusters of companies: Digital natives (e.g., Google, Facebook, Amazon) and large, IT-intensive enterprises (e.g., financial services companies).
In the mid-2000s, a new business need emerged: “Let’s make sense of the mountains of data we continue to accumulate,” focusing on the future and the mining of data to understand better the likely results of different courses of actions and decisions. Unlike the previous shift in the fortunes of the IT industry, the companies driving this shift were not old or new “IT vendors.” They were the companies born on the Web, the digital natives, with data (and ever more data) at the core of everything they did. Google, Facebook, Amazon, Netflix—all built from scratch their IT infrastructure to their own specifications, inventing and re-imagining in the process the entire IT stack. They were the first outsiders insisting that IT was such a core competency for them that they could not trust it to “IT vendors” and could do a more efficient and effective job on their own, thank you.
Google and Amazon later became insiders by offering their infrastructure as a service to small and large businesses. But other companies born on the Web, such as Netflix, simply continued to amass unparalleled knowledge about state-of-the-art IT, develop innovative IT solutions which they then made available to the world as open source software, and hire (or train) the best and the brightest IT professionals. This became a new IT ecosystem, a whole new world of IT far surpassing what was happening in the “IT industry” on any measure of innovation and competitiveness.
Some of the large enterprises which used to be the most important customers of the traditional IT vendors are now joining this new IT ecosystem, reinforcing the reinvention of how IT is developed, sold and bought. Consider these recent news items:
The new “IT vendors” are companies that see IT not only as an integral part of their business strategy but go even further to view IT—and the data they collect and mine–as what their company is all about. This new breed of IT vendors have emerged over the last decade from the ranks of digital natives and they are joined now by large companies that want to get further return—and possibly new sources of revenues—from their large investments in IT. Furthermore, just like the digital natives, these large established companies don’t want to be beholden anymore to the lock-in tactics of traditional IT vendors.
The size of the IT industry worldwide in 2010 was about $1.5 trillion. Gartner predicts that the industry (including telecom) will grow to $3.8 trillion this year. The real “IT industry,” however, is much larger than that, given all the IT-related activities happening outside the traditional boundaries of the industry. And if you include in the “IT industry” everything that’s being digitized—content, communications, consumer electronics, commerce—we are looking at an industry that will grow to at least $20 trillion by 2020. In this largely expanded industry there will probably be still room for the traditional IT players. But the growth spurts, innovation, and new skills will come from today’s outsiders, from the companies whose core competency is the collection, processing, analysis, distribution, and use of data.
Originally published on Forbes.com
[youtube https://www.youtube.com/watch?v=tlEuunxdIVM?rel=0]
Wefi, the Mobile Data Analytics company, analyzed data collected from over 70,000 devices driven over 3.5 million miles on six major interstates.
[youtube https://www.youtube.com/watch?v=PxxUAlm2Jnw?rel=0&w=560&h=315]
[youtube https://www.youtube.com/watch?v=wjmFrqyPnKc?rel=0]
You probably heard the news about the CFO of Morgan Stanley becoming the new Google CFO and the on-going migration of Wall Street bankers to Silicon Valley (here and here). But did you know that this is just a surface manifestation of an underlying secular trend, nothing less than a new restructuring of the IT industry and the emergence of completely new developers and sellers of information technology products and services?
You could argue that at any point in time in the last half a century you could say about the fast-changing IT industry that it is not your father’s IT industry. But remarkably, the structure of the IT industry has been very stable, with only one major redrawing of its landscape throughout its history. Today, the “IT industry” is being redefined with two major clusters of companies increasingly competing with the incumbents and reshaping the old IT world order.
First, what did your father’s IT industry look like? Until about 1990, the industry was dominated by IBM and a few smaller companies (e.g., DEC, HP, Wang, Prime). These companies were offering one-stop-shopping, providing to enterprises the entire IT stack, from hardware platforms (including proprietary chips) to peripherals to operating systems to networking to applications and to services.
The advent of the PC and, in particular, the networking of PCs in the 1980s, gave rise to a new tidal wave of digital data. As a result, between 1990 and 2000, the industry’s structure has expanded to include large vendors focused on one layer of the IT stack: Intel in semi-conductors, EMC in storage, Cisco in networking, Microsoft in operating systems, Oracle in databases. IBM saved itself from the fate of DEC, Wang, and Prime by focusing on services. DEC could have saved itself if it realized in time the value of focusing on its biggest competitive advantage—networking—instead of letting a number of focused players enter this market (which Cisco eventually dominated). At the same time, a number of focused PC vendors, primarily Compaq and Dell, carved a larger role for themselves by gradually expanding their reach into other layers of the IT stack, emulating the old model of a vertically-integrated IT vendor. By and large, they were less successful than the new, horizontally-focused IT vendors.
The restructured IT industry, and specifically, the focused, “best-in-class” vendors answered a pressing business need. Digitization and the rapid growth of data unleashed new business requirements and opportunities that called for new ways to sell and buy IT. The new competitive and business pressures to keep more and more data online and for longer duration, to mind and mine the data, to share and move it around, all contributed to the demand for a flexible IT infrastructure where buyers assemble together the pieces of their IT infrastructure from different vendors. Most important, in the late 1980s and early 1990s, businesses fundamentally changed their attitude towards and the scope of what they did with data: From primarily an internal, back-office bookkeeping, “how did we do last quarter?” focus on the past, to external, customer-, partner-, and supplier-oriented data creation, collection, and mining, with a focus on the present and “let’s understand how to serve our stakeholders better.”
Now, how is today’s or tomorrow’s IT industry different from what it was just fifteen years ago? We still have more or less the same dominant players—IBM, Cisco, EMC, Oracle, HP, a reinvigorated Dell, etc. If there has been any marked change, it has been a shift back to a vertically-integrated model, with all of these players providing the whole IT stack (or what they call, with their insatiable appetite for new buzzwords, “converged infrastructures”).
I would argue that this “back to the future” model is on its last legs and the real future belongs to two clusters of companies: Digital natives (e.g., Google, Facebook, Amazon) and large, IT-intensive enterprises (e.g., financial services companies).
In the mid-2000s, a new business need emerged: “Let’s make sense of the mountains of data we continue to accumulate,” focusing on the future and the mining of data to understand better the likely results of different courses of actions and decisions. Unlike the previous shift in the fortunes of the IT industry, the companies driving this shift were not old or new “IT vendors.” They were the companies born on the Web, the digital natives, with data (and ever more data) at the core of everything they did. Google, Facebook, Amazon, Netflix—all built from scratch their IT infrastructure to their own specifications, inventing and re-imagining in the process the entire IT stack. They were the first outsiders insisting that IT was such a core competency for them that they could not trust it to “IT vendors” and could do a more efficient and effective job on their own, thank you.
Google and Amazon later became insiders by offering their infrastructure as a service to small and large businesses. But other companies born on the Web, such as Netflix, simply continued to amass unparalleled knowledge about state-of-the-art IT, develop innovative IT solutions which they then made available to the world as open source software, and hire (or train) the best and the brightest IT professionals. This became a new IT ecosystem, a whole new world of IT far surpassing what was happening in the “IT industry” on any measure of innovation and competitiveness.
Some of the large enterprises which used to be the most important customers of the traditional IT vendors are now joining this new IT ecosystem, reinforcing the reinvention of how IT is developed, sold and bought. Consider these recent news items:
The new “IT vendors” are companies that see IT not only as an integral part of their business strategy but go even further to view IT—and the data they collect and mine–as what their company is all about. This new breed of IT vendors have emerged over the last decade from the ranks of digital natives and they are joined now by large companies that want to get further return—and possibly new sources of revenues—from their large investments in IT. Furthermore, just like the digital natives, these large established companies don’t want to be beholden anymore to the lock-in tactics of traditional IT vendors.
The size of the IT industry worldwide in 2010 was about $1.5 trillion. Gartner predicts that the industry (including telecom) will grow to $3.8 trillion this year. The real “IT industry,” however, is much larger than that, given all the IT-related activities happening outside the traditional boundaries of the industry. And if you include in the “IT industry” everything that’s being digitized—content, communications, consumer electronics, commerce—we are looking at an industry that will grow to at least $20 trillion by 2020. In this largely expanded industry there will probably be still room for the traditional IT players. But the growth spurts, innovation, and new skills will come from today’s outsiders, from the companies whose core competency is the collection, processing, analysis, distribution, and use of data.
Originally published on Forbes.com
[slideshare id=45132578&doc=pythondatascienceatzillow-150225104833-conversion-gate02]
