Friday, August 09, 2013

Was I A Victim of Big Data?

On a recent trip to Phoenix my flight was delayed then cancelled.  Through rerouting and other delays what should have been a 1.5 hour direct flight turned into a 12 hour ordeal.

I wonder was I a victim of Big Data?  Let me explain.

As a Big Data marketeer one of my favorite stories of how Big Data can force changes in the way you do business and cause you to call into question recommendations that are not intuitive is the following hypothetical dilemma.
You are the flight operations manager of an airline.  You have 2 planes about to depart.  It's snowing hard.  The airport calls down to inform you that only 1 of your 2 departures will be granted permission  to depart before the airport will shut down.  One plane has 4 passengers on board, the other is full with 200 passengers.  
What do you do?
You run your new "Flight Operations Optimizer" application.  It's a new "Big Data" application that calculates the down stream 72 hour impact of canceling a flight based on multiple data sets including all passengers impacted, expected weather delays at all downstream destinations, airplane maintenance schedules and crew schedules.
The Flight Operations Optimizer comes back and advises that you let the flight with 4 passengers depart.  That alternative has the least down stream impact to the airline - A better business outcome.
To which you say WHAT! That can't be right.  If I do that I'll have 200 pissed of passengers at the counter to deal with and furthermore I won't make my goal of - most passenger on time departures!
The story points out 2 major elements of Big Data.

  1. If you can analyze enough data often kept in different silos the results can well be counter intuitive.  Leading you to dismiss it and still take the decision that doesn't lead to the best business outcome.
  2. The employee goals you have in place to drive the best corporate outcomes might be driving behaviors that don't actually don't accomplish that.
Now back to my bad travel karma story.

Moments before boarding we were told our flight is now departing from another gate.  Quickly we hurried over to the new gate only to be told our flight was delayed 4 hours.  What happend?

From eavesdropping on the gate agents conversation I learned that the plane for the flight to LA had a mechanical and that our plane was allocated to the LA flight. Our flight was delayed while they looked for another plane.  2 hours later our flight was canceled. 

Could it be that the airline flight operations manager ran the "Flight Operations Optimizer" application the result being that giving our plane to the LA flight had the least down stream impact?

Then after most of the passengers on our flight found alternatives to waiting 4 hours there were very few left on our delayed Phoenix flight.  So the "Flight Operations Optimizer" application advised it be cancelled.

And that is how I suspect I fell victim to Big Data.  Best business outcomes don't always mean best personal outcomes.

Saturday, December 08, 2012

The Big Data Inflection Point

NetApp has offered Big Data Solutions since May of 2011.  We offer a portfolio of 10 solutions that address the major use cases of Big Data.   These solutions are based on both our storage platforms: FAS with Clustered Data ONTAP and E-Series with Santricity.

The Big Data market is loud and confusing.  In fact Big Data was named the most confusing term in IT this year surpassing Cloud which is now number 2.  Most of this confusion is the result of the fact that no two cases of Big Data are alike.  Additionally there are new technologies like Hadoop and it’s ecosystem of tools and applications that are causing a disruption to analytic technologies resulting a many innovations and considerable VC investment in start-up companies.

Big Data has captured the imagination of many enterprises as documented success stories point to new ways of doing business that change the game and result in considerable competitive advantage or significantly better business outcomes.

NetApp has a credible seat at any customer discussion about Big Data by way of our considerable experience in managing data at scale.  Our largest customer has over an Exabyte of data and we have hundreds of customers with over ten petabytes.  Many of the storage efficiency innovations that NetApp has lead such as deduplication and thin provisioning have lead to contemplation of “keep forever” data strategies.  The “delete key” is no longer the answer to Big Data.

What makes Big Data different is that customers reach an inflection point where they can no longer continue to what they did yesterday but just a little more.  Indeed they must fundamentally rethink their data storage strategies.   It is at that inflection point where without deployment of new approaches and technologies data growth and Big Data can become a liability or with the right approach become a propellant to the business.

It is at this inflection point that NetApp can be your trusted partner to help customers use Big Data to grow their businesses efficiently and flexibly. 

Saturday, March 31, 2012

Mike Olson - The Future of Hadoop

Last week I attended GigaOM Structure DATA in New York.  The conference is solely focused on Big Data with a mashup of start-ups, big companies, investors and thought leaders.  

Mike Olson did a masterful job in his interview with Jo Maitland painting a picture of Hadoop and the Big Data Analytics landscape while deftly handling numerous landmines tossed his way by Jo.  It's worth a watching.

Watch live streaming video from gigaombigdata at

Saturday, October 29, 2011

Big Data Starts with ABCs

If you haven't noticed Big Data has created a lot of buzz lately.  Much of the buzz is from the absolute wow factor of how big is big.  With the number of smart phones nearing 6 billion all creating content, Facebook generating over 30 billion pieces of content a month and data expected to grow at 40% year on year it's easy to imagine big really is BIG.

In fact the digital universe has recently broken the zettabyte barrier which is approximately equal to a thousand exabytes or a billion terabytes.  How big is that?  To give you an idea of scale it would take everyone on the planet posting to Twitter 7*24 for 100 years to generate a zettabybe.

So you get the idea - it’s really big. 

As an IT organization you may be thinking that your own data growth will soon be stretching the limits of your infrastructure. A way to define big data is to look at your existing infrastructure, the amount of data you have now, and the amount of growth you're experiencing.  Is it starting to break your existing processes? If so, where?

“Big” refers to a size that's beyond the ability of your current tools to affordably capture, store, manage,and analyze your data. This is a practical definition since “big” might be a different number for each person trying but unable to extract business advantage from their data.

When we talk to our customers, we find that their existing infrastructure is breaking on three major axes:

  1. Complexity.  Data is no longer about text and numbers; It includes real-time events and shared infrastructure. Data is now linked at high fidelity and includes multiple types. The sheer complexity of data is skyrocketing. Having to apply normal algorithms for search, storage and categorization is a lot more complex.
  2. Speed.  How fast is the data coming at you? High definition video, streaming over the Internet to storage devices, to player devices, full motion video for surveillance – all of these have very high ingestion rates. You have to be able to keep up with the data flow. You need the compute, network and storage to deliver high definition to thousands of people at once, with good viewing quality. For high performance computing you need systems that can perform trillions of operations and store pedabytes of data per second.
  3. Volume.  For all of the data you are collecting and generating you have store it securely and make it available for ever. IT teams today are having making decisions about what is “too much data”. They might flush all data each week and start again. But there are certain applications like healthcare where you can never delete the data. It has to live forever.

These trends in data growth are something we at NetApp have been following for quite a while now.  We’ve been enhancing ONTAP to deal with the scale needed to handle large repositories of data and we have also made strategic acquisitions anticipating the need for high density high performance (Engenio) and infinite content repositories (Bycast).

In conversations with our customers dealing with the onslaught of data we have noticed 3 important use cases that are stretching the limits of their existing infrastructure.

We’ve named these axis’ the ABCs of Big Data.

  • Analytics.  - Analytics for extremely large data sets to gain insight and take advantage of that digital universe, and turning it into information. Giving you insight about your business to make better decisions.
  • Bandwidth - Performance for data-intensive workloads at really high speeds.
  • Content - Boundless secure scalable data storage that allows you to keep in forever.

Thursday, June 23, 2011

Moving from Clouds to Big Data

As many of you know I joined NetApp about a year ago.  I've spent most of my time here developing the go to marketing strategy for NetApp's OnCommand Management Software.  Last week the new portfolio was announced as part of the our Cloud launch and I posted how OnCommand supports the Cloud with four fundamental elements in my NetApp blog called Clouds OnCommand.

It was a lot of fun and I learned a lot about the IT management market segment and how Virtualization is causing us all to look at new creative and innovative ways to manage the ensuing complexity.  Check out the OnCommand story on YouTube.

It's been a refreshing change working at NetApp, a really great company with innovative products that is growing and doing well. I am thoroughly enjoying it.

Now I have taken on a new assignment to define our Big Data marketing strategy.  Big Data is getting a lot of visibility as the proliferation of data dominates the landscape.  The best defination of Big Data I've heard is that “Big Data” refers to datasets whose size is beyond the ability of typical database software tools to capture, store, mange and analyze"

I'm looking forward to digging into this space and understanding how I can help NetApp define a winning strategy.  More to come soon.