Mike Olson - The Future of Hadoop

Last week I attended GigaOM Structure DATA in New York.  The conference is solely focused on Big Data with a mashup of start-ups, big companies, investors and thought leaders.  

Mike Olson did a masterful job in his interview with Jo Maitland painting a picture of Hadoop and the Big Data Analytics landscape while deftly handling numerous landmines tossed his way by Jo.  It's worth a watching.

Watch live streaming video from gigaombigdata at livestream.com

Big Data Starts with ABCs

If you haven't noticed Big Data has created a lot of buzz lately.  Much of the buzz is from the absolute wow factor of how big is big.  With the number of smart phones nearing 6 billion all creating content, Facebook generating over 30 billion pieces of content a month and data expected to grow at 40% year on year it's easy to imagine big really is BIG.

In fact the digital universe has recently broken the zettabyte barrier which is approximately equal to a thousand exabytes or a billion terabytes.  How big is that?  To give you an idea of scale it would take everyone on the planet posting to Twitter 7*24 for 100 years to generate a zettabybe.

So you get the idea - it’s really big. 

As an IT organization you may be thinking that your own data growth will soon be stretching the limits of your infrastructure. A way to define big data is to look at your existing infrastructure, the amount of data you have now, and the amount of growth you're experiencing.  Is it starting to break your existing processes? If so, where?

“Big” refers to a size that's beyond the ability of your current tools to affordably capture, store, manage,and analyze your data. This is a practical definition since “big” might be a different number for each person trying but unable to extract business advantage from their data.

When we talk to our customers, we find that their existing infrastructure is breaking on three major axes:

1. Complexity.  Data is no longer about text and numbers; It includes real-time events and shared infrastructure. Data is now linked at high fidelity and includes multiple types. The sheer complexity of data is skyrocketing. Having to apply normal algorithms for search, storage and categorization is a lot more complex.
2. Speed.  How fast is the data coming at you? High definition video, streaming over the Internet to storage devices, to player devices, full motion video for surveillance – all of these have very high ingestion rates. You have to be able to keep up with the data flow. You need the compute, network and storage to deliver high definition to thousands of people at once, with good viewing quality. For high performance computing you need systems that can perform trillions of operations and store pedabytes of data per second.
3. Volume.  For all of the data you are collecting and generating you have store it securely and make it available for ever. IT teams today are having making decisions about what is “too much data”. They might flush all data each week and start again. But there are certain applications like healthcare where you can never delete the data. It has to live forever.

These trends in data growth are something we at NetApp have been following for quite a while now.  We’ve been enhancing ONTAP to deal with the scale needed to handle large repositories of data and we have also made strategic acquisitions anticipating the need for high density high performance (Engenio) and infinite content repositories (Bycast).

In conversations with our customers dealing with the onslaught of data we have noticed 3 important use cases that are stretching the limits of their existing infrastructure.

We’ve named these axis’ the ABCs of Big Data.

• Analytics.  - Analytics for extremely large data sets to gain insight and take advantage of that digital universe, and turning it into information. Giving you insight about your business to make better decisions.
• Bandwidth - Performance for data-intensive workloads at really high speeds.
• Content - Boundless secure scalable data storage that allows you to keep in forever.

Moving from Clouds to Big Data

As many of you know I joined NetApp about a year ago.  I've spent most of my time here developing the go to marketing strategy for NetApp's OnCommand Management Software.  Last week the new portfolio was announced as part of the our Cloud launch and I posted how OnCommand supports the Cloud with four fundamental elements in my NetApp blog called Clouds OnCommand.

It was a lot of fun and I learned a lot about the IT management market segment and how Virtualization is causing us all to look at new creative and innovative ways to manage the ensuing complexity.  Check out the OnCommand story on YouTube.

It's been a refreshing change working at NetApp, a really great company with innovative products that is growing and doing well. I am thoroughly enjoying it.

Now I have taken on a new assignment to define our Big Data marketing strategy.  Big Data is getting a lot of visibility as the proliferation of data dominates the landscape.  The best defination of Big Data I've heard is that “Big Data” refers to datasets whose size is beyond the ability of typical database software tools to capture, store, mange and analyze"

I'm looking forward to digging into this space and understanding how I can help NetApp define a winning strategy.  More to come soon.

Season's Greetings

Enjoy this animated card from Jacquie Lawson.

Jet Lag Explained

See this Interesting study on Jet Lag. Now I know why I always feel like so bad after changing time zones.  However I have my own measurement - average MPH. You don't feel normal until your average MPH is below 4 (normal human speed) When you step of the plane after a 12 hour flight across 6 time zones your MPH is 600. It takes 2 days to get below 4 which is when you'll start to feel better.

Google Maps For Cycling

Google today introduced maps for cycling.  In my trial use some of the routes are lacking but it will only get better with time with crowd sourcing feedback.  I have been using a Google Maps mashup to store and share bicycle routes for years.  I've tagged my routes on delicious here.

Here's the video that shows how it works.

Broad-winged Hawks Nesting in San Jose

Today I discovered that we have a hawk family nesting in the tall palm tree on our street.  It was quite a sight.  I took these pictures this morning.  I tried to identify them and as best as I can tell they are Broad-winged Hawks.

Can anyone verify?

The Serendipitous Beauty of iPhone Use Cases

It seems not a day goes by that I don't discover a new use case for my iPhone.  My latest happened driving home from San Mateo.  It was 1:30PM,  Terry Gross was wrapping up her interview with James Cameron and I caught the last 5 minutes.  Having just watched the Avatar I really wanted to hear the whole interview.  Then it hit me. I could use the new NPR iPhone App to listen to it.  So I went to Programs - Fresh Air - James Cameron - Listen Now, connected my iPhone to the car stereo and VIOLA listened to the entire interview.

How cool is that?

When iPhone users meet, the conversation inevitable turns to "What cool applications have you discovered?".  That exploration is even more compelling as the apps can be downloaded and tried right there and then. It's hard to imagine how the iPhone challengers will ever catch up.  Efforts by Sony Ericsson, LG Electronics, Samsung and 21 others to create an App Store for all their divergent phones will be lost in complexity and fraught with political land mines.

As I noted in "Goodbye Laptop - Hello iPhone" the iPhone experience has changed the way I use computers in many profound ways. Now I can't wait for my next serendipitous iPhone-user meeting to learn another life changing use case.

Use Tactical Urgency to Introduce New Technologies

Recently Kelly Emo, SOA Product Marketing Manager at HP Software wrote in her post "Is your SOA in Action? Four ways to keep it that way.." One element she offers in her Obvious Insite # 3 is to use your SOA governance to drive adoption of new technologies such as RIA and Cloud Computing.

Much of my work over the last decade in getting new technologies adopted by enterprise IT has been under the proverbial banner of "Herding Cats". I have learned is that it is very hard to introduce new technology as part of the strategic plan and that SOA governance is often more of a roadblock than a driver.

Technologies that help integrate data between silos require the endorsement of too many chiefs and even with executive stakeholder support strategic efforts can fail under the urgency of immediate problems. In my experience with technology adoption of RIAs and EBSs from both the vendor and purchaser point of view I have found that even with a strong business imperative the inevitable urgency of tactical requirements derails the best strategic plans.

A better approach is to use a tactical urgency to demonstrate a real benefit. In this way it is possible to establish a beach head through a small project that demonstrates a believable ROI. From there you can position the technology successfully in the broader strategy.

Visualize Facebook Relationships

Facebook has seen tremendous growth over the year. The Mary Meeker: Facebook Is Eating Your Lunch And Dinner post at All Facebook singles out the statistic from her presentation at Web2.0 that Facebook is the largest share gainer of online usage over the past 3 years. Indeed at over 300M users, if Facebook were a country it would be the forth largest behind only China, India and the US.

With all the hype on social networking I thought it would be interesting to highlight again a dynamic application called the CurlGraph. The CurlGraph shows how your Facebook friends are related to each other and is especially fun when you see how your circles of friends are interrelated.

If you are on Facebook I urge you to give it a try and to suggest it to your friends.

I made a brief demo video you can post to Facebook and you'll find the Curl website has instructions on installation.

From the Desk

Richard Treadway

Risks In The Cloud

The Internet is buzzing today with the news that Microsoft subsidiary Danger has lost the server stored data on T-Mobile's Sidekick phones. See Did Microsoft Just Kill the Cloud? and Sidekick outage says more about the future of 'Pink' than Microsoft's cloud and The cloud: no place for amateurs

While there are not a lot of Sidekick users out there this incident has once a gain forced the question: How safe is your data in the cloud?

Have we developed a false sense of security as we depend more and more on data out of our immediate control on servers managed by Google, Salesforce, Yahoo, Facebook and Twitter? I dare say not many of us has thought through a disaster recovery plan if all our emails, contacts, photos and documents were to disappear overnight.

The Power Law of Social Networks

Yesterday Dion Hinchcliffe lists 22 Power Laws of the Emerging Economy. It's an interesting post worth a read but I think he omitted one of the most important "The Power Law of Social Networks" Social Networks themselves are defined by a power curve. Albert-Laszlo Barabasi illustrates this in his book "Linked, The New Science of Networks." This interconnectivity drives the information age where popular nodes can rise up quickly. Like all social networks the Internet has a few nodes with millions of connections and millions of nodes with very few connections. Increasingly 6 degrees of separation is becoming 3 degrees of separation.

iPhone 3G Network Slow?

Has your iPhone 3G been going slower and slower? Mine has. Now this from Adrian Kingsley-Hughes at ZDnet - iPhone users strangling the AT&T network.

What is dawning on AT&T is that the iPhone changes the way people use phones.

"Up until the iPhone, smartphones were cellphones with a browser clumsily attached. The iPhone is basically a pipe to the internet with a phone attached."

This is exactly what I experienced when I wrote "Goodbye Laptop - Hello iPhone"

It is precisely whythe 9 million iPhone users are using more of the network than AT&T's 20 million other cell phone users.

Hopefully AT&T can continue to build out the plumping or Apple can sign another provider that can keep up.

Can Social Media Sell Wine?

Steve Heimhoff asks the question "Has social media ever sold anything, besides itself?" It's a great question and has prompted a good discussion on his blog.

There is a lot of evidence that making social media a strategic part of a winery's overall marketing mix makes good business sense. I've been working to create a concrete ROI for social media's use in wineries and in my research thus far I've have found a lot of evidence that a dollar spent on social media can be much more effective than a dollar spent on traditional media.

According to Nielson online we trust the recommendations of friends (90%) and online reviews (70%) a lot more than traditional media (62% and lower). A good wine review from a friend is going to influence me more than a good review by a known wine critic. Forrester's recent Social Technology report found that the prime wine buying demographic of 35+ year olds grew their online participation by 60% last year prompting them to conclude:

"... marketers can now safely create social media marketing for people ages 35 and older."

In a recent survey by Viralvines.com on Twitter use in wineries, in answer to the question "How has your presence on Twitter helped your business? 96.4% responded that "It has helped us engage more with our customers"

Furthermore the conversation about a winery's brand is happening even if a winery chooses to ignore it and that carries a real risk of negative effects. According to Nielsen BuzzMetrics 25% of search results for the worlds largest brands are links to user generated content. This extends to niche brands as well and a bad review without the appropriate response can have long lasting effects.

Forrester's Social Media Profile Tool

Last week Forrester released this fun social media Consumer Profile Tool.

The tools lets you query their 2009 data by age, county and gender and shows the percentages of participants in each of their defined participation categories: creators, critics, collectors, joiners, spectators and inactives.

This reinforces the point that any social media strategy should start with a good understanding of your customers and how they communicate.




See Social Technologies Ladder for an explanation of the categories.

Thanks to Forrester for providing this fun way to access their data.

50% of US Online Adults Use Social Media Networks

According to a new research report on "The Broad Reach Of Social Technologies" by Sean Corcoran at Forrester 1 in 2 US online adults are in a social network and 4 in 5 use social media at least once a month.

Interestingly the 35 and older group saw the most growth at 60% over last year with 38% visit social media sites regularly. In the 35+ somethings group with 1 in 5 are creators and 70% spectators. The conclusion:

"... marketers can now safely create social media marketing for people ages 35 and older."

This should be another signal to brands whose primary segment is the 35+ that the time is now to invest in social media marketing.

Are You Ready For Socialnomics?

Some very enlightening statistics on social media in this video from Socialnomics-Social Media Blog posted by c.net. While social media is possibly at the top of the hype curve there is no denying that it is driving a fundamental change in how corporations sell and how people make buying decisions.

New San Jose Airport

After bad mouthing San Jose airport for years, the new terminal is pretty nice.

Richard

Posted via email from Richard Treadway's Live Stream

The Importance of the Long Tail

Nielson recently posted data on how engaging brand in the "long tail" are fairing. The data shows that we spend less time at long tail sites and more time at the top sites.



The conclusion is that long tail isn't where the future business is at least not just yet.

"What does this amount to? As much as anyone thinks the future is in the long tail, it’s just not the case-at least not yet. In fact, consumers feel more comfortable on large, mass media sites. We know the Internet is changing. We know there are more blogs, boards, tweets and social networks than ever before. But what’s also clear is that while the Internet itself is fragmenting (like all other media), people continue to spend their time on the sites that offer them the most options and functionality."

But the numbers don't support that conclusion. The numbers are simply a reflection of the fact the long tail is a power curve.

Chris Anderson’s theory of the long tail states that as the cost of distribution approaches zero the demand is infinitely eclectic. The long tail is a special power curve that stretches out forever. With a power curve you would expect people to spend the most time at the top sites and the least time in the less popular sites. Most people have more than one specialized interest and will spend a little time at each long tail site. The entry point to the long tail is usually the top sites explaining why there is more time spent there.

A more interesting measure would be total time in the long tail not the average time.

When Will Internet Access Be Like Electricity

My recent stay at the Westin Kierland reminded me we are still a long way from Internet access being treated like a service utility. The Internet service at the Westin was particularly expensive and very SLOOW. Additionally the business center charged for use of their computers by the minute requiring you to submit a credit card to a reader that held it for the duration of your session. It cost me $5 to login and print my boarding pass.



Additionally there was no free WIFI access in the lobby. When I complained to the concierge I was told the reason... to discourage Internet use. Well it worked, I barely used it. But that attitude is bucking what is surely the trend to a more readily accessible Internet. Think of it; Charging for Internet access in a hotel is like charging for use of the electricity. Think how many complaints there would if a hotel required you to swipe your credit card to turn on the lights? Why should Internet access be any different?