Makai's Blog

The trend of moving IT infrastructure from private data centers to public or hybrid cloud infrastructure appears inevitable. The media has discussed cloud trends for years and now other terms such as big data are taking cloud computing’s place as “the new new thing” in business IT. But it’s possible big data will actually reverse some of the movement towards cloud computing.

To see why big data and the cloud are not entirely compatible, we have to look at technology limitations. Daily transfers of large files in the terabyte, petabyte, and beyond range remains slow and expensive. Bandwidth will continually increase, but so will the amount of data generated and saved, especially after sensor networks come into widespread usage.

So a “big data + cloud computing” model for the next 5-10 years will likely involve the following pieces:

Multiple Local Data Centers

1. collect data
2. store all collected data locally
3. perform basic analysis
4. compression

Centralized Cloud Operations

1. aggregate locally-processed data
2. acquire data from a wide range of sources, both proprietary and public
3. perform complex analysis on the entire data set

The complexity is in determining which operations should be performed locally to minimize data transfer versus sending everything to the centralized infrastructure. But as long as there’s business value that can be gained from analyzing very large data sets to improve operations, the complexity will be overcome.

Further Resource: Big Data Is On A Collision Course With the Cloud (GigaOm)

High-end niche technology skill sets on the edge are expensive. That’s part of what contributes to Hadoop’s slow adoption rate, reports GigaOm.

Like all useful technologies, the Hadoop hiring problem will get better as the tools mature and move into mainstream skill sets. Pig helps, as does Amazon’s Elastic MapReduce by introducing (non-Lisp) programmers to the MapReduce paradigm.

A bigger question raised in GigaOm’s article on whether Groupon can compete with Facebook isn’t the money that is being thrown around to top-quality big data and Hadoop developers. It’s how much interesting data Groupon have to analyze. While every organization is collecting a dramatically increasing amount of data today, it’s questionable whether Groupon is as mature in its collection, analysis, and visualization efforts as Facebook. While some engineers would be excited to join a fledgling data analysis effort at Groupon, Facebook has both the data collection maturity and massive scale of data to make any data scientist excited.

The question of data maturity will come up for mainstream companies as they try to get a handle on their big data challenges. It will be difficult to lead top data scientists away from organizations diligent in collecting and analyzing their data to immature companies that “just don’t get it” when it comes to how data can inform a business strategy.

Link: Can Groupon Compete With Facebook in Hadoop Hiring? (GigaOm)

Enterprises need to differentiate between little and big data. “Big data” remains a moving target without an exact definition, but it has certain criteria:

1. Collected over an extended period of time, often years and even decades
2. A mix of unstructured, semi-structured, and structured data
3. Large enough in size that it is difficult for traditional business intelligence platforms to analyze it in real-time
4. Unknown data quality - often a mix of proprietary and public data sources

“Little data” is:

1. Collected over a short duration of days, weeks, or months
2. Mostly structured data
3. Easily analyzed by traditional business intelligence platforms
4. High data quality - usually proprietary data collected based on a well-defined business need

Traditional data warehousing and business intelligence approaches are not enough for leading-edge companies to stay in their positions. Companies who are not yet feeling the data deluge but will be as big data goes mainstream need to ensure their “little data” needs are covered by strong business intelligence practices. As the data deluge grows over the next several years, companies that have thought through the implications and can act will generate more business value than their competitors based on their data best practices.

Related link: Distinguishing Between Little and Big Data 

The folks at ReadWriteWeb collected three great presentations on R covering:

1. R for web development in 2011
2. R packages the experts use, and
3. R usage by the New York Times Graphics Department.

Link: 3 Presentations on R (ReadWriteWeb)