Makai's Blog
Month
The pointillist analogy used by Steve Mills in his keynote to describe the picture that can come from big data is fantastic.
I came across this insightful blog post while browsing Hacker News. This particular line struck me:
if you can identify a use case within the space that no one is attacking directly, then your chances of having a breakout success and a product that basically sells itself is much, much greater.
The case used in that post referred to 37Signals competing with email as a tool for project management. 37Signals’ software is not intended to be a stripped-down version of Microsoft Project. Instead, it is an upgrade to email because it provides structure and tracking in handling communications on projects, but with minimal complexity.
If people are still performing an inefficient method of business, such as using only email for project management, there is an opportunity to improve the situation. However, that opportunity’s solution must be less than or equal in complexity to the current method. Otherwise, users will either continuing using the existing method or demand much greater value from the new solution to make it worth the switching costs.
So if you are designing a solution with end users in mind, make sure to compare the ease of use not only to the vastly more powerful system, but the simpler system it could replace.
Fantastic summary by Information Arbitrage about the major trends and problems in big data. Key to the big data revolution is the idea that ubiquitous sensors will fuel unprecedented growth in the amount of data collected.
There’s also some other interesting points: Amazon Web Services (AWS) lock-in, the importance of simplicity in visualization, and that new algorithms are key to extracting value from large data sets.
One area where I disagree with the post is here:
With open source and Internet protocols commoditizing software, the advantage will be derived through data.
While open source continues to make certain classes of software a commodity (for example, web servers), new software will continue to provide value as long as there are application niches that remain unserved by the open source community.
A well-done set of slides showing potential uses of sifting through large data sets.
OpenStack made its first release this week. It’s important because as cloud computing usage continues to grow, there need to be common standards for storing data and computing resources. It will be interesting to see how this nascent project develops over the next several years as cloud computing continues to move into mainstream business operations.
A great blog post by Matt Welsh on how working at Google has changed his perspective on distributed systems and computing in general.
Business intelligence is moving from niche work for enterprise technology users to mainstream employees. These five software-as-a-service (SaaS) applications show just a few examples of how enterprise data can be visualized and used to positively inform business decisions. The applications are also significantly less expensive than most applications by Oracle, Teradata and similar enterprise-servicing firms.
Google, which created the MapReduce algorithm, has recently realized it is not sufficient for real-time response for most big data analytics. Instead, something else will be needed for real-time analysis of enterprise big data but it is still an open question as to what that replacement will be.
Hadoop certainly is not the only NoSQL tool around for analyzing big data. As with any technology project, you first have to understand the problem and the set of constraints you are under before recommending which tools to proceed with.
One of the topics at the Hadoop World Conference this week is how Hadoop can be used for enterprise business intelligence (BI). This article suggests that there are scenarios where enterprises have data that could be used for BI but that are not being kept because it is beyond their enterprise-class storage capacity. With a much cheaper Hadoop cluster of failure-resistant commodity storage servers, enterprises can instead save data for analysis with Hadoop instead of throwing it away.
I was thinking lately about all the computer programs I’ve used throughout my life that I no longer use. A good example is Winamp [1], an application for listening to MP3s. I used it almost every day from my freshman year of high school until about my junior year of college until I bought my first iPod and switched over to iTunes. The MP3 files I had (which themselves were ripped from my collection of CDs before I went off to college) are still on my computer and I use them frequently (The Offspring, Collective Soul, Green Day, etc. never get old).
But I haven’t touched Winamp since I switched to iTunes. And eventually something will cause me to move on from iTunes as well. Programs, like people, die off, but their legacy lives on in the data they create or were used for. That’s why concepts like digital rights management (DRM) and cloud computing data lock-in are so repulsive: eventually when those programs die off, data goes with them. People don’t take their worldly possessions with them when they die. Why should programs?
Eliminating DRM restrictions and open standards for cloud computing are important: the tools we’re using today will be gone tomorrow, but data will remain valuable.
[1] I know technically Winamp isn’t dead: you can download it over at http://www.winamp.com/, but I’m so accustomed to iTunes that there’s no reason for me to take a step backwards. Other programs truly are dead in that the technical knowledge to get an emulator up and running for say, a Zilog processor, is prohibitive to all but the most expert users with lots of time on their hands.
The massive increase in data produced during the past several years will be dwarfed in the upcoming decade by sensor networks. We’re just at the cusp of a major change in how we interact with the world. While living in Charlottesville, I took a course in Fall of ‘07 with a leading expert in the University of Virginia Computer Science department on sensor networks. Scientists are still trying to figure out how to control sensor networks with thousands, and even millions of nodes. New programming models are still being developed to interact with the sensors. When both the hardware and software matures, it will allow unprecedented interaction between people, computers, and the environments they live in. This article provides a fascinating look at some of the potential implications of sensor network data production.
This is a Wall Street Journal article on the business behind screen scraping websites for data. Nielsen is one of the major players in the field and has recently stopped a controversial practice of obtaining data from sources that are protected by user log ins and require agreement to not scrap the sites.
The article is biased towards showing the illegitimate side of collecting data from the Internet and completely avoids the use of application programming interfaces (APIs), which are designed for obtaining information from websites.
Blogs provided an explosion of information early in the 2000s that led to the popularity of RSS feeds and related readers. Twitter has taken the place of RSS feeds for most people today, but perhaps we are at the dawn of a new generation of tools to track the growing information on the Internet.
Hadoop is primarily used today for improved business intelligence and extraction, transformation, load (ETL) processes, but there will be opportunities in the future for integrating discrete data stores into integrated sets. This trend is just starting: most companies will not face this problem for several years because their usage of Hadoop and related NoSQL technologies is in its infancy.
Thingd is building a database of the world’s “things” which can be used by other companies as metadata for combining with other data sources. It’s an interesting concept, but it seems to match earlier work done with Cyc in the 80s and 90s that ultimately proved difficult because many objects are difficult to conceptually delineate from other objects. Still, if they stick to very specific product SKUs, it might work since there are a large, but ultimately finite set of products in existence.