Makai's Blog

Universities in Canada and elsewhere are rebelling against the high costs of proprietary, closed systems for publishing and retrieving scientific information. This development is a positive outgrowth of the reduction in government funding available to universities.

Open access to scientific research that is funded by taxpayer dollars is a critical component towards advancing our data-driven society. Currently large datasets and scientific research are not easily accessible outside of universities that pay hundreds of thousands of dollars in subscription fees a year. Not only does this limit the public’s ability to review scientific data themselves, it adds to the amount of fraudulent science performed.

The trend towards open access of scientific research and results is a very positive step that will help transform higher education institutions into a more transparent, accountable entities in the next several years.

It will be a great day for science when data produced by experiments are freely available on data websites such as InfoChimps.

The trend of moving IT infrastructure from private data centers to public or hybrid cloud infrastructure appears inevitable. The media has discussed cloud trends for years and now other terms such as big data are taking cloud computing’s place as “the new new thing” in business IT. But it’s possible big data will actually reverse some of the movement towards cloud computing.

To see why big data and the cloud are not entirely compatible, we have to look at technology limitations. Daily transfers of large files in the terabyte, petabyte, and beyond range remains slow and expensive. Bandwidth will continually increase, but so will the amount of data generated and saved, especially after sensor networks come into widespread usage.

So a “big data + cloud computing” model for the next 5-10 years will likely involve the following pieces:

Multiple Local Data Centers

1. collect data
2. store all collected data locally
3. perform basic analysis
4. compression

Centralized Cloud Operations

1. aggregate locally-processed data
2. acquire data from a wide range of sources, both proprietary and public
3. perform complex analysis on the entire data set

The complexity is in determining which operations should be performed locally to minimize data transfer versus sending everything to the centralized infrastructure. But as long as there’s business value that can be gained from analyzing very large data sets to improve operations, the complexity will be overcome.

Further Resource: Big Data Is On A Collision Course With the Cloud (GigaOm)

High-end niche technology skill sets on the edge are expensive. That’s part of what contributes to Hadoop’s slow adoption rate, reports GigaOm.

Like all useful technologies, the Hadoop hiring problem will get better as the tools mature and move into mainstream skill sets. Pig helps, as does Amazon’s Elastic MapReduce by introducing (non-Lisp) programmers to the MapReduce paradigm.

A bigger question raised in GigaOm’s article on whether Groupon can compete with Facebook isn’t the money that is being thrown around to top-quality big data and Hadoop developers. It’s how much interesting data Groupon have to analyze. While every organization is collecting a dramatically increasing amount of data today, it’s questionable whether Groupon is as mature in its collection, analysis, and visualization efforts as Facebook. While some engineers would be excited to join a fledgling data analysis effort at Groupon, Facebook has both the data collection maturity and massive scale of data to make any data scientist excited.

The question of data maturity will come up for mainstream companies as they try to get a handle on their big data challenges. It will be difficult to lead top data scientists away from organizations diligent in collecting and analyzing their data to immature companies that “just don’t get it” when it comes to how data can inform a business strategy.

Link: Can Groupon Compete With Facebook in Hadoop Hiring? (GigaOm)

Enterprises need to differentiate between little and big data. “Big data” remains a moving target without an exact definition, but it has certain criteria:

1. Collected over an extended period of time, often years and even decades
2. A mix of unstructured, semi-structured, and structured data
3. Large enough in size that it is difficult for traditional business intelligence platforms to analyze it in real-time
4. Unknown data quality - often a mix of proprietary and public data sources

“Little data” is:

1. Collected over a short duration of days, weeks, or months
2. Mostly structured data
3. Easily analyzed by traditional business intelligence platforms
4. High data quality - usually proprietary data collected based on a well-defined business need

Traditional data warehousing and business intelligence approaches are not enough for leading-edge companies to stay in their positions. Companies who are not yet feeling the data deluge but will be as big data goes mainstream need to ensure their “little data” needs are covered by strong business intelligence practices. As the data deluge grows over the next several years, companies that have thought through the implications and can act will generate more business value than their competitors based on their data best practices.

Related link: Distinguishing Between Little and Big Data 

The folks at ReadWriteWeb collected three great presentations on R covering:

1. R for web development in 2011
2. R packages the experts use, and
3. R usage by the New York Times Graphics Department.

Link: 3 Presentations on R (ReadWriteWeb)

GigaOm is reviewing topics from last week’s Structure Big Data conference. One subject is that horizontal markets for big data analysis and visualization are quickly filling up with offerings. Those offerings may not be the right answers to the big data challenge but there is certainly a lot of competition in the space. The author of this post, “Why Big Data Startups Should Take a Narrow View,” advocates looking at vertical markets.

I agree with looking into vertical markets for a specific fundamental reason. Data exploration, analysis, and visualization generate value only when combined with domain-specific knowledge. Data visualizations in isolation may look interesting but value is only created when the information is acted upon. It is much easier to act on specific information created by big data from your own industry than across industries that are not pertinent to your line of business.

DataStax’s new product Brisk is a merger of Cassandra, Hadoop, Hive. While Apache HBase is the most common NoSQL data stores used with Hadoop, Brisk attempts to merge Cassandra’s distributed data stores with Hadoop’s MapReduce capabilities (and Hive’s job tracking tools). The general concept is that Hadoop can be run against one of Cassandra’s multiple distributed data store without impacting the performance of the other data stores.

Brisk is an interesting concept but the proof is in whether the product delivers the best of both Cassandra and Hadoop without transferring the weaknesses. Since Cassandra is eventually consistent (trading consistency for availability and partition tolerance in the CAP Theorem), how that impacts Hadoop’s MapReduce jobs remains an open question.

I’ve requested a copy of Brisk through DataStax’s website and when I get my hands on it I’ll create a further technical write up.

Link: DataStax Brisk

This amazing 100-second video of world events during human history was created by scraping all geo-tagged articles from Wikipedia. The visualization is obviously heavily based on Western world history since most English Wikipedia articles are written by contributors in the United States and Western Europe.

The video is worth watching for the final outline of the continents based on events alone.

Link: A History of the World in 100 Seconds

Here is a summary with links to my MS MIT retrospective posts:

• Part 1 - Retrospective Introduction and Program Value Summary
• Part 2 - MS MIT Curriculum Overview
• Part 3 - Mod 0 and Mod 1
• Part 4 - Mod 2 and Mod 3
• Part 5 - Mod 4
• Part 6 - Post-Graduation, Program Value, and Conclusion

This is the final part of my retrospective on the UVA Master’s in Management of Information Technology program I graduated from in May 2010. See part 1, part 2, part 3, part 4, and part 5 for context.

Graduation and Beyond

Graduation at UVA is a beautiful tradition. Go to it! It provides closure to the MS MIT experience.

After wrapping up Mod 4 and graduating it takes time to transition back into a normal work/life balance. Here are some tips and lessons to ease the transition:

• When you finish your project defense in May and then graduate, it will take a couple of months to unwind. You will wake up on Saturday mornings with a sense of urgency until you realize you don’t have to read and analyze several case studies over the weekend. A vacation can help a lot.
• Keep applying the concepts learned in the program. It’s much easier to remember accounting and finance when you refresh your memory often.
• Go to alumni events. Nicole Fitzwater is the Director of Alumni Relations and does a fantastic job of setting up events for networking and catching up with classmates.

Program Value

The MS MIT program provides most of its value by teaching business school topics in the context of technology. If you are in a technology field and are interested in MBA programs but do not want to study unrelated topics, the MS MIT program is a great choice.

The class is composed of a range of experience levels. My classmates had between 3 and 25+ years of experience and averaged 14 years of experience. There was roughly a 50/50 split between commercial and government positions (either consulting or employees).

A big portion of value in the program is gained by learning from your classmates. The program, particularly the Charlottesville section, facilitates interaction between classmates to assist the learning process outside of the classroom.

My One Criticism of the Curriculum

Overall I had a great experience with the MS MIT program. However, I was disappointed in one subject area. Technology and entrepreneurship is not covered in the curriculum. We learned a disproportionate amount about issues in large organizations. Many of the challenges we discussed in class were symptoms of big bureaucratic organizations.

For example, one of our case studies was on implementing a CRM system at a Fortune 500 organization. The political challenges were more of an issue than the technology problems. Smaller companies would not face the same challenges because they would be more likely to implement a standard software-as-a-service solution.

However, most of the subject matter in the program could apply to established (not startup) organizations. If you are looking to learn more about applying technology to create startups, there are other Master’s programs out there with more of an emphasis on that subject.

Conclusion

That’s my retrospective on the MS MIT program. In hindsight it was a great boost to my career and such a pleasure to have been a part of despite the heavy workload.

I hope this retrospective is helpful for prospective and current MS MIT students. If you have further questions or feedback, please email me at [email protected].

This is part 5 of my retrospective on the UVA Master’s in Management of Information Technology program I graduated from in May 2010. See part 1, part 2, part 3, and part 4 for context.

Mod 4

Mod 4 is by far the most difficult, intense, and valuable experience in the MS MIT program. Prepare for emotional ups and downs. One day you will have everything lined up with your company and course work then the next day your group has to scramble because a project deliverable was not acceptable.

Besides the Mod 4 Capstone Project where you work with a company, you will study corporate strategy, managerial accounting and finance, marketing, and behavioral event interviewing. After completing Mod 4, you will be able to hold your own against any top MBA student in corporate strategy discussions. We learned Michael Porter’s Five Forces and similar corporate strategy subject material. We read and discussed Harvard Business Review case studies for context. All of the material was grounded in technology subject matter. I felt the focused subject matter approach worked very well because we analyzed case studies that we had experience dealing with.

Accounting and finance was also beneficial. I never studied accounting or finance before the MS MIT program. Learning finance was difficult but I now have working knowledge of calculating discounted cash flows, net present value, weighted average cost of capital, and related concepts. Reading balance sheets, income statements, and statements of cash flows is important when working with publicly traded companies because you can fully understand and appreciate the business challenges they face.

I found the Behavioral Event Interviewing (BEI) class very beneficial. Every time I go into an interview and get asked BEI questions, I know how to structure my answers so they are appropriate. Studying BEI boosted my interview confidence because I know more about what the interviewer is looking for.

It’s easy to get lost in the Mod 4 class work because there is so much of it. But the capstone project is ultimately the biggest part of the grade. There are two major presentations in addition to written reports:

1. Your company and its industry analysis. Tell a compelling story for what your company does, what aspects it does better and worse than rivals, why it is better or worse, and where the company’s strategy is taking it. This presentation takes place in late February or early March.
2. Your revenue-generating IT initiative for your company. Quickly review your company’s strategy then dive into your IT initiative, why it is critical to achieving the company’s strategy, how the initiative will be implemented, and the financials around implementation and execution. This presentation occurs the second to last day of the program.

After the final presentation, the group returns on the last day to defend its work to the professors. Prepare to answer questions that time did not allow for after the presentation the previous day.

Advice for Mod 4:

• Focus on the story to your capstone project. It’s easy to get lost in industry data analysis. You must tell a compelling story to create a truly great IT initiative for the company you are working with.
• If you’ve never studied corporate strategy before, learn the fundamentals of Michael Porter’s Five Forces and industry analysis then run with it on the capstone project. There is too much information to learn so you need to get the basics and move on because time is precious in Mod 4.
• Grab drinks once your group successfully defends its capstone project, you’ll need it!

My advice for adjusting to life after graduation, analysis of the MS MIT program’s value and conclusions are found in part 6.

This is part 4 of my retrospective on the UVA Master’s in Management of Information Technology program I graduated from in May 2010. See part 1, part 2, and part 3 for context.

Mod 2

Mod 2 covers managing information technology projects. A lot of the material is related to PMBoK (Project Management Body of Knowledge). Some people in the class earn their Project Management Professional (PMP) certification by taking the PMP test after finishing Mod 2. I cannot speak to how easy or difficult Mod 2 makes the PMBoK material since I did not take the test.

By far the most interesting part of Mod 2 is the group project. Groups are assigned by classmate geography to facilitate interaction. For example, I lived in Charlottesville at the time and my other four group members lived within a half an hour of me.

Groups are responsible for finding a completed (or failed) IT project and performing a retrospective. There is a laundry list of things that can go wrong on IT projects and groups have to analyze what went right, what went wrong, and what future projects can learn to do better.

Each group interviews project stakeholders, analyzes documentation and deliverables, and views demos of the system if it was completed. The analysis objective is to piece together the outcome and the project’s significant intermediate events. At the end of the three months, each group presents a retrospective on the project.

Things I wish I knew before going into Mod 2:

• Learn to work well with your group. Your group members will change in Mod 3 but the lessons learned while working under the stress on a difficult project will be very beneficial for the remainder of the program.
• Practice, practice, practice for the presentation. It makes a major difference in the Mod 2 grade! We ran through our presentation in full six or seven times in person and made adjustments as we went along. That’s at least four hours of time spent only on speaking our lines in the presentation plus several more hours for slide edits.
• Do not wait until the night before your presentation to practice it for the first time. You will make changes to the slide deck and if the changes are drastic you will not have enough time to become comfortable with the section you are speaking on.  

Mod 3

The main topics for Mod 3 are enterprise integration, data warehousing, and business intelligence. Although I found Mod 2 interesting, Mod 3 was where the classwork really became fascinating because it focused on enterprise-wide issues.

The Mod 3 topics are all major challenges that frustrate even the best IT organizations. Class discussions on enterprise integration and business intelligence were interesting because many of my classmates were working on these large projects. Professors provided best practices and case studies while classmates provided concrete examples.

The group project for Mod 3 comes from a list of choices on relevant issues in IT organizations, such as social networks, cloud computing, and the “data deluge” (how to process and make sense of the exponentially increasing amount of data organizations produce). As a side note, the data deluge and big data are the topics this blog usually focuses on so if you have further interest in that area, please check my archive for relevant posts.

Advice for Mod 3:

• Pick a topic everyone in the group is interested in. If one person dominates the topic choice then others will have less motivation to learn the subject matter.
• Start pinging contacts in your network to find out the level of access to C-level executives they have. You want to have several potential companies lined up for Mod 4 by mid-December.
• Take a few days off either during Thanksgiving or around Christmas. You will be working for four months straight on a very difficult project in Mod 4.

Mod 4 is introduced even before Mod 3 ends. Mentally prepare yourself for the most difficult part of the program.

My Mod 4 retrospective can be found in part 5.

This is part 3 of my retrospective on the UVA Master’s in Management of Information Technology program I graduated from in May 2010. See part 1 and part 2 for context.

Mod 0

May 2009 was the first weekend of my cohort’s program. It was more than just a meet and greet with classmates. Mod 0 set the tone for program weekends with three 8-hour class days.

Topics included an introduction to corporate strategy, IT relevance, and the program’s tag line, “Delivering business value through IT.” We learned that the program is about how IT works when done well. IT departments can be a critical piece of corporate strategy and not just a cost center.

If you leave Mod 0 feeling like you did not get any value out of the topics and discussions then you should consider dropping the program. It is a major commitment not only for yourself but also to your classmates who are paying $40,000 for an education. The cohort is only as strong as its weakest link. Everyone has to contribute for the program to produce maximum value.

Mod 1

Mod 1 is 10 days in June of rigorous class and project work. You get very little sleep. I slept 4 hours a night (2:30am to 6:30am) every day during Mod 1 and caught up as much as possible on the weekend break that divides the 5 day sections. Coffee was crucial.

Topics in Mod 1 included enterprise architecture, computer network fundamentals, computer and network security, database modeling, and an introduction to data warehousing (covered in further detail in Mod 3). The idea behind Mod 1 is to give non-technical students a grasp of technical fundamentals so the concepts are no longer intimidating. The topics are high-level. You will not be doing any Java or .NET programming.

If you are technical or have a technical background, the subject matter in Mod 1 is straightforward. You can fill in gaps in your knowledge or refresh areas you have forgotten. For example, I knew HTTP servers ran on port 80 but I forgot that a browser running on a client machine opens high number ports for communication to an HTTP server. I should have known that concept but had not thought about it in awhile so class corrected it for me. There were dozens of similar examples scattered throughout our coursework.

A few bits of advice for students about to experience Mod 1:

• Set expectations with family, friends, and your company that during Mod 1 you will be incommunicado. You will not have time for anything outside the program during these two weeks.
• Speak up in class when you have something valuable to add. Mod 1 is a combination of the Charlottesville and Northern Virginia sections so the classroom is crowded. Make your answers short and get to the point immediately.
• Go to the bars on the Corner with your classmates after finishing up your group project work each night. Get to know your classmates! You’ll learn a lot from discussions over beers.
• Plan to do a few hours of work on Saturday and Sunday during the weekend between the five day sections. Otherwise catch up on sleep as much as possible.
• Get started on your Mod 2 reading as soon as Mod 1 ends. I made the mistake of taking a week off from all class-related reading after Mod 1 and put myself in a huge hole.

Mod 1 is a fantastic and intense experience. Go into it with the mindset of working incredibly hard the entire time and you can recover when the two weeks are over.

See part 4 for Mods 2 and 3.

Arin Sime wrote a great piece on the iron triangle of software development. Arin proposes we add a fourth dimension to the iron triangle that represents employee morale. Adding employee morale really fills in the model and I’m surprised no one suggested it before.

But there was one question I was left with after reading Arin’s article. Why should business leaders care about employee morale, especially if the project is staffed with consultants?

Who cares if consultants burn out producing software as long as it’s on time, within budget, and fulfills the desired scope? Here are three quick reasons:

1. Negative reputation. Running your business by exploiting others instead of building trust and respect will come back at you. Your customers will find out. This is the Internet Age. Word travels fast. Your business reputation is difficult to build but easily destroyed.
2. Troublesome future projects. A consulting firm that respects its employees will no longer do business with you in the future. Ones that continue to do business with you will staff future projects with less productive workers because good consultants will refuse to work with you.
3. Poor software quality. What is “done” in application development? Software is only as good as the business value it produces. When your business changes, how easy is it to adapt that software? If the software foundation is built on sand and it requires a completely new architecture to handle any changes, the long term value of that software is jeopardized.

Software development is about people. If you screw over your people, it is unpredictable in what ways it will come back to negatively impact you later.

Google received 75,000 applications in a single week in a tight labor market for top technical talent. Here are several potential reasons that may be a bad thing [1]:

1. It takes a lot of time to sort, review, call, interview, decide, and potentially offer to that many applicants. That’s a lot of resources that could be used for something else. If you have developers performing interviews then “Maker’s Schedule, Manager’s Schedule” becomes an issue because the developer’s schedules are so broken up [2].
2. Is the Google reputation for only hiring the best and brightest weakened because they are on a hiring binge? It’s one thing to say “we need more great people!” It’s another to set a public mandate that you’re going to hire 6,000 new employees for an increase of ~23% in total headcount. That’s a big slow company mentality. Doesn’t Google want to figure out a better way to do things?
3. If the brand is damaged, is Google receiving many applications from people who are not qualified? That’s fine in a small company if you can weed out unqualified applicants. But at most big companies there’s people that fall through the cracks. Especially when recruiters are trying to make a hiring target for the month.

In addition to the volume of applications, why does Google need to hire that many new people? The company built their core business with less than 6% (1900/32000) of the total employees they hope to have by the end of 2011. Google’s product portfolio has expanded dramatically since their IPO. Yet it appears inertia is taking over a company that was a start up only 10 years ago.

[1] I am in Washington, DC but it’s really tough to find great available developers here. I’m assuming most other major U.S. and European cities are similar in that regard.

[2] Paul Graham - http://www.paulgraham.com/makersschedule.html

Consulting firms and their consultants are locked in a struggle. In this post I define that struggle as consulting career mismatch.

Have you seen an “I’m an IBM-er” commercial? We learn about the interesting project the speaker on the commercial is working on. He’s employed by IBM. But we never learn the consultant’s name! The speaker’s career benefits if he gains name recognition. Instead IBM takes credit for the consultant’s hard work. But IBM pays the consultant’s salary and for the commercial. IBM wants the positive recognition because it provided the resources to execute the work.

Consulting career mismatch appears when the best interest of your career is misaligned with your consulting firm’s strategic direction. For example, you want to build your reputation as an expert in business intelligence. You want executives to call you when they have a business intelligence question. But your firm, XYZ Consulting, wants you to be one of thousands of “XYZ Business Intelligence Consultants.” XYZ Consulting wants to be the firm that receives the phone call from executives for business intelligence solutions.

IBM is the most obvious example of where consulting career mismatch can occur but it applies to all firms. The mismatch is most poignant at large firms because they have greater leverage over individual consultants.

Large firms want you to be an interchangeable cog in their consulting machinery. If you are a nameless consultant whose proof of excellent work depends solely on the firm’s reputation then you are less valuable without employment at that firm. The firm won’t have to pay you a premium for your services because you do not create as much value on the open market without the firm’s name recognition.

Consulting career mismatch is not inherently bad. You can still have a great career at large consulting firms despite the mismatch. But mismatch can derail your ambitions when aiming to be a recognized expert at a large firm. There is less friction when building your niche reputation as an independent consultant or part of a small firm that depends upon your success to remain in business.

Career mismatch should be seriously considered by every consultant that wants to build a unique personal reputation. The environment at your firm impacts whether or not you are successful and how that reputation is received.