Big Data Center of Excellence – When you do, and don’t, need Big Data
By now, companies have not only heard about Big Data, but budgeted for it as well. Some have it in production, but most are still experimenting around with it in development and testing.
One of the top questions I get asked, by executive sponsors as well as senior architects, is the starter question:
When DO we need Big Data, and when DON’T we?
Part of the reason people ask this question is that Big Data is still a new field, and there is legitimate desire to truly understand what are the applicable use cases. Another reason is senior-level people do not want to throw millions of dollars into an 18-month experiment that results in a system worse than what already exists.
Here are some guidelines for situations when Big Data is, and is not, a viable solution:
Yes to Big Data!
- You are starting a new data collection system from Day Zero.
- New requirements are emerging that require a whole new league of storage and/or processing.
- You have many disparate sources of data, and have a real need to combine them into one location.
- As end user base grows and performance needs to be maintained as more users go on. .
- The Regulatory compliance or risk management dictates a new system be built and existing systems do not have the capacity to deliver.
- Your existing database or data warehouse is reaching its capacity limit.
- The non-database system will not support new features due to the custom coding.
- Performance is getting exponentially slower, as data volumes increase; what should take a few seconds requires 45 minutes.
No to Big Data!
- You have an existing system that works, and do not have the resources to build an entirely new system.
- Your data warehouse that works well enough to at least limp along.
- The existing systems does not need new functionality added.
- What you have is maybe not perfect, but good enough that there are more important priorities.
- The security requirements dictate that data sets be completely quarantined and so it must be stored separately from other data. This piece is not a good candidate for big data.
- There are already enough urgent projects that Big Data cannot be properly prioritized.
- There is not enough time or staff to plan and execute a major project.
To Summaries the Need of Big Data
Big Data requires a big commitment; not only in dollar amounts, although the costs for hardware and software are rapidly decreasing. The real commitment is in time, attention, and priority. If there are not resources to give a Big Data project the attention it needs, then it will not succeed.
Therefore, for situations where not everything needs to be built from scratch, and/or when time and budget are not unlimited, sometimes what is needed is not an entirely new Big Data purchase and installation; sometimes, what a “hybrid Big Data solution” will suffice. The next paper in this series will focus on hybrid Big Data.