Like I mentioned in one of my previous posts, I'm exploring the big data ecosystem. In this post, I will briefly talk about big data and Hadoop, and why they are needed. I'm envisioning this as a series of 4 blog posts, and here goes the first one.
What is big data?
What is huge amount of data today need not be considered big a few years from now. So, there are 3 important vectors to check upon if the problem needs a big data solution or not:
Big data comes with big problems
When the amount of data is huge, nodes may end up spending too much time in data transfer. While Grid Computing works well for high analysis with lower amount of data, it requires low level programming and thus may not prove as efficient.
Hadoop was built to overcome above shortcomings
Its key features are:
So is Hadoop better than other databases?
Well, it depends on the use case. There are some use cases where RDBMS solutions like MySQL, PostgreSQL, MSSQL etc shine, and then others where Hadoop is the better alternate. In general,
It is important to note here that newer databases like Cassandra and DynamoDB allow huge volume of data to be processed - millions of columns and billions of columns and give RDBMS competition. They still have limitations on querying on fields other than primary and secondary index, but for many practical purposes, can replace the RDBMS variant.
So what is Hadoop?
Hadoop is a framework for distributed processing of large data sets, across clusters of commodity computers (nodes). All the nodes that we need are commodity hardware - it is enterprise grade servers with no customisation needed, and thus can be bought off the shelf as is. In the world of cloud computing, these nodes can sit inside a VPC as well.
What is big data?
What is huge amount of data today need not be considered big a few years from now. So, there are 3 important vectors to check upon if the problem needs a big data solution or not:
- Volume of data
- Velocity at which data is being generated: depends on the growth rate
- Variety of data: structured vs unstructured vs multi factored vs linked vs dynamic
Big data comes with big problems
- We need efficiency in storage since data volume/velocity/variety is high
- Data losses due to corruption and hard disk failures get magnified when working with big data, and accordingly, the recovery strategies needs to be adapted.
- The time it takes to analyze data also goes up significantly, thus requiring better techniques for analysis
- Finally, the monetary cost of analysis also shoots up due to huge storage & computation needs
- Most traditional RDBMS have scalability issues - with larger databases, one will require denormalization, sharding, partitioning, change of indexes, optimizing queries and so on to get the best performance.
- At times, one can not simply add more nodes or computation power for execution time to go down which can limit the horizontal scalability of the database.
- Finally, RDBMS databases are designed to process structured data - Long texts, videos, images etc may not be good for it.
When the amount of data is huge, nodes may end up spending too much time in data transfer. While Grid Computing works well for high analysis with lower amount of data, it requires low level programming and thus may not prove as efficient.
Hadoop was built to overcome above shortcomings
Its key features are:
- Is cost effective
- Can handle huge volume of data
- Efficient in storage
- Has good recovery solutions
- Is horizontally scale
- Minimizes learning curve
So is Hadoop better than other databases?
Well, it depends on the use case. There are some use cases where RDBMS solutions like MySQL, PostgreSQL, MSSQL etc shine, and then others where Hadoop is the better alternate. In general,
- RDBMS work exceptionally well with low volume data, while Hadoop with larger datasets
- RDBMS models are static schema while Hadoop allows dynamic schemas
- RDBMS can scale vertically (you can improve the process itself) but won’t scale horizontally (can’t improve performance of query by adding more nodes)
- Database solutions require dedicated server requirements which can get more expensive quickly, Hadoop is made of commodity computers
- Hadoop is a batch interactive system, and so can’t expect millisecond latencies. Thus for most practical purposes where you need to return a response quickly, Hadoop won’t be the ideal choice.
- Hadoop encourages you to write data once into the storage and analyze it multiple times, while databases support both read and write multiple times.
It is important to note here that newer databases like Cassandra and DynamoDB allow huge volume of data to be processed - millions of columns and billions of columns and give RDBMS competition. They still have limitations on querying on fields other than primary and secondary index, but for many practical purposes, can replace the RDBMS variant.
So what is Hadoop?
Hadoop is a framework for distributed processing of large data sets, across clusters of commodity computers (nodes). All the nodes that we need are commodity hardware - it is enterprise grade servers with no customisation needed, and thus can be bought off the shelf as is. In the world of cloud computing, these nodes can sit inside a VPC as well.
Hadoop has two core components:
- HDFS (Hadoop Distributed File System): Takes care of all storage related complexities, which data goes where, replicating data. HDFS is virtual, so the local file system and HDFS co-exist
- Mapreduce: Takes care of all computation related complexities
In the next posts, we will explore HDFS, Mapreduce and Hadoop ecosystem in detail.
