10 Tips to Save Your Data Lakes from Becoming Data Bogs
The article presents the 10 best practices to save data lakes from becoming data bogs
A data lake is a central repository that offers you to store all data sources whether it is unstructured or semi-structured in a huge volume. Data lakes are usually developed on low-cost commodity hardware, making it economically feasible to store terabytes and petabytes of data. Data is generally stored in a raw format without first being fine-tuned or structured. From there, it can be scrubbed and optimized for the purpose at hand, be it a dashboard for interactive analytics, downstream machine learning, or analytics applications. Here, the data lake infrastructure provides users and developers with self-service access to siloed information. It also empowers your data team to work collectively on the same information, which can be curated and secured for the right team or operation. Today, the big problem arises that how to save data lakes for high performance. And using the data in a successful manner is a crucial step to save data lakes from becoming data bogs. There are many top tips to save data lakes that are being practiced for high performance. Here the article will discuss the top 10 tips to save data lakes from becoming data bogs.
Data ingestion can get tricky and needs an early planning
Data lake ingestion is the procedure of collecting or absorbing data into object storage. In data lakes architecture, ingestion is much simple in comparison to a data warehouse because data lakes allow you to store semi-structured data in its native format. However, data ingestion is important here, and you must think about it early. This is because if you do not store your data properly, it can be difficult to access later on. Moreover, proper data ingestion can help address functional challenges such as optimizing storage for analytic performance and ensuring exactly-once processing of streaming event data.
Make multiple copies of the data
The main function of a data lake in the first place is to store huge amounts of data with very low investment in both terms- financially and in engineering hours and since we store the data unstructured and decoupling storage from computing. You should take advantage of these newfound storage capabilities by storing both raw and processed data. Keeping a copy of the raw historical data, in its original form, can prove essential when you need to ‘replay’ a past state of affairs.
Set a retention policy
While this might seem different from the previous tip, the fact that you want to store some data for longer periods of time is not like you should store the data forever. The main reasons you might want to get rid of data are its Compliance and cost. You will need a way to enforce whatever retention policy you create. It means you need to be able to identify the data that you want to delete and the data you want to store for the longer term and know exactly where to find this data in your object storage layer (S3, Azure Blob, HDFS, etc.)
Understand the data you are bringing in
It is right that data lakes are all about “store now, analyze later”, but using it completely with a blindfold will not work well. You must understand the data as it is being ingested in terms of the schema of each data source, sparsely populated fields, etc. Gaining this visibility on reading rather than trying to infer it in write will save you from trouble by enabling you to generate ETL pipelines based on the most accurate and available data.
Partition your data
Partitioning your data is helpful in reducing data query engines such as Amazon Athena require to scan in order to return the results for a specific query. Partitions are logical entities referenced by Hive metastores, and which map to folders on Amazon S3 where the data is physically stored.
Data governance and access control
Data lakes have earned some notoriety among CISOs, who are rightfully suspicious of the idea of ‘dumping’ all your data into an unstructured repository, making it strenuous to set specific row, column, or table-based permissions as in a database. Although, this problem is very easy to address presently with multiple governance tools available to ensure you have control over who can see which data. In the Amazon cloud, the recently introduced Lake Formation creates a data catalog that permits you to set access control for data and metadata stored in S3.
Readable file formats
Columnar storage makes data easy to read, and that is the primary motto to store the data you plan to use for analytics in a format such as Apache Parquet or ORC. In addition to being optimized for reads, these file formats have the advantage of being open-source rather than proprietary, which means you will be able to access them using a variety of analytic services.
Merge small files
Data streams, logs, or change-data-capture will typically produce countless of small ‘event’ files every day. While you could try to query these small files directly, doing so will have a very negative impact on your performance over time, which is why you will want to merge small files in a process called compaction.
Leverage automation and AI
Due to the variety and velocity of data coming into a data lake, it is important to automate the data acquisition and transformation processes. Organizations can leverage next-generation data integration and enterprise data warehousing (EDW) tools along with artificial intelligence (AI) and machine learning that can help them classify, analyze and learn from the data at a high speed with better accuracy.
Identify and define the organization’s data goal
One of the most important preemptive steps to save data lakes from becoming data bogs is to set clear boundaries for the type of information organizations are trying to collect, and their intent of what they want to do with it. Organizations need to have clarity on what they want to attain from the data they are collecting.
The post 10 Tips to Save Your Data Lakes from Becoming Data Bogs appeared first on Analytics Insight.
The article presents the 10 best practices to save data lakes from becoming data bogs
A data lake is a central repository that offers you to store all data sources whether it is unstructured or semi-structured in a huge volume. Data lakes are usually developed on low-cost commodity hardware, making it economically feasible to store terabytes and petabytes of data. Data is generally stored in a raw format without first being fine-tuned or structured. From there, it can be scrubbed and optimized for the purpose at hand, be it a dashboard for interactive analytics, downstream machine learning, or analytics applications. Here, the data lake infrastructure provides users and developers with self-service access to siloed information. It also empowers your data team to work collectively on the same information, which can be curated and secured for the right team or operation. Today, the big problem arises that how to save data lakes for high performance. And using the data in a successful manner is a crucial step to save data lakes from becoming data bogs. There are many top tips to save data lakes that are being practiced for high performance. Here the article will discuss the top 10 tips to save data lakes from becoming data bogs.
Data ingestion can get tricky and needs an early planning
Data lake ingestion is the procedure of collecting or absorbing data into object storage. In data lakes architecture, ingestion is much simple in comparison to a data warehouse because data lakes allow you to store semi-structured data in its native format. However, data ingestion is important here, and you must think about it early. This is because if you do not store your data properly, it can be difficult to access later on. Moreover, proper data ingestion can help address functional challenges such as optimizing storage for analytic performance and ensuring exactly-once processing of streaming event data.
Make multiple copies of the data
The main function of a data lake in the first place is to store huge amounts of data with very low investment in both terms- financially and in engineering hours and since we store the data unstructured and decoupling storage from computing. You should take advantage of these newfound storage capabilities by storing both raw and processed data. Keeping a copy of the raw historical data, in its original form, can prove essential when you need to ‘replay’ a past state of affairs.
Set a retention policy
While this might seem different from the previous tip, the fact that you want to store some data for longer periods of time is not like you should store the data forever. The main reasons you might want to get rid of data are its Compliance and cost. You will need a way to enforce whatever retention policy you create. It means you need to be able to identify the data that you want to delete and the data you want to store for the longer term and know exactly where to find this data in your object storage layer (S3, Azure Blob, HDFS, etc.)
Understand the data you are bringing in
It is right that data lakes are all about “store now, analyze later”, but using it completely with a blindfold will not work well. You must understand the data as it is being ingested in terms of the schema of each data source, sparsely populated fields, etc. Gaining this visibility on reading rather than trying to infer it in write will save you from trouble by enabling you to generate ETL pipelines based on the most accurate and available data.
Partition your data
Partitioning your data is helpful in reducing data query engines such as Amazon Athena require to scan in order to return the results for a specific query. Partitions are logical entities referenced by Hive metastores, and which map to folders on Amazon S3 where the data is physically stored.
Data governance and access control
Data lakes have earned some notoriety among CISOs, who are rightfully suspicious of the idea of ‘dumping’ all your data into an unstructured repository, making it strenuous to set specific row, column, or table-based permissions as in a database. Although, this problem is very easy to address presently with multiple governance tools available to ensure you have control over who can see which data. In the Amazon cloud, the recently introduced Lake Formation creates a data catalog that permits you to set access control for data and metadata stored in S3.
Readable file formats
Columnar storage makes data easy to read, and that is the primary motto to store the data you plan to use for analytics in a format such as Apache Parquet or ORC. In addition to being optimized for reads, these file formats have the advantage of being open-source rather than proprietary, which means you will be able to access them using a variety of analytic services.
Merge small files
Data streams, logs, or change-data-capture will typically produce countless of small ‘event’ files every day. While you could try to query these small files directly, doing so will have a very negative impact on your performance over time, which is why you will want to merge small files in a process called compaction.
Leverage automation and AI
Due to the variety and velocity of data coming into a data lake, it is important to automate the data acquisition and transformation processes. Organizations can leverage next-generation data integration and enterprise data warehousing (EDW) tools along with artificial intelligence (AI) and machine learning that can help them classify, analyze and learn from the data at a high speed with better accuracy.
Identify and define the organization’s data goal
One of the most important preemptive steps to save data lakes from becoming data bogs is to set clear boundaries for the type of information organizations are trying to collect, and their intent of what they want to do with it. Organizations need to have clarity on what they want to attain from the data they are collecting.
The post 10 Tips to Save Your Data Lakes from Becoming Data Bogs appeared first on Analytics Insight.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.