Sales-Project-Data-Engineering-Project-1.0

Data Engineering in Hadoop using Cloudera Performed the principle tasks involved in managing, loading, extracting, and transforming data. This respository holds the scripst that I wrote during the whole project. The project was done in Cloudera using Hadoop.

Loading and Profiling the Data

1. We create a staging environment
2. Upload the Sales Data
3. Form the audit database where we track the changes made in the back_office database

Data Quality Testing

1. Identify the blank values
2. Find the missing locations