Unified View for HR, Airlines, Equipment and Vehicles
About the Customer
The client was facing challenges in terms of accurate reporting. Information was siloed from a variety of sources and stored in multiple location-specific databases. The complexity increased due to multiple data points spread across several countries. The client relied on a batch process to build and process data, which took upwards of 5 hours.
Data Lineage, Data Mapping, and changing business demands, combined with multiple data sources, including relational and non-relational databases, resulted in reporting errors.
The legacy reporting solution was managed on-premises at multiple data centres leading to costly management, maintenance, and latency issues of the reporting platform.
The client was looking for a partner to help design and develop a next-generation data platform.
What we did
Data Governance & Architecture
Master Data Management
Data Warehousing and BI
AWS Data Migration Service
CompuGain was selected as the Client’s Solution partner to address the aforementioned challenges. Following a 360-degree data assessment, CompuGain developed the following strategy:
- AWS was chosen as a cloud platform to reduce maintenance cost, latency issues and improve reporting performance.
- AWS was the recommended data ingestion platform for flexibility, reliability, and scalability.
- AWS Lambda functions were orchestrated to trigger and process the files and move the files to respective S3 buckets based on success/failure criteria.
- After analyzing the data, Glue jobs were recommended over other technologies and patterns as an inexpensive solution. AWS Glue was used to transform and load data from disparate data points.
- AWS Redshift was used to create a centralized repository with a standardized global solution.
- The transformed data was then stored in a centralized repository created in AWS Redshift to allow near-real-time access to operational data from various sources, each with its database.
- The global operational data in AWS RedShift enabled the organization to make educated business decisions on time to meet and exceed internal and external customer requirements.
- Version control was built using AWS Code Commit and AWS Code Pipeline so that all codes developed by different people could access the most recent version of the code.
- Power BI was used to have quick view to create the required reports, which allowed the client to have a quick access to view the historical data, assisting various departments such as security, compliance, transportation, and so on. This helped improve the efficiency across all units by facilitating users to make quicker decisions and allow proper reporting to customers.
- AWS Simple Email Service (AWS SES) and AWS Lambda functions were utilized for error notifications.
Results and Benefits
To Migrate Data from On-prem to AWS
Reduction in Manual processes
Improved Data Availability
- Data readiness to key end-users for better decisions on a real-time basis.
- Appropriate data accessibility was available on a user level to the required departments for analysis to make informed business decisions.
- Data was now available in 10 minutes which earlier took 5 hours.
- The Report provided served as a single source of truth for creating reports across all departments, and each department was given access to only the information pertinent to their department.
- Using the available Global Data, the data migration from on-premises to AWS S3 was completed in 4 months.
- Created adaptable sources, which helped the developer in attaining code standards.
- Version Control helped the developers across the organization to access the latest code.
- 90% Reduction in Manual Processes.