Maniganda

Use case to implement the Kafka schema registry to maintain the evolving schema from the application and make it into plug-and-play type of architecture.

• Design and implement the ETL Pipelines
• Implementing the Schema registry to address the evolving application schema
• Building Data Warehouses and Data modeling for the project
• Data migration activity to on premise to AWS cloud
• Leading and mentoring the team
• Conceptualize, Design, Develop and Productize ETL pipeline using Big Data stack of Python, Scala, Spark SQL, Hive and HBase.
• Spark code development via Scala or Python Expertise on working on different file formats ORC, Parquet and CSV.
• Analyze the performance of the Spark job and tune it by implementing best optimization technique to achieve better performance.
• Data Schema/Migration activity as per the feature requirement
• Research and implement evolving tools and libraries when use cases needs.

A project to migrate from the on-premise solution to the Serverless fashion approach in Microsoft Azure. It had a manual team of 4 members in size early on and has proposed and productized the solution in a Serverless fashion in Azure successfully.

• Solution and implementing of the ETL pipelines
• Data project migration to Azure cloud
• Lead and mentor the team
• Research and implement evolving technologies and libraries based on use case needs.
• Designed End to End ETL pipeline using Spark and Python
• Loading, Cleansing, Validating, and Transforming data on different layers of
• ETL process. Data Import and Export into HDFS and Hive using Sqoop jobs.
• Performed Data enrichments activities such as filtering, pivoting, format modeling, sorting, and aggregation through Spark transformations
• Optimized the hive queries via modeling the data using Partitions and Bucketing techniques

Code development and maintained continuous delivery through Jenkins pipelines. Fixed functional and performance defects.

Worked on new modules and features by following best practices of coding and process. POC preparation and technical support for production issues.