Senior Data Engineer with experience developing pipelines and solutions using Microservices modules to collect, process, aggregate, and provide batch and stream data to the user using GCP or AWS services. Previously spent 13 years designing Data Warehouse (DW) and Business Intelligence solutions in areas including finance, customer service, telecom, public and private health, mining, insurance, retail, sales, marketing and IT.Hire Alysson
Responsible for speaking with stakeholders to understand needs then create cloud solutions to solve problems involving large amounts of data. Also responsible for design, implementation, and support of platform providing secured access to large datasets, analyze logs, debug Python scripts, fix code, and customize development.
Responsible for creating, maintaining, and optimizing scalable data pipelines on AWS, writing code for manipulating data using both Python and complex SQL queries, cloud resources tuning adjustments, structured and unstructured data manipulation, both for batch and streaming data flow, as well as optimizing persistence on Data Lake.
Developed solution able to process huge volume data using Google Cloud DataFlow. Solution was able to copy 15 Terabytes of data on SQL Server and wrote new structure to Google Cloud. Extracted XML and JSON data stored in blobs column and saved to Cloud Storage. Also audited all processes to ensure the content at the source was the same at the destination and saved audit results in BigQuery tables.