Lead Data Engineer (Gcp)

for Lead Data Engineer (GCP)

The Google Lead Data Engineer will play a key role in designing and implementing the Data Hub using Google cloud components working closely with the enterprise data team and data architects, solution architects, business systems analyst and data engineers

Key accountabilities/ Experience

• Take a lead in designing and building production data pipelines from data ingestion to consumption using GCP services, Java, Python, Scala, BigQuery, DBT, SQLetc.
• Provide end-to-end technical guidance and expertise on how to effectively use Google Cloud to build solutions; creatively applying cloud infrastructure and platform services to help solve business problems; and communicating these approaches to different business users
• Building and managing data pipelines with a deep understanding of workflow orchestration, task scheduling and dependency management
• Designing, building, operationalizing the Wealth Data Hub (WDH) using Google Cloud Platform (GCP) data services such as DataProc, Dataflow, CloudSQL, BigQuery, CloudSpannerin combination with third parties such as Spark, Apache Beam/ composer, DBT, Cloud PubSub, Confluent Kafka, Cloud storage Cloud Functions & Github
• Designing and implementing data ingestion patterns that will support batch, streaming and API interface on both the Ingress and Egress.
• Provide guidance on Implementing application logging, notification, jobs monitoring and performance monitoring
• Guide a team of data engineers and work hands on in developing framework and custom code using best practices that will meet the demanding performance requirements
• Ability to do proof of technology using GCP technologies and work with data architects, solution architects to achieve the desired results and performance.
• Experience using Cloud Dataflowusing Java/Pythonfor deploying streaming jobs in GCP as well as batch jobs using text/JSON files and writing them to BigQuery

Functional Competencies and Skills

• Good communication and problem-solving skills. Ability to effectively convey ideas to business and technical teams
• Experience working with relational/NoSQL databases
• Experience with Google Cloud Platform Databases (SQL, Spanner, PostgreSQL)
• Knowledge of Java microservices and Spring Boot
• Strong architecture knowledge with experience in providing technical solutions for cloud infrastructure.
• Experience with data streaming and technologies such as Kafka, Spark-streaming etc.
• Experience with the primary managed data services within GCP, including DataProc, Dataflow, BigQuery/DBT, Cloud Spanner, Cloud SQL, Cloud Pub/Sub etc.
• 10+ years of experience in data engineering, performance optimization for large OLTP applications with a minimum of 4+ years of working experience as Google Cloud Platform (GCP) developer
• Experience with Infrastructure as Code (IaC) practices and frameworks like Terraform
• Working knowledge of developing and scaling JAVA REST services, using frameworks such as Spring

Nice-To-Have Skills:

• Experience in implementation of operational data store and involved in design of data models across multiple domains such as party, financial, non financial, securities, user domain etc
• Understanding of Wealth business line and the various data domains required for building an end to end solution

Education and Other requirements:

• Active Google Cloud Data Engineer certification or Google Professional Cloud Architect certification
• Degree in Computer Science or related field