Lead Data Engineer- Toronto, Canada (On-Site)

Job Title: Lead Data Engineer

Job Location: Toronto, Canada (on-site)

Job Type: Contract (Incorporated/ T4)

Description

The Data Engineering team is looking for people who are passionate about working in agile delivery environments and resolving the engineering challenges of building robust and scalable data systems aligned to enterprise data strategy.

As Lead, Data Engineer you will be responsible for developing, constructing, and testing large-scale data analytics systems based on AWS cloud that will help address the disparate analytics needs of a growing organization.

Responsibilities

• Design solutions aligned with long-term architecture and technology strategy using Amazon Web Services (AWS) for Cloud development.
• Work in a fast-paced environment collaborating with developers, data engineers, architects, researchers, and data scientists.
• Conceptualize, design, and implement analytics products that enhance Investments analytics capabilities.
• Participate in the development life cycle from start to completion - requirements analysis, development, testing, and deployment.
• Develop tools that prepare, transform, combine, and manage structured and unstructured data for use by Investments business users.
• Define and shape Investments' future technology and research process
• Ensure architecture will support the requirements of the Investments business.

Must Have Skills

• University degree in Engineering or Computer Science preferred.
• Overall IT experience 8+ years
• AWS EMR, S3, Glue.
• Experience of ETL pipelines, managing multiple datasets and providing necessary support.
• Hands-on experience building data exploratory interfaces using Notebooks (JupyterHub/Lab, Livy, SPARK, Hudi, Hive)
• Knowledge & experience with driving IaC concepts within an organization, leveraging Terraform, Ansible, GitHub actions.
• Deep proficiency in Python with experience using Spark, Pandas or PySpark.
• Hands-on expertise with building data pipelines and applications, leveraging Kubernetes, Airflow,

Nice to have

• (AWS SagemakerSparkmagic, Enterprise Gateway, matplotlib, Plotly/Dash, etc.).
• Demonstrated ability to easily deal with both abstract and concrete concepts and be able to reconcile them for the appropriate audience and context preferred.
• Familiar with cloud technology best practices to enable the distribution and analysis of big data on the cloud (formatting/partitioning/etc.).
• Quickly understand organizational dynamics and management priorities, and to be able to work effectively in a fast paced, results driven company.
• Experience with Cloud based data and analytics platforms, warehouses (Redshift/Spectrum, Databricks, Snowflake), BI Tools, OLAP systems (Clickhouse, Druid), including a mix of relational, non-relational, streaming and event-based architectures.
• Ability to work in an entrepreneurial environment and be a self-starter.
• Demonstrate strong facilitation, negotiation, interpersonal, communication and collaboration skills.
• Experience with front-end framework ie. Angular, React is a plus
• Interests in the financial industry.