Lead Data Engineer Jobs

Position : Lead Data Engineer

Duration: Contract

Location: Toronto, ON

Responsibilities

Must have

- University degree in Engineering or Computer Science preferred.

- Overall IT experience 8+ years

- Hands-on experience building data exploratory interfaces using Notebooks (JupyterHub/Lab, Livy, SPARK, Hudi, Hive)

- Hands-on expertise with building data pipelines and applications, leveraging Kubernetes, Airflow,

- Knowledge & experience with driving IaC concepts within an organization, leveraging Terraform, Ansible, GitHub actions.

- Deep proficiency in Python with experience using Spark, Pandas or PySpark.

- AWS EMR, S3, Glue.

- Experience of ETL pipelines, managing multiple datasets and providing necessary support.

Nice to have

- Experience with front-end framework ie. Angular, React is a plus

- Interests in the financial industry.

- Familiar with cloud technology best practices to enable the distribution and analysis of big data on the cloud (formatting/partitioning/etc.).

- Ability to work in an entrepreneurial environment and be a self-starter.

- Demonstrated ability to easily deal with both abstract and concrete concepts and be able to reconcile them for the appropriate audience and context preferred.

- Quickly understand organizational dynamics and management priorities, and to be able to work effectively in a fast paced, results driven company.

- Demonstrate strong facilitation, negotiation, interpersonal, communication and collaboration skills.

- Experience with Cloud based data and analytics platforms, warehouses (Redshift/Spectrum, Databricks, Snowflake), BI Tools, OLAP systems (Clickhouse, Druid), including a mix of relational, non-relational, streaming and event-based architectures.

(AWS SagemakerSparkmagic, Enterprise Gateway, matplotlib, Plotly/Dash, etc.).