Data Engineer, Technology Services Group

The Position

Continued growth has created an exciting opportunity for a Cloud Data Engineer to join our Technology Services Group.

The successful candidates will have the opportunity to collaborate, build and succeed as part of a team of high performing professionals, and the ability to share challenges and rewards that come from working with a dynamic group of entrepreneurial thinkers who come from diverse backgrounds and fields of specialty with a passion for growing businesses - both ours and our clients’.

As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions. Your focus will be on building scalable and efficient data pipelines, data lakes and warehouses on-prem or on cloud platforms like AWS, MS Azure or GCP based on needs and constraints of the end consumer / client. You will work with data scientists, front-end and back-end engineers, and domain experts to ensure data availability, integrity, and reliability for analytics, reporting and machine learning. Your primary area is data engineering, but you are comfortable working in a secondary area of expertise such as data presentation/visualization, backend engineering, or data modeling (SQL, NoSQL).

Responsibilities

Designing and Implementing Data Pipelines:

• Ensure data accuracy, integrity, privacy, consistency, security, and compliance through quality control procedures throughout the pipeline.
• Create and maintain optimal data pipeline architecture.
• Leverage cloud data platform services and technologies like those from AWS (AWS EMR, AWS Lambda functions, AWS Athena, DynamoDB, AWS Glue, Amazon Redshift), Azure Data Factory, Azure Databricks, Azure Data Lake, Google BigQuery or similar offerings
• Design, develop and maintain data ingestion and ETL pipelines to acquire, process and store large volumes of structured and unstructured data.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build automated ETL process, orchestrating multiple SSIS packages, SQL jobs and sources.
• Optimize pipelines for performance, scalability, and cost efficiency.
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Ensure data quality, consistency, and integrity throughout the pipeline.
• Integrate data from various sources including databases, APIs, and third-party systems to create a unified data ecosystem.
• Write unit/integration tests, and contribute to engineering wiki, and documents work.
• Design and optimize data schema and determine which storage formats (Parquet, Avro, ORC) to use for efficient data retrievals.
• Support development of data models (ERD diagrams, normalization, etc.)

Building and Managing Data Warehouses and Data Lakes

• Architect, implement and manage cloud-based data warehousing and data lake solutions.
• Implement data governance policies, access controls (row and cell level security, tag based, cross-account access) and security measures.

Qualifications

• In depth knowledge and experience with SQL (NoSQL is a plus)
• Exposure to distributed data processing frameworks like Apache Spark or Hadoop
• Exposure to Tableau / PowerBI solutions.
• Proficiency with SSIS, SSAS, SSRS
• Experience in T-SQL Data-Modeling
• Proficiency in programming languages such as Python, Scala or Java
• Proficiency in one of the cloud platforms – AWS, Azure or GCP and their associated data services
• Experience with / exposure to data pipelines and workflow management tools (e.g. Airflow)
• Innovative and dynamic. Brings an entrepreneurial spirit to our business
• Proven experience of working as a Data Engineer (working with T-SQL)
• Proven ability to work independently and with a team
• Results-oriented individual with the ability to meet aggressive timelines
• Familiarity with data warehousing concepts and strong understanding of data integration, data modeling
• Familiarity with ML Ops (deploying machine learning models to production and subsequent maintenance and monitoring) is a plus.
• Strong work ethic and someone who is energetic, enthusiastic and committed to developing their career
• Excellent problem-solving, communication, and organizational skills
• Self-directed, with the ability to thrive in a fast-paced and dynamic environment
• Experience working with cloud Data Warehouse solutions (e.g., Snowflake, Redshift, BigQuery, etc.).
• Experience with version control and CI/CD pipelines
• Proven experience (1-3 years) as a Data Engineer with a focus on cloud-based data engineering
• Knowledge of different types of databases, warehouses, and analytical systems
• Highly developed writing, communication, and presentation skills

Required Education: University Degree in Computer Science, Information Systems, or related field