Requirements

Data Engineer – Candidate Requirements

Intermediate Level

3–5 years' experience

• 3–5 years of hands-on experience in data engineering.
  

• Strong proficiency in Python and/or SQL, including query optimisation.
  

• Experience working with both relational and non-relational databases.
  

• Experience designing and building data pipelines and data models.
  

• Understanding and practical experience with lakehouse architectures, including the medallion pattern.
  

• Practical experience with at least one major cloud platform, including:
  

  • Microsoft Azure
    

  • AWS
    

  • Google Cloud Platform (GCP)
    

• Familiarity with:
  

  • Databricks
    

  • Snowflake
    

  • Delta Lake
    

  • PySpark
    

• Understanding of data transformation frameworks such as dbt.
  

• Experience with version control using Git.
  

• Understanding of CI/CD practices for data workflows.
  

• Strong analytical and problem-solving skills.
  

• Ability to perform root-cause analysis on complex data issues.
  

• Good communication and stakeholder engagement skills.
  

Senior Level

6–8+ years' experience

• 6–8+ years of hands-on experience in data engineering.
  

• All intermediate-level technical requirements, together with demonstrable experience in:
  

  • Leading end-to-end data platform delivery.
    

  • Architecting enterprise-grade lakehouse environments.
    

  • Implementing data mesh patterns.
    

  • Infrastructure-as-code using tools such as Terraform, Bicep, AWS CDK or Pulumi.
    

  • DevOps and CI/CD pipelines.
    

  • Working effectively with cross-functional teams in a dynamic consulting environment.
    

  • Mentoring junior engineers.
    

  • Contributing to technical strategy and solution direction.
    

Qualifications

• Bachelor's degree in:
  

  • Computer Science
    

  • Information Systems
    

  • Information Technology
    

  • or a related field.
    

• Master's degree in a relevant field is advantageous.
  

Certifications

One or more of the following certifications would be advantageous:

• Microsoft Fabric Data Engineer Associate
  

• Microsoft Azure Data Engineer Associate
  

• Databricks Certified Data Engineer Associate
  

• Google Professional Data Engineer
  

• AWS Certified Data Engineer – Associate
  

• Databricks Certified Data Engineer Professional
  

Technology Experience

Languages & Frameworks

• Python
  

• PySpark
  

• SQL
  

• dbt
  

Microsoft Fabric & Azure

• Microsoft Fabric Lakehouses
  

• Fabric Pipelines
  

• Fabric Semantic Models
  

• Direct Lake
  

• Azure Data Factory
  

• Azure Data Lake Storage Gen2
  

• Azure Synapse Analytics
  

• Azure Databricks
  

• Azure Event Hubs
  

Google Cloud Platform

• BigQuery
  

• Cloud Storage
  

• Dataflow
  

• Dataproc
  

• Pub/Sub
  

Amazon Web Services

• Amazon S3
  

• AWS Glue
  

• Amazon Redshift
  

• Amazon EMR
  

• Amazon Kinesis
  

Databricks & Data Platforms

• Databricks
  

• Delta Lake
  

• Unity Catalog
  

• MLflow
  

• Databricks Workflows
  

Databases

• Azure SQL
  

• Azure Cosmos DB
  

• PostgreSQL
  

• Snowflake
  

• BigQuery
  

• Amazon Redshift
  

DevOps & Infrastructure as Code

• Git
  

• Azure DevOps
  

• GitHub Actions
  

• Terraform
  

• Bicep
  

• AWS CDK
  

• CI/CD pipelines
  

Streaming & Messaging

• Azure Event Hubs
  

• Azure Stream Analytics
  

• Apache Kafka
  

• Amazon Kinesis
  

• Google Pub/Sub
  

Visualisation & Analytics

• Microsoft Power BI
  

• Microsoft Fabric Real-Time Dashboards
  

• Looker / Looker Studio
  

• Amazon QuickSight