Job Summary
A company that runs a privacy-preserving data collaboration platform is seeking a AI Developer who will replatform legacy T-SQL systems onto modern Spark/Delta Lake pipelines, build robust FastAPI services around them, and integrate AI features.Responsibilities:
Data Engineering: Replatform T-SQL into Spark SQL and build production PySpark pipelines on Delta Lake.
API Development: Build secure, idempotent FastAPI services with robust job-status semantics and authentication.
Quality & Validation: Write type-hinted, tested Python (pytest) with CI gates; validate migrated pipelines against legacy systems with parity evidence.
Spec-First Delivery: Work autonomously from design docs and ADRs, keeping documentation current alongside code changes.
Minimum Requirements:
Experience: 4+ years in professional Python development, with 2+ years building production Spark/distributed data pipelines.
Education: Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
Certifications: Databricks Certified Developer for Apache Spark (or equivalent) is advantageous.
Python (3.12+): Modern idiomatic Python, strict static analysis (mypy/pyright), Pydantic, ABC patterns, modern packaging (
uv/Poetry), and CLI tooling.Apache Spark / PySpark: Production OSS Spark experience (DataFrame API, Spark SQL, tuning, driver/executor architecture, Spark Connect).
Lakehouse Formats: Delta Lake (or Iceberg/Hudi) including
MERGE INTO, schema evolution, time travel, and idempotent writes.SQL: Advanced, dialect-portable SQL skills to translate complex legacy T-SQL logic into Spark SQL.
Testing & CI/CD: Pytest (fixtures, tiered suites), Git/GitHub flow, PR-driven workflows with linting (
ruff) and type-check gates.APIs & Containers: FastAPI development and daily development using Docker Compose stacks.
Preferred Skills
AI/LLM Engineering: Provider-neutral LLM integration, structured output validation, local/hosted inference, and guard-railing.
Governance & Security: PII tokenization, hashing schemes, and data privacy awareness (POPIA/GDPR).
Cloud & Platform: Azure data estate (ADLS Gen2, Key Vault), Trino, Hive Metastore, and OpenTelemetry observability.
Benefits:
Competitive salary based on experience (salary can potentially be more based on experience/skills)
IF you meet the above requirements and want to make a career-changing move, apply today by emailing your CV to itcareers@hireresolve.za.com
About
Specialists in Civil, Structural, Mechanical Engineering, Information Technology, Mining, Manufacturing and Finance Careers! Hire Resolve is one of the larger and more agile South African recruitment companies that focus on placing professionals and skilled people in permanent employment and contract employment. We prefer and focus on working with top quality professionals and candidates in South Africa and Africa. Hire Resolve has successfully placed Engineering, Mining, IT, Manufacturing and Finance professionals with top firms across the Western Cape, Eastern Cape, KwaZulu Natal, Gauteng and in Africa. Hire Resolve has assisted candidates to find jobs at over 100 JSE listed companies of which many are global companies with offices and operations in South Africa and Africa. It is for this reason that we are well respected in the industries we operate in and in the recruitment industry.
Error!
Success!
You have successfully created your alert.
You will receive an email when a new job matching your criteria is posted.
Please check your email. It looks like you haven't verified your account yet. Here's what you're missing out on:
- Response from employers
- Being able to log in and update your profile
- Apply for more jobs
Didn't receive the link? Resend Verification Link