Reports to: Software Architect
This person is responsible for creating data pipelines to support downstream data delivery.
A data pipeline is a series of automated, programmatic steps. These data pipelines gather data from numerous, potentially disparate data sources and bring everything together into a central place (data lake & warehouse) where it is made readily available for downstream consumption.
The incumbent works with a variety of technologies on a day to day basis. The nature of the job requires constant upskilling and personal development.
There are two main duties:
- Develop & maintain data pipelines in support of downstream data delivery.
- Perform analysis on organisational data & create reports to communicate the results.
Other potential duties include, but are not limited to:
- Communicate and advocate for a data-driven mindset in the larger organisational community, especially the IT department.
- Investigate and evaluate emerging technologies like robotic process automation (RPA), new AWS services, new data management approaches, machine learning advancements, etc.
- Liaise with individuals in the organisation in support of all the above duties.
Qualifications & Skills required:
- Good programming skills in both Python and SQL.
- Good database design skills and an understanding of data warehousing techniques and approaches.
- An understanding of and ability to access data from sources such as:
- Microsoft SQL Server
- Amazon S3
- Windows and Linux file systems (Excel, delimited files, etc.)
- HTTP APIs
- SFTP sites
- An ability to engineer cloud-first solutions in the Amazon Web Services cloud.
- An interest in machine learning and advanced analytics, and the development of data pipelines in support thereof.
- An understanding of the software development process, with proficiency in Git for version control and build & deployment pipelines in GitLab.
- An ability to create reports and BI dashboards to communicate data & analysis results, using a variety of technologies, which might include:
- Microsoft SQL Server Reporting Services
- Microsoft PowerBI
- Microsoft Excel