Role - Data Engineer

Location - London (Hybrid)

Pay - £600 outside IR35

Join a leading organisation as they build a next-generation big data infrastructure powered by cloud technologies. This initiative enables AI/ML-driven automation to enhance trading operations and optimise physical assets.

As a Data Engineer, you will play a key role in developing and maintaining scalable data pipelines, supporting advanced analytics, and working alongside data scientists and business teams to drive critical insights.

Key Responsibilities

• Develop, construct, test, and maintain robust ETL/ELT data pipelines.
• Support users in navigating the big data platform, ensuring seamless data ingestion and troubleshooting issues.
• Solve complex data integration challenges, sourcing data from structured and unstructured sources.
• Clean, preprocess, and deliver high-quality data for AI/ML models and predictive analytics.
• Perform data quality testing and monitoring to ensure data integrity.
• Collaborate with data scientists to develop scalable AI/ML models.
• Build data visualisation dashboards and business intelligence solutions.
• Optimise pipelines, dashboards, and frameworks for efficient data workflows.
• Take ownership of assigned projects—from requirements gathering to solution delivery.
• Work with CI/CD tools (Azure DevOps, Jenkins) to streamline deployment.
• Follow best practices in coding standards (DRY principles, Git, unit/integration testing, code reviews).

Required Skills & Experience

• Strong experience building ETL and ELT data pipelines.
• Proficiency in Python and data tools such as Pandas.
• Hands-on experience with PySpark or similar distributed data processing tools.
• Advanced SQL skills (T-SQL, ANSI SQL, Spark SQL, etc.).
• Experience working with structured, semi-structured, and unstructured data.
• Strong understanding of data concepts and ability to communicate insights effectively.
• Hands-on experience with cloud data services (Azure Data Factory, Synapse, Databricks, AWS Redshift, Glue, Athena, BigQuery, etc.).
• Solid analytical and problem-solving skills, with strong communication abilities.
• Experience in trading or commodity markets is a plus.