Data Engineer Lead (OT Data)

Job Description:

Data Engineer (OT Data) (Category - Engineer)

Sector:

Oil and Gas

Location:

Doha, Qatar

Experience:

10 years

The Opportunity:

The Data Engineer will be a foundational architect of our delivery strategy, responsible for building and maintaining the data ecosystem required for our scoped-out use cases. You will be the crucial link between our vast, complex data sources and our advanced analytics ambitions. This involves tackling the unique challenges of the oil and gas sector, bridging the worlds of operational technology (OT)—with its real-time drilling data, SCADA systems, and seismic surveys—and information technology (IT). Your mission is to build the specific, high-performance data pipelines required to deliver our portfolio of projects. This means going beyond simple data movement; you will be creating reliable, scalable, and secure data products that serve as the fuel for our most critical AI models and analytics dashboards, directly enabling tangible business value and competitive advantage.

Key Responsibilities:

• Architect & Build Data Pipelines: Design, construct, install, test, and maintain highly scalable data management systems and ETL/ELT pipelines.
• Integrate Diverse Data Sources: Develop processes to ingest and integrate high-volume, high-velocity data from SCADA systems, historians (like OSIsoft PI, Aspen InfoPlus.21), DCS, PLC, and IoT sensors.
• Cloud Data Platform Development: Implement and manage data solutions on the Microsoft Azure cloud platform, leveraging services like Azure IoT Hub, Azure Event Hubs, and Azure Stream Analytics for real-time ingestion and processing of operational technology (OT) data.
• Data Modelling & Warehousing: Design and implement data models optimized for time-series data from industrial assets, supporting operational dashboards and real-time analytics.
• Enable Advanced AI: Build the data infrastructure to support AI/ML models for predictive maintenance, operational anomaly detection, and process optimization using real-time OT data.
• Champion Master Data Management (MDM): Design and implement MDM strategies and solutions to create a single, authoritative source of truth for critical data domains such as wells, equipment, and assets, ensuring data consistency across the enterprise.
• Ensure Data Quality & Governance: Implement robust data quality checks, validation rules, and monitoring to ensure the accuracy, consistency, and reliability of our data. Adhere to and help shape our data governance policies.
• Embrace Industry Standards: Champion and implement industry-specific data standards and models, such as the OSDU™ Data Platform, to ensure interoperability and a unified data view across the upstream lifecycle.
• Collaborate & Innovate: Work closely with a cross-functional team of geoscientists, drilling engineers, data scientists, and business analysts to understand their data needs and deliver effective solutions.
• Automate & Optimise: Identify opportunities for process automation and infrastructure optimisation to improve data delivery, scalability, and cost-effectiveness.
• Security First: Implement and maintain security best practices to protect our sensitive and proprietary data assets.

Required Skills and Qualifications:

• Bachelor's in engineering, Information Systems, or a related quantitative field.
• 5+ years of proven experience in a data engineering role.
• Experience within oil and gas industry is highly preferred.
• Demonstrable experience building and operationalizing large-scale data pipelines and applications.

Technical Proficiencies:

• Expert-level proficiency in SQL and Python for data manipulation and pipeline development.
• Big Data Technologies: Hands-on experience with distributed computing frameworks like Apache Spark (PySpark). Experience with streaming technologies like Kafka is a plus.
• Cloud Platforms: Deep experience with Microsoft Azure (Azure Data Lake Storage, Azure Data Factory, Azure Databricks, Azure Synapse).
• Data Warehousing/Lakehouse: Proven experience with modern data platforms Databricks Delta Lake.
• AI & ML Knowledge: Understanding of machine learning lifecycles and the data requirements for training and deploying AI/ML models.
• Orchestration: Experience with workflow orchestration tools like Airflow, Dagster, or Azure Data Factory.
• Databases: Strong understanding of both relational (e.g., PostgreSQL, SQL Server) and NoSQL databases. Experience with graph databases (e.g., Neo4j) and vector databases is highly desirable.
• Version Control: Proficiency with Git and CI/CD best practices.

Preferred Qualifications:

• Familiarity with historian systems (e.g., OSIsoft PI System) and their data structures.
• Hands-on experience with the OSDU™ Data Platform.
• Experience working with industrial communication protocols (e.g., OPC UA, Modbus TCP/IP).
• Understanding of cybersecurity considerations for OT environments and data segregation.
• Experience integrating data from ERP systems like SAP.
• Professional certifications in Azure or Databricks.
• Advanced SQL skills, including query optimisation and performance tuning.
• Knowledge of containerization technologies (Docker, Kubernetes).
• Experience of constructing and maintaining enterprise knowledge graphs.