AI Lead (Doha , Qatar)

Codvo is a global empathy-led technology services company where product innovation and mature software engineering are embedded in our core DNA. Our core values are Respect, Fairness, Growth, Agility, and Inclusiveness, guiding everything we do. We continually expand our expertise in digital strategy, design, architecture, and product management to offer measurable results and outside-the-box thinking.

About the Role

We are seeking a highly skilled Senior AI Engineer with a deep understanding of asset-heavy industries such as oil & gas, manufacturing, or utilities to lead the design, development, and implementation of robust and scalable pipelines and backend systems for our Generative AI applications. In this role, you will be responsible for orchestrating the flow of data, integrating AI services, developing RAG pipelines, working with LLMs, and ensuring the smooth operation of the backend infrastructure that powers our GenAI solutions.

You will also apply modern LLMOps practices, handle schema-constrained generation, optimize cost and latency trade-offs, mitigate hallucinations, and ensure robust safety, personalization, and observability across GenAI systems.

Responsibilities

• Generative AI Pipeline Development
• Design and implement scalable and modular pipelines for data ingestion, transformation, and orchestration across GenAI workloads.
• Manage data and model flow across LLMs, embedding services, vector stores, SQL sources, and APIs.
• Build CI/CD pipelines with integrated prompt regression testing and version control.
• Use orchestration frameworks like LangChain or LangGraph for tool routing and multi-hop workflows.
• Monitor system performance using tools like Langfuse or Prometheus.
• Data and Document Ingestion
• Develop systems to ingest unstructured (PDF, OCR) and structured (SQL, APIs) data.
• Apply preprocessing pipelines for text, images, and code.
• Ensure data integrity, format consistency, and security across sources.
• AI Service Integration
• Integrate external and internal LLM APIs (OpenAI, Claude, Mistral, Qwen, etc.).
• Build internal APIs for smooth backend-AI communication.
• Optimize performance through fallback routing to classical or smaller models based on latency or cost budgets.
• Use schema-constrained prompting and output filters to suppress hallucinations and maintain factual accuracy.
• Retrieval-Augmented Generation (RAG) Pipelines
• Build hybrid RAG pipelines using vector similarity (FAISS/Qdrant) and structured data (SQL/API).
• Design custom retrieval strategies for multi-modal or multi-source documents.
• Apply post-retrieval ranking using DPO or feedback-based techniques.
• Improve contextual relevance through re-ranking, chunk merging, and scoring logic.
• LLM Integration and Optimization
• Manage prompt engineering, model interaction, and tuning workflows.
• Implement LLMOps best practices: prompt versioning, output validation, caching (KV store), and fallback design.
• Optimize generation using temperature tuning, token limits, and speculative decoding.
• Integrate observability and cost-monitoring into LLM workflows.
• Backend Services Ownership
• Design and maintain scalable backend services supporting GenAI applications.
• Implement monitoring, logging, and performance tracing.
• Build RBAC (Role-Based Access Control) and multi-tenant personalization.
• Support containerization (Docker, Kubernetes) and autoscaling infrastructure for production.

Required Skills and Qualifications

Education

• Bachelor’s or Master’s in Computer Science, Artificial Intelligence, Machine Learning, or related field.

Experience

• 5+ years of experience in AI/ML engineering with end-to-end pipeline development.
• Hands-on experience building and deploying LLM/RAG systems in production.
• Strong experience with public cloud platforms (AWS, Azure, or GCP).
• 2+ years of experience in asset-heavy industries (e.g., oil & gas, utilities, manufacturing).

Technical Skills

• Proficient in Python and libraries such as Transformers, SentenceTransformers, PyTorch.
• Deep understanding of GenAI infrastructure, LLM APIs, and toolchains like LangChain/LangGraph.
• Experience with RESTful API development and version control using Git.
• Knowledge of vector DBs (Qdrant, FAISS, Weaviate) and similarity-based retrieval.
• Familiarity with Docker, Kubernetes, and scalable microservice design.
• Experience with observability tools like Prometheus, Grafana, or Langfuse.

Generative AI Specific Skills

• Knowledge of LLMs, VAEs, Diffusion Models, GANs.
• Experience building structured + unstructured RAG pipelines.
• Prompt engineering with safety controls, schema enforcement, and hallucination mitigation.
• Experience with prompt testing, caching strategies, output filtering, and fallback logic.
• Familiarity with DPO, RLHF, or other feedback-based fine-tuning methods.

Soft Skills

• Strong analytical, problem-solving, and debugging skills.
• Excellent collaboration with cross-functional teams: product, QA, and DevOps.
• Ability to work in fast-paced, agile environments and deliver production-grade solutions.
• Clear communication and strong documentation practices.

Preferred Qualifications

• Experience with OCR, document parsing, and layout-aware chunking.
• Hands-on with MLOps and LLMOps tools for Generative AI.
• Contributions to open-source GenAI or AI infrastructure projects.
• Knowledge of GenAI governance, ethical deployment, and usage controls.
• Experience with hallucination suppression frameworks like Guardrails.ai, Rebuff, or Constitutional AI.