Data Pipeline Specialist
hace 3 semanas
As a Data Pipeline Specialist, you will be responsible for designing and implementing efficient data pipelines using Airflow, Python, and various data storage solutions.Key Responsibilities- Data Pipeline Development:- Design and build DAGs (TaskFlow, dynamic DAGs, deferrable operators) to manage data flow between systems.- Develop Python-based ETL/ELT pipelines for APIs, storage, Kafka, and databases.- Operational Excellence:- Ensure smooth operation of Airflow in Azure/Kubernetes with blue/green or canary releases.- Quality Assurance:- Implement data quality checks, unit tests, and CI/CD DAG validation to ensure reliability and accuracy.- Real-time Processing:- Build near–real-time event pipelines with proper schema management to support business decision-making.- Performance Optimization:- Model SQL/blob datasets with optimal partitioning and retention to improve query performance.- Incident Management:- Drive observability (metrics, logs, alerts, SLAs) and lead incident reviews to minimize downtime.- Security Best Practices:- Enforce security best practices: IAM, secrets, RBAC, PII handling, data contracts to protect sensitive information.- Infrastructure Management:- Build CI/CD pipelines, containerize with Docker, and use Terraform/Helm for infrastructure as code.- Collaboration:- Collaborate with Android/backend teams on data interfaces and documentation to ensure seamless integration.Requirements- Experience: 8+ years in data/backend engineering with strong Python skills.- Airflow Experience: 2+ years with Airflow 2.X and advanced operators/hooks/scheduler tuning.- Data Storage: Experience with reliable large-scale ETL/ELT (batch + streaming) solutions like BigQuery/Redshift/Snowflake.- SQL and Data Modeling: Strong SQL and data modeling skills with experience in production-ready debugging, tuning, and monitoring.- Security and Compliance: Knowledge of RBAC, secrets management, OAuth/OIDC, and API gateways to ensure secure data exchange.- Communication and Documentation: Strong communication and documentation skills to collaborate effectively with cross-functional teams.
-
Senior Data Pipelines Specialist
hace 3 semanas
Guadalajara, México Bebeedataengineer A tiempo completoJob Title: Senior Data EngineerDescriptionWe're looking for a skilled Senior Data Engineer to lead the design, build, and maintenance of our data pipelines.You'll work closely with cross-functional teams to define interfaces and data contracts.Our ideal candidate has strong Python expertise and experience designing reliable ETL/ELT at scale.Familiarity with...
-
Data Pipeline Engineer
hace 2 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow)Location: Guadalajara, MXJob-Type: Full-timeJob Description:We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Pipeline - Data Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow) Location: Guadalajara, MXJob-Type: Full-timeJob Description:We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Data Pipeline Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow)Location: Guadalajara, MXJob-Type: Full-timeJob Description:We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Data Pipeline Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow) Location: Guadalajara, MX Job-Type: Full-time Job Description: We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Pipeline - Data Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow) Location: Guadalajara, MX Job-Type: Full-time Job Description: We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Data Pipeline Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow) Location: Guadalajara, MX Job-Type: Full-time Job Description: We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Data Pipeline Engineer
hace 3 semanas
Guadalajara, México Global Connect Technologies A tiempo completoJob Title: Data Pipeline Engineer (Python/Airflow) Location: Guadalajara, MX Job-Type: Full-time Job Description: We are seeking an experienced Senior Data Pipeline Engineer to design, build, and operate secure, reliable, and cost-efficient data pipelines supporting our Android connected and infotainment experiences. You will lead Python-based ETL/ELT...
-
Data Pipeline Expert
hace 3 semanas
Guadalajara, México Bebeedatapipelineengineer A tiempo completoWe are seeking an experienced Data Pipeline Engineer to own the design, development, and operation of reliable, secure, and cost-efficient data pipelines.- Key Responsibilities:- Design and build Airflow DAGs (TaskFlow, dynamic DAGs, deferrable operators, Secrets backend).- Develop Python ETL/ELT for APIs, storage, message buses, and databases; create...
-
DevOps Engineer Senior Specialist
hace 4 días
Guadalajara, México NTT DATA A tiempo completo**Req ID**: 343454 We are currently seeking a DevOps Engineer Senior Specialist to join our team in GDL, Jalisco (MX-JAL), Mexico (MX). **Technical skills required** - Proficiency in AWS Cloud Platform - Strong understanding of Cloud Networking - Experience with Infrastructure as Code using Terraform - Hands-on with CI/CD pipelines via GitHub Actions -...
