Data Engineer/Integration Lead
hace 4 semanas
Company
Requisition ID: 122859BR
Required Qualifications:
- 5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark.
- Proven track record of building scalable data pipelines in cloud environments.
- Proficiency in flow design, processors, and data provenance in Apache NiFi.
- Strong expertise in Spark, Hadoop, and distributed computing on AWS EMR.
- In-depth knowledge of AWS services (S3, Glue, Redshift, RDS, Lambda, Step Functions).
- Experience with data formats (JSON, CSV, Parquet, Avro) and transformation techniques.
- Strong problem-solving skills and ability to troubleshoot complex data processing issues.
- Excellent communication skills with the ability to document and explain technical details clearly.
Preferred Qualifications:
- AWS Certified Solutions Architect or Data Analytics Specialty.
- Experience with data governance frameworks and compliance requirements.
- Familiarity with CI/CD pipelines and version control (GitLab, Jenkins).
Key Responsibilities:
Design & Develop Data Pipelines:
- Architect and implement end-to-end data pipelines using AWS S3, EMR, Glue, Step Functions, Apache NiFi, Spark.
- Manage data ingestion processes from AWS S3, ensuring secure and efficient data transfer.
- Implement initial data routing, validation, and transformations using Apache NiFi processors and Spark Data Engines.
Data Processing & Transformation:
- Integrate using AWS EMR, Apache NiFi, Spark to perform complex data transformations and analytics.
- Optimize Spark jobs for processing large-scale datasets with a focus on performance and resource utilization.
- Handle both historical and incremental data loads, ensuring data consistency and integrity.
Data Storage & Management:
- Define and implement data storage strategies across S3, RDS, and Redshift, adhering to business requirements.
- Manage data catalog creation and schema management using AWS Glue.
Automation & Orchestration:
- Develop and manage workflows using Apache Airflow, AWS Step Functions to automate data processing tasks.
- Implement monitoring, error handling, and retries within the orchestration framework.
Security & Compliance:
- Ensure data security with encryption (AES-256, TLS) and IAM role-based access controls.
- Implement data governance policies using AWS Glue Data Catalog to ensure compliance with regulatory requirements.
Performance Monitoring & Optimization:
- Utilize AWS CloudWatch to monitor the performance of EMR clusters, NiFi flows and data storage.
- Continuously optimize Spark job configurations and NiFi data flows for maximum throughput and minimal latency.
About Us
Infosys is a global leader in next-generation digital services and consulting. We enable clients in more than 50 countries to navigate their digital transformation. With over four decades of experience in managing the systems and workings of global enterprises, we expertly steer our clients through their digital journey. We do it by enabling the enterprise with an AI-powered core that helps prioritize the execution of change. We also empower the business with agile digital at scale to deliver unprecedented levels of performance and customer delight. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.
Equal Employment Opportunity:
Infosys provides equal employment opportunities to applicants and employees without regard to race; color; sex; gender identity; sexual orientation; religious practices and observances; national origin; pregnancy, childbirth, or related medical conditions; status as a protected veteran or spouse/family member of a protected veteran; or disability.
#J-18808-Ljbffr-
Lead Data Engineer
hace 4 semanas
distrito federal, México Thomson Reuters A tiempo completoAre you passionate about the chance to bring your experience to a world-class company that is market-leading for both content and technology? If yes, we are looking for you! Join our team! We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing, developing, and...
-
Senior Data Integration Engineer
hace 1 semana
distrito federal, México EPAM Systems A tiempo completoWe are in search of a Senior Data Integration Engineer with expertise in ETL/ELT solutions, SQL, data visualization, Google Cloud Platform, and Pentaho BI. Your primary role will involve supporting the entire data pipeline, identifying and resolving issues with data sources, understanding the logic behind queries, and communicating effectively with key...
-
Data Engineer
hace 4 semanas
distrito federal, México Enroutesystems A tiempo completoWe love technology, and we enjoy what we do. We are always looking for innovation. We have social awareness and try to improve it daily. We make things happen. You can trust us. Our Enrouters are always up for a challenge. We ask questions, and we love to learn. We pride ourselves on having great benefits and compensations, a fantastic work environment,...
-
Lead Integration Engineer
hace 3 días
distrito federal, México Global Payments A tiempo completoLead Integration Engineer Apply locations Cuajimalpa, Mexico City, Mexico time type Full time posted on Posted 3 Days Ago job requisition id R0055927 Every day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team...
-
Cloud Data Engineer Lead
hace 4 semanas
distrito federal, México MX003 Marsh And Mclennan Servicios S.A. De Cv A tiempo completoDescription : MMC is seeking candidates for the following position based in the Mexico City office and be onsite 3 days a week : Cloud Data Engineer Lead / Data Engineering Lead What can you expect? The Cloud Data Engineer Lead will lead a small team, and be part of a very talented large team building data ingestion and transformation pipelines in the...
-
Lead Integration Engineer
hace 1 semana
distrito federal, México Global Payments A tiempo completoEvery day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team helps over 3 million companies, more than 1,300 financial institutions and over 600 million cardholders grow with confidence and achieve amazing results....
-
Data Engineer/Integration Lead
hace 4 semanas
distrito federal, México Infosys A tiempo completoRequired Qualifications: 5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark. Proven track record of building scalable data pipelines in cloud environments. Proficiency in flow design, processors, and data provenance in Apache NiFi. Strong expertise in Spark, Hadoop, and...
-
Digital Engineering Lead Engineer
hace 3 horas
distrito federal, México NTT DATA, Inc. A tiempo completoNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Mexico City, México (MX-MEX), Mexico (MX). Who We Are: At NTT DATA America's ,...
-
Data Ingestion Engineer
hace 4 semanas
distrito federal, México Encora A tiempo completoImportant Information: Years of Experience : 3+ years of experience in data engineering, ingestion pipelining, and ETL/ELT. Job Mode : Full-time. Work Mode : Remote. Job Summary: The Data Ingestion Engineer will be responsible for managing data ingestion processes, platform management, and optimizing ETL/ELT pipelines on the Databricks platform. The role...
-
QA Engineer
hace 4 semanas
distrito federal, México Data Privacy A tiempo completoYou will be part of the QA team that is focused on quality on front end embedded client/TV as well as backend cloud services that reports high quality data to customers. The client QA team is responsible to test various embedded clients and firmware for a variety of Vizio TVs. The backend QA team is responsible to validate our award-winning Inscape data set...
-
Senior Integration Software Engineer
hace 3 semanas
distrito federal, México Workyard A tiempo completoWorkyard is a fast-growing venture backed startup that is developing an innovative workforce management platform for the construction market. In an industry where $300 billion is spent annually on labor, we are fundamentally changing the experience for companies and workers by adding trust, transparency, and technology to workforce management and enable...
-
Senior Data Engineer/Data Architect
hace 4 semanas
distrito federal, México LAAgencia A tiempo completoSenior Data Engineer/Data Architect (Azure) Mexico Tasks: Implement data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. Develop scalable and re-usable, self-service frameworks for data ingestion and processing. Design, build, and manage SQL Server databases in the Azure cloud. Perform data...
-
Lead Software Engineer
hace 4 semanas
distrito federal, México Mheducation A tiempo completoOverview Build the Future Do you enjoy testing the limits of possibility? At McGraw Hill, our Lead Software Engineers drive progress and help build the future of learning. If you have the passion and technical expertise to thrive in an innovative and agile environment, we want to learn more about you. Your impact on the team The Content Acceleration team ...
-
Clinical Data Lead
hace 2 semanas
distrito federal, México ICON A tiempo completoClinical Data Lead, Imaging - Mexico City - Hybrid ICON plc is a world-leading healthcare intelligence and clinical research organization. We’re proud to foster an inclusive environment driving innovation and excellence, and we welcome you to join us on our mission to shape the future of clinical development. JR124806 Clinical Data Lead Mexico City -...
-
DevOps Engineer
hace 2 semanas
distrito federal, México NTT DATA, Inc. A tiempo completoNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a DevOps Engineer to join our team in Guadalajara, Jalisco (MX-JAL), Mexico (MX). Responsibilities: DevOps Engineer with over 5 years...
-
Data Platform Engineer
hace 4 semanas
distrito federal, México Infostretch Corporation A tiempo completoCompany Description: Apexon is a digital-first technology services firm backed by Goldman Sachs Asset Management and Everstone Capital. We specialize in accelerating business transformation and delivering human-centric digital experiences. For over 17 years, Apexon has been meeting customers wherever they are in the digital lifecycle and helping them...
-
Middleware (Web/App Servers) SME Lead
hace 4 semanas
distrito federal, México NTT DATA, Inc. A tiempo completoNTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Middleware (Web/App Servers) SME Lead to join our team in Mexico, México (MX-MEX), Mexico (MX). Short Role Description: The...
-
Lead Architect
hace 3 semanas
distrito federal, México NTT DATA A tiempo completoBuscamos un Lead Architect con experiencia en Java y microservicios para diseñar, construir y publicar APIs y integrar sistemas legados. NTT Data es un equipo de más de 139.000 profesionales con presencia en 50 países y diferentes sectores. Nuestra misión es ofrecer soluciones tecnológicas y de negocio. Las actividades principales a realizar incluyen:...
-
Data Team Lead
hace 1 semana
distrito federal, México World Business Lenders, LLC A tiempo completoAbout World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...
-
Data Team Lead
hace 3 semanas
distrito federal, México WBL California, LLC A tiempo completoAbout World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...