Data Engineer/Integration Lead

hace 4 semanas


distrito federal, México Infosys Limited A tiempo completo

Company

Requisition ID: 122859BR

Required Qualifications:

  • 5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark.
  • Proven track record of building scalable data pipelines in cloud environments.
  • Proficiency in flow design, processors, and data provenance in Apache NiFi.
  • Strong expertise in Spark, Hadoop, and distributed computing on AWS EMR.
  • In-depth knowledge of AWS services (S3, Glue, Redshift, RDS, Lambda, Step Functions).
  • Experience with data formats (JSON, CSV, Parquet, Avro) and transformation techniques.
  • Strong problem-solving skills and ability to troubleshoot complex data processing issues.
  • Excellent communication skills with the ability to document and explain technical details clearly.

Preferred Qualifications:

  • AWS Certified Solutions Architect or Data Analytics Specialty.
  • Experience with data governance frameworks and compliance requirements.
  • Familiarity with CI/CD pipelines and version control (GitLab, Jenkins).

Key Responsibilities:


Design & Develop Data Pipelines:

  • Architect and implement end-to-end data pipelines using AWS S3, EMR, Glue, Step Functions, Apache NiFi, Spark.
  • Manage data ingestion processes from AWS S3, ensuring secure and efficient data transfer.
  • Implement initial data routing, validation, and transformations using Apache NiFi processors and Spark Data Engines.

Data Processing & Transformation:

  • Integrate using AWS EMR, Apache NiFi, Spark to perform complex data transformations and analytics.
  • Optimize Spark jobs for processing large-scale datasets with a focus on performance and resource utilization.
  • Handle both historical and incremental data loads, ensuring data consistency and integrity.

Data Storage & Management:

  • Define and implement data storage strategies across S3, RDS, and Redshift, adhering to business requirements.
  • Manage data catalog creation and schema management using AWS Glue.

Automation & Orchestration:

  • Develop and manage workflows using Apache Airflow, AWS Step Functions to automate data processing tasks.
  • Implement monitoring, error handling, and retries within the orchestration framework.

Security & Compliance:

  • Ensure data security with encryption (AES-256, TLS) and IAM role-based access controls.
  • Implement data governance policies using AWS Glue Data Catalog to ensure compliance with regulatory requirements.

Performance Monitoring & Optimization:

  • Utilize AWS CloudWatch to monitor the performance of EMR clusters, NiFi flows and data storage.
  • Continuously optimize Spark job configurations and NiFi data flows for maximum throughput and minimal latency.

About Us

Infosys is a global leader in next-generation digital services and consulting. We enable clients in more than 50 countries to navigate their digital transformation. With over four decades of experience in managing the systems and workings of global enterprises, we expertly steer our clients through their digital journey. We do it by enabling the enterprise with an AI-powered core that helps prioritize the execution of change. We also empower the business with agile digital at scale to deliver unprecedented levels of performance and customer delight. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.


Equal Employment Opportunity:

Infosys provides equal employment opportunities to applicants and employees without regard to race; color; sex; gender identity; sexual orientation; religious practices and observances; national origin; pregnancy, childbirth, or related medical conditions; status as a protected veteran or spouse/family member of a protected veteran; or disability.

#J-18808-Ljbffr
  • Lead Data Engineer

    hace 4 semanas


    distrito federal, México Thomson Reuters A tiempo completo

    Are you passionate about the chance to bring your experience to a world-class company that is market-leading for both content and technology? If yes, we are looking for you! Join our team! We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will be responsible for designing, developing, and...


  • distrito federal, México EPAM Systems A tiempo completo

    We are in search of a Senior Data Integration Engineer with expertise in ETL/ELT solutions, SQL, data visualization, Google Cloud Platform, and Pentaho BI. Your primary role will involve supporting the entire data pipeline, identifying and resolving issues with data sources, understanding the logic behind queries, and communicating effectively with key...

  • Data Engineer

    hace 4 semanas


    distrito federal, México Enroutesystems A tiempo completo

    We love technology, and we enjoy what we do. We are always looking for innovation. We have social awareness and try to improve it daily. We make things happen. You can trust us. Our Enrouters are always up for a challenge. We ask questions, and we love to learn. We pride ourselves on having great benefits and compensations, a fantastic work environment,...


  • distrito federal, México Global Payments A tiempo completo

    Lead Integration Engineer Apply locations Cuajimalpa, Mexico City, Mexico time type Full time posted on Posted 3 Days Ago job requisition id R0055927 Every day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team...

  • Cloud Data Engineer Lead

    hace 4 semanas


    distrito federal, México MX003 Marsh And Mclennan Servicios S.A. De Cv A tiempo completo

    Description : MMC is seeking candidates for the following position based in the Mexico City office and be onsite 3 days a week : Cloud Data Engineer Lead / Data Engineering Lead What can you expect? The Cloud Data Engineer Lead will lead a small team, and be part of a very talented large team building data ingestion and transformation pipelines in the...


  • distrito federal, México Global Payments A tiempo completo

    Every day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team helps over 3 million companies, more than 1,300 financial institutions and over 600 million cardholders grow with confidence and achieve amazing results....


  • distrito federal, México Infosys A tiempo completo

    Required Qualifications: 5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark. Proven track record of building scalable data pipelines in cloud environments. Proficiency in flow design, processors, and data provenance in Apache NiFi. Strong expertise in Spark, Hadoop, and...


  • distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Mexico City, México (MX-MEX), Mexico (MX). Who We Are: At NTT DATA America's ,...

  • Data Ingestion Engineer

    hace 4 semanas


    distrito federal, México Encora A tiempo completo

    Important Information: Years of Experience : 3+ years of experience in data engineering, ingestion pipelining, and ETL/ELT. Job Mode : Full-time. Work Mode : Remote. Job Summary: The Data Ingestion Engineer will be responsible for managing data ingestion processes, platform management, and optimizing ETL/ELT pipelines on the Databricks platform. The role...

  • QA Engineer

    hace 4 semanas


    distrito federal, México Data Privacy A tiempo completo

    You will be part of the QA team that is focused on quality on front end embedded client/TV as well as backend cloud services that reports high quality data to customers. The client QA team is responsible to test various embedded clients and firmware for a variety of Vizio TVs. The backend QA team is responsible to validate our award-winning Inscape data set...


  • distrito federal, México Workyard A tiempo completo

    Workyard is a fast-growing venture backed startup that is developing an innovative workforce management platform for the construction market. In an industry where $300 billion is spent annually on labor, we are fundamentally changing the experience for companies and workers by adding trust, transparency, and technology to workforce management and enable...


  • distrito federal, México LAAgencia A tiempo completo

    Senior Data Engineer/Data Architect (Azure) Mexico Tasks: Implement data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. Develop scalable and re-usable, self-service frameworks for data ingestion and processing. Design, build, and manage SQL Server databases in the Azure cloud. Perform data...

  • Lead Software Engineer

    hace 4 semanas


    distrito federal, México Mheducation A tiempo completo

    Overview Build the Future Do you enjoy testing the limits of possibility? At McGraw Hill, our Lead Software Engineers drive progress and help build the future of learning. If you have the passion and technical expertise to thrive in an innovative and agile environment, we want to learn more about you. Your impact on the team The Content Acceleration team ...

  • Clinical Data Lead

    hace 2 semanas


    distrito federal, México ICON A tiempo completo

    Clinical Data Lead, Imaging - Mexico City - Hybrid ICON plc is a world-leading healthcare intelligence and clinical research organization. We’re proud to foster an inclusive environment driving innovation and excellence, and we welcome you to join us on our mission to shape the future of clinical development. JR124806 Clinical Data Lead Mexico City -...

  • DevOps Engineer

    hace 2 semanas


    distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a DevOps Engineer to join our team in Guadalajara, Jalisco (MX-JAL), Mexico (MX). Responsibilities: DevOps Engineer with over 5 years...

  • Data Platform Engineer

    hace 4 semanas


    distrito federal, México Infostretch Corporation A tiempo completo

    Company Description: Apexon is a digital-first technology services firm backed by Goldman Sachs Asset Management and Everstone Capital. We specialize in accelerating business transformation and delivering human-centric digital experiences. For over 17 years, Apexon has been meeting customers wherever they are in the digital lifecycle and helping them...


  • distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Middleware (Web/App Servers) SME Lead to join our team in Mexico, México (MX-MEX), Mexico (MX). Short Role Description: The...

  • Lead Architect

    hace 3 semanas


    distrito federal, México NTT DATA A tiempo completo

    Buscamos un Lead Architect con experiencia en Java y microservicios para diseñar, construir y publicar APIs y integrar sistemas legados. NTT Data es un equipo de más de 139.000 profesionales con presencia en 50 países y diferentes sectores. Nuestra misión es ofrecer soluciones tecnológicas y de negocio. Las actividades principales a realizar incluyen:...

  • Data Team Lead

    hace 1 semana


    distrito federal, México World Business Lenders, LLC A tiempo completo

    About World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...

  • Data Team Lead

    hace 3 semanas


    distrito federal, México WBL California, LLC A tiempo completo

    About World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...