Data Engineer/Integration Lead

hace 2 meses


distrito federal, México Infosys Limited A tiempo completo

Company

Requisition ID: 122859BR

Required Qualifications:

  • 5+ years of experience in data engineering using Python with a focus on AWS S3, EMR, Glue, Step Functions, Apache NiFi and Spark.
  • Proven track record of building scalable data pipelines in cloud environments.
  • Proficiency in flow design, processors, and data provenance in Apache NiFi.
  • Strong expertise in Spark, Hadoop, and distributed computing on AWS EMR.
  • In-depth knowledge of AWS services (S3, Glue, Redshift, RDS, Lambda, Step Functions).
  • Experience with data formats (JSON, CSV, Parquet, Avro) and transformation techniques.
  • Strong problem-solving skills and ability to troubleshoot complex data processing issues.
  • Excellent communication skills with the ability to document and explain technical details clearly.

Preferred Qualifications:

  • AWS Certified Solutions Architect or Data Analytics Specialty.
  • Experience with data governance frameworks and compliance requirements.
  • Familiarity with CI/CD pipelines and version control (GitLab, Jenkins).

Key Responsibilities:


Design & Develop Data Pipelines:

  • Architect and implement end-to-end data pipelines using AWS S3, EMR, Glue, Step Functions, Apache NiFi, Spark.
  • Manage data ingestion processes from AWS S3, ensuring secure and efficient data transfer.
  • Implement initial data routing, validation, and transformations using Apache NiFi processors and Spark Data Engines.

Data Processing & Transformation:

  • Integrate using AWS EMR, Apache NiFi, Spark to perform complex data transformations and analytics.
  • Optimize Spark jobs for processing large-scale datasets with a focus on performance and resource utilization.
  • Handle both historical and incremental data loads, ensuring data consistency and integrity.

Data Storage & Management:

  • Define and implement data storage strategies across S3, RDS, and Redshift, adhering to business requirements.
  • Manage data catalog creation and schema management using AWS Glue.

Automation & Orchestration:

  • Develop and manage workflows using Apache Airflow, AWS Step Functions to automate data processing tasks.
  • Implement monitoring, error handling, and retries within the orchestration framework.

Security & Compliance:

  • Ensure data security with encryption (AES-256, TLS) and IAM role-based access controls.
  • Implement data governance policies using AWS Glue Data Catalog to ensure compliance with regulatory requirements.

Performance Monitoring & Optimization:

  • Utilize AWS CloudWatch to monitor the performance of EMR clusters, NiFi flows and data storage.
  • Continuously optimize Spark job configurations and NiFi data flows for maximum throughput and minimal latency.

About Us

Infosys is a global leader in next-generation digital services and consulting. We enable clients in more than 50 countries to navigate their digital transformation. With over four decades of experience in managing the systems and workings of global enterprises, we expertly steer our clients through their digital journey. We do it by enabling the enterprise with an AI-powered core that helps prioritize the execution of change. We also empower the business with agile digital at scale to deliver unprecedented levels of performance and customer delight. Our always-on learning agenda drives their continuous improvement through building and transferring digital skills, expertise, and ideas from our innovation ecosystem.


Equal Employment Opportunity:

Infosys provides equal employment opportunities to applicants and employees without regard to race; color; sex; gender identity; sexual orientation; religious practices and observances; national origin; pregnancy, childbirth, or related medical conditions; status as a protected veteran or spouse/family member of a protected veteran; or disability.

#J-18808-Ljbffr

  • distrito federal, México EPAM Systems A tiempo completo

    We are in search of a Senior Data Integration Engineer with expertise in ETL/ELT solutions, SQL, data visualization, Google Cloud Platform, and Pentaho BI. Your primary role will involve supporting the entire data pipeline, identifying and resolving issues with data sources, understanding the logic behind queries, and communicating effectively with key...

  • Data Engineer

    hace 2 meses


    distrito federal, México Enroutesystems A tiempo completo

    We love technology, and we enjoy what we do. We are always looking for innovation. We have social awareness and try to improve it daily. We make things happen. You can trust us. Our Enrouters are always up for a challenge. We ask questions, and we love to learn. We pride ourselves on having great benefits and compensations, a fantastic work environment,...

  • Lead Integration Engineer

    hace 3 semanas


    distrito federal, México Global Payments A tiempo completo

    Lead Integration Engineer Apply locations Cuajimalpa, Mexico City, Mexico time type Full time posted on Posted 3 Days Ago job requisition id R0055927 Every day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team...


  • distrito federal, México NTT Data A tiempo completo

    Req ID: 291166 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Hyderabad, Telangana (IN-TG), India (IN)....


  • distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Mexico City, México (MX-MEX), Mexico (MX). Who We Are: At NTT DATA America's ,...

  • Lead Integration Engineer

    hace 4 semanas


    distrito federal, México Global Payments A tiempo completo

    Every day, Global Payments makes it possible for millions of people to move money between buyers and sellers using our payments solutions for credit, debit, prepaid and merchant services. Our worldwide team helps over 3 million companies, more than 1,300 financial institutions and over 600 million cardholders grow with confidence and achieve amazing results....


  • distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Mexico City, México (MX-MEX), Mexico (MX). Who We Are: At NTT DATA America's ,...

  • QA Engineer

    hace 2 meses


    distrito federal, México Data Privacy A tiempo completo

    You will be part of the QA team that is focused on quality on front end embedded client/TV as well as backend cloud services that reports high quality data to customers. The client QA team is responsible to test various embedded clients and firmware for a variety of Vizio TVs. The backend QA team is responsible to validate our award-winning Inscape data set...


  • distrito federal, México LAAgencia A tiempo completo

    Senior Data Engineer/Data Architect (Azure) Mexico Tasks: Implement data ingestion pipelines from multiple data sources using Azure Data Factory, Azure Databricks and other ETL tools. Develop scalable and re-usable, self-service frameworks for data ingestion and processing. Design, build, and manage SQL Server databases in the Azure cloud. Perform data...

  • Lead Software Engineer

    hace 2 meses


    distrito federal, México Mheducation A tiempo completo

    Overview Build the Future Do you enjoy testing the limits of possibility? At McGraw Hill, our Lead Software Engineers drive progress and help build the future of learning. If you have the passion and technical expertise to thrive in an innovative and agile environment, we want to learn more about you. Your impact on the team The Content Acceleration team ...

  • Clinical Data Lead

    hace 4 semanas


    distrito federal, México ICON A tiempo completo

    Clinical Data Lead, Imaging - Mexico City - Hybrid ICON plc is a world-leading healthcare intelligence and clinical research organization. We’re proud to foster an inclusive environment driving innovation and excellence, and we welcome you to join us on our mission to shape the future of clinical development. JR124806 Clinical Data Lead Mexico City -...

  • DevOps Engineer

    hace 1 mes


    distrito federal, México NTT DATA, Inc. A tiempo completo

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a DevOps Engineer to join our team in Guadalajara, Jalisco (MX-JAL), Mexico (MX). Responsibilities: DevOps Engineer with over 5 years...

  • Data Team Lead

    hace 4 semanas


    distrito federal, México World Business Lenders, LLC A tiempo completo

    About World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...

  • Data Team Lead

    hace 1 mes


    distrito federal, México WBL California, LLC A tiempo completo

    About World Business Lenders ( World Business Lenders (WBL) provides general purpose short-term real estate collateralized commercial loans to a broad customer base comprised of small and medium sized businesses throughout the United States that lack access to traditional funding. WBL is a U.S.-based company with a 100% remote workforce. This is a remote...


  • distrito federal, México Aitopics A tiempo completo

    Alcon Our mission is to provide innovative vision products that enhance quality of life by helping people see better. From vision research to eye health, learn more at Alcon.com. At Alcon, we are driven by the meaningful work we do to help people see brilliantly. We innovate boldly, champion progress, and act with speed as the global leader in eye care....

  • Lead Security Engineer

    hace 4 semanas


    distrito federal, México HERE Technologies A tiempo completo

    Join Us to Shape the Future of Digital Security at a Global Scale! As a Lead Security Engineer, you will be at the forefront of safeguarding our critical infrastructure and data. Your role involves dynamic challenges including: Security Incident Analysis and Resolution : Leverage your expertise to manage and resolve security incidents from diverse sources...


  • distrito federal, México Nova Credit A tiempo completo

    Mid/Senior Software Engineer - API Integration (LatAm) At Nova Credit, our mission is to power a more fair and inclusive financial system for the world. We are on our way to accomplishing this mission by rewiring the financial industry with better credit infrastructure, analytics, and workflows, enabling more people to access credit opportunities. Our...

  • Data Architect SR

    hace 2 meses


    distrito federal, México Data Privacy A tiempo completo

    NEORIS is seeking a Data Architect to ensure that technology solutions for data are aligned with business needs and comply with established guidelines and security regulations. The role involves directing the technical implementation of projects and proposing transitional architectures for incremental value delivery. Additionally, the architect will oversee...


  • distrito federal, México Fortrea A tiempo completo

    As a leading global contract research organization (CRO) with a passion for scientific rigor and decades of clinical development experience, Fortrea provides pharmaceutical, biotechnology, and medical device customers a wide range of clinical development, patient access, and technology solutions across more than 20 therapeutic areas. Conducting operations in...


  • distrito federal, México Salesforce, Inc. A tiempo completo

    Sr. Data Engineer, Email Platform Location: Mexico - Mexico City Type: Full time Posted: 28 Days Ago About Salesforce: We’re Salesforce, the Customer Company, inspiring the future of business with AI + Data + CRM. Leading with our core values, we help companies across every industry blaze new trails and connect with customers in a whole new way. ...