Big Data Engineer

Overview At Avenga, we believe that human creativity empowers technology that matters. Operating globally, our 6000+ specialists provide a full spectrum of services, including business and tech advisory, enterprise solutions, CX, UX and Ul design, managed services, product development, and software development. This is the job In CDMX within the TMT industry, we are actively seeking a professional to strengthen our team dedicated to Big Data ecosystem management and data architecture. You'll be responsible for building and maintaining scalable Big Data solutions, working with cutting-edge technologies including Cloudera, Spark, and modern data lakehouse architectures. This role combines technical expertise with strategic vision to design Big Data architectures applicable across various business areas. This position is more office-based than remote, requiring a hybrid approach of 2 to 4 days per week at offices located in Bosques and/or Constituyentes. This is you Responsibilities Maintain and optimize the Big Data ecosystem using Cloudera platform. Handle, process, and manage unstructured data from multiple sources. Implement and maintain scripts using Python for data processing and automation. Design and build scalable Big Data architectures with strategic vision applicable to various business domains. Configure and optimize the Big Data ecosystem for performance and reliability. Conduct configuration and functional testing of ecosystem components. Design and develop comprehensive data models for information storage. Implement data models for storing both structured and unstructured information. Create efficient data flows for seamless information processing. Execute loading, validation, homogenization, transformation, and modeling of data from diverse sources. Lead data homogenization processes to ensure consistency across systems. Ensure and maintain high data quality standards throughout the pipeline. Develop and implement control mechanisms to manage and monitor data quality. Design and maintain publisher-subscriber services for real-time data distribution. Build and optimize data pipelines and streaming solutions using Flume, Sqoop, and Kafka. Create and maintain dashboards and visualizations using Kibana, Tableau, or Grafana. Great candidates will have: Qualifications At least 3 year implementing Big Data solutions for data management using HBase, Hive, and Spark. 2+ years of hands-on SQL experience. Proven track record handling unstructured data (minimum 2 years). Solid background in publisher-subscriber service design (2+ years required). 2+ years maintaining Big Data ecosystems, specifically Cloudera platform. Experience with data transformation in Big Data ecosystems including PIG, Spark, and ETL processes (minimum 2 years). 3+ years of programming expertise in Python, R, Java, or Scala. Demonstrated ability implementing dashboards using Kibana, Tableau, or Grafana (at least 3 years). Strong background in pipeline and streaming design with Flume, Sqoop, and Kafka (3+ years required). Strategic vision and logic for building Big Data architectures applicable to diverse business areas. Practical experience constructing Data Lakehouse solutions. Spanish speaker. Nice-to-have skills: Knowledge of Artificial Intelligence solutions, models, and tools. Experience with AI model implementation and integration. Advanced metadata analysis capabilities for feature selection. This is your role Maintain and optimize the Big Data ecosystem using Cloudera platform. Handle, process, and manage unstructured data from multiple sources. Implement and maintain scripts using Python for data processing and automation. Design and build scalable Big Data architectures with strategic vision applicable to various business domains. Configure and optimize the Big Data ecosystem for performance and reliability. Conduct configuration and functional testing of ecosystem components. Design and develop comprehensive data models for information storage. Implement data models for storing both structured and unstructured information. Create efficient data flows for seamless information processing. Execute loading, validation, homogenization, transformation, and modeling of data from diverse sources. Lead data homogenization processes to ensure consistency across systems. Ensure and maintain high data quality standards throughout the pipeline. Develop and implement control mechanisms to manage and monitor data quality. Design and maintain publisher-subscriber services for real-time data distribution. Build and optimize data pipelines and streaming solutions using Flume, Sqoop, and Kafka. Create and maintain dashboards and visualizations using Kibana, Tableau, or Grafana. At Avenga, everyone matters. We provide equal opportunities in recruitment, career development, and leadership, regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, or any other characteristic. We are committed to fostering a work environment where our diverse community of employees, candidates, and business partners actively shapes our growth. By bringing together people from different backgrounds and experiences, we build a workplace where everyone feels free to be themselves while honoring the boundaries of others. #J-18808-Ljbffr