Data Engineer

Who We AreBuild. Scale. Sustain.PALO IT is a global technology consultancy that crafts tech as a force for good. We design develop and scale digital and sustainable products and services to unlock value across the triple bottom line : people planet profit. We do the right thing and we do it right. We're proud to be a World Economic Forum New Champion and a B Corp-certified company.We are small enough to care locally big enough to deliver globally (5 continents 18 offices 650 experts from 50 nationalities)We are robust and resilient (100% independent and 0 debt)We are entrepreneurs and passionate experts : we invest in what we believe genuinely and work as a collective intelligenceWe are positive courageous caring doers and committed to excellenceAbout Gen-e2While the market is still largely AI-augmenting delivery we have reinvented the SDLC to be AI First. Our approach is a game-changer in productivity and quality with a strong collaboration between generative AI and our best Talents :We now generate 95% of the entire product code documentation infrastructure as code and even design with GitHub Copilot.The quality consistently exceeds the output of our best traditional engineering teams.A product repository houses all product artefacts giving AI full project context for higher-quality generation.A library of rules and prompts defines coding standards design principles and security guidelines ensuring enterprise-grade quality and scalability.With Gen-e2 we deliver end-to-end products 23 faster than traditional approaches while raising the bar for engineering excellence.Your RoleWe're looking for a Data Engineer to support our partner in managing the influx of weather-related datasets that are critical for their business transformation in the insurance domain. You'll be the go-to expert for ingesting organizing and preparing weather-based data pipelines that drive insights and automation.Key ResponsibilitiesDesign develop and manage scalable and secure data pipelines to ingest and process large volumes of weather data from various external sources.Leverage APIs batch and streaming ingestion mechanisms for real-time and historical weather datasets.Collaborate with data scientists solution architects and engineering teams to integrate weather data into products and machine learning workflows.Work with stakeholders to define data models and ensure weather data is cleaned structured and usable for downstream applications.Implement data quality checks validation routines and anomaly detection specific to weather information (e.g. temperature precipitation wind patterns).Optimize storage and compute costs for high-volume weather datasets ensuring data pipelines are cost-effective and resilient.Maintain documentation and metadata to provide transparency and governance over data usage.Who You AreProven experience working as a Data Engineer ideally within insurance climate tech or industries where weather data plays a pivotal role.Strong proficiency in Python SQL and at least one major cloud platform (AWS preferred; GCP or Azure also welcome).Experience working with weather data APIs such as NOAA OpenWeatherMap Weather Company Meteomatics or similar.Proficient in ETL tools data orchestration frameworks (Airflow Prefect Dagster) and streaming platforms (Kafka Kinesis).Familiar with geospatial data formats and processing tools (GeoJSON raster data GDAL PostGIS).Solid understanding of data lakes data warehouses (Snowflake BigQuery Redshift) and data modeling best practices.Strong problem-solving skills with attention to detail and a proactive mindset.B2-C1 level in EnglishMore About PALO ITWe're eager to adapt to change learn from our experiences and move to meet our planet's urgent needs. We are continuously taking action to :Become a climate net-zero companyDeliver projects with a positive impactTrain 100% of our workforce on impactAchieve B Corp certification among all our offices across the globeContinuously measure & improve employee happinessOur clients are amongst the world's most successful companies. We innovate with both established Fortune 1000s SMEs and start-ups who aim to make an impact become global leaders and address the worlds most complex challenges.What We OfferStimulating working environmentsUnique career pathInternational mobilityInternal R&D projectsKnowledge sharingPersonalized trainingEntrepreneurship & intrapreneurship opportunitiesFor more on our team culture and benefitsCheck out our careers pageAnd our PALOCast with direct testimonies from our PalowansKey SkillsApache HiveS3HadoopRedshiftSparkAWSApache PigNoSQLBig DataData WarehouseKafkaScalaEmployment Type : Full TimeExperience : yearsVacancy : 1 #J-18808-Ljbffr