Azure Data Engineer

Become a Derevian and release your Superpower

Derevo

Science, art & human power applied to your information.

We are a consulting firm specialized in data discovery projects adding meaninig to your data through our disruptive methodology which combines elements of design thinking + user experience with technological platforms for information analytics.

**Summary**:
The desired profile should have hands-on experience in designing, establishing, and maintaining data management and storing systems. Skilled in collecting, processing, cleaning, and deploying large datasets, understanding ER data models, and integrating with multiple data sources. Efficient in analyzing, communicating, and proposing different ways of building Data Warehouses, Data Lakes, End-to-End Pipelines, and Big Data solutions to clients, either in batch or streaming strategies.

**Technical Proficiencies**:

- SQL:
Data Definition Language, Data Manipulation Language, Intermediate/advanced queries for analytical purpose, Subqueries, CTEs, Data types, Joins with business rules applied, Grouping and Aggregates for business metrics, Indexing and optimizing queries for efficient ETL process, Stored Procedures for transforming and preparing data, SSMS, DBeaver
- Python:
Experience with object-oriented, manage and processing large datasets, optimizing memory footprint, data structures, ingesting data through multiple structured or unstructured data sources, best practices for writing efficient code. Knowledge in pandas, numpy, pyspark libraries
- Azure:
Intermediate/Advanced knowledge in

Azure Storage Account:
Provision Azure Blob Storage or Azure Data Lake instances

Build efficient file systems for storing data into folders with static or parametrized names, considering possible security rules and risks

Experience identifying use cases for open-source file formats like parquet, AVRO, ORC

Understanding optimized column-oriented file formats vs optimized row-oriented file formats

Implementing security configurations through Access Keys, SAS, AAD, RBAC, ACLs

Azure Data Factory / Azure Synapse Pipelines:
Provision Azure Data Factory instances

Create ETL/ELT processes ingesting on-premises or cloud data sources

Use Azure IR, Self-Hosted IR, Azure-SSIS to establish connections to distinct data sources

Use of Copy or Polybase activities for loading data

Build efficient and optimized ADF Pipelines using linked services, datasets, parameters, triggers, data movement activities, data transformation activities, control flow activities and mapping data flows

Build Incremental and Re-Processing Loads

Azure Synapse Analytics:
Experience identifying and implementing use cases for dedicated and serverless SQL Pools

Experience identifying and implementing use cases for Spark Pools

Understanding MPP architectures and how to work with big data workflows

Experience identifying use cases for Copy or Polybase loads using different structured and semi-structured data sources

Create and manage internal, external and temporary tables in Azure Synapse Analytics

Use of staging, bronze, silver and gold layers. Load structured and semi-structured data in batch using incremental or reprocess strategies

Optimizing the Data Warehouse: strong knowledge of best practices to optimize query performance using replicate, round robin and hash distributions, data partitions, table indexing and table statistics

Optimizing Apache Spark cluster: strong knowledge of best practices to optimize spark job executions, using optimal data format, caching, partitioning, and bucketing data, optimizing joins and shuffles, broadcast variables, hyperspace indexes. Understanding of lazy executions, narrow and wide transformations. Work with the correct executor size in Spark cluster

Investigate optimal data skews in dedicated SQL and Spark pools

Monitoring and analyzing query execution plans to optimize response times and credit consumption

Manage workloads classes to assign different compute resource levels

Understanding of star and snowflake schemas, Watermark date columns, Slowly changing dimensions, ACID compliance and big data massively parallel processing architectures
- Desirable

Azure Databricks, Azure Purview, Azure Event Hubs, Azure Streaming Analytics
- Desired Certifications

Microsoft Certified: DP203 Azure Data Engineer Associate

Databricks Certified Developer

La contratación es inmediata, los beneficios que ofrecemos son superiores a los de ley, esquema 100% nominal, trabajo 100% remoto, trabajamos con diferentes clientes internacionales y nacionales por lo que los proyectos son muy interesantes para el desarrollo profesional, adicional apoyamos en certificaciones de nuestros colaboradores para continuar con el crecimiento.