Cloud Data Architect

Responsibilities:

• Implement and manage data ingestion pipelines from diverse sources such as Kafka, RDBMS (Postgres) using CDC (Change Data Capture), and file systems (CSV) following Medalion Architecture principles
• Develop and optimize data transformations using PySpark and SQL to handle data ranging from MB to GB, depending on the source
• Conduct unit testing and integration testing to ensure the accuracy and reliability of data transformations and pipelines
• Work with AWS technologies, including S3 for data storage and Docker on AWS for containerized applications
• Implement and manage infrastructure using Terraform, such as creating S3 buckets, managing Databricks Service Principals, and deploying infrastructure as code
• Deploy and manage solutions using CI/CD pipelines, particularly with CircleCI, to ensure seamless and automated deployment processes

Requirements:

• Minimum 8 years of a professional experience as Data Engineer
• Proficiency in Python/PySpark
• Strong experience with AWS cloud services (3S, Docker)
• Knowledge of ETL Processing
• Strong experience of DataBricks
• Effective communication skills in English (minimum B2 level)
• Knowledge of system design
• Proficiency in SQL
• Basic knowledge of Azure
• Familiarity with Kedro and Airbyte - nice to have
• Knowledge of Machine Learning - nice to have
• Experience in Unity Catalog

Offer:

• Private medical care
• Co-financing for the sport card
• Training & learning opportunities
• Constant support of dedicated consultant
• Team-building events organized by DCG
• Employee referral program