Юзтех 12 дней назад
Data Engineer
- Удалённо
- Полная занятость
PythonSQLCI/CDGitKafkaJiraData EngineeringApache SparkHadoopS3Apache AirflowApache IcebergParquet
Описание
Responsibilities
- Analysis of existing solutions and data marts
- Migration of data marts from Hadoop stack to Apache Iceberg + Spark 3.5 + Apache Airflow stack
- Implementation of missing replicas for data marts
- Reworking existing solutions on the target stack
- Building and developing data marts, participation in testing
- Filling and updating technical documentation
Requirements
- Confident knowledge of SQL and Python
- Experience with Apache Spark
- Practical experience with data storage based on Hadoop / S3
- Knowledge and experience with Apache Iceberg
- Experience using Apache Airflow
- Understanding of data storage formats, including Parquet
- Experience with Git, CI/CD, Jira, Confluence
- Basic knowledge of Kafka
We offer
- Career and professional opportunity in a stable, accredited IT company
- Extended DMI policy with dentistry
- Remote work and flexible schedule
- Training, certification, English language club
- Gamified loyalty program