Юзтех 12 дней назад

Data Engineer

  • Удалённо
  • Полная занятость
PythonSQLCI/CDGitKafkaJiraData EngineeringApache SparkHadoopS3Apache AirflowApache IcebergParquet

Описание

Responsibilities

  • Analysis of existing solutions and data marts
  • Migration of data marts from Hadoop stack to Apache Iceberg + Spark 3.5 + Apache Airflow stack
  • Implementation of missing replicas for data marts
  • Reworking existing solutions on the target stack
  • Building and developing data marts, participation in testing
  • Filling and updating technical documentation

Requirements

  • Confident knowledge of SQL and Python
  • Experience with Apache Spark
  • Practical experience with data storage based on Hadoop / S3
  • Knowledge and experience with Apache Iceberg
  • Experience using Apache Airflow
  • Understanding of data storage formats, including Parquet
  • Experience with Git, CI/CD, Jira, Confluence
  • Basic knowledge of Kafka

We offer

  • Career and professional opportunity in a stable, accredited IT company
  • Extended DMI policy with dentistry
  • Remote work and flexible schedule
  • Training, certification, English language club
  • Gamified loyalty program