Weekday AI вчера
Lead Data Engineer
4 000 000 – 6 000 000 ₽в год
- Delhi, NCR, Delhi, India
- Опыт от 6 лет
PythonSQLJavaDockerKafkaAWSAzureAirflowGCPSparkHadoopScala
Описание
Responsibilities
- Design and implement robust, end-to-end batch data pipelines using tools such as Apache Spark, Hadoop, and Airflow.
- Develop real-time streaming pipelines with technologies like Apache Kafka, Flink, or Spark Structured Streaming to deliver low-latency analytics.
- Optimize pipeline performance through partitioning, indexing, and efficient data format selection (e.g., Parquet, ORC).
- Establish data governance, quality checks, and automated monitoring to ensure data reliability and compliance.
- Lead capacity planning and scaling strategies for data infrastructure, including cluster management and cost optimization.
- Mentor and coach junior data engineers, fostering a collaborative and growth-oriented team culture.
- Collaborate with Data Scientists to translate analytical requirements into scalable pipeline solutions.
- Drive integration of data pipelines with downstream applications, dashboards, and reporting tools.
- Stay current with emerging data engineering technologies and evaluate their applicability to business needs.
- Participate in incident response and root cause analysis to quickly resolve data pipeline failures.
Requirements
- 6+ years of experience in data engineering with a proven track record in both batch and real-time processing.
- Deep knowledge of distributed computing frameworks such as Hadoop, Spark, and Flink.
- Proficiency in message queuing and stream processing systems like Kafka, Pulsar, or RabbitMQ.
- Strong scripting skills in Python, Scala, or Java, and experience with SQL on large datasets.
- Hands-on experience with cloud data services (AWS Glue, Redshift, BigQuery, Snowflake, or Azure Data Factory).
- Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines for data workloads.
- Demonstrated ability to architect and scale data pipelines to handle petabyte-scale volumes.
- Excellent communication skills, capable of translating technical concepts for non-technical stakeholders.
- A proactive problem-solver with a passion for data quality, governance, and automation.
- Leadership aptitude, with experience building, mentoring, and growing data engineering teams.
Nice to have
We offer
- Salary range: Rs 4000000 - Rs 6000000 (ie INR 40-60 LPA)