CareeRoom вчера
Lead Data Platform Engineer
от 400 000 ₽в месяц
- Москва
- Гибрид
- Полная занятость
PythonKubernetesGitLab CIFastAPIData EngineeringAirflowleadData PlatformCephбиоинформатика
Описание
Responsibilities
- Design a scalable, reliable, and performant architecture for a unified data platform
- Develop and maintain the platform's technical roadmap, synchronized with business goals and researchers' needs
- Analyze and ensure the scalability, reliability, and performance of key components
- Design and implement systems for orchestrating multi-stage data processing pipelines
- Develop key platform components: control panel, orchestrators, agents, API, SDK
- Create and develop self-service tools for data engineers and analysts
- Ensure multi-tenancy, access control, and audit logging
- Ensure data integrity, reliability, and security at all stages
- Optimize the use of computing and storage resources (capacity planning, performance benchmarking)
- Develop observability: monitoring, APM, alerting
- Formulate and implement technical standards for the platform team
- Systematize development processes, unify the stack and tools
- Maintain technical documentation for platform services and architectural solutions
Requirements
- Experience in designing horizontally scalable, fault-tolerant data-intensive systems on bare-metal/VM and/or in Kubernetes
- Understanding of architectural trade-offs and the ability to choose solutions for specific loads
- Strong proficiency in Python
- Experience developing backend services and APIs using FastAPI
- Experience with pipeline orchestrators: Airflow, Dagster, and others
- Understanding of HTC/HPC systems
- Experience with distributed storage systems: Ceph (CephFS), S3-compatible storage
- Understanding of object and file storage for data-intensive workloads
- Experience building CI/CD pipelines in GitLab
- Ability to make and justify system architectural decisions
- Experience in formulating technical standards: architectural principles, code review, engineering documentation, operational practices
Nice to have
- Experience developing in Go
- Deep understanding of Kubernetes: workloads, operators, network policies, security
- Experience with schedulers like Slurm
- Experience with Singularity/Apptainer containers
- Experience in the biotechnology/bioinformatics domain and understanding of workload specifics
We offer
- Formal employment according to the Labor Code of the Russian Federation
- Competitive salary, discussed individually based on skills
- Work in a rapidly growing, stable organization that is not afraid of crisis
- A team of professionals at the intersection of advanced IT and science
- Modern equipment for workstations
- Opportunities for professional growth and training
- 5/2 schedule, office in Moscow, hybrid possible after the probation period
Hiring process
- Respond with a CV to ••• with the subject DATA