Job Opportunities API

This role is for one of the Weekday's clients Min Experience: 3+ years Location: Bengaluru JobType: full-time We are looking for a highly skilled Big Data Engineer with 3+ years of experience in building scalable data pipelines and distributed systems. The ideal candidate will have strong expertise in Apache Spark (Scala), experience working across on-premise and AWS environments, and a solid understanding of large-scale data processing in AdTech ecosystems.  This role involves working on high-volume datasets (billions of records), optimizing distributed jobs, and contributing to the design of robust data infrastructure powering analytics and identity-driven use cases.  Requirements Key Responsibilities: Design, develop, and optimize large-scale batch data pipelines using Apache Spark (Scala)  Process and transform high-volume datasets (TBs of data) in distributed environments  Build and maintain data pipelines across hybrid infrastructure (On-Prem + AWS)  Work with object storage systems such as S3 and MinIO for efficient data access and storage  Develop reusable and scalable data processing frameworks  Optimize Spark jobs for performance (memory tuning, partitioning, shuffling, etc.)  Manage and orchestrate workloads using HashiCorp Nomad  Integrate data pipelines with PostgreSQL and other downstream systems  Ensure data quality, consistency, and reliability across pipelines  Troubleshoot production issues and perform root cause analysis  Contribute to system design discussions, especially for high-scale AdTech use cases (identity resolution, user profiling, etc.)    Required Skills:  Strong programming experience in Scala  Good working knowledge of Python (for auxiliary tasks, scripting, or ML integration)  Deep expertise in Apache Spark (Core and SQL)  Strong understanding of distributed data processing and large-scale systems  Experience working with AWS (S3, EMR or equivalent ecosystem) and on-prem clusters  Hands-on experience with object storage systems (S3 / MinIO)  Experience with HashiCorp Nomad or similar orchestration tools  Solid understanding of data modeling and ETL pipeline design  Experience working with PostgreSQL or similar relational databases  Strong debugging and performance tuning skills for Spark jobs  Familiarity with Unix/Linux environments and shell scripting    Good to Have: Experience in AdTech, Identity Graph, or User Profiling systems  Exposure to machine learning pipelines or feature engineering workflows  Experience with data lake architectures  Understanding of cost optimization and resource management in AWS    Tech Stack Summary: Languages: Scala (Primary), Python (Secondary)  Processing: Apache Spark (Core and SQL)  Infrastructure: AWS and On-Prem  Storage: S3, MinIO  Orchestration: HashiCorp Nomad  Database: PostgreSQL Must-have skills Spark, SQL, Scala Good-to-have skills Python, AWS, Big Data