Job Opportunities API

Job Description Job Description: • We are looking for a skilled Operations Engineer to support our Azure-based Data Engineering platform powered by Databricks. • This role blends DevOps practices, system reliability, and security auditing to ensure our data infrastructure is robust, secure, and scalable. • You’ll be instrumental in automating workflows, monitoring system health, and enforcing security best practices across our data ecosystem. Key Responsibilities: • Maintain and monitor Databricks-based data pipelines and Azure data services (e.g., Data Lake, Synapse, Key Vault). • Implement CI/CD pipelines using Azure DevOps for data engineering workflows • Conduct regular system health checks and performance tuning of clusters and jobs. • Automate infrastructure provisioning using tools like Terraform or ARM templates. • Use monitoring tools (e.g., Azure Monitor, Log Analytics, Prometheus) to track system metrics and alert on anomalies. • Perform security audits using tools like Databricks Security Analysis Tool (SAT) to ensure compliance with best practices. • Manage secrets and credentials securely using Azure Key Vault and Databricks secret scopes. • Collaborate with security teams to enforce network security, access control, and data encryption policies. • Maintain operational runbooks and incident response documentation. Qualifications: Bachelor’s degree in computer science, Engineering, or related field. • • 5+ years of experience in DevOps, cloud operations, or data platform support. • • Proficiency in Azure services, Databricks, and scripting languages (Python, Bash). • • Experience with CI/CD tools (Azure DevOps, GitHub Actions) and infrastructure-as-code. • • Familiarity with security frameworks and auditing tools in cloud environments. • • Strong troubleshooting skills and a proactive approach to system reliability. Preferred Skills: • • Experience with Databricks SAT for security posture assessments. • • Knowledge of Unity Catalog, RBAC, and data governance in Azure Databricks. • • Exposure to MLOps and operationalizing machine learning pipelines. • • Understanding of cost optimization strategies in Azure.