Job Opportunities API

Senior Data Engineer Location: Herndon, VA (Remote Work) Must have an Public Trust Clearance KEY RESPONSIBILITIES •      Provide authoritative expertise on data engineering methods and best practices, including code first development approaches and modern pipeline design patterns. •      Design, implement, and maintain the data architecture that supports products and end users, with all assets managed under source control. •      Design, implement, and maintain ELT and ETL pipelines for efficient processing of source data in Azure Synapse and Azure Machine Learning, using both SDK V1 and SDK V2. •      Migrate source data identified by SBA OIG into Azure Data Lake Storage. •      Normalize entity attributes such as addresses, phone numbers, and other common fields. •      Review, maintain, and improve existing architecture and pipelines, including periodic audits addressing bottlenecks, deprecated dependencies, and architecture drift. •      Establish quality controls across all pipelines and introduce error handling, logging mechanisms, and validation checks. •      Incorporate source control across all pipelines and analytics codebases so code can evolve iteratively without destabilizing the architecture. •      Optimize ingestion, processing, and storage across a wide variety of datasets and data types, including modern columnar formats such as Parquet. •      Develop self service capabilities that let SBA OIG analysts query and export data for investigations and audits. •      Author robust standard operating procedures governing the authoring, development, validation, publishing, execution, and monitoring of all data pipelines and assets in the Azure environment. •      Produce detailed documentation of the data architecture, including data dictionaries, entity relationship diagrams, and pipeline process maps. •      Maintain and expand the environment with additional datasets and services on request, following a defined intake and testing process before production deployment. •      Stay current with emerging AI tooling relevant to data engineering and contribute to exploratory work evaluating automation and language model assisted capabilities.