Job Opportunities API

ABOUT US Dragonfly is building the world's first Automated Solutions Architect. We help businesses navigate the complex landscape of modern tools (SaaS, AI, Infrastructure) by using an AI-powered platform that understands their unique context and recommends the optimal tech stack. Our platform is powered by a proprietary knowledge graph of 230K+ products, 4M+ companies, and the relationships between them — built from 100+ external data sources, LLM-driven research, and human curation. The architecture is designed but much of it still runs as manually-triggered pipelines. We need someone to turn it into a production data platform. THE ROLE We don't have a separate data team with its own stack. The data platform lives in the same monorepo as the product, follows the same engineering practices, and ships through the same CI pipeline. You'll be working alongside product engineers in Python and SQL — the same codebase, the same review process, the same standards. The best way to describe this role is a software engineer who builds data systems. You'll be as comfortable making a product change as you are writing a pipeline. We've designed the data architecture: a medallion-layer pipeline in BigQuery via Dataform, a signal-based quality scoring model, an append-only ontology, and an AI-driven research pipeline. We need someone to own it — productionise what exists, extend it as the product evolves, and keep it running day-to-day. We operate a high-autonomy, high-trust environment. You'll be given a problem and the space to solve it — not a task list. AI is your first port of call for everything: understanding the codebase, exploring data, drafting implementations, debugging. Tickets describe what needs doing, not how. We expect you to think beyond the immediate task — consider knock-on effects, integration points, and how your work fits into the broader product and business strategy. Curiosity matters: you should want to understand how everything connects, not just the bit you're working on. THE DATA PIPELINE Dragonfly's core asset is a proprietary knowledge graph — every SaaS product, AI tool, and infrastructure service, what it does, who makes it, how it compares, and how confident we are in that data. Building this knowledge graph is a multi-stage pipeline: raw data from external sources is normalised, scored for quality, matched and enriched by AI agents, curated by humans, and served to downstream systems (search, recommendations, product catalogue). The architecture follows a medallion pattern (Bronze → Silver → Gold) in BigQuery via Dataform, with an append-only ontology at the centre. What's missing is the production engineering: scheduling, monitoring, incremental processing, source onboarding automation, and the platform layer that makes it all self-service for product engineers. WHAT YOU'LL DO 1. PRODUCTION DATA PIPELINES The pipeline exists as ~200 Dataform files across multiple datasets. Today, builds are triggered manually. Incremental processing has been validated in design but not hardened for production. You'll own making this run reliably. - Scheduled builds, retry logic, and failure alerting - Harden incremental processing — source refreshes need to flow through without full rebuilds - Monitoring: volume checks per layer, freshness SLAs, broken source contract detection - Maintain and evolve the data marts that feed search, recommendations, and the product catalogue - Manage versioned dataset transitions with rollback capability 2. SOURCE ONBOARDING & INGESTION We ingest from several sources. Adding a new source today means writing a silver view by hand. You'll build a repeatable process so that onboarding a new vendor is a config change, not a project. - Evaluate and implement managed ingestion tooling (Airbyte, Meltano, or similar) for batch sources - New sources land in Bronze and flow through the existing Silver standardisation layer automatically -