# Keeping data current

Four jobs you will end up writing: a daily country pull, an incremental sync, closure detection, and a bulk export — with the record cost of each.

**What this covers:** A daily pull for one country; An incremental sync; Detecting closures cheaply; A bulk export, done properly.

**Assumed knowledge:** [Pagination](./api-pagination.md) and [the delta feed](./endpoints-changes.md).

**Canonical HTML:** https://jobopportunitiesapi.org/docs/recipes/sync  
**Machine-readable index:** https://jobopportunitiesapi.org/docs/ai/index.md  
**Last verified:** 2026-08-22  
**Superseded by:** the live API at https://api.jobopportunitiesapi.org and its spec at https://jobopportunitiesapi.org/openapi.json — where this file and the API disagree, the API is right.

---

> **Choose by record cost, not by convenience** — Re-reading a listing endpoint charges a record for every row you look at, whether or not it changed. The delta feed charges only for rows that moved. On any slice of size that is the difference between a plan you can afford and one you cannot.

---

<a id="recipe-daily-country-pull"></a>

## 1. A daily pull for one country

The simplest useful job: fetch everything in a market once a day, resumably, and write it somewhere.

### 1.1 In depth

Use this to seed a store, or when your market is small enough that a full re-read is cheaper than the machinery of an incremental sync. Once the slice is more than a few tens of thousands of rows, switch to [the incremental sync](#recipe-incremental-sync) — the record cost of a daily full read grows linearly and the delta feed's does not.

Two details make it resumable rather than merely repeatable. The cursor is written to disk after each page is processed, so a crash restarts from the last complete page rather than from the beginning. And rows are keyed on `id`, so re-processing a page is harmless.

### 1.2 Exact contract

daily_pull.py — one country, resumable, standard library only

```python
"""Fetch every live row for one country and write NDJSON.

    export JOA_KEY=...
    python3 daily_pull.py IE

Resumable: the cursor is persisted after each page, so an interrupted run
continues rather than restarting. Rows are keyed on id, so re-processing a
page costs records but corrupts nothing.
"""
import json, os, pathlib, sys, urllib.parse, urllib.request

API     = "https://api.jobopportunitiesapi.org"
KEY     = os.environ["JOA_KEY"]
country = sys.argv[1] if len(sys.argv) > 1 else "IE"
state   = pathlib.Path(f".joa-{country}.cursor")
out     = pathlib.Path(f"{country}.ndjson")

def get(params):
    req = urllib.request.Request(
        f"{API}/v1/jobs?" + urllib.parse.urlencode(params),
        headers={"Authorization": f"Bearer {KEY}"},
    )
    with urllib.request.urlopen(req, timeout=90) as r:
        return json.load(r)

cursor = state.read_text().strip() if state.exists() else ""
rows = 0
with out.open("a") as f:
    while True:
        params = {"country": country, "limit": 200}
        if cursor:
            params["cursor"] = cursor
        page = get(params)
        for job in page["data"]:
            f.write(json.dumps(job, separators=(",", ":")) + "\n")
            rows += 1
        f.flush()
        if not page.get("has_more") or not page.get("next_cursor"):
            state.unlink(missing_ok=True)   # finished: forget the cursor
            break
        cursor = page["next_cursor"]
        state.write_text(cursor)           # AFTER writing the page
print(f"{rows} rows -> {out}")
```

**Record cost:** one per row, every run. A market with 60,000 live rows costs 60,000 records a day, which is the whole Growth allowance in a single run — this is precisely the calculation that should push you to the delta feed.

**See also**

- [An incremental sync](./recipes-sync.md#recipe-incremental-sync)
- [What counts as a record](./account-record-meter.md#meter-what-counts)
- [Resuming an interrupted pull](./api-pagination.md#resumable-pulls)

<a id="recipe-incremental-sync"></a>

## 2. An incremental sync

Seed once, then poll /v1/changes. You pay for rows that moved, not for rows you look at, and you cannot miss a row to a shared timestamp.

### 2.1 In depth

This is the shape almost every serious integration ends up with. Seed the store once — with a full pull, or with `/v1/export` if your plan has it — then poll the delta feed every fifteen minutes or so and apply the changes.

It is not only cheaper, it is more correct. A nightly diff cannot see a row that was created and closed between two runs, and it cannot distinguish a row that closed from one that fell off the end of your pagination. The feed reports both, explicitly, in order.

> **Your default branch must retract, not ignore** — `withdrawn` was a new change kind once, and clients whose switch statement ignored unknown values went on serving rows that had been retracted — including rows removed at an employer's request. Treat an unrecognised kind as “stop serving this row and re-fetch it by id”.

### 2.2 Exact contract

The full loop, with the state handling and the change-kind switch, is on [the delta feed page](./endpoints-changes.md#changes-loop). The summary:

1. Seed with `/v1/jobs` or `/v1/export`; record the instant you started.
2. `GET /v1/changes?since=<that instant>`; then `since=<next_since>` forever after.
3. `created` / `updated` → upsert. `delisted` / `withdrawn` / **anything else** → retract.
4. Persist `next_since` **after** applying the batch.
5. Loop while `count == limit`; a short page means you are current.

|  | Daily full pull | Incremental sync |
| --- | --- | --- |
| Records charged | every row, every day | only rows that moved |
| Sees short-lived rows | no | yes |
| Distinguishes closed from missing | no | yes |
| Sees employer opt-outs | only as an absence | explicitly, as `withdrawn` |
| Plan needed | any | `delta_feed` |
| Complexity | a loop | a loop and a cursor in your database |

**See also**

- [The sync loop, written correctly](./endpoints-changes.md#changes-loop)
- [The four change kinds — and the one that breaks integrations](./endpoints-changes.md#change-kinds)
- [What counts as a record](./account-record-meter.md#meter-what-counts)

<a id="recipe-closures"></a>

## 3. Detecting closures cheaply

/v1/jobs/expired returns ids, dates and reasons only — about ninety per cent smaller than the same page of full rows.

### 3.1 In depth

If all you need is to mark your own rows stale, you do not need the rows back. This endpoint gives you the id, when it closed and why, which is enough to update your store — and it is the smallest possible response for that job.

Note that it charges the same one record per id as a full row would. The saving is in bandwidth, latency and parsing, not in the meter. If you are already running the delta feed you do not need this endpoint at all — closures arrive there as `delisted`.

### 3.2 Exact contract

A cron-friendly closure sweep

```bash
#!/bin/bash
# Mark rows stale in our own store. Idempotent; safe to run hourly.
set -euo pipefail
A=https://api.jobopportunitiesapi.org
STATE=${STATE:-.joa-expired-cursor}
since=$(cat "$STATE" 2>/dev/null || echo '2026-08-01T00:00:00Z')

while : ; do
  page=$(curl -s --fail-with-body -H "Authorization: Bearer $JOA_KEY" \
         --get "$A/v1/jobs/expired" \
         --data-urlencode "since=$since" --data-urlencode 'limit=1000')
  count=$(jq -r '.count' <<<"$page")
  jq -r '.data[] | "\(.id)\t\(.closed_at)\t\(.closed_reason)"' <<<"$page" \
    | while IFS=$'\t' read -r id at reason; do
        echo "UPDATE jobs SET closed_at='$at', closed_reason='$reason' WHERE id='$id';"
      done
  # Persist AFTER emitting, so a crash re-reads rather than skips.
  jq -r '.next_since' <<<"$page" > "$STATE"
  [ "$count" -eq 1000 ] || break   # short page: current
done
```

An empty page echoes your cursor back rather than returning null, so this loop keeps working once it is current — no special case needed.

**See also**

- [GET /v1/jobs/expired](./endpoints-jobs.md#endpoint-jobs-expired)
- [Closed roles](./ledger-data-model.md#closed-roles)
- [The sync loop, written correctly](./endpoints-changes.md#changes-loop)

<a id="recipe-bulk-export"></a>

## 4. A bulk export, done properly

Stream NDJSON, resume with after, and check the last line — because the error can arrive inside a 200.

### 4.1 In depth

Reach for the export rather than a paged listing whenever you want a whole slice: it streams, it resumes to the exact row, and it does not re-run an ordering query per page. It needs a plan with `bulk_export`.

> **The last line is not necessarily a row** — If the record allowance runs out mid-stream, the response has already returned 200 and cannot become a 402 — so the error arrives as the final line, with the `after` cursor to resume from. A consumer that assumes a clean end of stream silently truncates its own dataset.

### 4.2 Exact contract

Export, resume, verify

```bash
#!/bin/bash
set -euo pipefail
A=https://api.jobopportunitiesapi.org
out=${1:-corpus.ndjson}
touch "$out"

for attempt in 1 2 3 4 5; do
  after=$(tail -1 "$out" 2>/dev/null | jq -r '.id // empty' 2>/dev/null || true)
  if curl -sN --fail-with-body -H "Authorization: Bearer $JOA_KEY" \
       --get "$A/v1/export" \
       --data-urlencode 'country=IE' \
       ${after:+--data-urlencode "after=$after"} >> "$out"; then
    break
  fi
  echo "attempt $attempt interrupted after ${after:-start}" >&2
  sleep $((attempt * 5))
done

# The stream can end on an error object inside a 200. Silence here is a
# truncated dataset that looks complete.
if tail -1 "$out" | jq -e 'has("error")' >/dev/null 2>&1; then
  echo "INCOMPLETE: $(tail -1 "$out" | jq -r '.error')" >&2
  echo "resume with after=$(tail -1 "$out" | jq -r '.after')" >&2
  exit 1
fi
echo "$(wc -l < "$out") rows"
```

Rows may repeat across a resume if the connection dropped mid-line — upsert on `id` rather than appending blindly into a store that cares.

**See also**

- [GET /v1/export](./endpoints-export.md#endpoint-export)
- [The failure mode that catches people](./endpoints-export.md#export-quota)
- [Queries that can time out, and how to make them fast](./api-filtering.md#expensive-queries)

---

## Where to go next

This file is part of **Recipes**. Others in the same group:

- [Client code](./recipes-languages.md) — A working client in curl, Python, JavaScript and Go. No SDK, no dependencies, and every request re-sent against the live API before this page ships.
- [Things to build](./recipes-build.md) — Three complete builds — a job board, an employer watcher, a CRM enrichment job — and an honest list of which integrations exist today.

Always useful:

- [index.md](./index.md) — the map of every file here
- [BUILD-A-SITE.md](./BUILD-A-SITE.md) — the paste-whole brief for building against this API
- [quickstart.md](./quickstart.md) — zero to a first authenticated response
- [api-errors.md](./api-errors.md) — every status code and whether to retry it