harmony Data API developer docs Get an API key

Keeping a copy in sync

Load the full dataset once. After that, each request returns only what was added, changed or removed since the last one, so you never need to delete and reload everything.

The three steps

  1. Load everything once. Download GET /v1/exports/conferences (gzipped JSON Lines, one record per line) and insert every record into your table, keyed by id. Save the response's X-Sync-Token header.
  2. Ask for what changed. As often as you like (daily, hourly), call GET /v1/conferences/changes?since=<saved token>.
  3. Apply each entry, then save the new token. An entry with "deleted": true means remove that id; anything else is the full current record, so replace your row (insert it if it's new). Then save next_sync_token. If has_more is true, call again straight away with it.

The same works for journals. Keep one token per dataset.

Example (Python)

import gzip, json, os, requests

API = "https://api.orbytelgroup.com/v1"
HEADERS = {"Authorization": "Bearer " + os.environ["ORBYTEL_KEY"]}

def initial_load(db):
    r = requests.get(f"{API}/exports/conferences", headers=HEADERS, timeout=300)
    r.raise_for_status()
    for line in gzip.decompress(r.content).splitlines():
        db.upsert(json.loads(line))              # keyed by record["id"]
    db.save_token(r.headers["X-Sync-Token"])

def sync(db):
    token = db.load_token()
    while True:
        r = requests.get(f"{API}/conferences/changes",
                         params={"since": token}, headers=HEADERS, timeout=60)
        if r.status_code == 410:                 # license changed: start again
            return initial_load(db)
        r.raise_for_status()
        page = r.json()
        for record in page["data"]:
            if record.get("deleted"):
                db.delete(record["id"])          # fine if you never had it
            else:
                db.upsert(record)
        token = page["next_sync_token"]
        db.save_token(token)                     # after applying the page
        if not page["has_more"]:
            break

Good to know

Sizes and limits