Research and public data

data.worldbank.org

Updated Apr 28, 2026

World Bank Open Data - Scraping & Data Extraction

Notes an AI agent reads before it works on data.worldbank.org: what to open, what to call, what breaks. Written by browser-harness contributors after real runs, and kept on GitHub under the MIT license.

Run it in Browser Use Cloud Source on GitHub

Give it to your agent

Any coding agent

Start the task with this line. The Markdown version is the same notes without the page around them.

Read https://browser-use.com/skills/world-bank.md, then <your task>

Browser Harness

Save the files into your workspace and turn domain skills on. The code below calls its helpers, such as http_get and js. What Browser Harness is.

d="${BH_AGENT_WORKSPACE:-$HOME/.config/browser-harness/agent-workspace}/domain-skills/world-bank"
mkdir -p "$d" && cd "$d"
curl -sLO https://raw.githubusercontent.com/browser-use/browser-harness/main/agent-workspace/domain-skills/world-bank/scraping.md
export BH_DOMAIN_SKILLS=1

https://api.worldbank.org/v2 - free REST API for global development indicators. No API key, no auth, no browser needed. All data via http_get.

Do this first

Every response is a 2-element JSON array: [metadata, data]. The metadata element is always at index 0 (pagination info); the data array is at index 1. This is the single biggest gotcha - json.loads(raw) gives you a list, not a dict.

from helpers import http_get
import json
 
raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
d = json.loads(raw)
meta = d[0]   # {"page": 1, "pages": 2, "per_page": 50, "total": 66, ...}
rows = d[1]   # list of data records

Always append ?format=json - default response is XML.

Common workflows

Single country, single indicator

from helpers import http_get
import json
 
raw = http_get("https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD?format=json")
d = json.loads(raw)
meta, rows = d[0], d[1]
 
for r in rows:
    if r["value"] is not None:   # recent years often have null values
        print(r["date"], r["value"])
# Confirmed output (2026-04-18):
# 2024 28750956130731.2
# 2023 27292170793214.4
# 2022 25604848907611.0
# ...

Most recent N values (`mrv` param)

mrv (most recent values) skips null years and returns the N most recent non-provisional points.

from helpers import http_get
import json
 
raw = http_get(
    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
    "?format=json&mrv=5"
)
d = json.loads(raw)
for r in d[1]:
    print(r["date"], r["value"])
# Confirmed output (2026-04-18):
# 2024 28750956130731.2
# 2023 27292170793214.4
# 2022 25604848907611.0
# 2021 23315080560000.0
# 2020 21060473613000.0

Multiple countries, date range

Semicolon-delimit country codes in the URL path. Use date=YYYY:YYYY for a range.

from helpers import http_get
import json
 
raw = http_get(
    "https://api.worldbank.org/v2/country/US;CN;GB/indicator/SP.POP.TOTL"
    "?format=json&date=2000:2023&per_page=100"
)
d = json.loads(raw)
meta, rows = d[0], d[1]
print(f"Total records: {meta['total']}, pages: {meta['pages']}")
 
for r in rows:
    print(r["country"]["value"], r["date"], r["value"])
# Confirmed: returns 8 records per page (50 default), date range honored exactly
# Countries: ['China', 'United States', 'United Kingdom']
# Dates: ['2000', '2001', ..., '2023']

All countries, latest value only

Use mrv=1 with per_page=1000 to get all 266 countries in a single call.

from helpers import http_get
import json
 
raw = http_get(
    "https://api.worldbank.org/v2/country/all/indicator/NY.GDP.PCAP.CD"
    "?format=json&mrv=1&per_page=1000"
)
d = json.loads(raw)
meta, rows = d[0], d[1]
print(f"Countries returned: {len(rows)}")  # 266 (includes aggregates)
 
# Filter out regional aggregates — they have no iso2Code or have aggregate ids
countries_only = [r for r in rows if len(r["country"]["id"]) == 2]
for r in sorted(countries_only, key=lambda x: -(x["value"] or 0))[:5]:
    print(r["country"]["value"], r["date"], f"${r['value']:,.0f}")
# Confirmed output (2026-04-18):
# Luxembourg 2024 $135,605
# Norway 2024 $105,056
# ...

Full pagination (fetch all pages)

from helpers import http_get
import json
 
def fetch_all_pages(base_url):
    """Fetch all pages of a World Bank API endpoint."""
    all_rows = []
    page = 1
    while True:
        url = f"{base_url}&page={page}" if "?" in base_url else f"{base_url}?page={page}"
        d = json.loads(http_get(url))
        meta, rows = d[0], d[1]
        all_rows.extend(rows)
        if page >= meta["pages"]:
            break
        page += 1
    return all_rows
 
# Example: all US GDP data (66 years, 2 pages)
rows = fetch_all_pages(
    "https://api.worldbank.org/v2/country/US/indicator/NY.GDP.MKTP.CD"
    "?format=json&per_page=50"
)
print(f"Total rows: {len(rows)}")  # 66
non_null = [(r["date"], r["value"]) for r in rows if r["value"] is not None]
print(f"Non-null: {len(non_null)}, range: {non_null[-1][0]}–{non_null[0][0]}")

Indicators list (discover available indicators)

from helpers import http_get
import json
 
raw = http_get("https://api.worldbank.org/v2/indicator?format=json&per_page=50")
d = json.loads(raw)
meta = d[0]
print(f"Total indicators: {meta['total']}, pages: {meta['pages']}")
# Confirmed: 29,511 indicators across 591 pages
 
for ind in d[1][:3]:
    print(ind["id"], "-", ind["name"])
# 1.0.HCount.1.90usd - Poverty Headcount ($1.90 a day)
# ...

Indicators by topic

from helpers import http_get
import json
 
# Topic 3 = Economy & Growth
raw = http_get("https://api.worldbank.org/v2/topic/3/indicator?format=json&per_page=50")
d = json.loads(raw)
print(f"Economy & Growth indicators: {d[0]['total']}")  # 306
 
for ind in d[1][:5]:
    print(ind["id"], "-", ind["name"])

Country metadata

from helpers import http_get
import json
 
raw = http_get("https://api.worldbank.org/v2/country/US?format=json")
d = json.loads(raw)
c = d[1][0]
print(c["name"], c["capitalCity"], c["region"]["value"], c["incomeLevel"]["value"])
# United States  Washington D.C.  North America  High income
 
# Filter countries by income level
raw = http_get("https://api.worldbank.org/v2/country?format=json&incomeLevel=LIC&per_page=300")
d = json.loads(raw)
print(f"Low-income countries: {d[0]['total']}")  # 25

Topics list

from helpers import http_get
import json
 
raw = http_get("https://api.worldbank.org/v2/topics?format=json")
d = json.loads(raw)
for t in d[1]:
    print(t["id"], t["value"])
# 1  Agriculture & Rural Development
# 2  Aid Effectiveness
# 3  Economy & Growth
# ... (21 topics total)

Parallel fetch for multiple indicators (ThreadPoolExecutor)

from helpers import http_get
from concurrent.futures import ThreadPoolExecutor
import json
 
INDICATORS = {
    "NY.GDP.MKTP.CD": "GDP (current US$)",
    "SP.POP.TOTL": "Population",
    "NY.GDP.PCAP.CD": "GDP per capita",
}
 
def fetch_indicator(ind_id):
    url = (
        f"https://api.worldbank.org/v2/country/US/indicator/{ind_id}"
        f"?format=json&mrv=5"
    )
    d = json.loads(http_get(url))
    return ind_id, d[1]
 
with ThreadPoolExecutor(max_workers=3) as ex:
    results = dict(ex.map(lambda i: fetch_indicator(i), INDICATORS))
 
for ind_id, rows in results.items():
    latest = next((r for r in rows if r["value"] is not None), None)
    if latest:
        print(f"{INDICATORS[ind_id]}: {latest['date']} = {latest['value']:,.2f}")

URL reference

Base URL

https://api.worldbank.org/v2

HTTP redirects to HTTPS (302). Always use HTTPS directly.

Endpoint patterns

Endpoint	Description
`/country/{code}/indicator/{id}`	Single country + indicator time series
`/country/{c1};{c2};{c3}/indicator/{id}`	Multi-country (semicolon-delimited)
`/country/all/indicator/{id}`	All countries
`/country/{code}`	Country metadata
`/country`	All countries metadata (filterable)
`/indicator`	All indicators list
`/indicator/{id}`	Single indicator metadata
`/topic/{id}/indicator`	Indicators for a topic
`/topics`	All topics

Query parameters

Parameter	Values	Notes
`format`	`json`, `xml` (default)	Always set `format=json`
`per_page`	integer, default 50, max 1000	Higher is faster for bulk
`page`	integer, default 1	For paginating results
`date`	`2020`, `2000:2023`	Single year or colon-separated range
`mrv`	integer	N most recent non-null values
`gapfill`	`Y`	Forward-fill nulls when used with `mrv`
`incomeLevel`	`LIC`, `MIC`, `HIC`, `LMC`, `UMC`	Filter countries by income
`region`	`EAS`, `ECS`, `LAC`, `MEA`, `NAC`, `SAS`, `SSF`	Filter countries by region

Common indicator IDs (confirmed working, 2026-04-18)

ID	Name
`NY.GDP.MKTP.CD`	GDP (current US$)
`NY.GDP.PCAP.CD`	GDP per capita (current US$)
`SP.POP.TOTL`	Population, total
`SL.UEM.TOTL.ZS`	Unemployment (% of labor force)
`FP.CPI.TOTL.ZG`	Inflation, consumer prices (%)
`NE.EXP.GNFS.ZS`	Exports of goods and services (% of GDP)
`SP.DYN.LE00.IN`	Life expectancy at birth
`SE.ADT.LITR.ZS`	Literacy rate, adult total (%)
`EG.USE.PCAP.KG.OE`	Energy use per capita (kg of oil equiv.)

Find more: https://api.worldbank.org/v2/indicator?format=json&per_page=50&page=N

Country codes (ISO2)

Standard ISO 3166-1 alpha-2 codes: US, CN, GB, DE, JP, IN, BR, etc. Special: all for all countries.

Response structure

Every endpoint returns a 2-element array:

[
  {
    "page": 1,
    "pages": 2,
    "per_page": 50,
    "total": 66,
    "sourceid": "2",
    "lastupdated": "2026-04-08"
  },
  [
    {
      "indicator": {"id": "NY.GDP.MKTP.CD", "value": "GDP (current US$)"},
      "country": {"id": "US", "value": "United States"},
      "countryiso3code": "USA",
      "date": "2024",
      "value": 28750956130731.2,
      "unit": "",
      "obs_status": "",
      "decimal": 0
    },
    ...
  ]
]

Country metadata endpoint returns same 2-element shape but with country objects (not indicator rows) at index 1.

Gotchas

Response is always a 2-element array, not a dict. d = json.loads(raw) gives a list. d[0] is pagination metadata, d[1] is the data list. Accessing d["page"] raises TypeError. This is the most common mistake.
value can be null. Recent years (e.g. 2025) and data-sparse countries frequently have null values. Always check if r["value"] is not None before using. Use mrv=N to skip nulls automatically.
Always append ?format=json. The default response format is XML. Without format=json, you get an XML string that fails json.loads.
all/indicator/{id} includes regional aggregates. The "all countries" endpoint returns 266 entries including aggregated regions like "Africa Eastern and Southern" (id: "ZH"). Filter to real countries with len(r["country"]["id"]) == 2 (ISO2 codes are always 2 chars; aggregate codes are 2-3 chars but with non-standard values).
Semicolons in URL path, not query string. Multi-country requests use country/US;CN;GB/indicator/... not ?countries=US,CN,GB. Commas do not work.
HTTP 302 redirects HTTP to HTTPS. Always use https:// directly to avoid an extra round trip.
per_page in metadata is sometimes a string, sometimes an integer. The API returns "per_page": "50" (string) for some endpoints and "per_page": 50 (int) for others. Don't compare with == without casting: int(meta["per_page"]).
Invalid country codes return an error object, not a 2-element array. A bad code gives [{"message": [{"id": "120", "key": "Invalid value", ...}]}] - a 1-element list with an error dict. Check if isinstance(d[0], dict) and "message" in d[0] before accessing d[1].
mrv + gapfill=Y forward-fills the latest value into future years. If 2024 is the latest data point and mrv=3, gapfill=Y returns 2025 (the current year) with the 2024 value copied in. Useful for "current" lookups, but the filled date is misleading.
No rate limit documented, but 3 req/s sustained is safe. The API handles bursts (parallel ThreadPoolExecutor with max_workers=3) without issue. For crawling thousands of indicators, add time.sleep(0.5) between pages.
date range returns records newest-first. Results within a date range are sorted descending by year. If you need ascending order, sort after fetching: sorted(rows, key=lambda r: r["date"]).
Indicator IDs are case-sensitive. ny.gdp.mktp.cd returns an error; use the uppercase dot-separated form NY.GDP.MKTP.CD.

scraping.md · updated Apr 28, 2026 · 1,537 words

Sites change. These notes describe what worked when they were last updated, not a guarantee they still do. If something broke, the fix is a pull request to browser-harness.