Browser Use
All site skills

Research and public data

sec.gov

SEC EDGAR - Scraping & Data Extraction

Notes an AI agent reads before it works on sec.gov: what to open, what to call, what breaks. Written by browser-harness contributors after real runs, and kept on GitHub under the MIT license.

Run it in Browser Use CloudSource on GitHub

Give it to your agent

Any coding agent

Start the task with this line. The Markdown version is the same notes without the page around them.

Read https://browser-use.com/skills/sec-edgar.md, then <your task>

Browser Harness

Save the files into your workspace and turn domain skills on. The code below calls its helpers, such as http_get and js. What Browser Harness is.

d="${BH_AGENT_WORKSPACE:-$HOME/.config/browser-harness/agent-workspace}/domain-skills/sec-edgar"
mkdir -p "$d" && cd "$d"
curl -sLO https://raw.githubusercontent.com/browser-use/browser-harness/main/agent-workspace/domain-skills/sec-edgar/scraping.md
export BH_DOMAIN_SKILLS=1

https://www.sec.gov / https://data.sec.gov / https://efts.sec.gov - all public data, no auth required. Every workflow here is pure http_get - no browser needed.

Do this first

SEC.gov requires a custom User-Agent on www.sec.gov and data.sec.gov. Always pass headers=UA or you get 403.

import json
UA = {"User-Agent": "browser-harness research@example.com"}
# Format required: "CompanyName contact@email.com"
# "Mozilla/5.0" (http_get default) works on efts.sec.gov and data.sec.gov
# but FAILS on www.sec.gov (company_tickers.json, Archives/, etc.)

Start with company_tickers.json to resolve any ticker → CIK in one call, then branch to whichever endpoint you need.

import json
UA = {"User-Agent": "browser-harness research@example.com"}
tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
# 10,391 public companies, ~50KB, always fresh
# Entry format: {"cik_str": 320193, "ticker": "AAPL", "title": "Apple Inc."}
 
# Look up by ticker (exact, case-sensitive in the data)
aapl = next(v for v in tickers.values() if v['ticker'] == 'AAPL')
# {'cik_str': 320193, 'ticker': 'AAPL', 'title': 'Apple Inc.'}
 
# CIK is an int here; pad to 10 digits for API URLs
cik = str(aapl['cik_str']).zfill(10)  # "0000320193"

Common workflows

Ticker / name → CIK lookup

import json
UA = {"User-Agent": "browser-harness research@example.com"}
tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
 
# By ticker
tsla = next((v for v in tickers.values() if v['ticker'] == 'TSLA'), None)
# {'cik_str': 1318605, 'ticker': 'TSLA', 'title': 'Tesla, Inc.'}
 
# By partial name match
apples = [v for v in tickers.values() if 'APPLE' in v['title'].upper()]
# [{'cik_str': 320193, 'ticker': 'AAPL', 'title': 'Apple Inc.'}, ...]

Company submissions (metadata + recent filings list)

import json
UA = {"User-Agent": "browser-harness research@example.com"}
cik = "0000320193"  # Apple - always zero-pad to 10 digits
data = json.loads(http_get(f"https://data.sec.gov/submissions/CIK{cik}.json", headers=UA))
 
print(data['name'])             # "Apple Inc."
print(data['cik'])              # "0000320193"
print(data['sic'])              # "3571"
print(data['sicDescription'])   # "Electronic Computers"
print(data['tickers'])          # ["AAPL"]
print(data['exchanges'])        # ["Nasdaq"]
 
# Most recent ~1,000 filings are in data['filings']['recent']
recent = data['filings']['recent']
# Fields per filing (parallel arrays, same index):
# accessionNumber, filingDate, reportDate, form, primaryDocument,
# primaryDocDescription, size, isXBRL, items, fileNumber
 
# Filter for 10-K and 10-Q only
filings_10k = [
    (f, d, a, doc)
    for f, d, a, doc in zip(
        recent['form'], recent['filingDate'],
        recent['accessionNumber'], recent['primaryDocument']
    )
    if f in ('10-K', '10-Q')
]
# Result: [('10-Q', '2026-01-30', '0000320193-26-000006', 'aapl-20251227.htm'), ...]

Build direct filing document URL

# Given accessionNumber and primaryDocument from submissions JSON:
accn = "0000320193-25-000079"
doc  = "aapl-20250927.htm"
cik  = "320193"  # int part only (no leading zeros) for Archives path
 
accn_nodash = accn.replace("-", "")
url = f"https://www.sec.gov/Archives/edgar/data/{cik}/{accn_nodash}/{doc}"
# https://www.sec.gov/Archives/edgar/data/320193/000032019325000079/aapl-20250927.htm
 
# Full 10-K is 1.5MB of XBRL-tagged HTML — use http_get for text extraction
content = http_get(url, headers=UA)  # UA required on www.sec.gov

XBRL financial data - single company, one concept over time

import json
UA = {"User-Agent": "browser-harness research@example.com"}
cik_padded = "0000320193"
 
# companyconcept: one metric, all reported values (quarterly + annual)
data = json.loads(http_get(
    f"https://data.sec.gov/api/xbrl/companyconcept/CIK{cik_padded}/us-gaap/Assets.json",
    headers=UA
))
# data keys: cik, taxonomy, tag, label, description, entityName, units
# data['units']['USD'] -> list of {end, val, accn, fy, fp, form, filed}
 
entries = data['units']['USD']
 
# Deduplicate: same period re-reported across multiple filings — keep latest
def annual_series(entries):
    seen = {}
    for e in entries:
        if e.get('form') == '10-K' and e.get('fp') == 'FY':
            end = e['end']
            if end not in seen or e['filed'] > seen[end]['filed']:
                seen[end] = e
    return [seen[k] for k in sorted(seen)]
 
assets = annual_series(entries)
for e in assets[-5:]:
    print(f"{e['end']}  ${e['val']/1e9:.1f}B")
# 2021-09-25  $351.0B
# 2022-09-24  $352.8B
# 2023-09-30  $352.6B
# 2024-09-28  $365.0B
# 2025-09-27  $359.2B

XBRL financial data - all US-GAAP metrics for a company

import json
UA = {"User-Agent": "browser-harness research@example.com"}
 
# companyfacts: all reported XBRL concepts in one ~5MB call
data = json.loads(http_get(
    "https://data.sec.gov/api/xbrl/companyfacts/CIK0000320193.json",
    headers=UA
))
# data['entityName'] = "Apple Inc."
# data['facts'] = {'us-gaap': {...503 concepts...}, 'dei': {...}}
 
usgaap = data['facts']['us-gaap']
print(len(usgaap))   # 503 concepts for Apple
 
# Common concept names (companies vary — check what's available):
# Revenue:     RevenueFromContractWithCustomerExcludingAssessedTax  (post-2018 standard)
#              SalesRevenueNet                                       (older filings)
#              Revenues                                              (some companies still use)
# Net income:  NetIncomeLoss
# Assets:      Assets
# Cash:        CashAndCashEquivalentsAtCarryingValue
# EPS:         EarningsPerShareBasic, EarningsPerShareDiluted
 
# Find all revenue-related concepts this company reported:
revenue_keys = [k for k in usgaap if 'Revenue' in k]
 
# Extract annual revenue — handle company-specific concept name
for concept in ['RevenueFromContractWithCustomerExcludingAssessedTax', 'SalesRevenueNet', 'Revenues']:
    if concept in usgaap:
        entries = usgaap[concept]['units'].get('USD', [])
        annual = {}
        for e in entries:
            if e.get('form') == '10-K' and e.get('fp') == 'FY':
                end = e['end']
                if end not in annual or e['filed'] > annual[end]['filed']:
                    annual[end] = e
        if annual:
            print(f"Using: {concept}")
            for end in sorted(annual)[-3:]:
                print(f"  {end}  ${annual[end]['val']/1e9:.1f}B")
            break
# Apple output:
# Using: RevenueFromContractWithCustomerExcludingAssessedTax
#   2023-09-30  $383.3B
#   2024-09-28  $391.0B
#   2025-09-27  $416.2B

Cross-company financial comparison (XBRL frames)

import json
UA = {"User-Agent": "browser-harness research@example.com"}
 
# frames: one concept, one period, all companies that reported it
# Period formats:
#   CY2024           = calendar year 2024 (annual)
#   CY2024Q4I        = Q4 2024 instantaneous (balance sheet items)
#   CY2024Q4         = Q4 2024 duration (income statement items)
 
# Top companies by annual revenue (2024)
data = json.loads(http_get(
    "https://data.sec.gov/api/xbrl/frames/us-gaap/RevenueFromContractWithCustomerExcludingAssessedTax/USD/CY2024.json",
    headers=UA
))
companies = sorted(data['data'], key=lambda x: x['val'], reverse=True)
# data['data'] entries: {accn, cik, entityName, loc, start, end, val}
for c in companies[:5]:
    print(f"{c['entityName']:<40}  ${c['val']/1e9:.0f}B")
# Walmart Inc.                              $675B
# AMAZON.COM, INC.                          $638B
# Apple Inc.                                $391B
# McKESSON CORPORATION                      $359B
# Alphabet Inc.                             $350B
 
# Total assets snapshot end of 2024 (balance sheet = instantaneous)
data2 = json.loads(http_get(
    "https://data.sec.gov/api/xbrl/frames/us-gaap/Assets/USD/CY2024Q4I.json",
    headers=UA
))
# 6,229 companies for this frame

Full-text search across all filings

import json
UA = {"User-Agent": "browser-harness research@example.com"}
 
# Search for any phrase across filing documents
# Params: q (quoted phrase), forms (comma-separated), dateRange=custom,
#         startdt, enddt, size (max 100), from (offset for pagination)
url = (
    "https://efts.sec.gov/LATEST/search-index"
    "?q=%22climate+risk%22"
    "&forms=10-K"
    "&dateRange=custom&startdt=2024-01-01"
    "&size=10&from=0"
)
data = json.loads(http_get(url, headers=UA))
# Note: default http_get UA (Mozilla/5.0) works fine on efts.sec.gov
 
print(data['hits']['total']['value'])   # e.g. 1438 matching documents
hits = data['hits']['hits']             # up to 100 per call
 
for h in hits:
    src = h['_source']
    # Key fields: display_names, ciks, form, file_date, adsh (accession), period_ending
    name = src['display_names'][0] if src.get('display_names') else '?'
    cik  = src['ciks'][0] if src.get('ciks') else '?'
    print(f"{name}  form={src['form']}  filed={src['file_date']}  accn={src['adsh']}")
 
# Pagination: max 100 per page, use from= to walk through results
# Page 2: from=100, Page 3: from=200, etc.
for page in range(0, 300, 100):
    page_url = url + f"&from={page}"
    page_data = json.loads(http_get(page_url, headers=UA))
    if not page_data['hits']['hits']:
        break
    # process...
 
# Aggregations — group hits by entity, SIC, state
aggs = data['aggregations']
top_entities = aggs['entity_filter']['buckets']   # [{key: "Name (CIK...)", doc_count: N}, ...]
top_sics     = aggs['sic_filter']['buckets']
top_states   = aggs['biz_states_filter']['buckets']

Find a company's CIK by name search (via search aggregations)

import json, re
UA = {"User-Agent": "browser-harness research@example.com"}
 
# Best method: company_tickers.json (fastest, all tickers)
tickers = json.loads(http_get("https://www.sec.gov/files/company_tickers.json", headers=UA))
msft = next(v for v in tickers.values() if v['ticker'] == 'MSFT')
# CIK = msft['cik_str']  → 789019
 
# Alternative: full-text search aggregations (finds CIK from company name)
data = json.loads(http_get(
    "https://efts.sec.gov/LATEST/search-index?q=%22microsoft+corporation%22&forms=10-K",
    headers=UA
))
buckets = data['aggregations']['entity_filter']['buckets']
# [{'key': 'MICROSOFT CORP  (MSFT)  (CIK 0000789019)', 'doc_count': 11}, ...]
for b in buckets[:3]:
    m = re.search(r'\(CIK (\d+)\)', b['key'])
    if m:
        print(f"{b['key'][:50]}  →  CIK {m.group(1)}")

Parallel fetching (multiple companies)

import json
from concurrent.futures import ThreadPoolExecutor
 
UA = {"User-Agent": "browser-harness research@example.com"}
 
def get_company_meta(ticker_cik):
    ticker, cik = ticker_cik
    subs = json.loads(http_get(f"https://data.sec.gov/submissions/CIK{cik}.json", headers=UA))
    return {"ticker": ticker, "name": subs['name'], "sic": subs['sic']}
 
companies = [("AAPL", "0000320193"), ("TSLA", "0001318605"), ("MSFT", "0000789019")]
with ThreadPoolExecutor(max_workers=3) as ex:
    results = list(ex.map(get_company_meta, companies))
# Confirmed: 3 requests complete in ~0.28s
# SEC rate limit: 10 req/sec — stay at max_workers ≤ 8 to be safe

API reference

EndpointWhat it returnsUA required
www.sec.gov/files/company_tickers.jsonAll 10,391 tickers → CIK mappingYES
data.sec.gov/submissions/CIK{10-digit}.jsonCompany meta + ~1000 recent filingsYES
data.sec.gov/api/xbrl/companyfacts/CIK{10-digit}.jsonAll XBRL facts (~5MB)YES
data.sec.gov/api/xbrl/companyconcept/CIK{10-digit}/{taxonomy}/{concept}.jsonOne concept, all valuesYES
data.sec.gov/api/xbrl/frames/{taxonomy}/{concept}/{unit}/{period}.jsonAll companies for one periodYES
efts.sec.gov/LATEST/search-index?q=...Full-text search across filingsNO (Mozilla/5.0 works)
www.sec.gov/Archives/edgar/data/{cik}/{accn-nodash}/{doc}Actual filing documentYES

data.sec.gov accepts Mozilla/5.0 (the http_get default). www.sec.gov (Archives, company_tickers) requires the "CompanyName email@example.com" format.

Rate limits

SEC documents a 10 requests/second limit. In practice:

  • 12 rapid sequential calls to data.sec.gov completed in 2.4s (5 req/s) with no throttling.
  • 3 parallel calls completed in 0.28s without issue.
  • Stay at max_workers ≤ 8 for ThreadPoolExecutor to respect the 10 req/s ceiling.
  • No per-day or per-hour cap documented; the 10/s limit is the only stated constraint.

Gotchas

  • www.sec.gov returns 403 with Mozilla/5.0 UA - The http_get default ("Mozilla/5.0") works on data.sec.gov and efts.sec.gov but is blocked on www.sec.gov. Always pass headers=UA where UA includes your company name and email. Confirmed: "python-requests/2.28" → 403.

  • data.sec.gov is more permissive - Mozilla/5.0 works on data.sec.gov (submissions, xbrl). But always use the proper UA anyway - SEC's policy page explicitly requires it and they can add stricter checks at any time.

  • XBRL contains duplicate entries per period - The same fiscal year end date appears multiple times when a company restates or re-files. Each entry has a filed date and accn (accession). To get the canonical value, deduplicate by end keeping the entry with the latest filed date.

  • Revenue concept name varies by company and era - There is no single canonical "revenue" concept. Apple uses RevenueFromContractWithCustomerExcludingAssessedTax. Microsoft uses the same for recent years, but older filings used SalesRevenueNet. Always check which concepts are actually present: [k for k in usgaap if 'Revenue' in k].

  • fp field for annual filings is 'FY', but quarterly values also appear in 10-K - A 10-K re-reports each quarter (fp=Q1, Q2, Q3) plus the full year (fp=FY). Filter on both form == '10-K' AND fp == 'FY' to get only annual totals.

  • companyfacts is ~5MB per company - For a single metric, use companyconcept instead (much smaller). Only use companyfacts when you need multiple concepts from the same company.

  • submissions recent filings cap at ~1,000 - Very old filings don't appear. If you need historical data before that window, use the filings.files array in submissions JSON to find older filing JSON pages (data.sec.gov/submissions/CIK{cik}-submissions-001.json, etc.).

  • adsh in search results is the accession number - The search index returns adsh (no dashes). To build the document URL, insert dashes: adsh[:10] + '-' + adsh[10:12] + '-' + adsh[12:], or use the accessionNumber field from submissions JSON (which already has dashes).

  • size param is capped at 100 - Requesting size=200 silently returns 100 hits. Walk results with from=0, from=100, etc. Maximum reachable index is 10,000 (Elasticsearch default).

  • Search total 'gte' relation means >10,000 hits - When total['relation'] == 'gte', there are more than 10,000 results (only first 10,000 accessible). Narrow with dateRange or forms filters.

  • company_tickers.json covers only exchange-listed companies - ~10,391 entries. Many SEC filers (private companies, bond issuers, FHLBs) have CIKs but no ticker. Find them via the full-text search aggregations or submissions lookup if you have the CIK.

  • Filing document is XBRL-tagged HTML, 1–2MB - Retrieving the actual 10-K HTML works but is large. For financial data extraction, always prefer the XBRL API endpoints over parsing the document.

  • CIK format gotcha - company_tickers.json returns cik_str as an int (320193). The submissions and xbrl APIs require a 10-digit zero-padded string in the filename (CIK0000320193). Always use str(cik).zfill(10) when building URLs.

scraping.md · updated Apr 28, 2026 · 1,899 words

Sites change. These notes describe what worked when they were last updated, not a guarantee they still do. If something broke, the fix is a pull request to browser-harness.

More research and public data skills