Research and public data

github.com

Updated Apr 28, 2026

GitHub - Scraping & Data Extraction

Notes an AI agent reads before it works on github.com: what to open, what to call, what breaks. Written by browser-harness contributors after real runs, and kept on GitHub under the MIT license.

Run it in Browser Use Cloud Source on GitHub

Give it to your agent

Any coding agent

Start the task with this line. The Markdown version is the same notes without the page around them.

Read https://browser-use.com/skills/github.md, then <your task>

Browser Harness

Save the files into your workspace and turn domain skills on. The code below calls its helpers, such as http_get and js. What Browser Harness is.

d="${BH_AGENT_WORKSPACE:-$HOME/.config/browser-harness/agent-workspace}/domain-skills/github"
mkdir -p "$d" && cd "$d"
curl -sLO https://raw.githubusercontent.com/browser-use/browser-harness/main/agent-workspace/domain-skills/github/scraping.md
export BH_DOMAIN_SKILLS=1

https://github.com - public data, mix of REST API (fast, rate-limited) and browser (trending page only).

Do this first

Use the REST API for repo/user/release data - it's one call, no browser, fully parsed JSON.

import json
data = json.loads(http_get("https://api.github.com/repos/{owner}/{repo}"))
# Key fields: stargazers_count, forks_count, description, language, topics,
#             open_issues_count, created_at, updated_at, pushed_at,
#             watchers_count, subscribers_count, network_count,
#             default_branch, license, homepage, visibility

Use raw.githubusercontent.com for file contents - no rate limit, no auth, no base64 decode:

readme = http_get("https://raw.githubusercontent.com/owner/repo/main/README.md")
content = http_get("https://raw.githubusercontent.com/owner/repo/main/pyproject.toml")

Use the browser only for the trending page - it's server-side rendered HTML, no API equivalent.

Common workflows

Repo metadata (API)

import json
data = json.loads(http_get("https://api.github.com/repos/browser-use/browser-use"))
print(data['stargazers_count'], data['forks_count'], data['description'])
# returns: 88349  10136  '🌐 Make websites accessible for AI agents.'

User / org profile (API)

import json
user = json.loads(http_get("https://api.github.com/users/browser-use"))
print(user['type'], user['followers'], user['public_repos'], user['blog'])
# returns: 'Organization'  3046  39  'https://browser-use.com'

The trending page is JS-rendered. article.Box-row selector confirmed working (15 results for today/all-languages, 12 for filtered). All fields work in a single JS call - must navigate and wait in the same script run, as each run is a separate exec context.

import json
goto_url("https://github.com/trending")          # or /trending/python?since=weekly
wait_for_load()
wait(2)                                       # extra 2s — React hydration completes after readyState
 
result = js("""
(function(){
  var rows = Array.from(document.querySelectorAll('article.Box-row'));
  return JSON.stringify(rows.map(function(el){
    var h2link = el.querySelector('h2 a');
    var starLink = el.querySelector('a[href*="/stargazers"]');
    var forkLink = el.querySelector('a[href*="/forks"]');
    var langEl = el.querySelector('[itemprop="programmingLanguage"]');
    var todayEl = el.querySelector('.d-inline-block.float-sm-right');
    var descEl = el.querySelector('p');
    return {
      name: h2link ? h2link.innerText.trim().replace(/\\s+/g,' ') : null,
      url: h2link ? 'https://github.com' + h2link.getAttribute('href') : null,
      stars_total: starLink ? starLink.innerText.trim() : null,
      stars_period: todayEl ? todayEl.innerText.trim() : null,
      forks: forkLink ? forkLink.innerText.trim() : null,
      language: langEl ? langEl.innerText.trim() : null,
      desc: descEl ? descEl.innerText.trim() : null
    };
  }));
})()
""")
repos = json.loads(result)
# stars_period text is e.g. "737 stars today" or "47,053 stars this week"

Supported URL params:

/trending - all languages, today
/trending/python - filtered to Python
/trending?since=weekly or ?since=monthly
/trending/python?since=weekly - combined

Search repositories (API)

import json
results = json.loads(http_get(
    "https://api.github.com/search/repositories?q=browser+automation+language:python&sort=stars&per_page=10"
))
print(results['total_count'])   # e.g. 3250
for r in results['items']:
    print(r['full_name'], r['stargazers_count'])

Search API rate limit is 10 req/min unauthenticated (separate from the 60/hour core limit). Runs out fast if called in a loop.

Commits, releases, issues (API)

import json
# Commits
commits = json.loads(http_get("https://api.github.com/repos/owner/repo/commits?per_page=10"))
# Fields: sha, commit.message, commit.author.date, author.login
 
# Releases
releases = json.loads(http_get("https://api.github.com/repos/owner/repo/releases?per_page=5"))
# Fields: tag_name, name, published_at, body, assets
 
# Issues
issues = json.loads(http_get("https://api.github.com/repos/owner/repo/issues?state=open&per_page=10"))
# Fields: number, title, labels, state, created_at, user.login
 
# Contributors
contribs = json.loads(http_get("https://api.github.com/repos/owner/repo/contributors?per_page=10"))
# Fields: login, contributions

File contents via API (base64)

import json, base64
resp = json.loads(http_get("https://api.github.com/repos/owner/repo/contents/path/to/file.py"))
content = base64.b64decode(resp['content']).decode()
# resp also has: size, sha, html_url
# Prefer raw.githubusercontent.com for large files — no base64, no rate limit hit

Parallel fetching (multiple repos)

import json
from concurrent.futures import ThreadPoolExecutor
 
def fetch_repo(name):
    data = json.loads(http_get(f"https://api.github.com/repos/{name}"))
    return {"name": name, "stars": data['stargazers_count'], "lang": data['language']}
 
repos = ["owner/repo1", "owner/repo2", "owner/repo3"]
with ThreadPoolExecutor(max_workers=3) as ex:
    results = list(ex.map(fetch_repo, repos))
# Confirmed working; watch rate limit — 60 unauthenticated calls/hour total

Gotchas

Rate limits are per IP, unauthenticated - Core API: 60 req/hour. Search API: 10 req/min. These are separate pools. Check /rate_limit endpoint: http_get("https://api.github.com/rate_limit"). With a GITHUB_TOKEN, both limits increase to 5,000/hour.

Token header format - Use Authorization: Bearer <token> (not token <token>), plus X-GitHub-Api-Version: 2022-11-28:

import os
token = os.environ.get('GITHUB_TOKEN', '')
headers = {"Authorization": f"Bearer {token}", "X-GitHub-Api-Version": "2022-11-28"} if token else {}
data = json.loads(http_get("https://api.github.com/repos/owner/repo", headers=headers))

404 raises HTTPError, not a JSON error - Wrap API calls for missing repos:

try:
    data = json.loads(http_get("https://api.github.com/repos/owner/repo"))
except Exception as e:
    print("Not found or rate limited:", e)

Code search requires auth - GET /search/code returns HTTP 401 without a token. Repo/user/issues search works unauthenticated.
Trending page selectors only work if navigation is in the same script run - Each uv run browser-harness exec is fresh. Selectors that returned 0 results were run in a separate invocation after the page had navigated away. Always include goto_url() + wait_for_load() + wait(2) in the same script.
wait(2) after wait_for_load() on trending - document.readyState == 'complete' fires before React finishes painting repo cards. Without the extra 2s sleep, article.Box-row count was 0 even though the DOM technically loaded.
Trending stars field is a string with commas - stars_total comes back as "4,548" not 4548. Parse with int(r['stars_total'].replace(',', '')) if you need to sort.
stars_period text includes the period - Value is "737 stars today" or "47,053 stars this week" - strip the trailing word if you want just the number.
Repo page DOM is React-heavy, API is better - Extracting star counts from the repo HTML page (github.com/owner/repo) is unreliable because GitHub uses React with server-side hydration and component IDs change. The REST API returns all the same data cleanly.
raw.githubusercontent.com has no rate limit and no auth - Use it for any public file. It serves the raw bytes, no JSON wrapping or base64.
Trending page article count varies - Today filter returned 15 articles, weekly Python filter returned 12. Don't assume 25 results; iterate document.querySelectorAll('article.Box-row') and take what's there.

scraping.md · updated Apr 28, 2026 · 887 words

Sites change. These notes describe what worked when they were last updated, not a guarantee they still do. If something broke, the fix is a pull request to browser-harness.