YouTube playback and transcript extraction
Use this playbook for public desktop video pages when you need reliable metadata or the visible transcript. YouTube lazy-renders the transcript control, so a newly loaded watch page usually does not expose it until the description is expanded.
Open and hydrate the watch page
Open a new tab rather than replacing the user's active tab, then give the Polymer app a few seconds to hydrate.
tab = new_tab("https://www.youtube.com/watch?v=<VIDEO_ID>")
wait_for_load()
wait(4)Keep using target_id=tab for later calls. YouTube pages can create or reuse additional targets, and target-free JavaScript may otherwise run in a different watch tab.
Read stable metadata
The hydrated player response is more stable than scraping rendered title and channel elements:
metadata = js("""
(() => {
const response = window.ytInitialPlayerResponse;
if (!response) return null;
const details = response.videoDetails || {};
const microformat = response.microformat?.playerMicroformatRenderer || {};
return {
id: details.videoId,
title: details.title,
author: details.author,
duration_seconds: Number(details.lengthSeconds || 0),
views: Number(details.viewCount || 0),
publish_date: microformat.publishDate || null,
description: details.shortDescription || "",
caption_tracks:
response.captions?.playerCaptionsTracklistRenderer?.captionTracks || []
};
})()
""", target_id=tab)If the response is briefly absent, wait for hydration and retry before falling back to DOM scraping.
Open the transcript panel
- Expand the description using the visible
#expandcontrol. YouTube may keep hidden duplicates in the DOM, so filter by rendered geometry. - After expansion, find a leaf whose trimmed text is exactly
Show transcript; click its closest semantic button. - Wait for the side panel to render.
expanded = js("""
(() => {
const transcriptButton = [...document.querySelectorAll('button, tp-yt-paper-button')]
.find(el => el.getClientRects().length && el.innerText.trim() === 'Show transcript');
if (transcriptButton) return true;
const expand = [...document.querySelectorAll('tp-yt-paper-button#expand')]
.find(el => el.getClientRects().length);
if (!expand) return false;
expand.click();
return true;
})()
""", target_id=tab)
if not expanded:
raise RuntimeError("Inspect the description before opening its transcript")
wait(1)
opened = js("""
(() => {
const leaf = [...document.querySelectorAll('body *')]
.find(el => el.children.length === 0 &&
(el.innerText || '').trim() === 'Show transcript' &&
el.getClientRects().length);
const button = leaf?.closest('button, tp-yt-paper-button');
if (!button) return false;
button.click();
return true;
})()
""", target_id=tab)
if not opened:
raise RuntimeError("The transcript control is absent; inspect the page before retrying")
wait(2)Avoid coordinate clicks here. The description height and transcript-button position vary with viewport, localization, experiments, and description length.
Extract the rendered transcript
Read only the visible transcript panel. Keep its timestamps and caption lines in display order:
transcript = js("""
(() => {
const panel = [...document.querySelectorAll('ytd-transcript-renderer')]
.find(el => el.getClientRects().length);
return panel ? panel.innerText : null;
})()
""", target_id=tab)If transcript is empty, inspect the panel and retry before reporting a transcript.
Treat auto-generated captions as fallible, especially for names, dates, numbers, and technical terms. Preserve timestamps when a questionable claim needs to be checked against the audio.
Common failure modes
Show transcriptis missing: expand the description first, then wait for the action row to render.- JavaScript returns metadata from the wrong video: pass the watch tab's
target_idon every call. - A hidden duplicate receives the click: require
getClientRects().lengthfor both the expansion control and transcript text leaf. - The transcript panel is absent: the UI may be localized, captions may be unavailable, or the panel may still be loading. Inspect the screenshot before changing selectors.
YouTube - Data Extraction
Field-tested against youtube.com on 2026-04-21. No authentication required for any approach documented here.
Approach 1 (Fastest): oEmbed API - No Auth, No Browser
https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={VIDEO_ID}&format=json
Returns JSON in ~0.3s. Works for any public video. Does not require login.
from helpers import http_get
import json
def youtube_oembed(video_id):
"""Fetch oEmbed metadata for a YouTube video.
Returns title, author, thumbnail URL, and embed iframe HTML.
"""
url = f"https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={video_id}&format=json"
return json.loads(http_get(url))
data = youtube_oembed("dQw4w9WgXcQ")
# {
# "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
# "author_name": "Rick Astley",
# "author_url": "https://www.youtube.com/@RickAstleyYT",
# "type": "video",
# "thumbnail_url": "https://i.ytimg.com/vi/dQw4w9WgXcQ/hqdefault.jpg",
# "thumbnail_width": 480,
# "thumbnail_height": 360,
# "width": 200,
# "height": 113,
# "version": "1.0",
# "provider_name": "YouTube",
# "html": '<iframe width="200" height="113" src="https://www.youtube.com/embed/dQw4w9WgXcQ?feature=oembed" ...>'
# }Bulk oEmbed (ThreadPoolExecutor)
from concurrent.futures import ThreadPoolExecutor
import json
from helpers import http_get
video_ids = ["dQw4w9WgXcQ", "jNQXAC9IVRw", "9bZkp7q19f0"]
def fetch_oembed(vid):
url = f"https://www.youtube.com/oembed?url=https://www.youtube.com/watch?v={vid}&format=json"
try:
return json.loads(http_get(url))
except Exception as e:
return {"error": str(e), "id": vid}
with ThreadPoolExecutor(max_workers=5) as ex:
results = list(ex.map(fetch_oembed, video_ids))
# 3 videos: ~4.1s total (YouTube oEmbed is slower than Spotify's; don't use >5 workers)Approach 2: Watch Page - Full Metadata via ytInitialPlayerResponse
Every youtube.com/watch?v={ID} page embeds two JSON blobs in the HTML:
ytInitialPlayerResponse- video details, microformat, caption track listytInitialData- comments section structure, related videos
Extract all video metadata
from helpers import http_get
import json, re
def scrape_video(video_id):
html = http_get(f"https://www.youtube.com/watch?v={video_id}")
# ---- ytInitialPlayerResponse ----
m = re.search(r'var ytInitialPlayerResponse = (\{.*?\});(?:var|</script>)', html, re.DOTALL)
if not m:
raise ValueError(f"ytInitialPlayerResponse not found for video {video_id} — video may be private, deleted, or region-blocked")
pr = json.loads(m.group(1))
# Check playability before parsing
status = pr.get("playabilityStatus", {}).get("status")
if status == "LOGIN_REQUIRED":
raise ValueError(f"Video {video_id} is age-restricted or login-gated (playabilityStatus: LOGIN_REQUIRED)")
if status == "ERROR":
reason = pr.get("playabilityStatus", {}).get("reason", "unknown")
raise ValueError(f"Video {video_id} is unavailable: {reason}")
vd = pr["videoDetails"]
mf = pr["microformat"]["playerMicroformatRenderer"]
caps = pr.get("captions", {}) \
.get("playerCaptionsTracklistRenderer", {}) \
.get("captionTracks", [])
return {
# Core
"video_id": vd["videoId"],
"title": vd["title"],
"author": vd["author"],
"channel_id": vd["channelId"],
"description": vd["shortDescription"],
"duration_s": int(vd["lengthSeconds"]),
"view_count": int(vd["viewCount"]),
"keywords": vd.get("keywords", []),
"is_live": vd.get("isLiveContent", False),
"is_private": vd.get("isPrivate", False),
# Microformat (richer publishing data)
"publish_date": mf.get("publishDate"), # ISO 8601, e.g. "2009-10-25T00:57:33-07:00"
"upload_date": mf.get("uploadDate"),
"category": mf.get("category"), # e.g. "Music", "Gaming", "Education"
"like_count": int(mf.get("likeCount", 0)),
"is_family_safe": mf.get("isFamilySafe"),
"is_unlisted": mf.get("isUnlisted"),
"available_countries": mf.get("availableCountries", []), # list of ISO 3166-1 alpha-2 codes
"channel_name": mf.get("ownerChannelName"),
"channel_url": mf.get("ownerProfileUrl"),
"embed_url": mf.get("embed", {}).get("iframeUrl"),
# Thumbnails (all publicly accessible, no auth)
"thumbnail_hq": f"https://i.ytimg.com/vi/{video_id}/hqdefault.jpg", # 480×360, always exists
"thumbnail_max": f"https://i.ytimg.com/vi/{video_id}/maxresdefault.jpg", # 1280×720, may 404
# Caption tracks — baseUrl is included for reference but returns empty in practice;
# use the Show Transcript UI flow in the browser instead (see playback.md)
"caption_tracks": [
{
"lang": t.get("languageCode"),
"name": t.get("name", {}).get("simpleText"),
"kind": t.get("kind", "manual"), # "manual" or "asr" (auto-generated)
"base_url": t.get("baseUrl"),
}
for t in caps
],
}
video = scrape_video("dQw4w9WgXcQ")
# {
# "video_id": "dQw4w9WgXcQ",
# "title": "Rick Astley - Never Gonna Give You Up (Official Video) (4K Remaster)",
# "author": "Rick Astley",
# "channel_id": "UCuAXFkgsw1L7xaCfnd5JJOw",
# "duration_s": 213,
# "view_count": 1764468859,
# "publish_date": "2009-10-24T23:57:33-07:00",
# "category": "Music",
# "like_count": 16942558,
# "caption_tracks": [
# {"lang": "en", "name": "English", "kind": "manual"},
# {"lang": "en", "name": "English (auto-generated)", "kind": "asr"},
# {"lang": "de-DE", "name": "German (Germany)", "kind": "manual"},
# ...
# ],
# }Approach 3: Search Results - No Auth
youtube.com/results?search_query={QUERY} is server-side rendered. The ytInitialData blob contains up to ~20 video results.
from helpers import http_get
import json, re
from urllib.parse import quote_plus
def youtube_search(query, max_results=20):
"""Search YouTube videos without a browser or API key."""
url = f"https://www.youtube.com/results?search_query={quote_plus(query)}"
html = http_get(url)
m = re.search(r'var ytInitialData = (\{.*?\});</script>', html, re.DOTALL)
data = json.loads(m.group(1))
# Walk the nested structure to find videoRenderer items
section_contents = (
data.get("contents", {})
.get("twoColumnSearchResultsRenderer", {})
.get("primaryContents", {})
.get("sectionListRenderer", {})
.get("contents", [])
)
results = []
for section in section_contents:
for item in section.get("itemSectionRenderer", {}).get("contents", []):
vr = item.get("videoRenderer", {})
if not vr:
continue
snippet = vr.get("detailedMetadataSnippets", [])
desc = (
"".join(r.get("text", "") for r in snippet[0]["snippetText"]["runs"])
if snippet else None
)
results.append({
"video_id": vr["videoId"],
"url": f"https://www.youtube.com/watch?v={vr['videoId']}",
"title": vr.get("title", {}).get("runs", [{}])[0].get("text"),
"channel": vr.get("ownerText", {}).get("runs", [{}])[0].get("text"),
"channel_url": (
"https://www.youtube.com"
+ vr.get("ownerText", {}).get("runs", [{}])[0]
.get("navigationEndpoint", {})
.get("browseEndpoint", {})
.get("canonicalBaseUrl", "")
),
"duration": vr.get("lengthText", {}).get("simpleText"), # e.g. "3:32"
"views": vr.get("viewCountText", {}).get("simpleText"), # e.g. "1,764,468,859 views"
"published": vr.get("publishedTimeText", {}).get("simpleText"), # e.g. "7 years ago"
"description_snippet": desc,
"thumbnail": f"https://i.ytimg.com/vi/{vr['videoId']}/hqdefault.jpg",
})
if len(results) >= max_results:
return results # exit both loops immediately
return results
results = youtube_search("python tutorial", max_results=5)
# Returns up to ~14-20 results (YouTube serves fewer than 20 on first page)
# [
# {
# "video_id": "K5KVEU3aaeQ",
# "title": "Python Full Course for Beginners",
# "channel": "Programming with Mosh",
# "duration": "2:02:21",
# "views": "6,056,121 views",
# "published": "1 year ago",
# }, ...
# ]Approach 4: Channel Metadata - No Auth
Channel pages (youtube.com/@handle or youtube.com/channel/{CHANNEL_ID}) embed metadata in ytInitialData.
from helpers import http_get
import json, re
def scrape_channel(handle_or_id):
"""
handle_or_id: "@RickAstleyYT" (handle, with @)
"UCuAXFkgsw1L7xaCfnd5JJOw" (channel ID)
"""
if handle_or_id.startswith("UC"):
url = f"https://www.youtube.com/channel/{handle_or_id}"
else:
url = f"https://www.youtube.com/{handle_or_id}"
html = http_get(url)
m = re.search(r'var ytInitialData = (\{.*?\});</script>', html, re.DOTALL)
data = json.loads(m.group(1))
# Canonical metadata (always present)
cmd = data.get("metadata", {}).get("channelMetadataRenderer", {})
# Subscriber count + handle from pageHeaderViewModel
ph = (
data.get("header", {})
.get("pageHeaderRenderer", {})
.get("content", {})
.get("pageHeaderViewModel", {})
)
meta_parts = [
part.get("text", {}).get("content", "")
for row in ph.get("metadata", {})
.get("contentMetadataViewModel", {})
.get("metadataRows", [])
for part in row.get("metadataParts", [])
]
# meta_parts is typically: ["@handle", "4.48m subscribers", "N videos"]
# Avatar (take the largest source)
avatar_sources = (
ph.get("image", {})
.get("decoratedAvatarViewModel", {})
.get("avatar", {})
.get("avatarViewModel", {})
.get("image", {})
.get("sources", [])
)
avatar_url = avatar_sources[-1]["url"] if avatar_sources else None
# Channel banner
banner_sources = (
ph.get("banner", {})
.get("imageBannerViewModel", {})
.get("image", {})
.get("sources", [])
)
banner_url = banner_sources[-1]["url"] if banner_sources else None
return {
"channel_id": cmd.get("externalId"),
"title": cmd.get("title"),
"description": cmd.get("description"),
"channel_url": cmd.get("channelUrl"),
"keywords": cmd.get("keywords", ""),
"handle": meta_parts[0] if len(meta_parts) > 0 else None,
"subscribers": meta_parts[1] if len(meta_parts) > 1 else None, # e.g. "4.48m subscribers"
"avatar_url": avatar_url,
"banner_url": banner_url,
}
channel = scrape_channel("@RickAstleyYT")
# {
# "channel_id": "UCuAXFkgsw1L7xaCfnd5JJOw",
# "title": "Rick Astley",
# "description": "2026 UK & Ireland Reflection Tour...",
# "channel_url": "https://www.youtube.com/channel/UCuAXFkgsw1L7xaCfnd5JJOw",
# "handle": "@RickAstleyYT",
# "subscribers": "4.48m subscribers",
# "avatar_url": "https://yt3.googleusercontent.com/...",
# "banner_url": "https://yt3.googleusercontent.com/...",
# }Thumbnail URLs - All Sizes
All thumbnail sizes are publicly accessible without auth. Construct directly from video_id:
def thumbnail_urls(video_id):
base = f"https://i.ytimg.com/vi/{video_id}"
return {
"default": f"{base}/default.jpg", # 120×90, always exists
"medium": f"{base}/mqdefault.jpg", # 320×180, always exists
"high": f"{base}/hqdefault.jpg", # 480×360, always exists
"standard": f"{base}/sddefault.jpg", # 640×480, always exists
"maxres": f"{base}/maxresdefault.jpg", # 1280×720, may 404 on older/low-res videos
}Extract Video ID from Any URL
import re
def extract_video_id(url):
"""Extract YouTube video ID (11-char) from any YouTube URL format."""
m = re.search(r'(?:v=|/v/|youtu\.be/|/embed/|/shorts/)([A-Za-z0-9_-]{11})', url)
return m.group(1) if m else None
extract_video_id("https://www.youtube.com/watch?v=dQw4w9WgXcQ") # "dQw4w9WgXcQ"
extract_video_id("https://youtu.be/dQw4w9WgXcQ") # "dQw4w9WgXcQ"
extract_video_id("https://www.youtube.com/shorts/dQw4w9WgXcQ") # "dQw4w9WgXcQ"
extract_video_id("https://www.youtube.com/embed/dQw4w9WgXcQ") # "dQw4w9WgXcQ"What Requires a Browser
The following are not accessible via http_get and require the CDP browser (see playback.md):
- Trending / Explore (
/feed/trending) -ytInitialDataloads but video items are empty without cookies - Playlist contents -
ytInitialDatareturns only microformat; full video list requires session cookies - Comments - loaded lazily via XHR, not present in initial HTML
- Shorts feed - requires JS hydration
- Channel Videos tab - video list requires cookies for consistent results
- Caption text content -
captionTracks[].baseUrlURLs return empty bytes regardless of session state; use the browser's Show Transcript UI flow instead (seeplayback.md) - Age-restricted videos - oEmbed returns HTTP 401;
scrape_video()raisesValueError("LOGIN_REQUIRED")
Watch-page DOM hydration - the wait you need
When you do fall through to the browser for watch-page DOM (e.g. because you need a
rendered UI state, not just metadata), wait_for_load() is not enough. The load
event fires before YouTube's Polymer components hydrate - h1.ytd-watch-metadata yt-formatted-string,
ytd-video-owner-renderer #channel-name a, and ytd-watch-info-text all return null for
~2s after load. Add a wait(3) after wait_for_load() before querying any watch-page
selector.
Field-tested 2026-04-24 on Brave; same behavior observed on ungoogled-chromium. Prefer
the http_get + ytInitialPlayerResponse path above - the browser path exists for flows
that need live UI state, not for reading metadata.
URL Patterns
| Resource | URL pattern |
|---|---|
| Video | https://www.youtube.com/watch?v={VIDEO_ID} |
| Short URL | https://youtu.be/{VIDEO_ID} |
| Shorts | https://www.youtube.com/shorts/{VIDEO_ID} |
| Channel handle | https://www.youtube.com/@{HANDLE} |
| Channel ID | https://www.youtube.com/channel/{CHANNEL_ID} |
| Playlist | https://www.youtube.com/playlist?list={PLAYLIST_ID} |
| Search | https://www.youtube.com/results?search_query={QUERY} |
| oEmbed | https://www.youtube.com/oembed?url={VIDEO_URL}&format=json |
| Thumbnail (HQ) | https://i.ytimg.com/vi/{VIDEO_ID}/hqdefault.jpg |
Gotchas
ytInitialPlayerResponseregex must use non-greedy match with lookahead -(\{.*?\});(?:var|</script>)withre.DOTALLis reliable. Do not use\{.*\}(greedy) - it consumes the entire rest of the page.viewCountandlengthSecondsare strings, not ints -vd["viewCount"]returns"1764468859". Always cast withint().likeCountlives inmicroformat, notvideoDetails-videoDetailsdoes not expose like count.microformat.playerMicroformatRenderer.likeCountis a string integer.availableCountriesis a list of ISO 3166-1 alpha-2 codes - 249 entries for globally available videos. An empty list means region data is unavailable, not that the video is globally blocked.- oEmbed thumbnail is always
hqdefault(480×360) - if you need 1280×720, construct themaxresdefault.jpgURL directly, but check for 404 on older videos. - Search returns ~14–20 results - YouTube does not guarantee 20 results. Always iterate
itemSectionRenderer.contentsrather than assuming a fixed count. - Channel subscriber count is a rounded string -
"4.48m subscribers", not an integer. Parse with regex if sorting:re.search(r'([\d.]+)\s*([km]?)', text, re.I). meta_partsorder is[handle, subscribers, video_count]- the third element is not always present. Index defensively.- Caption
baseUrlis not fetchable -captionTracks[].baseUrlcontainsexpire=andsignature=params but returns empty bytes in all tested conditions (plainhttp_get, XHR from within the page, andfetch()with cookies). Use the Show Transcript UI in the browser for caption text (seeplayback.md). - Age-restricted videos -
scrape_video()raisesValueErrorwith aLOGIN_REQUIREDmessage.oEmbedreturns HTTP 401 (raisesurllib.error.HTTPError). Neither approach can access age-restricted content without login. - Private / deleted videos - oEmbed returns HTTP 404 (raises
urllib.error.HTTPError). Wrap intry/except. ytInitialDatablob terminator is;</script>- usingre.DOTALLwith(\{.*?\});</script>is safe; the blob does not contain;</script>internally.