doc-scraper — community doc-scraper, snowflake-dbt-demo, community, ide skills, Claude Code, Cursor, Windsurf

v1.0.0
GitHub

About this Skill

Perfect for Data Analysis Agents needing automated documentation scraping from Snowflake Data Cloud This project demonstrates many of dbt's features when used with the Snowflake Data Cloud

sfc-gh-dflippo sfc-gh-dflippo
[0]
[0]
Updated: 3/5/2026

Agent Capability Analysis

The doc-scraper skill by sfc-gh-dflippo is an open-source community AI agent skill for Claude Code and other IDE workflows, helping agents execute tasks with better context, repeatability, and domain-specific guidance.

Ideal Agent Persona

Perfect for Data Analysis Agents needing automated documentation scraping from Snowflake Data Cloud

Core Value

Empowers agents to scrape Snowflake documentation into Markdown format with SQLite caching, utilizing Python scripts and command-line options like --output-dir and --spider-depth, while handling 7-day expiration for cached data

↓ Capabilities Granted for doc-scraper

Scraping specific sections of Snowflake documentation
Automating documentation updates with periodic runs
Generating localized documentation sets by specifying base paths like /en/sql-reference/

! Prerequisites & Limits

  • Requires Python 3 environment
  • Limited to scraping docs.snowflake.com
  • Dependent on SQLite for caching
Labs Demo

Browser Sandbox Environment

⚡️ Ready to unleash?

Experience this Agent in a zero-setup browser environment powered by WebContainers. No installation required.

Boot Container Sandbox

doc-scraper

Install doc-scraper, an AI agent skill for AI agent workflows and automation. Works with Claude Code, Cursor, and Windsurf with one-command setup.

SKILL.md
Readonly

Snowflake Documentation Scraper

Scrapes docs.snowflake.com sections to Markdown with SQLite caching (7-day expiration).

Usage

First time setup (auto-installs uv and doc-scraper):

bash
1python3 .claude/skills/doc-scraper/scripts/doc_scraper.py

Subsequent runs:

bash
1doc-scraper --output-dir=./snowflake-docs
2doc-scraper --output-dir=./snowflake-docs --base-path="/en/sql-reference/"
3doc-scraper --output-dir=./snowflake-docs --spider-depth=2

Command Options

OptionDefaultDescription
--output-dirRequiredOutput directory for scraped docs
--base-path/en/migrations/URL section to scrape
--spider-depth1Link depth: 0=seeds, 1=+links, 2=+2nd
--limitNoneCap URLs (for testing)
--dry-run-Preview without writing

Output

sql
1output-dir/
2├── SKILL.md              # Auto-generated index
3├── scraper_config.yaml   # Editable config (auto-created)
4├── .cache/               # SQLite cache (auto-managed)
5└── en/migrations/*.md    # Scraped pages with frontmatter

Configuration

Auto-created at {output-dir}/scraper_config.yaml:

yaml
1rate_limiting:
2  max_concurrent_threads: 4
3spider:
4  max_pages: 1000
5  allowed_paths: ["/en/"]
6scraped_pages:
7  expiration_days: 7

Troubleshooting

IssueSolution
Too many pagesLower --spider-depth or edit config
Missing pagesIncrease --spider-depth
Cache corruptionDelete {output-dir}/.cache/ (rare)

FAQ & Installation Steps

These questions and steps mirror the structured data on this page for better search understanding.

? Frequently Asked Questions

What is doc-scraper?

Perfect for Data Analysis Agents needing automated documentation scraping from Snowflake Data Cloud This project demonstrates many of dbt's features when used with the Snowflake Data Cloud

How do I install doc-scraper?

Run the command: npx killer-skills add sfc-gh-dflippo/snowflake-dbt-demo/doc-scraper. It works with Cursor, Windsurf, VS Code, Claude Code, and 19+ other IDEs.

What are the use cases for doc-scraper?

Key use cases include: Scraping specific sections of Snowflake documentation, Automating documentation updates with periodic runs, Generating localized documentation sets by specifying base paths like /en/sql-reference/.

Which IDEs are compatible with doc-scraper?

This skill is compatible with Cursor, Windsurf, VS Code, Trae, Claude Code, OpenClaw, Aider, Codex, OpenCode, Goose, Cline, Roo Code, Kiro, Augment Code, Continue, GitHub Copilot, Sourcegraph Cody, and Amazon Q Developer. Use the Killer-Skills CLI for universal one-command installation.

Are there any limitations for doc-scraper?

Requires Python 3 environment. Limited to scraping docs.snowflake.com. Dependent on SQLite for caching.

↓ How To Install

  1. 1. Open your terminal

    Open the terminal or command line in your project directory.

  2. 2. Run the install command

    Run: npx killer-skills add sfc-gh-dflippo/snowflake-dbt-demo/doc-scraper. The CLI will automatically detect your IDE or AI agent and configure the skill.

  3. 3. Start using the skill

    The skill is now active. Your AI agent can use doc-scraper immediately in the current project.

Related Skills

Looking for an alternative to doc-scraper or another community skill for your workflow? Explore these related open-source skills.

View All

widget-generator

Logo of f
f

f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

★ 149.6k
⑂ 0
AI

flags

Logo of vercel
vercel

flags is a Next.js feature management skill that enables developers to efficiently add or modify framework feature flags, streamlining React application development.

★ 138.4k
⑂ 0
Browser

zustand

Logo of lobehub
lobehub

The ultimate space for work and life — to find, build, and collaborate with agent teammates that grow with you. We are taking agent harness to the next level — enabling multi-agent collaboration, effortless agent team design, and introducing agents as the unit of work interaction.

★ 72.8k
⑂ 0
AI

data-fetching

Logo of lobehub
lobehub

The ultimate space for work and life — to find, build, and collaborate with agent teammates that grow with you. We are taking agent harness to the next level — enabling multi-agent collaboration, effortless agent team design, and introducing agents as the unit of work interaction.

★ 72.8k
⑂ 0
AI