Website Content Crawler

100 credits / 100 leads Limited permissions

metudo/website-content-crawler 4.6 (892) 892 156K 156K Crafted by Metudo

Crawl websites and extract clean text for AI/LLM use. Provide start URLs and configure depth, format, and content filters.

Start URLs

Start URL(s) ? One URL per line. Crawling begins from each.

One URL per line. Crawling begins from each.

Include URL patterns ? Glob patterns. Leave empty to follow all links on domain.

Glob patterns. Leave empty to follow all links on domain.

Exclude URL patterns

Max crawl depth

levels

Max pages

pages

Crawler type

Respect robots.txt

Output format

Markdown (recommended for LLMs) Plain text Cleaned HTML

Remove elements

Navigation menus Footers Ads & banners Cookie banners Comment sections

Save raw HTML alongside text

Run options

Maximum credit per run

Unlimited

Build

latest

Timeout

604800s

Memory

4 GB

Actor running slowly or crashing? Try increasing memory.

Build

Timeout ? Maximum run duration in seconds

seconds

No timeout

Memory ? RAM allocated to the actor run

Maximum credit per run (optional)

credits

No maximum limit

These settings only apply to Actor runs started from Metudo Console. Learn how to provide input when starting the Actor via the API.

Crawl websites and extract text content for AI and LLM applications. Get clean markdown from any page.

Full documentation and changelog will appear here.

No data yet. Run this actor to populate this tab.

Paste items separated by new line.

0 items