Skip to content
botshub

Crawl4AI

by Crawl4AI

Singapore

Agent infrastructure and tools

Visit Crawl4AI

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that p...

About Crawl4AI

Crawl4AI is an open-source web crawler built for language model workflows. It visits pages, cleans them up and returns Markdown, JSON or cleaned HTML that is ready to feed into retrieval and agent pipelines, rather than the raw HTML a general crawler would give you.

It can crawl many URLs at once, work with Chromium, Firefox and WebKit, and extract structured data with any model, whether open-source or hosted. Content can be split into chunks by topic, regular expression or sentence, and media tags and links are captured as well. It installs with pip and also ships as Docker images with a command-line interface.

The project was started by a developer known as UncleCode, who built the first version in a few days and released it openly. It has grown into one of the more popular crawlers for AI use, with a cloud API announced as a closed beta.

Key features

  • Returns clean Markdown, JSON or cleaned HTML
  • Crawls many URLs in parallel
  • Chromium, Firefox and WebKit support
  • Structured extraction with any language model
  • Chunking by topic, regex or sentence
  • Captures media tags and links
  • Install with pip or run in Docker
  • Apache 2.0 licence

All content and agents from Crawl4AI

Frequently asked questions about Crawl4AI

What is Crawl4AI?
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that p... It is listed on botshub under agent infrastructure and tools.
Where is Crawl4AI from?
Crawl4AI comes from Singapore.
Is Crawl4AI free or open source?
Crawl4AI is tagged as open source on botshub. Check the official site for current terms.
What can Crawl4AI do?
Key features include returns clean Markdown, JSON or cleaned HTML, crawls many URLs in parallel, chromium, Firefox and WebKit support, structured extraction with any language model and chunking by topic, regex or sentence.
What are alternatives to Crawl4AI?
Other agent infrastructure and tools on botshub include Agent Reach, ai-artifacts, HOL Standards SDK and LangServe. See all alternatives to Crawl4AI.
Suggest an improvement
Report this listing
Is this your product? Claim this listing