Build software better, together

A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

ruby crawler scraper web spider web-crawler web-scraper web-scraping web-spider spider-links

Updated May 10, 2022
Ruby

USCDataScience / sparkler

Star

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

search search-engine distributed-systems information-retrieval big-data spark solr web-crawler nutch tika sparkles

Updated Jun 22, 2022
Java

VIDA-NYU / ache

Star

ACHE is a web crawler for domain-specific search.

web-crawler web-scraping hacktoberfest web-spider focused-crawler domain-specific-search web-search

Updated Jun 20, 2022
Java

brendonboshell / supercrawler

Star

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

sitemap crawler robot web-crawler distributed-crawler

Updated Apr 8, 2022
JavaScript

rivermont / spidy

Star

The simple, easy to use command line web crawler.

python crawler web-crawler crawling python3 web-spider

Updated Apr 28, 2022
Python

hyunwoongko / kochat

Star

Opensource Korean chatbot framework

deep-learning web-crawler chatbot korean deeplearning sentence-classification korean-chatbot sequance-tagging

Updated Jun 22, 2022
Python

lucasxlu / LagouJob

Star

Job data mining repo for lagou.com

nlp machine-learning data-mining web-crawler python3 data-analysis lagou

Updated Apr 19, 2019
Python

yields / ant

Star

A web crawler for Go

go golang scraper spider web-crawler

Updated Jun 6, 2022
Go

microfisher / Strong-Web-Crawler

Star

基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

crawler phantomjs web-crawler sellenium

Updated Oct 25, 2019
C#

antchfx / antch

Star

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

golang crawler framework web-crawler scraping crawling web-spider

Updated May 31, 2020
Go

crawler-commons / crawler-commons

Star

A set of reusable Java components that implement functionality common to any web crawler

java open-source library web-crawler robots-txt sitemaps

Updated Apr 25, 2022
Java

elliotxx / zhihu-crawler-people

Star

A simple distributed crawler for zhihu && data analysis

python crawler spider web-crawler python-crawler web-spider

Updated Nov 11, 2019
Python

Hecate2 / Ignareo-ISML-auto-voter

Star

Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns!

python http microservice high-performance web-crawler concurrency distributed asyncio gevent web-spider isml sukasuka chtholly sukamoka ignareo tiat

Updated Feb 21, 2022
Python

duyet / awesome-web-scraper

Star

A collection of awesome web scaper, crawler.

php awesome spider storage phantomjs web-crawler web-scraper scrapy awesome-list goutte slimerjs

Updated Jun 2, 2022

TurnerSoftware / InfinityCrawler

Star

A simple but powerful web crawler library for .NET

crawler spider web-crawler robots-txt web-crawling

Updated Jun 2, 2022
C#

crawlab-team / crawlab-lite

Star

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

platform crawler spider web-crawler scrapy scrapyd scrapy-ui scrapyd-ui crawling-tasks crawlab crawler-management

Updated Jun 19, 2022
Vue

commoncrawl / news-crawl

Star

News crawling with Storm-crawler - stores content as WARC

crawler news web-crawler apache-storm warc commoncrawl common-crawl

Updated Mar 31, 2022
Java

Algebra-FUN / WeReadScan

Star

扫描“微信读书”已购图书并下载本地PDF的爬虫

web-crawler selenium pdf-converter weread

Updated Aug 23, 2021
Python

Norconex / collector-http

Star

Norconex Web Crawler (or spider) is a flexible web crawler for collecting, parsing, and manipulating data from the Internet (or Intranet) to various data repositories such as search engines.

java search-engine flexible web-crawler norconex-http-collector collector-http