crawler

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Jul 17, 2020
PHP

s0md3v / Photon

Star

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Jul 22, 2020
Python

crawlab-team / crawlab

Star

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management

Updated Jul 26, 2020
Go

injetlee / Python

Star

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

python crawler excel wechat

Updated Feb 10, 2020
Python

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Jun 24, 2020
JavaScript

yujiosaka / headless-chrome-crawler

Star

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

Updated Jul 7, 2020
JavaScript

chyroc / WechatSogou

Star

基于搜狗微信搜索的微信公众号爬虫接口

python crawler pypi dict scrapy wechat sogou wsapi

Updated Apr 23, 2020
Python

rmax / scrapy-redis

Star

Redis-based components for Scrapy.

redis crawler distributed scrapy

Updated Jul 9, 2020
Python

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Nov 1, 2019
Python

MontFerret / ferret

Star

Declarative web scraping

go cli golang crawler chrome data-mining scraper library tool dsl scraping crawling query-language scraping-websites cdp

Updated Jul 24, 2020
Go

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated May 24, 2020

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Feb 29, 2020
Python

symfony / dom-crawler

Star

The DomCrawler component eases DOM navigation for HTML and XML documents.

php crawler symfony component symfony-component dom-crawler

Updated Jul 24, 2020
PHP

imWildCat / scylla

Star

Intelligent proxy pool for Humans™ (Maintainer needed)

python crawler scylla python3 proxy-pool

Updated Jul 18, 2020
Python

Arachni / arachni

Star

Web Application Security Scanner Framework

javascript ruby crawler security-audit modular hack dom analysis scanner detection hacking xss audit web-application penetration-testing sql-injection vulnerability-detection arachni scanners

Updated Jan 28, 2020
Ruby

dotnetcore / DotnetSpider

Star

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

crawler csharp cross-platform dotnetcore distributed

Updated Jun 25, 2020
C#

NikolaiT / GoogleScraper

Star

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

python search-engine crawler scraping search-engines search-engine-optimization

Updated May 11, 2020
HTML

constverum / ProxyBroker

Star

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

crawler privacy proxy proxy-server http-proxy socks proxies anonymity anonymous proxypool proxy-list proxy-checker

Updated Jul 2, 2020
Python

jae-jae / QueryList

Star

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

crawler scraper spider querylist

Updated Jul 18, 2020
PHP

xtuhcy / gecco

Star

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

java crawler dynamic jsoup gecco fastjson

Updated Jul 23, 2020
Java

gaojiuli / gain

Star

Web crawling framework based on asyncio.

python crawler spider aiohttp asyncio uvloop

Updated Jun 1, 2019
Python

PuerkitoBio / gocrawl

Star

Polite, slim and concurrent web crawler.

crawler robots-txt

Updated Feb 2, 2020
Go

Improve this page

Add a description, image, and links to the crawler topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the crawler topic, visit your repo's landing page and select "manage topics."

Learn more