Build software better, together

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated May 27, 2021
PHP

henrylee2cn / pholcus

Star

Pholcus is a distributed high-concurrency crawler software written in pure golang

spider crowler

Updated Feb 15, 2020
Go

luyishisi / Anti-Anti-Spider

Star

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

python spider geek

Updated Jun 29, 2020
Python

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Mar 19, 2021
JavaScript

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。

chrome automation spider selenium python3 crawl tkinter outlook wxpython csdn hotmail

Updated Jun 8, 2021
Python

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Jun 10, 2021
Python

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated May 29, 2021

tophubs / TopList

Star

今日热榜，一个获取各大热门网站热门头条的聚合网站，使用Go语言编写，多协程异步快速抓取信息，预览:https://mo.fish

golang spider hot today-s-hot-list hotlist

Updated Jul 9, 2021
Go

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Jun 28, 2021
Python

JAVClub / core

Star

🔞 JAVClub - 让你的大姐姐不再走丢

spider japanese google-drive jav video-streaming magnet adult porn adult-content javbus javiewer

Updated Jun 14, 2021
JavaScript

DormyMo / SpiderKeeper

Star

admin ui for scrapy/open source scrapinghub

spider dashboard scrapy scrapyd scrapy-ui scrapyd-ui scrapyd-dashboard

Updated Mar 19, 2021
Python

201206030 / fiction_house

Star

小说精品屋是一个多平台（web、安卓app、微信小程序）、功能完善的屏幕自适应小说漫画连载系统，包含精品小说专区、轻小说专区和漫画专区。包括小说/漫画分类、小说/漫画搜索、小说/漫画排行、完本小说/漫画、小说/漫画评分、小说/漫画在线阅读、小说/漫画书架、小说/漫画阅读记录、小说下载、小说弹幕、小说/漫画自动采集/更新/纠错、小说内容自动分享到微博、邮件自动推广、链接自动推送到百度搜索引擎等功能。

cms spider book download read novel cartoon soft-novel

Updated Jun 4, 2021
Java

Gerapy / Gerapy

Star

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

docker vuejs django spider dashboard vue distributed scrapy scrapyd webspider gerapy

Updated Jun 10, 2021
Python

shiyanhui / dht

Star

BitTorrent DHT Protocol && DHT Spider.

go spider dht bittorrent-dht-protocol

Updated Dec 19, 2020
Go

jae-jae / QueryList

Star

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

crawler scraper spider querylist

Updated Jul 18, 2021
PHP

my8100 / scrapydweb

Star

Open

User Guide | Q&A | 用户指南 | 问答

78

LWsmile commented Nov 27, 2018

linux：HTTPConnectionPool(host='192.168.0.24', port=6801): Max retries exceeded with url: /listprojects.json (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0a78b2d828>: Failed to establish a new connection: [Errno 111] Connection refused',))
windows：HTTPConnectionPool(host='localhost', port=6801): Max retries exceeded with url: /jobs (Caused by Ne

jumper2014 / lianjia-beike-spider

Star

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新房），稳定可靠快速！支持csv,MySQL, MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

crawler spider lianjia

Updated Jul 24, 2019
Python

lorien / grab

Star

Web Scraping Framework

python framework spider asynchronous network http-client web-scraping pycurl urllib3

Updated Feb 22, 2021
Python

howie6879 / owllook

Star

owllook-小说搜索引擎

spider schedule book sanic aiohttp reader asyncio novels uvloop novels-search owllook

Updated Mar 20, 2021
Python

wkunzhi / Python3-Spider

Star

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

python crawler spider selenium crawl scrapy splash geek taobao scrapy-crawler meituan dianping pyppeteer

Updated Jul 24, 2020
Python

gaojiuli / gain

Star

Web crawling framework based on asyncio.

python crawler spider aiohttp asyncio uvloop

Updated Jun 1, 2019
Python

sjdirect / abot

Star

Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

c-sharp unit-testing crawler spider csharp parsing cross-platform web-crawler netcore log4net takes-care flexibility pluggable spiders csharp-library abot netcore2 netstandard20 netcore3 javascript-renderer netstandard21 abot-nuget icrawldecisionmaker netsta

Updated Jul 16, 2021
C#

hu17889 / go_spider

Star

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

go crawler spider schedule pipeline

Updated Nov 16, 2017
Go

CharlesPikachu / DecryptLogin

Star

APIs for loginning some websites by using requests.

Updated Jun 18, 2021
Python

spider

Here are 2,317 public repositories matching this topic...

facert / awesome-spider

gocolly / colly

jhao104 / proxy_pool

shengqiangzhang / examples-of-web-crawlers

s0md3v / Photon

crawlab-team / crawlab

使用docker-compose部署，访问前端页面请求404

不能使用非crawlab里面mongodb么?

可配置爬虫，在界面上对进行修改，提示保存成功，然而并没有

guyueyingmu / avbook

henrylee2cn / pholcus

luyishisi / Anti-Anti-Spider

bda-research / node-crawler

kangvcar / InfoSpider

SpiderClub / haipproxy

BruceDone / awesome-crawler

tophubs / TopList

gaojiuli / toapi

JAVClub / core

DormyMo / SpiderKeeper

201206030 / fiction_house

Gerapy / Gerapy

shiyanhui / dht

jae-jae / QueryList

my8100 / scrapydweb

User Guide | Q&A | 用户指南 | 问答

jumper2014 / lianjia-beike-spider

lorien / grab

howie6879 / owllook

wkunzhi / Python3-Spider

gaojiuli / gain

sjdirect / abot

hu17889 / go_spider

CharlesPikachu / DecryptLogin

Improve this page

Add this topic to your repo