A collection of awesome web crawler,spider in different languages
-
Updated
Nov 3, 2021
#
web-scraper
Here are 573 public repositories matching this topic...
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api
php
http
client
json
framework
curl
xml
proxy
restful
class
http-client
http-proxy
api-client
web-scraper
requests
web-scraping
php-curl
web-service
php-curl-library
-
Updated
Mar 17, 2022 - PHP
Web Scraper in Go, similar to BeautifulSoup
-
Updated
Jan 18, 2022 - Go
A list of practical knowledge-building projects.
javascript
python
processing
c
java
search-engine
music-player
programming
csharp
projects
web-scraper
cpp11
-
Updated
Jun 4, 2021
Faster requests on Python 3
python
curl
high-performance
cython
python-library
web-scraper
python3
speed
open-data
http-requests
web-scraping
scrapy
ndjson
python-requests
urllib
download-file
urllib3
faster-than-requests
requests3
requests-toolbelt
-
Updated
May 27, 2022 - Nim
-
Updated
Mar 1, 2022 - JavaScript
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
-
Updated
May 10, 2022 - Ruby
Generate and download e-books from online sources.
-
Updated
May 30, 2022 - Python
Bulk download your favourite anime episodes from your favourite anime websites
ffmpeg
anime
web-scraper
anime-search
anime-fans
anime-downloader
anime-scraper
9anime
hls-downloader
animeultima
animepahe-downloader
animepahe
4anime
monkey-dl
-
Updated
May 30, 2021 - Python
A framework for creating semi-automatic web content extractors
python
crawler
tutorial
extractor
scraping
web-scraper
selector
css-selector
web-scraping
scrapy
scrapers
beautifulsoup
xpath-expression
lxml
selector-expression
-
Updated
Oct 24, 2020 - Python
A list of scrapers from around the web.
-
Updated
Mar 16, 2022
bug
Something isn't working
help wanted
Extra attention is needed
good first issue
Good for newcomers
OnlyFans content downloader
-
Updated
May 12, 2022 - Python
NBA Stats API via Basketball Reference
-
Updated
Mar 7, 2022 - HTML
A simple browser/client-side web scraper.
-
Updated
Apr 24, 2017 - TypeScript
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
Updated
Mar 2, 2022 - Python
spekulatius
commented
Aug 28, 2020
It would be beneficial to return the URL of the sitemap.xml file directly.
A collection of awesome web scaper, crawler.
-
Updated
Mar 7, 2021
Fetch user's data across social media
python
social-media
pinterest
web-scraper
web-scraping
request
instagram-scraper
twitter-scraper
facebook-scraper
scrapping-python
selenium-python
reddit-scraper
quora-scraper
tiktok-scraper
medium-scraper
pinterest-scrapper
-
Updated
Mar 18, 2022 - Python
MetaData html scraper and parser for Node.js (supports Promises and callback style)
-
Updated
Feb 1, 2022 - JavaScript
Go cascadia package command line CSS selector
tsv
command-line
curl
extract
web-scraper
css-selector
web-scraping
html-source
command-line-tool
csv-table
cascadia
html-text
-
Updated
Jan 2, 2022 - Go
Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!
python
instagram
spam
winning
selenium
macros
web-scraper
python3
cheating
posts
instagram-scraper
comments
mentions
selenium-webdriver
hacktoberfest
giveaways
instagram-bot
selenium-python
instagram-script
hacktoberfest2020
-
Updated
Nov 29, 2021 - Python
PHP Library for detecting CMS
-
Updated
Feb 24, 2020 - PHP
Powerful web scraping framework for Crystal
-
Updated
Jan 24, 2022 - Crystal
A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.
-
Updated
Feb 21, 2022 - Python
sudiptosarkar
commented
Oct 24, 2017
Yes, I know the class name needs refactor. We need a test class made for this class.
Tests should not make real calls. Instead, calls must be mocked with Mockito and/or PowerMockito.
Examples of the testing pattern can be found in the other three dependency repos mentioned in the README.md.
A command line interface for downloading Bollywood and punjabi songs
python
music
youtube
mp3
python-script
songs
web-scraper
download-songs
music-download
singer
bollywood
tqdm
hollywood
song-downloader
song-download-script
music-download-script
song-pypi
mr-jatt
top-songs
song-download
-
Updated
Jan 9, 2020 - Python
Lightweight scraper for Google News
crawler
news
web-crawler
web-scraper
news-articles
news-crawler
google-news
google-news-scraper
news-scraper
google-crawler
-
Updated
May 12, 2022 - JavaScript
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
python
scrapy-spider
web-scraper
craigslist
web-scraping
scrapy
web-crawling
scrapy-crawler
scrapy-tutorial
-
Updated
Aug 5, 2017 - Python
TRScraper, doğal dil işleme uygulamalarında kullanılmak amacıyla geliştirilmiş, Türkçe içerik girilen büyük platformlarda metin madenciliği yapma imkanı sunan bir uygulamadır.
nlp
text-mining
scraper
turkish
selenium
web-scraper
dataset
datasets
dataset-generation
turkce
selenium-python
turkce-kaynak
dogal-dil-isleme
turkish-nlp
-
Updated
Feb 16, 2021 - Python
Improve this page
Add a description, image, and links to the web-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraper topic, visit your repo's landing page and select "manage topics."
It seems the scrapper doesn't work at all anymore to retrieve experiences for linkedIn profiles.