A collection of awesome web crawler,spider in different languages
-
Updated
Mar 16, 2021
#
web-scraper
Here are 461 public repositories matching this topic...
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api
php
http
client
json
framework
curl
xml
proxy
restful
class
http-client
http-proxy
api-client
web-scraper
requests
web-scraping
php-curl
web-service
php-curl-library
-
Updated
Mar 23, 2021 - PHP
Web Scraper in Go, similar to BeautifulSoup
-
Updated
Sep 24, 2020 - Go
A list of practical knowledge-building projects.
javascript
python
processing
c
java
search-engine
music-player
programming
csharp
projects
web-scraper
cpp11
-
Updated
Oct 18, 2020
-
Updated
Mar 20, 2021 - JavaScript
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
-
Updated
Mar 1, 2021 - Ruby
Faster requests on Python 3
python
curl
high-performance
cython
python-library
web-scraper
python3
speed
open-data
http-requests
web-scraping
scrapy
ndjson
python-requests
urllib
download-file
urllib3
faster-than-requests
requests3
requests-toolbelt
-
Updated
Feb 24, 2021 - Nim
A framework for creating semi-automatic web content extractors
python
crawler
tutorial
extractor
scraping
web-scraper
selector
css-selector
web-scraping
scrapy
scrapers
beautifulsoup
xpath-expression
lxml
selector-expression
-
Updated
Oct 24, 2020 - Python
Bulk download your favourite anime episodes from your favourite anime websites
ffmpeg
anime
web-scraper
anime-search
anime-fans
anime-downloader
anime-scraper
9anime
hls-downloader
animeultima
animepahe-downloader
animepahe
4anime
monkey-dl
-
Updated
Mar 9, 2021 - Python
A list of scrapers from around the web.
-
Updated
Aug 20, 2020
Download and generate e-books from online sources.
-
Updated
Mar 21, 2021 - Python
NBA Stats API via Basketball Reference
-
Updated
Mar 23, 2021 - HTML
A simple browser/client-side web scraper.
-
Updated
Apr 24, 2017 - TypeScript
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
Updated
Oct 31, 2020 - Python
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, SHFE and news data crawlers on BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
web-scraper
web-scraping
newsletter
reuters
bloomberg
futures
web-scrapers
scrapper
financial-data
news-websites
data-scraping
news-scraper
futures-historical-data
data-scraper
sraping
python-web-scraper
financial-times
options-data
wall-street-journal
wallstreetbets
-
Updated
Feb 9, 2021 - Python
spekulatius
commented
Dec 6, 2020
The image URL should be returned, if this the only part of the anchor texts.
A collection of awesome web scaper, crawler.
-
Updated
Mar 7, 2021
MetaData html scraper and parser for Node.js (supports Promises and callback style)
-
Updated
Aug 15, 2020 - JavaScript
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
nlp
crawler
text-mining
scraper
news
scraping
web-scraper
text-extraction
web-scraping
readability
tei-xml
news-articles
html2text
article-extractor
news-scraper
text-cleaning
text-preprocessing
-
Updated
Mar 18, 2021 - Python
Open
Add Estate Agent
AnthonyBloomer
commented
Sep 3, 2018
Scrape estate agent information. Example estate agent page: https://www.daft.ie/dublin/estate-agents/dublin-2/herlihy/
PHP Library for detecting CMS
-
Updated
Feb 24, 2020 - PHP
OnlyFans content downloader
-
Updated
Mar 18, 2021 - Python
Powerful web scraping framework for Crystal
-
Updated
Jun 21, 2020 - Crystal
Go cascadia package command line CSS selector
tsv
command-line
curl
extract
web-scraper
css-selector
web-scraping
html-source
command-line-tool
csv-table
cascadia
html-text
-
Updated
Apr 19, 2020 - Go
Fetch user's data across social media
python
social-media
pinterest
web-scraper
web-scraping
request
instagram-scraper
twitter-scraper
facebook-scraper
scrapping-python
selenium-python
reddit-scraper
quora-scraper
tiktok-scraper
medium-scraper
pinterest-scrapper
-
Updated
Mar 6, 2021 - Python
Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!
python
instagram
spam
winning
selenium
macros
web-scraper
python3
cheating
posts
instagram-scraper
comments
mentions
selenium-webdriver
hacktoberfest
giveaways
instagram-bot
selenium-python
instagram-script
hacktoberfest2020
-
Updated
Mar 18, 2021 - Python
A command line interface for downloading Bollywood and punjabi songs
python
music
youtube
mp3
python-script
songs
web-scraper
download-songs
music-download
singer
bollywood
tqdm
hollywood
song-downloader
song-download-script
music-download-script
song-pypi
mr-jatt
top-songs
song-download
-
Updated
Jan 9, 2020 - Python
Adult XXX Addons (18+) for the Kodi Media Center - Kodi is a registered trademark of the XBMC Foundation. We are not connected to or in any other way affiliated with Kodi - DMCA: legal@tvaddons.co
-
Updated
Jan 21, 2021 - Python
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
python
scrapy-spider
web-scraper
craigslist
web-scraping
scrapy
web-crawling
scrapy-crawler
scrapy-tutorial
-
Updated
Aug 5, 2017 - Python
Improve this page
Add a description, image, and links to the web-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraper topic, visit your repo's landing page and select "manage topics."
unable to get the company website
Company Attrs: ['overview', 'jobs', 'life', 'insights']
Error!!!
Traceback (most recent call last):
File "linkedin_scrapper_chrome.py", line 72, in
company_overview_dict = company.to_dict()
File "/xxx/xxxx/linkedin_scrapper/scrape_linkedin/ResultsObject.py", line 13, in to_dict
return dict(zip(self.attributes, vals))
File "/xxxx/xxxx