A collection of awesome web crawler,spider in different languages
-
Updated
May 29, 2021
#
web-scraper
Here are 525 public repositories matching this topic...
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api
php
http
client
json
framework
curl
xml
proxy
restful
class
http-client
http-proxy
api-client
web-scraper
requests
web-scraping
php-curl
web-service
php-curl-library
-
Updated
Jul 24, 2021 - PHP
Web Scraper in Go, similar to BeautifulSoup
-
Updated
Sep 24, 2020 - Go
A list of practical knowledge-building projects.
javascript
python
processing
c
java
search-engine
music-player
programming
csharp
projects
web-scraper
cpp11
-
Updated
Jun 4, 2021
-
Updated
Jul 25, 2021 - JavaScript
Faster requests on Python 3
python
curl
high-performance
cython
python-library
web-scraper
python3
speed
open-data
http-requests
web-scraping
scrapy
ndjson
python-requests
urllib
download-file
urllib3
faster-than-requests
requests3
requests-toolbelt
-
Updated
Jun 23, 2021 - Nim
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
-
Updated
Jun 23, 2021 - Ruby
Bulk download your favourite anime episodes from your favourite anime websites
ffmpeg
anime
web-scraper
anime-search
anime-fans
anime-downloader
anime-scraper
9anime
hls-downloader
animeultima
animepahe-downloader
animepahe
4anime
monkey-dl
-
Updated
May 30, 2021 - Python
A framework for creating semi-automatic web content extractors
python
crawler
tutorial
extractor
scraping
web-scraper
selector
css-selector
web-scraping
scrapy
scrapers
beautifulsoup
xpath-expression
lxml
selector-expression
-
Updated
Oct 24, 2020 - Python
Download and generate e-books from online sources.
-
Updated
Jul 16, 2021 - Python
A list of scrapers from around the web.
-
Updated
Jul 17, 2021
NBA Stats API via Basketball Reference
-
Updated
Jul 20, 2021 - HTML
A simple browser/client-side web scraper.
-
Updated
Apr 24, 2017 - TypeScript
A Reddit bot that summarizes news articles written in Spanish or English. It uses a custom built algorithm to rank words and sentences.
-
Updated
Apr 17, 2021 - Python
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
web-scraper
web-scraping
newsletter
reuters
bloomberg
futures
web-scrapers
scrapper
financial-data
news-websites
data-scraping
news-scraper
futures-historical-data
data-scraper
sraping
python-web-scraper
financial-times
options-data
wall-street-journal
wallstreetbets
-
Updated
Jun 28, 2021 - Python
Web scraping library and command-line tool for text discovery and extraction (main content, metadata, comments)
nlp
crawler
text-mining
scraper
news
scraping
web-scraper
text-extraction
web-scraping
readability
tei-xml
news-articles
html2text
article-extractor
news-scraper
text-cleaning
text-preprocessing
-
Updated
Jul 23, 2021 - Python
spekulatius
commented
Dec 6, 2020
The image URL should be returned, if this the only part of the anchor texts.
A collection of awesome web scaper, crawler.
-
Updated
Mar 7, 2021
MetaData html scraper and parser for Node.js (supports Promises and callback style)
-
Updated
Aug 15, 2020 - JavaScript
Instagram Bot which when given a post url will spam mentions to increase the chances of winning. Win Instagram Giveaways!
python
instagram
spam
winning
selenium
macros
web-scraper
python3
cheating
posts
instagram-scraper
comments
mentions
selenium-webdriver
hacktoberfest
giveaways
instagram-bot
selenium-python
instagram-script
hacktoberfest2020
-
Updated
Mar 18, 2021 - Python
Fetch user's data across social media
python
social-media
pinterest
web-scraper
web-scraping
request
instagram-scraper
twitter-scraper
facebook-scraper
scrapping-python
selenium-python
reddit-scraper
quora-scraper
tiktok-scraper
medium-scraper
pinterest-scrapper
-
Updated
Mar 6, 2021 - Python
PHP Library for detecting CMS
-
Updated
Feb 24, 2020 - PHP
Go cascadia package command line CSS selector
tsv
command-line
curl
extract
web-scraper
css-selector
web-scraping
html-source
command-line-tool
csv-table
cascadia
html-text
-
Updated
Jul 17, 2021 - Go
Powerful web scraping framework for Crystal
-
Updated
Jun 21, 2020 - Crystal
Adult XXX Addons (18+) for the Kodi Media Center - Kodi is a registered trademark of the XBMC Foundation. We are not connected to or in any other way affiliated with Kodi - DMCA: legal@tvaddons.co
-
Updated
Jul 22, 2021 - Python
A command line interface for downloading Bollywood and punjabi songs
python
music
youtube
mp3
python-script
songs
web-scraper
download-songs
music-download
singer
bollywood
tqdm
hollywood
song-downloader
song-download-script
music-download-script
song-pypi
mr-jatt
top-songs
song-download
-
Updated
Jan 9, 2020 - Python
A simple python library that allows for easy access of the SEC website so that someone can parse filings, collect data, and query documents.
-
Updated
Jul 25, 2021 - Python
Web Scraping Craigslist's Engineering Jobs in NY with Scrapy
python
scrapy-spider
web-scraper
craigslist
web-scraping
scrapy
web-crawling
scrapy-crawler
scrapy-tutorial
-
Updated
Aug 5, 2017 - Python
Improve this page
Add a description, image, and links to the web-scraper topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraper topic, visit your repo's landing page and select "manage topics."
Hello,
Thanks for new update in personal_info section,
I found out that the attribute 'certifications' return empty list []
Test url:
https://www.linkedin.com/in/an-nguyen-9b3248122/Results:
`{'personal_info': {'name': 'An Nguyen',
'headline': 'Data Scientist/Machine Learning Engineer',
'company': 'PERSOL PROCESS & TECHNOLOGY CO., LTD.',
'school': 'National Chiao Tung University',