#

web-scraping

Here are 1,839 public repositories matching this topic...

lorien / awesome-web-scraping

Star

List of libraries, tools and APIs for web scraping and data processing.

Updated Oct 30, 2020
Makefile

autoscraper

alirezamika / autoscraper

Star

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

Updated Nov 27, 2020
Python

php-curl-class / php-curl-class

Star

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

Updated Nov 1, 2020
PHP

apify-js

apify / apify-js

Star

Open

Update main examples to include DOM manipulation

1

mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

Read more

good first issue

Open

dataset.delete() throws unfriendly error when empty

2

Open

Improve error messages

1

Find more good first issues →

lorien / grab

Star

Web Scraping Framework

python framework asynchronous network http-client web-scraping pycurl

Updated Oct 1, 2020
Python

justmarkham / DAT8

Star

General Assembly's 2015 Data Science course in Washington, DC

python data-science machine-learning natural-language-processing course clustering naive-bayes linear-regression scikit-learn jupyter-notebook pandas data-visualization web-scraping data-analysis ensemble-learning logistic-regression decision-trees regular-expressions data-cleaning model-evaluation

Updated Apr 18, 2016
Jupyter Notebook

codingforentrepreneurs / 30-Days-of-Python

Star

A New Version of 30 Days of Python is nearly here. Get started today.

python api flask automation tutorial csv jupyter rest-api selenium pandas python3 web-scraping selenium-webdriver fastapi

Updated Nov 18, 2020
HTML

tidyverse / rvest

Star

Simple web scraping for R

html r web-scraping

Updated Nov 20, 2020
R

go-rod / rod

Star

A Devtools driver for web automation and scraping

scraper automation chrome-devtools headless devtools web-scraping cdp chrome-headless chrome-devtools-protocol devtools-protocol puppeteer

Updated Nov 23, 2020
Go

snoop

snooppr / snoop

Star

Snoop — инструмент разведки на основе открытых данных (OSINT world)

Updated Nov 25, 2020
Python

vprusso / youtube_tutorials

Star

Collection of scripts corresponding to LucidProgramming YouTube tutorials

python python3 web-scraping youtube-tutorial python-tutorial ctci-solutions lucidprogramming python3-tutorial technical-interview

Updated Oct 16, 2020
Python

dinubs / coolqlcool

Star

Nextjs server to query websites with GraphQL

javascript graphql schema nextjs web-scraping

Updated Sep 11, 2020
JavaScript

juancarlospaco / faster-than-requests

Star

Faster requests on Python 3

Updated Nov 11, 2020
Python

alecxe / scrapy-fake-useragent

Star

Random User-Agent middleware based on fake-useragent

python web-scraping scrapy

Updated Sep 17, 2020
Python

AlexMathew / scrapple

Star

A framework for creating semi-automatic web content extractors

python crawler tutorial extractor scraping web-scraper selector css-selector web-scraping scrapy scrapers beautifulsoup xpath-expression lxml selector-expression

Updated Oct 24, 2020
Python

intoli / user-agents

Star

A JavaScript library for generating random user agents with data that's updated daily.

javascript user-agent random randomization navigator web-scraping browsers browser-automation user-agent-spoofer

Updated Nov 24, 2020
JavaScript

A9T9 / RPA

Star

UI.Vision: Open-Source RPA Software (formerly Kantu) - Modern Robotic Process Automation with Selenium IDE++

opencv automation webassembly web-scraping autohotkey browser-extension imacros selenium-ide browser-automation visual-recognition sikulix web-automation ui-tests uipath data-driven-tests

Updated Oct 28, 2020
JavaScript

x4nth055 / pythoncode-tutorials

Star

The Python Code Tutorials

python python-tutorials machine-learning natural-language-processing computer-vision text-classification tutorials python3 web-scraping face-detection scapy network-analysis network-programming programming-tutorial ethical-hacking network-security socket-programming scapy-tutorials

Updated Nov 25, 2020
Jupyter Notebook

rushter / selectolax

Star

Python binding to Modest engine (fast HTML5 parser with CSS selectors).

css python parser html5 web-scraping modest-engine

Updated Nov 1, 2020
Python

VIDA-NYU / ache

Star

ACHE is a web crawler for domain-specific search.

web-crawler web-scraping web-spider focused-crawler domain-specific-search web-search

Updated Nov 16, 2020
Java

infinitbyte / gopa

Star

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

lightweight elasticsearch crawler spider web-crawler scraping crawling web-scraping web-spider

Updated Nov 24, 2019
Go

jaebradley / basketball_reference_web_scraper

Star

NBA Stats API via Basketball Reference

python nba web-scraper web-scraping basketball-reference

Updated Nov 18, 2020
HTML

csu / quora-api

Star

An unofficial API for Quora.

python api flask rest-api web-api web-scraping quora quora-api

Updated Oct 9, 2016
Python

sangaline / wayback-machine-scraper

Star

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

python web-scraping command-line-tool wayback-machine wayback-archiver archive-dot-org

Updated Apr 26, 2019
Python

amoudgl / short-jokes-dataset

Star

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

python humor beautiful-soup jokes dataset web-scraping scrapers oneliners

Updated Oct 1, 2020
Python

hailoc12 / docbao

Star

Công cụ quét và phân tích từ khóa các trang báo mạng Việt Nam

python3 web-scraping newspaper-crawler facebook-crawler made-in-vietnam

Updated Sep 25, 2020
Python

City-Bureau / city-scrapers

Star

Open

Looking for issues to work on? Start here!

pjsier commented Nov 30, 2019

The City Scrapers project is split into multiple repos for different locations. You can see a list of these projects by checking out the city-scrapers topic tag. Issues that are open for contributors are labeled "help wanted".

Check out our [contributing guidelines](https://github.com

Read more

Hacktoberfest good first issue help wanted

Open

Spider: Illinois Finance Authority

18

Open

Spider: Chicago Special Service Area #20 South Western Avenue

9

Find more good first issues →

austinoboyle / scrape-linkedin-selenium

Star

Open

Scrape linkedin posts

3

alaimia commented Feb 26, 2020

Hello,
I need to scrape linkedin POSTS: extract coments, views, frofiles of peoples who interact wth the post...
So please, Austin or anyone else, have you any idea to do it using scrape company !!

Read more

enhancement good first issue help wanted

Open

How to run MyConnectionScrapper ?

1

Open

Feature request - Scrape Alumni, Followers, School URL

justmarkham / trump-lies

Star

Tutorial: Web scraping in Python with Beautiful Soup

python data-science tutorial beautiful-soup pandas requests dataset web-scraping

Updated Nov 18, 2018
Jupyter Notebook

yusuzech / r-web-scraping-cheat-sheet

Star

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

r web-scraping cheatsheet webscraping rvest scrape-websites httr rselenium

Updated May 27, 2020
R

Improve this page

Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.