#

web-scraping

Here are 1,657 public repositories matching this topic...

lorien / awesome-web-scraping

Star

List of libraries, tools and APIs for web scraping and data processing.

Updated Jun 28, 2020
Makefile

php-curl-class / php-curl-class

Star

PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

Updated May 23, 2020
PHP

apify / apify-js

Star

Open

Update main examples to include DOM manipulation

1

mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

Read more

good first issue

Open

dataset.delete() throws unfriendly error when empty

2

Open

Improve error messages

1

Find more good first issues →

lorien / grab

Star

Web Scraping Framework

python framework asynchronous network http-client web-scraping pycurl

Updated May 7, 2020
Python

justmarkham / DAT8

Star

General Assembly's 2015 Data Science course in Washington, DC

python data-science machine-learning natural-language-processing course clustering naive-bayes linear-regression scikit-learn jupyter-notebook pandas data-visualization web-scraping data-analysis ensemble-learning logistic-regression decision-trees regular-expressions data-cleaning model-evaluation

Updated Apr 18, 2016
Jupyter Notebook

codingforentrepreneurs / 30-Days-of-Python

Star

A New Version of 30 Days of Python is nearly here. Get started today.

python api flask automation tutorial csv jupyter rest-api selenium pandas python3 web-scraping selenium-webdriver fastapi

Updated Jul 29, 2020
Jupyter Notebook

tidyverse / rvest

Star

Simple web scraping for R

html r web-scraping

Updated Jul 27, 2020
R

go-rod / rod

Star

A Devtools driver to make web automation and scraping easy

golang chrome scraper automation chrome-devtools headless devtools web-scraping cdp chrome-headless rod chrome-devtools-protocol puppeteer chromedp chrome-devtools-driver

Updated Aug 28, 2020
Go

snoop

snooppr / snoop

Star

Snoop — инструмент разведки на основе открытых данных (OSINT world)

windows linux osint python3 web-scraping ctf termux pentest username-checker intelligence-service username-search search-magic

Updated Aug 12, 2020
Python

vprusso / youtube_tutorials

Star

Collection of scripts corresponding to LucidProgramming YouTube tutorials

python python3 web-scraping youtube-tutorial python-tutorial ctci-solutions lucidprogramming python3-tutorial technical-interview

Updated Aug 18, 2020
Python

dinubs / coolqlcool

Star

Nextjs server to query websites with GraphQL

javascript graphql schema nextjs web-scraping

Updated Jul 30, 2020
JavaScript

alecxe / scrapy-fake-useragent

Star

Random User-Agent middleware based on fake-useragent

python web-scraping scrapy

Updated Aug 15, 2020
Python

juancarlospaco / faster-than-requests

Star

Faster requests on Python 3

Updated Aug 12, 2020
Python

AlexMathew / scrapple

Star

A framework for creating semi-automatic web content extractors

python crawler tutorial extractor scraping web-scraper selector css-selector web-scraping scrapy scrapers beautifulsoup xpath-expression lxml selector-expression

Updated Oct 12, 2019
Python

intoli / user-agents

Star

A JavaScript library for generating random user agents with data that's updated daily.

javascript user-agent random randomization navigator web-scraping browsers browser-automation user-agent-spoofer

Updated Aug 2, 2020
JavaScript

A9T9 / Kantu

Star

UI.Vision RPA (formerly Kantu) - Modern Robotic Process Automation plus Selenium IDE++

opencv automation webassembly web-scraping autohotkey browser-extension imacros selenium-ide browser-automation visual-recognition sikulix web-automation ui-tests uipath data-driven-tests

Updated Jul 30, 2020
JavaScript

rushter / selectolax

Star

Python binding to Modest engine (fast HTML5 parser with CSS selectors).

css python parser html5 web-scraping modest-engine

Updated Aug 16, 2020
Python

VIDA-NYU / ache

Star

ACHE is a web crawler for domain-specific search.

web-crawler web-scraping web-spider focused-crawler domain-specific-search web-search

Updated Aug 18, 2020
Java

x4nth055 / pythoncode-tutorials

Star

The Python Code Tutorials

python python-tutorials machine-learning natural-language-processing computer-vision text-classification tutorials python3 web-scraping face-detection scapy network-analysis network-programming programming-tutorial ethical-hacking network-security socket-programming scapy-tutorials

Updated Aug 23, 2020
Jupyter Notebook

infinitbyte / gopa

Star

[WIP] GOPA, a spider written in Golang, for Elasticsearch. DEMO: http://index.elasticsearch.cn

lightweight elasticsearch crawler spider web-crawler scraping crawling web-scraping web-spider

Updated Nov 24, 2019
Go

csu / quora-api

Star

An unofficial API for Quora.

python api flask rest-api web-api web-scraping quora quora-api

Updated Oct 9, 2016
Python

jaebradley / basketball_reference_web_scraper

Star

NBA Stats API via Basketball Reference

python nba web-scraper web-scraping basketball-reference

Updated Aug 27, 2020
HTML

amoudgl / short-jokes-dataset

Star

Python scripts for building 'Short Jokes' dataset, featured on Kaggle

python humor beautiful-soup jokes dataset web-scraping scrapers oneliners

Updated Oct 24, 2019
Python

sangaline / wayback-machine-scraper

Star

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

python web-scraping command-line-tool wayback-machine wayback-archiver archive-dot-org

Updated Apr 26, 2019
Python

City-Bureau / city-scrapers

Star

Open

Looking for issues to work on? Start here!

pjsier commented Nov 30, 2019

The City Scrapers project is split into multiple repos for different locations. You can see a list of these projects by checking out the city-scrapers topic tag. Issues that are open for contributors are labeled "help wanted".

Check out our [contributing guidelines](https://github.com

Read more

Hacktoberfest good first issue help wanted

Open

Spider: Illinois Finance Authority

15

Open

Spider: Chicago Special Service Area #20 South Western Avenue

5

Find more good first issues →

justmarkham / trump-lies

Star

Tutorial: Web scraping in Python with Beautiful Soup

python data-science tutorial beautiful-soup pandas requests dataset web-scraping

Updated Nov 18, 2018
Jupyter Notebook

yusuzech / r-web-scraping-cheat-sheet

Star

Guide, reference and cheatsheet on web scraping using rvest, httr and Rselenium.

r web-scraping cheatsheet webscraping rvest scrape-websites httr rselenium

Updated May 27, 2020
R

jrbadiabo / Bet-on-Sibyl

Star

Machine Learning Model for Sport Predictions (Football, Basketball, Baseball, Hockey, Soccer & Tennis)

python machine-learning algorithms scikit-learn machine-learning-algorithms selenium web-scraping beautifulsoup machinelearning predictive-analysis python-2 web-crawling sports-stats sportsanalytics

Updated Feb 12, 2017
Jupyter Notebook

austinoboyle / scrape-linkedin-selenium

Star

Open

Scrape linkedin posts

3

alaimia commented Feb 26, 2020

Hello,
I need to scrape linkedin POSTS: extract coments, views, frofiles of peoples who interact wth the post...
So please, Austin or anyone else, have you any idea to do it using scrape company !!

Read more

enhancement good first issue help wanted

Open

How to run MyConnectionScrapper ?

1

Open

Feature request - Scrape Alumni, Followers, School URL

batuhaniskr / twitter-intelligence

Star

Twitter Intelligence OSINT project performs tracking and analysis of the Twitter

python flask osint analysis google-maps web-scraping tweet social-network-analysis

Updated Mar 6, 2020
Python

Improve this page

Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.