Skip to content
#

web-crawling

Star

Here are 176 public repositories matching this topic...

Language: All
Filter by language
All 176 Python 85 Jupyter Notebook 21 JavaScript 13 Java 12 HTML 10 Go 4 C# 3 C++ 3 Julia 3 PHP 3
Sort: Best match
Sort options
Best match Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated
apify-js

apify / apify-js

Star
Open

Update main examples to include DOM manipulation

1
mtrunkat
mtrunkat commented Sep 17, 2019

Main examples at Apify SDK webpage, Github repo and CLI templates should demonstrate how to manipulate with DOM and retrieve data from it.

Also add one example of scraping with Apify SDK + jQuery to https://sdk.apify.com/docs/examples/basiccrawler

Feedback from: https://medium.com/better-programming/do-i-need-python-scrapy-to-build-a-web-scraper-7cc7cac2081d

I lost an hour trying to make

Read more
good first issue
Find more good first issues
ayakashi

ayakashi-io / ayakashi

Star
Open

Add support for delivering results to a URL (webhook)

zisismaras
zisismaras commented May 28, 2019

There is already builtin support for saving to SQL, JSON, CSV and printing to the console.
We should also have a webhook script that delivers the results to a specified url in the config.

More info about the existing scripts can be found here:
https://ayakashi.io/docs/guide/builtin-saving-scripts.html

A good starting point is the printToConsole script:
https://github.com/ayakashi-io/a

Read more
enhancement good first issue

mirkomantovani / web-search-engine-UIC

Star

CS 582 Information Retrieval at University of Illinois at Chicago. Multithreaded crawling of UIC domain, inverted index, page rank, SEO with Context Pseudo-Relevance Feedback

python search-engine data-science information-retrieval research seo crawling pagerank inverted-index tf-idf cosine-similarity web-crawling query-expansion retrieve-documents search-engine-optimization pseudo-relevance-feedback page-rank
  • Updated Dec 15, 2018
  • Python

Improve this page

Add a description, image, and links to the web-crawling topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the web-crawling topic, visit your repo's landing page and select "manage topics."

Learn more