web-crawler-python

Web Crawler

Parser and database to index the terpene profile of different strains of Cannabis from online databases

Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.

Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart 💰 📊

A utility for crawling websites and building frequency lists of words

Explore a website recursively and download all the wanted documents (PDF, ODT…)

Web Scraping using Python Data mining , Data Analyzing & Data Visualization of the collected Data, The python script is written to fetch all the individual categories the website , The code is written for fetching the data from the first page and it iterates to each and every pages of website ( activities, categories, count of bought), and I used statistical techniques for mathematically analysis and presenting the data into visualization

📻 An OLX Scraper using Scrapy + MongoDB. It Scrapes recent ads posted regarding requested product and dumps to NOSQL MONGODB.

Zoominfo scraper with using of rotating proxies and headless Chrome from ScrapingAnt

This repo is mainly for dynamic web (Ajax Tech) crawling using Python, taking China's NSTL websites as an example.

a fully functional spider for aliexpress.com

Web Crawler and Scraper

Open-source web crawler

This script will dump youtube video comments to a CSV from youtube video links. Video links can be placed inside a variable or list or CSV

텍스트 분석용 데이터 수집을 위한 웹스크래핑 도구를 제공합니다.

A Web Crawler developed in Python.

A web crawler which crawls the stackoverflow website.

Traverses DA Intranet for file

Alibaba scraper with using of rotating proxies and headless Chrome from ScrapingAnt

Crawler for Cantonese pronunciation data on LSHK Jyutping Word List (香港語言學學會粵拼詞表)

Nemoscan is a script For Get Information About Targets Using Online API That Perform Speed Nmap, geoip ,dnslookup,whois,reverse_ip_lookup include In a directory-fuzzer

Automated script that navigates the World Wide Web in a methodical and automated way for automatic searches on Google

Image-scrapper-from-Google-image-search

A scraper built using Scrapy+Python that can quickly get a list of mot law enforcement agencies in the US using the PoliceOne.com directory

Web crawling application which finds the most efficient internal wikipedia webpage connection between two wikipedia webpages

Scrape each of the Natural Product Present on drugs.com

manga_down is a tool to download manga from mangareader and mangapanda

CookieBlade is a platform for users to keep track of their own or other’s social media statistics.

𝐓𝐡𝐞 𝐩𝐮𝐫𝐩𝐨𝐬𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐜𝐨𝐝𝐞 𝐢𝐧 𝐭𝐡𝐢𝐬 𝐫𝐞𝐩𝐨𝐬𝐢𝐭𝐨𝐫𝐲 𝐢𝐬 𝐟𝐨𝐫 𝐝𝐞𝐦𝐨𝐧𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐧𝐥𝐲, 𝐭𝐡𝐞 𝐬𝐜𝐫𝐢𝐩𝐭𝐬 𝐛𝐲 𝐭𝐡𝐞𝐦𝐬𝐞𝐥𝐯𝐞𝐬 𝐝𝐨 𝐧𝐨𝐭 𝐝𝐨 𝐚𝐧𝐲𝐭𝐡𝐢𝐧𝐠, 𝐛𝐞𝐜𝐚𝐮𝐬𝐞 𝐩𝐮𝐫𝐩𝐨𝐬𝐞𝐥𝐲 𝐦𝐚𝐧𝐲 𝐨𝐭𝐡𝐞𝐫 𝐟𝐨𝐥𝐝𝐞𝐫𝐬 𝐚𝐧𝐝 𝐝𝐚𝐭𝐚 𝐟𝐢𝐥𝐞𝐬 𝐚𝐫𝐞 𝐧𝐨𝐭 𝐢𝐧𝐜𝐥𝐮𝐝𝐞𝐝 𝐢𝐧 𝐨𝐫𝐝𝐞𝐫 𝐭𝐨 𝐧𝐨𝐭 𝐯𝐢𝐨𝐥𝐚𝐭𝐞 𝐚𝐧𝐲 𝐈𝐏𝐎 𝐨𝐫 𝐩𝐫𝐢𝐯𝐚𝐭𝐞 𝐝𝐚𝐭𝐚 𝐨𝐟 𝐚𝐧 𝐨𝐫𝐠𝐚𝐧𝐢𝐬𝐚𝐭𝐢𝐨𝐧. 𝐇𝐨𝐰𝐞𝐯𝐞𝐫, 𝐭𝐡𝐞 𝐜𝐨𝐝𝐞 𝐢𝐬 𝐬𝐭𝐢𝐥𝐥 𝐮𝐬𝐞𝐟𝐮𝐥 𝐟𝐨𝐫 𝐭𝐞𝐱𝐭 𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐢𝐝𝐞𝐚𝐬 𝐚𝐧𝐝 𝐨𝐩𝐞𝐧 𝐭𝐨 𝐭𝐡𝐞 𝐩𝐮𝐛𝐥𝐢𝐜, 𝐭𝐨 𝐚𝐬𝐬𝐢𝐬𝐭 𝐚𝐧𝐝 𝐦𝐨𝐭𝐢𝐯𝐚𝐭𝐞 𝐚𝐧𝐲𝐨𝐧𝐞 𝐢𝐧𝐭𝐞𝐫𝐞𝐬𝐭𝐞𝐝 𝐢𝐧 𝐭𝐞𝐱𝐭 𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠.

Web search engine to retrieve most relevant web-pages for user search query from web-pages crawled on the UIC domain