Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Affiliate Marketer – Crypto Education Platform

I'm looking for experienced affiliate marketers to promote Coiniq, an online crypto education platform built for beginner and intermediate traders. About Coiniq: Users get full access to structured crypto courses, AI-assisted learning, live trading examples,...

The Delicate Balance of Hygiene and Aroma

  SanMar Building Services provides professional commercial and office cleaning services in New York City for small and mid-size buildings, retail spaces, and offices. We are licensed, insured, and bonded, delivering reliable cleaning solutions tailored...

Blockchain Integration for Unified Comms

About Unified Comms: Unified Comms is a multi-service communication platform that brings together popular messaging tools like Slack, Gmail, WhatsApp, and more into a single, AI-driven workspace. Our mission is to streamline communication, improve productivity,...

Frontend Developer / Layout Developer (Elementor)

MATCH PAGE REDESIGN + FRONTEND DEVELOPMENT IN ELEMENTOR (Desktop & Mobile Versions) A redesign and frontend implementation of the match page is required in Elementor. After approval of your candidacy, we will provide: the full...

Full Stack Web3 Developer

We are seeking a highly skilled Full Stack Web3 Developer to design, develop, and scale a modern blockchain-powered waitlist and engagement platform built with Next.js, Supabase, and BSC Testnet integration.   The ideal candidate has...

Premium Companionship and Service Discovery in Thailand

When navigating the diverse world of local services and personal companionship, finding a platform that bridges the gap between digital listings and real-world reliability is paramount. Whether you are searching for a social companion for...

Seeking a partner to bring users to our site

Hello everyone, We have a website where we organize paid raffles/giveaways. We are looking for teammates who have marketing knowledge or an audience/communit and can attract people to join these raffles. Once the required number...

TokenTrack: Real-Time Market & Strategy Dashboard

A full-stack crypto trading platform with real-time market intelligence and automated trading bots. Tirixa is a production-ready Next.js application that brings institutional-grade trading tools to individual crypto traders. Monitor multi-chain markets, automate trading strategies, backtest...

Remote Person to Make Phone Calls | For Web Development Firm

We are hiring a person with an American accent to make phone calls for our US-based web development firm. Your Position: We are a growing web development firm seeking to fill a commission-based position. The...

Looking for a Virtual Assistant

💼 Looking for a Virtual Assistant We are currently looking for a reliable Virtual Assistant to help with email communication and basic online tasks. 🌍 Locations: United States Europe Canada Brazil Philippines Mexico Australia 📧...

Affiliate Marketer – Crypto Education Platform

I'm looking for experienced affiliate marketers to promote Coiniq, an online crypto education platform built for beginner and intermediate traders. About Coiniq: Users get full access to structured crypto courses, AI-assisted learning, live trading examples,...

The Delicate Balance of Hygiene and Aroma

  SanMar Building Services provides professional commercial and office cleaning services in New York City for small and mid-size buildings, retail spaces, and offices. We are licensed, insured, and bonded, delivering reliable cleaning solutions tailored...

Blockchain Integration for Unified Comms

About Unified Comms: Unified Comms is a multi-service communication platform that brings together popular messaging tools like Slack, Gmail, WhatsApp, and more into a single, AI-driven workspace. Our mission is to streamline communication, improve productivity,...

Frontend Developer / Layout Developer (Elementor)

MATCH PAGE REDESIGN + FRONTEND DEVELOPMENT IN ELEMENTOR (Desktop & Mobile Versions) A redesign and frontend implementation of the match page is required in Elementor. After approval of your candidacy, we will provide: the full...

Full Stack Web3 Developer

We are seeking a highly skilled Full Stack Web3 Developer to design, develop, and scale a modern blockchain-powered waitlist and engagement platform built with Next.js, Supabase, and BSC Testnet integration.   The ideal candidate has...

Premium Companionship and Service Discovery in Thailand

When navigating the diverse world of local services and personal companionship, finding a platform that bridges the gap between digital listings and real-world reliability is paramount. Whether you are searching for a social companion for...

Seeking a partner to bring users to our site

Hello everyone, We have a website where we organize paid raffles/giveaways. We are looking for teammates who have marketing knowledge or an audience/communit and can attract people to join these raffles. Once the required number...

TokenTrack: Real-Time Market & Strategy Dashboard

A full-stack crypto trading platform with real-time market intelligence and automated trading bots. Tirixa is a production-ready Next.js application that brings institutional-grade trading tools to individual crypto traders. Monitor multi-chain markets, automate trading strategies, backtest...

Remote Person to Make Phone Calls | For Web Development Firm

We are hiring a person with an American accent to make phone calls for our US-based web development firm. Your Position: We are a growing web development firm seeking to fill a commission-based position. The...

Looking for a Virtual Assistant

💼 Looking for a Virtual Assistant We are currently looking for a reliable Virtual Assistant to help with email communication and basic online tasks. 🌍 Locations: United States Europe Canada Brazil Philippines Mexico Australia 📧...