Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Extending Structural Rejuvenation to the Neck and Chest

A highly rigorous, carefully planned facial skincare routine is a staple for most individuals invested in their physical appearance. We meticulously apply heavy sunscreens, highly potent antioxidant serums, and rich moisturisers to the face every...

Advertisement for website

Hello everyone, Today We have a website where we organize paid raffles/giveaways. We are looking for teammates who have marketing knowledge or an audience/community and can attract people to join these raffles. Once the required...

The Saas Hub

  Meta Description: Learn why relying on manual processes is costing your online store time and money and how modern apps can streamline your workflow for better profitability.

Smart Sat Connect

Expert Starlink Installation, Saorview Installation, Satellite Dish Installation, Free Satellite TV Installation, WiFi Distribution, TV mounting, and Mobile Phone Signal Booster Ireland services. We ensure that your home or business stays connected through our fast,...

Megan Moura Photography

Oahu wedding photographer Honolulu. Wedding photographer Hawaii. Kauai, Maui & Oahu. Photography for weddings, elopements, special events, branding and family.

مترجم ومحلل في أسواق مالية

Hello, I am a motivated and active person with a strong interest in crypto and Web3 projects. I have experience using Telegram and Discord daily, and I am comfortable communicating with community members and helping...

Broker / Sales Agent – CryptoTax-Pro (Digital Asset)

Category: Broker, Crypto Broker, Web3 Sales Agent, M&A Broker   Objectives Identify and qualify crypto traders, investors, and small fund managers as potential clients Present CryptoTax-Pro using provided sales materials Build relationships with decision makers...

B2B Sales Representative — Review Management SaaS

Title: B2B Sales Representative — Review Management SaaS ($500 Per Closed Sale) Objectives Identify and qualify agencies, consultants, and business service providers as potential clients Present our reputation management SaaS using provided sales materials Close...

B2B Sales Representative — Accounting SaaS Product

Title: B2B Sales Representative — Accounting SaaS ($1,000 Per Closed Sale) Objectives Identify and qualify CPA firms and bookkeeping agencies as potential clients Present our SaaS solution using provided sales materials Build relationships with decision...

Create\edit video and photos

Job Title: Creative Content Producer (Video & Image Ads for Gaming Platform) Job Description: We are looking for a talented and creative individual to join our team as a Content Producer. The primary responsibility of...

Extending Structural Rejuvenation to the Neck and Chest

A highly rigorous, carefully planned facial skincare routine is a staple for most individuals invested in their physical appearance. We meticulously apply heavy sunscreens, highly potent antioxidant serums, and rich moisturisers to the face every...

Advertisement for website

Hello everyone, Today We have a website where we organize paid raffles/giveaways. We are looking for teammates who have marketing knowledge or an audience/community and can attract people to join these raffles. Once the required...

The Saas Hub

  Meta Description: Learn why relying on manual processes is costing your online store time and money and how modern apps can streamline your workflow for better profitability.

Smart Sat Connect

Expert Starlink Installation, Saorview Installation, Satellite Dish Installation, Free Satellite TV Installation, WiFi Distribution, TV mounting, and Mobile Phone Signal Booster Ireland services. We ensure that your home or business stays connected through our fast,...

Megan Moura Photography

Oahu wedding photographer Honolulu. Wedding photographer Hawaii. Kauai, Maui & Oahu. Photography for weddings, elopements, special events, branding and family.

مترجم ومحلل في أسواق مالية

Hello, I am a motivated and active person with a strong interest in crypto and Web3 projects. I have experience using Telegram and Discord daily, and I am comfortable communicating with community members and helping...

Broker / Sales Agent – CryptoTax-Pro (Digital Asset)

Category: Broker, Crypto Broker, Web3 Sales Agent, M&A Broker   Objectives Identify and qualify crypto traders, investors, and small fund managers as potential clients Present CryptoTax-Pro using provided sales materials Build relationships with decision makers...

B2B Sales Representative — Review Management SaaS

Title: B2B Sales Representative — Review Management SaaS ($500 Per Closed Sale) Objectives Identify and qualify agencies, consultants, and business service providers as potential clients Present our reputation management SaaS using provided sales materials Close...

B2B Sales Representative — Accounting SaaS Product

Title: B2B Sales Representative — Accounting SaaS ($1,000 Per Closed Sale) Objectives Identify and qualify CPA firms and bookkeeping agencies as potential clients Present our SaaS solution using provided sales materials Build relationships with decision...

Create\edit video and photos

Job Title: Creative Content Producer (Video & Image Ads for Gaming Platform) Job Description: We are looking for a talented and creative individual to join our team as a Content Producer. The primary responsibility of...