Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Chief Financial Officer (CFO)

We’re looking for a strategic Chief Financial Officer (CFO) to design financial architecture and lead fundraising across private and public rounds. This role goes far beyond bookkeeping — you’ll be a strategic partner to the...

Solana Blackchain Project - Review Code / U.S. Citizen Only

This is a Solana chain project.The coin is already built.Just need someone to review the code and to ensure it meets specs. Must be a U.S. Citizen. Must speak english fluently.

copy and paste

I’m currently looking for talented copywriters — beginners, mid-level, or experienced — to join upcoming projects.   Available Roles:   Website Copywriter Sales & Marketing Copywriter Social Media Copywriter Content Writer / Article Writer General...

Edit , Draw Plan and details

I want Architectural drawing assistant remotely who helps me edit or re draw some documents ;  ***The editor must be proficient exclusively in Vectorworks Software this is non-negotiable. The workload is high: each shot will...

Таргетолог

Ищем таргетолога Facebook / Instagram для запуска рекламы лид-формЧто нужно сделать:— корректно подключить заявки к AMО CRM;— обеспечить стабильную работу рекламных кабинетов (опыт обхода блокировок обязателен, мы находимся в РФ);— показать примеры кампаний по Казахстану,...

Hiring Professional Video Editor (Project-Based, Remote)

  We are looking for a creative, skilled video editor proficient with AI tools to deliver high-quality results on professional projects. Type of collaboration: Project-based | Fully remote | Competitive pay | Potential for long-term...

Programamer

I would like to find work as a back end developer 

Video Editor for 5 Vertical Geopolitical Explainers

🔹 ABOUT THE PROJECT We are producing five short, vertical videos focused on global issues, U.S. policy, and international stability. Each video is around 55–60 seconds and designed for TikTok, Instagram Reels, and YouTube Shorts....

Twilio API Bot

"I need the back end for my CRM made so I can perform mass sms campaigns to people that respond to my Instagram and Facebook ads" I've been having an issue where my twilio stops...

Python Crypto Trading Bot Developer (GUI + Dashboard)

I’m looking for an experienced Python developer to build a crypto trading bot with a modern GUI and dashboard. The trading strategy is already fully defined — I will provide all the logic, indicators, and...

Chief Financial Officer (CFO)

We’re looking for a strategic Chief Financial Officer (CFO) to design financial architecture and lead fundraising across private and public rounds. This role goes far beyond bookkeeping — you’ll be a strategic partner to the...

Solana Blackchain Project - Review Code / U.S. Citizen Only

This is a Solana chain project.The coin is already built.Just need someone to review the code and to ensure it meets specs. Must be a U.S. Citizen. Must speak english fluently.

copy and paste

I’m currently looking for talented copywriters — beginners, mid-level, or experienced — to join upcoming projects.   Available Roles:   Website Copywriter Sales & Marketing Copywriter Social Media Copywriter Content Writer / Article Writer General...

Edit , Draw Plan and details

I want Architectural drawing assistant remotely who helps me edit or re draw some documents ;  ***The editor must be proficient exclusively in Vectorworks Software this is non-negotiable. The workload is high: each shot will...

Таргетолог

Ищем таргетолога Facebook / Instagram для запуска рекламы лид-формЧто нужно сделать:— корректно подключить заявки к AMО CRM;— обеспечить стабильную работу рекламных кабинетов (опыт обхода блокировок обязателен, мы находимся в РФ);— показать примеры кампаний по Казахстану,...

Hiring Professional Video Editor (Project-Based, Remote)

  We are looking for a creative, skilled video editor proficient with AI tools to deliver high-quality results on professional projects. Type of collaboration: Project-based | Fully remote | Competitive pay | Potential for long-term...

Programamer

I would like to find work as a back end developer 

Video Editor for 5 Vertical Geopolitical Explainers

🔹 ABOUT THE PROJECT We are producing five short, vertical videos focused on global issues, U.S. policy, and international stability. Each video is around 55–60 seconds and designed for TikTok, Instagram Reels, and YouTube Shorts....

Twilio API Bot

"I need the back end for my CRM made so I can perform mass sms campaigns to people that respond to my Instagram and Facebook ads" I've been having an issue where my twilio stops...

Python Crypto Trading Bot Developer (GUI + Dashboard)

I’m looking for an experienced Python developer to build a crypto trading bot with a modern GUI and dashboard. The trading strategy is already fully defined — I will provide all the logic, indicators, and...