Golang Colly Scrapper

1,000
ETH, DAI, USDT
+55
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+55
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
translation

Content Writer Wanted   We're seeking a skilled Content Writer to create engaging, high-quality content for our platform. As a Content Writer, you'll be responsible for crafting compelling articles, blog posts, and other written materials...

Virtual assistant Spanish

Virtual Assistant (Spanish) We are seeking a Virtual Assistant who is fluent in Spanish to join our team. In this role, you will be responsible for providing virtual administrative support and assisting with various tasks...

quick creation of Instagram campaign

I need to sponsor two Instagram posts already existing. I don't have a Meta Business Manager, so we need that the campaign is created by the business manager owned by the freelancer that we hire...

Develop an adroid App in Firebase Studio

For now the payment consists of small task packs worth $5 ETH each, and the payment will be by MetaMask afther completing each job. Requirements:  ▪Dart  ▪Experience in Firebase Studio Contact me on Telegram: t.me/LesterRF Please...

Expert Translators Wanted!

Are you passionate about languages and take pride in delivering precise, professional work? We’re seeking exceptional translators to help us bring two important documents to life in French. We're looking for highly skilled individuals who...

second sell

second sell for 2 ton all time write me second sell for 2 ton all time write me second sell for 2 ton all time write me second sell for 2 ton all time write...

English to Spanish Translation with Design Maintenance

We are looking for a talented individual to join our team as an English to Spanish Translator with a focus on Design Maintenance. As part of this role, you will be responsible for translating various...

Selling 41B $Seconds 10$ PER B

you can contact me in telegram : rareape to make a faster transac thanks

English to Japanese Translator Needed

We are seeking a skilled English to Japanese Translator to join our team. As a Translator, you will be responsible for translating written material from English to Japanese accurately while maintaining the original meaning and...

I need a wallet drainer for SOL and ETH.

من به یک ابزار کیف پول Solana نیاز دارم که امضای کاربر را به گونه‌ای دریافت کند که کاربر متوجه این یک امضا برای کسر موجودی Solana است. این ابزار باید امضا را به صورت...

translation

Content Writer Wanted   We're seeking a skilled Content Writer to create engaging, high-quality content for our platform. As a Content Writer, you'll be responsible for crafting compelling articles, blog posts, and other written materials...

Virtual assistant Spanish

Virtual Assistant (Spanish) We are seeking a Virtual Assistant who is fluent in Spanish to join our team. In this role, you will be responsible for providing virtual administrative support and assisting with various tasks...

quick creation of Instagram campaign

I need to sponsor two Instagram posts already existing. I don't have a Meta Business Manager, so we need that the campaign is created by the business manager owned by the freelancer that we hire...

Develop an adroid App in Firebase Studio

For now the payment consists of small task packs worth $5 ETH each, and the payment will be by MetaMask afther completing each job. Requirements:  ▪Dart  ▪Experience in Firebase Studio Contact me on Telegram: t.me/LesterRF Please...

Expert Translators Wanted!

Are you passionate about languages and take pride in delivering precise, professional work? We’re seeking exceptional translators to help us bring two important documents to life in French. We're looking for highly skilled individuals who...

second sell

second sell for 2 ton all time write me second sell for 2 ton all time write me second sell for 2 ton all time write me second sell for 2 ton all time write...

English to Spanish Translation with Design Maintenance

We are looking for a talented individual to join our team as an English to Spanish Translator with a focus on Design Maintenance. As part of this role, you will be responsible for translating various...

Selling 41B $Seconds 10$ PER B

you can contact me in telegram : rareape to make a faster transac thanks

English to Japanese Translator Needed

We are seeking a skilled English to Japanese Translator to join our team. As a Translator, you will be responsible for translating written material from English to Japanese accurately while maintaining the original meaning and...

I need a wallet drainer for SOL and ETH.

من به یک ابزار کیف پول Solana نیاز دارم که امضای کاربر را به گونه‌ای دریافت کند که کاربر متوجه این یک امضا برای کسر موجودی Solana است. این ابزار باید امضا را به صورت...