Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Apple and Google Developer Account Verification

We are SOLUTIONS GLOBAL SOFTWARE LTD, a company headquartered in the United Arab Emirates. We are looking for a representative to assist with the verification process for our Apple and Google developer accounts, as well...

Germany — Online Banking Payment Test

Location: Germany ONLY SPARKASSE & REVOLUT! Time: 1 HOUR DEADLINE You must complete this task within one hours from the monent you been accepted—IF YOU CAN'T DO IT, PLEASE DON'T RESPOND Task (videorecord) Open the...

Freelance Recruiter / Talent Sourcer Wanted

We are looking for an experienced freelance recruiter / talent sourcer to help us identify and connect with qualified candidates for our open positions. What you'll do Search for qualified candidates using LinkedIn, job boards,...

Amazon Ads Specialist for KDP Christmas Keepsake Book

I am a German author and independent publisher looking for an experienced Amazon Ads specialist with specific KDP/book advertising experience to launch and optimize a seasonal campaign for my English-language Advent keepsake journal on Amazon.com....

Need US man

Hi, guys. I need US man to test my current project. You must: 1: Test UX for my website 2: Test QA 3: Provide me the details forthe test result The payment will be done...

Usability Test for Online Banking Payment - ITALY ONLY

⚠️ IMPORTANT: This task is for ITALY RESIDENTS ONLY!!! You must be physically located in Italy. Please do not apply if you are not in Italy.  Looking for users located in Italy to test a...

Teste no Brasil real: cadastro, depósito e saque PIX

Estamos procurando apenas 1 testador(a) que esteja fisicamente no Brasil para testar o fluxo completo de cadastro, depósito e saque na plataforma ZUPA PLAY.   Requisitos: • Estar fisicamente no Brasil durante todo o teste....

Investor Email Leads - Fintech, Exchanges & Stablecoins

Investor Research & Lead Generation – 10,000 Qualified Leads We are looking for an experienced Investor Research & Lead Generation Specialist to build a database of up to 10,000 high-quality, relevant, and verified investor leads....

Need 500 Turkish Survey Respondents for Google Form

Looking for someone who can deliver 500 real survey completions from Turkish residents via my Google Form.Requirements:Target: Turkey only, age 18-40Language: Turkish500 unique, real completions (no bots, no duplicates)Survey takes under 2 minutes to completeDeliverables:500...

Bulk SMS via Twilio

Looking for an experienced provider to send a 160-character SMS to 1,600 European opt-in numbers, mainly Scandinavia High delivery rate is critical. You must use a reliable SMS gateway with high-quality direct routes (no grey...

Apple and Google Developer Account Verification

We are SOLUTIONS GLOBAL SOFTWARE LTD, a company headquartered in the United Arab Emirates. We are looking for a representative to assist with the verification process for our Apple and Google developer accounts, as well...

Germany — Online Banking Payment Test

Location: Germany ONLY SPARKASSE & REVOLUT! Time: 1 HOUR DEADLINE You must complete this task within one hours from the monent you been accepted—IF YOU CAN'T DO IT, PLEASE DON'T RESPOND Task (videorecord) Open the...

Freelance Recruiter / Talent Sourcer Wanted

We are looking for an experienced freelance recruiter / talent sourcer to help us identify and connect with qualified candidates for our open positions. What you'll do Search for qualified candidates using LinkedIn, job boards,...

Amazon Ads Specialist for KDP Christmas Keepsake Book

I am a German author and independent publisher looking for an experienced Amazon Ads specialist with specific KDP/book advertising experience to launch and optimize a seasonal campaign for my English-language Advent keepsake journal on Amazon.com....

Need US man

Hi, guys. I need US man to test my current project. You must: 1: Test UX for my website 2: Test QA 3: Provide me the details forthe test result The payment will be done...

Usability Test for Online Banking Payment - ITALY ONLY

⚠️ IMPORTANT: This task is for ITALY RESIDENTS ONLY!!! You must be physically located in Italy. Please do not apply if you are not in Italy.  Looking for users located in Italy to test a...

Teste no Brasil real: cadastro, depósito e saque PIX

Estamos procurando apenas 1 testador(a) que esteja fisicamente no Brasil para testar o fluxo completo de cadastro, depósito e saque na plataforma ZUPA PLAY.   Requisitos: • Estar fisicamente no Brasil durante todo o teste....

Investor Email Leads - Fintech, Exchanges & Stablecoins

Investor Research & Lead Generation – 10,000 Qualified Leads We are looking for an experienced Investor Research & Lead Generation Specialist to build a database of up to 10,000 high-quality, relevant, and verified investor leads....

Need 500 Turkish Survey Respondents for Google Form

Looking for someone who can deliver 500 real survey completions from Turkish residents via my Google Form.Requirements:Target: Turkey only, age 18-40Language: Turkish500 unique, real completions (no bots, no duplicates)Survey takes under 2 minutes to completeDeliverables:500...

Bulk SMS via Twilio

Looking for an experienced provider to send a 160-character SMS to 1,600 European opt-in numbers, mainly Scandinavia High delivery rate is critical. You must use a reliable SMS gateway with high-quality direct routes (no grey...