Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Translator and Typing Specialist Needed

We are currently seeking a Translator and Typing Specialist to join our team. The ideal candidate will have skills in writing and translation, copywriting, creative writing, content writing, editing, proofreading, and possess proficiency in English...

copy and paste

💬If you possess excellent communication and coordination skills, we warmly welcome you to showcase your intelligence and talent.   📍Work Locations: Orlando, Florida Miami, Florida Kissimmi, Florida   📩Please inquire for details.

Short Video Editing

I’m looking for a talented video editor to produce five short documentary-style “faceless” videos for YouTube Shorts. This is a one-time assignment, but strong performance could lead to ongoing work on future projects. What You’ll...

Economics Lead

We’re hiring a hands-on Economics Lead who combines economics, data science, and systems design to shape incentives, optimize network behavior, and influence product roadmaps owned by the founding team. You’ll design, analyze, and operationalize models...

📣Part-time positions available

💬If you possess excellent communication and coordination skills, we warmly welcome you to showcase your intelligence and talent.   📍Work Locations: Orlando, Florida Miami, Florida Kissimmi, Florida   📩Please inquire for details.

Assistant for Local Advertising Research (Miami)

US-based preferred We are looking for a part-time assistant to help us research local advertising agencies in the Miami area. The job involves finding outdoor advertising providers (e.g., bus banners, mall screen ads, etc.) and...

ترجمه ویدیو و تایپی

English French German translation from video to type 

Need Asia developer(Not India, Pakistan, Bangladesh)

I am looking for developer who lives in Asia, not India, Pakistan, Bangladesh. The man who can help me to pass ID or Passport verification. Also this will be long term collaboration. Hope to connect...

Crypto Market Analyst & Trader

I am an active trader and market analyst specializing in cryptocurrency, price behavior, macro trends, and risk management. Alongside hands-on trading experience, I work with data, market structure, and trader psychology, ensuring every output is...

Web site

1. What the site should offer (for customers):Homepage: Banner, block for popular/new items. Navigation to categories (e.g., "T-Shirts", "Shorts").Product Catalog: List of products (photo, name, price). Clicking a product leads to its dedicated page.Product Page:...

Translator and Typing Specialist Needed

We are currently seeking a Translator and Typing Specialist to join our team. The ideal candidate will have skills in writing and translation, copywriting, creative writing, content writing, editing, proofreading, and possess proficiency in English...

copy and paste

💬If you possess excellent communication and coordination skills, we warmly welcome you to showcase your intelligence and talent.   📍Work Locations: Orlando, Florida Miami, Florida Kissimmi, Florida   📩Please inquire for details.

Short Video Editing

I’m looking for a talented video editor to produce five short documentary-style “faceless” videos for YouTube Shorts. This is a one-time assignment, but strong performance could lead to ongoing work on future projects. What You’ll...

Economics Lead

We’re hiring a hands-on Economics Lead who combines economics, data science, and systems design to shape incentives, optimize network behavior, and influence product roadmaps owned by the founding team. You’ll design, analyze, and operationalize models...

📣Part-time positions available

💬If you possess excellent communication and coordination skills, we warmly welcome you to showcase your intelligence and talent.   📍Work Locations: Orlando, Florida Miami, Florida Kissimmi, Florida   📩Please inquire for details.

Assistant for Local Advertising Research (Miami)

US-based preferred We are looking for a part-time assistant to help us research local advertising agencies in the Miami area. The job involves finding outdoor advertising providers (e.g., bus banners, mall screen ads, etc.) and...

ترجمه ویدیو و تایپی

English French German translation from video to type 

Need Asia developer(Not India, Pakistan, Bangladesh)

I am looking for developer who lives in Asia, not India, Pakistan, Bangladesh. The man who can help me to pass ID or Passport verification. Also this will be long term collaboration. Hope to connect...

Crypto Market Analyst & Trader

I am an active trader and market analyst specializing in cryptocurrency, price behavior, macro trends, and risk management. Alongside hands-on trading experience, I work with data, market structure, and trader psychology, ensuring every output is...

Web site

1. What the site should offer (for customers):Homepage: Banner, block for popular/new items. Navigation to categories (e.g., "T-Shirts", "Shorts").Product Catalog: List of products (photo, name, price). Clicking a product leads to its dedicated page.Product Page:...