Golang Colly Scrapper

1,000
ETH, DAI, USDT
+55
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+55
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Expert Translators Wanted

Are you a language enthusiast with a passion for precision and professionalism? We are on the lookout for top-tier translators to help us bring two important documents to life in Portuguese.   We need highly...

Expert Translators Needed!

Are you a language enthusiast with a passion for precision and professionalism? We are on the lookout for top-tier translators to help us bring two important documents to life in Spanish.   We need highly...

🚀 NFT Partner Wanted (60/40 Profit Share)

We are seeking a talented and creative NFT Partner to collaborate with in selling digital art pieces online. As an NFT Partner, you will work closely with the team to showcase and market a collection...

Help with VIDEO STREAMING

I need a script that masks the OBS virtual camera to splitcam to output stream.Many services I use block the stream from obs through splitcam and is not detected as the native camera so I...

Developer to Build Telegram Bot P2E Game MVP

Hi! I'm launching an early-stage startup and looking for a developer (or small team) to help build the MVP of a Telegram-based play-to-earn (P2E) game. The idea is to create an engaging Telegram bot where...

Translation Job

I need fluent English and Persian speakers to help me translate my english blog writings into persian for my persian audience

Upgrade MERN App to Web3 NFT Real Estate Marketplace

Include your GitHub profile showing senior-level experience. Applications without this or including with others will not be accepted. Hi, I have an in-progress MERN stack project with working UI and integrated payment features (PayPal, Stripe)....

Need to update frontend part using react

I have a Crypto NFT Game and need to update frontend part based on Figma design Candidate must has expertise in react and node  Will provide Figma design

SEO & Web Traffic Specialist

i'm seeking a skilled SEO web traffic expert who can increase sales in my store within 7 days. this is a performance based job, if you do not achieve the agreed traffic growth within the...

Senior Dev with Trading Experience

Project & Role Snapshot Scenium is a Web3 RWA‑trading startup that tokenizes private‑fund assets, stocks and crypto so anyone can trade 24/7 with instant settlement and zero dividend tax. You’ll collaborate directly with our CTO...

Expert Translators Wanted

Are you a language enthusiast with a passion for precision and professionalism? We are on the lookout for top-tier translators to help us bring two important documents to life in Portuguese.   We need highly...

Expert Translators Needed!

Are you a language enthusiast with a passion for precision and professionalism? We are on the lookout for top-tier translators to help us bring two important documents to life in Spanish.   We need highly...

🚀 NFT Partner Wanted (60/40 Profit Share)

We are seeking a talented and creative NFT Partner to collaborate with in selling digital art pieces online. As an NFT Partner, you will work closely with the team to showcase and market a collection...

Help with VIDEO STREAMING

I need a script that masks the OBS virtual camera to splitcam to output stream.Many services I use block the stream from obs through splitcam and is not detected as the native camera so I...

Developer to Build Telegram Bot P2E Game MVP

Hi! I'm launching an early-stage startup and looking for a developer (or small team) to help build the MVP of a Telegram-based play-to-earn (P2E) game. The idea is to create an engaging Telegram bot where...

Translation Job

I need fluent English and Persian speakers to help me translate my english blog writings into persian for my persian audience

Upgrade MERN App to Web3 NFT Real Estate Marketplace

Include your GitHub profile showing senior-level experience. Applications without this or including with others will not be accepted. Hi, I have an in-progress MERN stack project with working UI and integrated payment features (PayPal, Stripe)....

Need to update frontend part using react

I have a Crypto NFT Game and need to update frontend part based on Figma design Candidate must has expertise in react and node  Will provide Figma design

SEO & Web Traffic Specialist

i'm seeking a skilled SEO web traffic expert who can increase sales in my store within 7 days. this is a performance based job, if you do not achieve the agreed traffic growth within the...

Senior Dev with Trading Experience

Project & Role Snapshot Scenium is a Web3 RWA‑trading startup that tokenizes private‑fund assets, stocks and crypto so anyone can trade 24/7 with instant settlement and zero dividend tax. You’ll collaborate directly with our CTO...