Golang Colly Scrapper

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

Price is negotiable

 

Project Description:

Develop a high-performance crawler capable of traversing millions of domains quickly, extracting links and navigating up to a specified depth for each domain. Speed is essential.

Technologies:

  • Programming Language: Golang, with no specific preference for libraries as long as they are among the fastest and best suited for the project's needs.
  • Database: Redis, used as a queue to temporarily store the results of each crawled page in JSON format.

Key Features:

  • Retrieve a list of domains via an API returning domain names in JSON format. The frequency of checking for new domains available should be configurable (in minutes or seconds).
  • For each domain, extract links (CSS, JS, FORM, LINK, JSON) up to a specified crawl depth and a maximum number of pages per host. Configurations related to depth and maximum pages should be easily adjustable in a configuration file.
  • Identify for each host its IP address, TLD (e.g., .com), the port used (e.g., 80), and for each page, capture the source, URL, navigation depth, and presumed file type (based on content-type or extension).
  • Manage a blacklist for extensions and content-types, avoid duplicates through URL normalization, and ensure that the source code size of pages does not exceed a predefined limit (configurable).
  • Save crawl results to Redis in JSON format in batches, ready to be integrated into Elasticsearch. The number of documents per batch should be configurable.
  • The configuration of the number of workers should be automatic based on server capacity but adjustable via a configuration file for maximum flexibility. The number of requests per domain should be limited to avoid DDOS.
  • Plan rigorous memory leak management to ensure long-term performance.
  • A logging and performance monitoring system using Prometheus.
  • Handling of redirects, crawling both http and https. If http://amazon.com does not work or redirects, try https://amazon.com; otherwise, skip.
  • Ignore the robots.txt.

Other Requirements:

  • The HTTP response header must be accessible at the beginning of the source code for easy data processing.
  • Ensure clear documentation on how to modify configurations for easy and rapid adjustments as needed.
  • For each page, make an md5 hash of the source code after removing spaces.
  • Retrieve links from these specified attributes and tags.

List of attributes to handle:

  • href
  • src
  • data-src
  • data-href
  • data-url
  • action

List of tags to handle:

  • a
  • link
  • script
  • img
  • iframe
  • frame
  • source
  • form

Blacklist extensions:

  • .pdf
  • .mp3
  • .avi
  • .mp4
  • .ogg
  • .zip
  • .rar
  • .7z
  • .exe
  • .apk
  • .iso
  • .img
  • .jpg
  • .png
  • .gif
  • .bmp
  • .ico
  • .svg
  • .xml
  • .woff
  • .woff2
  • .ttf
  • .eot
  • .otf
  • .csv
  • .xls
  • .xlsx
  • .doc
  • .docx
  • .ppt
  • .pptx
  • .odt
  • .ods
  • .odp
  • .odg
  • .odf

Expected JSON on Redis per page: { "md5": "3ab3e9f9fcf1a62edfca58f00f1a4a8c", "domain": "example.com", "host": "subdomain.example.com", "url": "http://www.subdomain.example.com/page1", "tld": "com", "ip": "192.0.2.1", "port": 80, "depth": 1, "rank": 12345, "source": "HTTP/2 200 <html><head>...</head><body>...</body></html>", "filetype": "html" }

1,000
ETH, DAI, USDT
+53
0 days (till Mar 10th, 2024)

More Jobs like this

Show more
Blockchain-Integrated Decentralized Web Application(DApp)

I'm now seeking skilled and experienced developer that specialize in decentralized applications (DApps) and Web3.js I'm developing DApp using the MERN stack for the frontend and backend, while integrating blockchain technology to provide decentralized functionality....

Article Writing

We are seeking a talented and creative individual to join our team as an Article Writer. The ideal candidate will have a passion for writing and the ability to create engaging and informative content across...

General Translators Needed

We are urgently in need of general translators that can translate from English into various languages.  

Article writing

We are seeking a talented Article Writer to join our team. The ideal candidate will have a passion for writing and exceptional creativity. The Article Writer will be responsible for producing high-quality, engaging content for...

Monad Pixels On-Chain Pixel Art Platform

I am looking for a skilled blockchain and full-stack developer to help improve and expand my project, Monad Pixels, which is an on-chain pixel art platform where every pixel is an NFT, built on the...

Finish NFT trading platform

I am looking for someone who can help me finish my NFT trading platform. I wanna add chatting room and blog page. Before that first landing page has to be completed. If you are interested,...

Sales Specialist Needed for Digital Product

We are looking for a sales specialist to find buyers for our digital product. Tasks include outreach, negotiation, and helping build a sales process. Independence and communication skills are essential.

Article writing needed

We are seeking a talented Article Writer to join our team. The ideal candidate will have strong writing and translation skills with a focus on translating content effectively. Responsibilities: - Research and create engaging and...

YES-Fam Project Ambassador

YES Protocol is building a long-term, transparent, community-driven Web3 ecosystem.   We are looking for Ambassadors who are willing to learn, participate, and co-build the community.   This is not a short-term task or a...

Indoor Messy Winter Outfit Video

I need a video about 15 minutes. A female model wears a nice winter outfit. A friend of her pours a lot of different sauces all over her head and clothes.  Just real reactions -...

Blockchain-Integrated Decentralized Web Application(DApp)

I'm now seeking skilled and experienced developer that specialize in decentralized applications (DApps) and Web3.js I'm developing DApp using the MERN stack for the frontend and backend, while integrating blockchain technology to provide decentralized functionality....

Article Writing

We are seeking a talented and creative individual to join our team as an Article Writer. The ideal candidate will have a passion for writing and the ability to create engaging and informative content across...

General Translators Needed

We are urgently in need of general translators that can translate from English into various languages.  

Article writing

We are seeking a talented Article Writer to join our team. The ideal candidate will have a passion for writing and exceptional creativity. The Article Writer will be responsible for producing high-quality, engaging content for...

Monad Pixels On-Chain Pixel Art Platform

I am looking for a skilled blockchain and full-stack developer to help improve and expand my project, Monad Pixels, which is an on-chain pixel art platform where every pixel is an NFT, built on the...

Finish NFT trading platform

I am looking for someone who can help me finish my NFT trading platform. I wanna add chatting room and blog page. Before that first landing page has to be completed. If you are interested,...

Sales Specialist Needed for Digital Product

We are looking for a sales specialist to find buyers for our digital product. Tasks include outreach, negotiation, and helping build a sales process. Independence and communication skills are essential.

Article writing needed

We are seeking a talented Article Writer to join our team. The ideal candidate will have strong writing and translation skills with a focus on translating content effectively. Responsibilities: - Research and create engaging and...

YES-Fam Project Ambassador

YES Protocol is building a long-term, transparent, community-driven Web3 ecosystem.   We are looking for Ambassadors who are willing to learn, participate, and co-build the community.   This is not a short-term task or a...

Indoor Messy Winter Outfit Video

I need a video about 15 minutes. A female model wears a nice winter outfit. A friend of her pours a lot of different sauces all over her head and clothes.  Just real reactions -...