I will develop a web crawler using ruby or python with mechanize

Crawl the web or specific websites and extract relevant information to database and integrate to your internal systems. Make sure that the client is not blocked by the server.