CrawlGPT

  • Information gathering tool designed to crawl through obscure web pages, then uses a large language model to gather textual and statistical data.
  • For example, there are a lack of centralized data sources in BMA. Can we build a crawler software to gather information at scale?
  • 2018: https://github.com/heypoom/darkcrawl

#idea