Sponsored by

The Initial Data Offering (IDO) community is a place for data enthusiasts to discover new datasets daily. Check out the new Premium Access! https://initialdataoffering.com/offering/

The mission is to build a community of data enthusiasts and curate high-quality, unique datasets for businesses, researchers, and organizations worldwide.

Web scraping is one of those topics we are going to continue to hear about for the next few years. Every AI company is scraping. Everyone is trying to get access to data and the question of what on the web you can scrape vs. have to pay for continues to be debated.

Bright Data, the Israeli data scraping company, just won their lawsuit against X. This is important because Bright Data uses sophisticated scraping technology that allows them to get around anti-scraping attempts. One of the key things that Bright Data supposedly is doing is only scraping data that doesn’t require a login. This is something I have written about in the past, and I continue to believe holds true - the idea that you can stop people from legally scraping your site if you require a click thru or login.

As web scraping technology continues to get more sophisticated, it is going to be interesting to see if we get a bunch of acquisitions in the space. There are some very sophisticated web scraping technology companies that on their own aren’t as valuable. But if you take this technology, and put it in-house with a data-hungry company, you give yourself a ton of upside. One acquisition we recently saw was Vertical Knowledge getting acquired by Babel Street.

Selling data, using data for training, and scraping data for combining with other data assets, is a real advantage and can help you build a moat if you have sophisticated web scraping technology in-house. Maintaining scraping is a pain when you don’t have the right team and technology and it can get very expensive very quickly if you have to rely on 3rd parties.

The scraping roll-ups I think are right around the corner!

Looking for visuals and charts, rather than words, to understand the daily news?

Bay Area Times is a visual-based newsletter on business and tech, with 250,000+ subscribers.

Click once to sign up for free.