Web Scraping with ChatGPT

Introduction

Each and every single decision that you make is driven by data. For instance, every recommendation that’s suggested to you on social media platforms is tailored to your own preferences based on the data that the app is made aware about you.

There are several ways in which companies collect data, and web scraping is the most popular method among them.

As a developer, it’s important to understand web scraping and its working to ensure that you employ the right services for a business to succeed. So, this article aims on exploring web scraping, and how you can easily scrape for data using nothing but large language models (LLMs) like ChatGPT.

---

What is Web Scraping?

Web scraping is the process of extracting data from a website that can be used to make data-driven decisions. Simply put, it collects data from a site, and outputs it into a format that is more useful to a user.

How Does Web Scraping Work?

A simple web scraping service can work as follows:

1. You can provide a URL that you’d like to extract data from. For example, this could be an Amazon search result URL that you can use to extract information on products.
2. The scraper will visit the URL and will download the HTML content.
3. The HTML content is then traversed to pick out the required data through DOM manipulation.
4. The required data can then be accessed through DOM and can be parsed into a format that’s usable
5. to the user. This could be in the form of a CSV or a JSON file
6. This output can then be used to do so many things like machine learning model training to business intelligence.

Sometimes, these simple steps won’t work – especially when the site is backed by a web application firewall, or a CAPTCHA. These services tend to block scrapers from accessing the content on their site as these scrapers can exert a significant load on the servers. So, you will need to customize your scraper to leverage techniques like IP Rotation to bypass such restrictions.

---

Use Cases of Web Scraping

But, using web scraping can be a significant benefit. In fact, it can bring about benefits in many industries:

Market Research

Web scraping is a powerful tool for market research, enabling companies to gather large amounts of data from various online sources, such as e-commerce sites, social media platforms, and competitor websites. This data can be used to analyze trends, understand consumer behavior, track pricing strategies, and monitor product reviews.

For instance, a company can scrape product reviews to gain an insight on customer sentiment, or they can track competitors’ pricing to adjust their own strategies. This allows businesses to stay competitive and make informed decisions based on real-time data.

Academic Research

Web scraping allows researchers to collect vast amounts of data from online sources, including scientific publications, forums, and databases. This data can be used to perform large-scale analyses, such as tracking the prevalence of certain topics in research papers or gathering data on social trends over time.

For example, a researcher studying climate change might scrape data from various environmental websites and databases to analyze changes in global temperature patterns or the frequency of climate-related news articles.

Simply put, web scraping enables researchers to access and analyze data that would otherwise be difficult or time-consuming to collect manually.

Business Intelligence

Web scraping is essential for business intelligence, as it allows companies to extract data from multiple online sources to gain insights into market trends, customer preferences, and competitor activities.

This information can be used to create detailed reports, dashboards, and predictive models that help businesses make strategic decisions.

For instance, a retail company might use web scraping to monitor competitors’ product offerings and pricing, allowing them to adjust their inventory and pricing strategies accordingly.

---

ChatGPT in Web Scraping

What is ChatGPT?

But, here’s where it gets really interesting. You no longer have to code out a scraper using a chunk of Python code. Those days are gone. Right now, all you need is a ChatGPT Pro Account and a few lines of English to get started.

| Yep, it’s that simple.

For those of you who aren’t familiar, ChatGPT is a product of OpenAI that’s capable of understanding natural language and providing human-like responses to questions that you might ask, about anything. Not only that, but if you’re a Pro user, it has the capability of executing Python code directly within the conversation.

Good Prompts in ChatGPT

All you really need is a really good prompt. A prompt is something that you provide ChatGPT in order to get a response. For example, a simple prompt might look something like this

| “Hey, can you write me a Python code that sums two digits?”

You will get an output similar to this:

| Pretty cool isn’t it?

You can take this one step further and ask it to demo it by using 3 and 5 as the input:

As you can see, it’s capable of executing Python code directly inside the conversation. This let’s us gain a significant boost in terms of flexibility.

You can take this further by using a Pro license as it’s capable of accessing the internet. For example, you can ask it to summarize links:

So, with that being said, you can definitely use natural language like this to define a web scraper that can be used to extract data on basically any source that you can think of. But not only that, if your code runs into an error, you can use the vast knowledge base that GPT models are trained on to get the best troubleshooting tips out there!

Benefits of Using ChatGPT To Build Scrapers

So, by using ChatGPT to build your scraper, there’s tons of benefits. Some include:

1. Improved development times: You don’t need to code anything from scratch. All you need to do is instruct the model what to do and it generates the output for you.
2. Better troubleshooting: You can show GPT your error and gain inputs on what’s causing the error and even get fixes to the error.
3. Easy to get started: Anyone without programming knowledge can build a scraper on their own.

---

Using ChatGPT To Generate Web Scraping Code

So, let’s build our very own scraper with ChatGPT.

Step 01: Creating the prompt to building a scraper

You’ll need to make sure that you create the perfect prompt for GPT to build your scraper easily. To do so, you will need a prompt that is:

1. Descriptive: Your prompt should clearly describe what the action you’re looking to do
2. Concise: Your prompt should not take a long time to jump to the task at hand. Instead, get started right away.
3. Include details in your query to get more relevant answers: Make sure that requests provide any important details or context. Otherwise you are leaving it up to the model to guess what you mean.

In addition to that, you can also tell GPT to:

1. Adopt a persona: You can tell GPT to assume the role of a senior developer. By doing so, it’s able to assume the mindset of the senior developer and generate outputs based on the way the adopted persona would think
2. Specify the steps required to complete a task: You can instruct GPT how to generate the code rather than letting it figure it out on its own. This gives GPT a clear path on what to do.

So, in the end, you can leverage a prompt like this:

Hey GPT, You are now a Senior Software Engineer.

You are going to build a web scraper for me using Python.

The code you write should be of a Senior Developer scale and it should:

1. Accept a URL
2. Visit the given URL
3. Scrape for data based on the criteria that the consumer defines
4. Return a CSV of scraped data based on the required output.

For example, let's say I give you this URL - https://brightdata.com/blog/how-tos/how-to-rotate-an-ip-address. You're supposed to return the content of the post as a JSON.

The prompt that we generated is concise, detailed, and has steps outlining what the program should do and a persona to assume.

Step 02: Coding out the scraper

Next, provide GPT the prompt and take a look at its response:

As you can see, it really generated a scraper that’s customizable in nature!

---

Troubleshooting with ChatGPT

Here’s the conversation in which the code resides – https://chatgpt.com/share/2e9f2a80-246e-4490-a479-d1d5ea7d17b3

Extract the code that was generated and try running it locally:

As you can see, we ran into an error when trying to execute this. You can ask GPT to troubleshoot this error:

Based on the information given by GPT, we will need to install the package called requests, or try the other two fixes.

So, let’s first install the library:

Next, re-run the script to see the output:

As you can see, it output the article onto the console. But, this isn’t what we wanted. Let’s ask GPT to redo it and export the output as a JSON file:

GPT will generate new code again that updates the export process:

Update your snippet and re-run the script:

As you can see, data has been exported to a JSON file. Go ahead and open the JSON file:

And viola! GPT has successfully exported the article onto a JSON file!

That was very simple wasn’t it? You were able to customize the initial block that GPT generated and tailor it to your own requirements without even writing a single line of code! Now, that’s a no-code scraping solution!

---

Concluding Thoughts

And, that’s pretty much it for this guide. With tools like ChatGPT, building a web scraper is as simple as writing in English. All it takes is a few minutes to build a fully functional scraping service that’s capable of outputting data for future use as well.

By using ChatGPT to build your scrapers, you’re able to generate your code in under a few commands, and troubleshoot for errors and even get guidance on fixing errors. This significantly boosts development productivity and simplifies the overall workflow.

Interested to know who offers the best web scraping services? Read out Best Web Scrapers review of the top scraping service providers.