The web scraper is a feature you can use in both the Chat and the Workshop. In short, it reads all of the text on a single web page and then sends it back to the LLM to use as a prompt. Read more to learn how to use it to its fullest potential.
Chat
Add URLs
This feature lets the user drop in a specific URL (or multiple) that the AI can ingest. It will not click on any links or navigate through a website. It will only read the content on the specific page that the URL directs to.
This is great if you know exactly which pages you want to include in your prompt, but you don't want to rely on AI-Powered Web Search to find them.
Workshop
Web Scraper Step
What Does It Do?
When adding a new step in a Workflow, you can choose to add either a Prompt Step (the "normal" step) or a Web Scraper Step.
The Web Scraper Steps allows the AI to ingest the text from a single web page and insert it into the automation you're building.
It is essentially the same as copying and pasting the contents a single webpage into a prompt.
Unlike the AI-Powered Search, which has the ability to click on links, read multiple pages, and use AI to analyze the content, this feature can ONLY ingest the text content on a single webpage and insert the copied markdown into the workflow.
This is helpful if you:
Already know the exact URL you need the automation to ingest
Want the AI to search a URL that comes from a previous step
Do not want the AI-Powered Search to search multiple pages
How To Use It
First, you can manually enter a URL into the step. Just copy and paste a webpage into the prompt field.
Second, you can drop in the output of a previous step.
Let's say step 1 analyzes a file that you upload. Step 1's output is a summary of the file, and it includes key data like a company name and URL. The web scraper will check the output for a URL, and then go find that webpage.
Another use case might be that step 1 uses the AI-Powered Search to find a particular website. It will include the URL as part of its output. Using the AI-Powered Search to find ALL of the webpage info can be difficult, so you need to use the Web Scraper to read it in the next step.
Third, you can have the user paste a specific URL as part of a user input called "URL"
This step will scrape the webpage, copy all of the text, and treat it just like a prompt. This can save the user time by dropping in a webpage and not having to copy and past parts of the page.
