Run a web scraper on a schedule, without your laptop

A web scraper is a small program that visits web pages and copies out the facts you care about: prices, job listings, new posts. Your AI agent, such as Claude Code, Codex or Cursor, writes one from a paragraph you paste and puts it online on aidrop.it, where it runs on a timetable with your laptop closed. It needs a database, so after the 7-day free trial plan on Pro at $20 a month.

What you get is a table that grows with every run, for example:

You can ask your agent for it as a spreadsheet at any time. And when a run fails, or a page suddenly yields nothing, you get a message.

What you will have at the end

• A scraper that checks your chosen pages on schedule, around the clock.
• Everything it collects, saved in a database with the time and the page it came from.
• An alert by email or Slack when a run fails or finds nothing.
• The code in a repository, so your agent can change what it collects later.

Words you will meet

• Scraper — a program that reads web pages for you and writes down what it finds.
• Headless browser — a real web browser running without a screen, for pages that only show their content after their scripts run.
• robots.txt — a public file where a website says which pages automated visitors should leave alone.
• Database — the table above lives here. aidrop.it runs it next to your scraper.
• Secret — where the key for your alerts, such as an email service key, is saved on aidrop.it, out of the scraper’s code. You give it to your agent once, and the agent saves it.

Two ways to read a page

Most pages can be read the simple way: the scraper downloads the page and picks out the parts it needs. This is fast and fits the smallest size aidrop.it offers.

Some pages are nearly empty until the browser runs their scripts, as on many shops and dashboards. For those, the scraper drives a headless browser. It works, but it is slower and needs much more memory: plan for the 1 GB size, which starts at the Pro plan. Your agent tries the simple way first and switches only if the page comes back empty. Playwright’s own team says its browser image is not recommended for visiting untrusted websites, so keep the browser for sites you know.

How often to run it

Run it as often as the page really changes, and no more.

• Once a day suits most prices, listings and news pages. Pick a quiet hour, such as early morning.
• Every hour only for things that move fast and where an hour matters to you, such as a limited stock alert.
• Once a week for slow pages, such as a competitor’s pricing page.

More runs mean more load on someone else’s website and a bigger chance of being blocked. The timetable works as on the cron job page: the scraper is one always-on app with its own clock, and a run due during a restart is caught up when the app starts again.

How not to get blocked: scrape politely

• Read the site’s robots.txt and its terms. If the site offers an official API or data export, use that instead.
• Slow down. Wait a few seconds between pages, fetch only the pages you need, and do not fetch the same page twice in one run.
• Say who you are. The scraper should name you and a contact address in every request.
• Skip personal data you do not need. Privacy laws may apply to it.
• If a site blocks you or shows a “prove you are human” check, stop. Do not try to get around it.

aidrop.it logs where each of your app’s outgoing requests goes.

What you need

• An aidrop.it account — start free: a 7-day trial, no card, with a database. To keep the scraper running, Pro at $20/month; it also has the larger size that pages needing a headless browser use. Starter has no database to keep the results in. Plans are on the pricing page.
• Your agent connected to aidrop.it. The Quickstart shows how.
• The pages to watch, what to collect from each, and how often.
• Somewhere for alerts: an email service such as Resend, or a Slack incoming webhook. For alerts in Telegram, build a Telegram bot with Claude Code first.

Say this to your agent

Replace the two parts in square brackets before you paste: what to collect from which pages, and how often.

Build me a web scraper that runs on a schedule on aidrop.it. Use my existing aidrop.it Project, or create one called "Scraper" if I have none.

What to collect: [PAGES AND FIELDS, e.g. "name and price of each product on https://example.com/sale"].
How often: [WHEN, e.g. "every day at 7 am London time"].

- Python. Plain HTTP first (httpx with selectolax or BeautifulSoup). Use Playwright with headless Chromium only if the data is missing from the raw HTML. Then use the official Playwright image pinned to the library's version, run it as its non-root user, and pass the HTTPS_PROXY value to Playwright's proxy launch option.
- Read aidrop.it's runtime rules first and follow them. One always-on app with an in-process scheduler (APScheduler) and a small HTTP health endpoint. Schedule in my time zone.
- Be polite: obey robots.txt, a few seconds between requests, a User-Agent with my name and a contact address. Stop on 403, 429 or a CAPTCHA and alert me; never work around them.
- Save results in the Project's PostgreSQL with the time and source URL, plus one row per run (status, items, error). On startup, run once if a scheduled run was missed.
- Alert me over HTTPS (email API or Slack webhook) when a run fails or finds nothing. Secrets: ask me for each key the alerts need, store each one with project_secret_set under a clear name, and list those names in required_secret_names when you build. Never print a value back to me, and never write one into code, the repository, a commit or a log.

Build it, check it runs, do one test run, then show me the first results and when the next run is due.

How your agent builds the scraper

1. Picks your project, checks for a database, and asks you before creating one.
2. Looks at your pages and picks the simple way or the headless browser.
3. Asks you for the keys the alerts need and saves each one as a secret in your project. It never shows them again or writes them into the code.
4. Writes the scraper, its timetable, the results table and a small “I’m alive” page that aidrop.it checks.
5. Builds it on aidrop.it. If the scraper does not start, the agent reads the log, fixes it and builds again.
6. Does one test run and shows you the first rows of your table.

Check that the scraper collects the right data

• The first results match what you see on the site yourself.
• The next run time your agent reports matches what you asked for.
• After the next scheduled run, ask your agent “show me the last runs”. Each says how many items it found.
• Ask for the results as a spreadsheet file. It opens in Excel or Google Sheets.

To look at the results as charts in your browser instead of a spreadsheet, your agent can add a small dashboard next to the scraper: see deploy a Streamlit app.

When something goes wrong

Find what you see on the left and paste the right-hand text to your agent.

FAQ

Is web scraping legal?

It depends on where you are, what you collect and the site’s terms. Collecting public facts such as prices is common. Trouble usually starts with copying copyrighted content, collecting personal data, or getting around logins and blocks. Read the site’s terms first, prefer an official API, and ask a lawyer when the stakes are real.

What is the best language for a web scraper?

Python is the most common choice. It has simple tools for picking parts out of a page, such as Beautiful Soup, and Playwright for pages that need a real browser. JavaScript works as well. When an agent writes the code, the language matters less than scraping politely.

Can AI build a web scraper for me?

Yes. An AI coding agent can write the scraper, test it on your pages and fix it when a site changes its layout. On aidrop.it the same agent also puts it online, sets up the database next to it and checks it runs, so the scraper keeps working after you close the chat.

Related

• What is a cron job, and how to run one without a server
• Self-host n8n without managing a server
• Deploy a Streamlit app
• Taking your work with you
• Quickstart
• All playbooks