Skip to content

Scraping Pipeline

Turns public web data into validated output.

Extracts records into typed schemas and delivers CSV/JSONL files.

For
Analytics teams using public web data
Engine
Python · asyncio
Source code
Open · MIT
Buying
Written, fixed-price pilot

What is Scraping Pipeline?

Scraping Pipeline is an open-source Python pipeline that extracts validated, schema-typed data from public web pages. It fetches pages with asyncio, parses them with selectolax or BeautifulSoup, validates with Pydantic and Pandera, and delivers CSV, JSONL or Parquet. Its tests run offline against saved HTML.

50 pages, 1,000 validated records, 27 seconds.

New site = new ScraperConfig, no new code

0 pages0 records0 failed pages
ScraperConfig
sourcebooks.toscrape.com
pages1–50
parserselectolax
columnstitle · price_gbp · rating · in_stock · …

A replay of the real run in the README (run 0606b9bb…, 50 pages, 1,000 records, 27 s). Source-site permissions and rate limits are reviewed per project.

How it works

From the product's own documentation. Each number says where and on which version it was measured.

  1. Selectors live in config

    A new site needs a new ScraperConfig, not new code.

  2. Polite async HTTP

    Retries, timeouts and pacing. Failed pages are logged separately.

  3. Validated before export

    Each record through Pydantic, the whole table through Pandera.

  4. A traceable delivery

    CSV, JSONL, Excel or Parquet, with a manifest of the source pages.

1,000 records
50 pages in 27 seconds. Output and manifest are committed. Source
117 listings
One page in 2.8 seconds. A second site, same pipeline. Source
Offline tests
Tests run on saved HTML fixtures and never hit the site.
Two parsers
selectolax or BeautifulSoup + lxml, chosen from the command line.

Stack

  • Python
  • asyncio
  • selectolax
  • Pydantic v2
  • Pandera
  • Parquet

Try it

pip install -e ".[dev]" "selectolax<1.0"
scrape-books --pages 50 --out examples/books.csv

What it does not claim

Source-site permissions, rate limits and data use are reviewed separately for every project.

Current boundary

Source-site permissions, rate limits and data use are reviewed per project.

What a pilot covers

A sample starting scope. Final deliverables and acceptance criteria are fixed in the written proposal.

Question
How can source pages yield consistent data?
Likely input
Source URLs, access terms and target schema
Deliverables
  • Typed extraction sample
  • Validation and error report
  • CSV or JSONL delivery sample
Send your problem

Before you decide

Can I buy this product today?

A paid pilot and setup proposal can be prepared. Scope, data access, deliverables, timeline and a fixed price are agreed in writing after a discovery call.

Why is there no price?

Data volume, integration and success criteria differ from job to job. After the call you get a fixed-price written proposal; no paid work starts without your approval.

Do I need to upload live data here?

No. This site has no file upload or accounts. Data access, retention and permissions are set only in the pilot agreement.

What is the first step?

Choose Scraping Pipeline in the form and describe the problem you need to solve. The data and pilot boundaries are then set together.

  • CRM Cleaning API

    Prepares messy lead lists for CRM import.

    Rust (Axum) · Python
  • ProofFrame

    Data contracts with verifiable quality evidence.

    Rust · Python · WASM
  • Fuzzy Dedupe

    Near-duplicate detection for names and organizations.

    Rust · Python