# Permission first web data kit for Scrapling

Use this kit before a program reads another site's pages and copies selected fields into a table. Scrapling supplies the page reader, the parser that picks out the fields, and spiders that follow a planned set of links. Your business supplies the permission, limits, and person responsible for the result. Most ordinary uses on your own site or permitted public pages are lawful. This kit keeps the work outside login areas, excludes personal data, and stops when the owner says no. We reviewed the documentation and source; we did not install or run Scrapling.

## The decision flow

```text
┌────────────────────────────────────────────┐
│ Approved page list                         │
│ → Scrapling reads and selects fields       │
│ → Table or clean text → A person checks it │
└────────────────────────────────────────────┘
                     ▼
Own site, written permission, or terms allowing this use?
  No  → Get written permission first
  Yes ▼
Login, paywall, CAPTCHA, Turnstile, IP block, or stop letter?
  Yes → Stop and ask a lawyer
  No  ▼
Personal data involved, or reuse rights unclear?
  Yes → Stop and ask a lawyer
  No  ▼
Approved API or feed available?
  Yes → Prefer that API or feed within its terms
  No  → Read only the approved public pages
                     ▼
Obey robots.txt → Low rate → Identify the business
                     ▼
Go ahead with logs and human review
Any new refusal → Stop the whole target job
```

An API is a site's designated machine entrance for data. A feed is a published stream or file of updates. Prefer either when available and permitted. A public endpoint does not itself grant permission to reuse everything it returns.

## One permission record per target

Complete a separate record for every site, even when several sites use the same store software. A subcontractor signs the same scope. Keep the completed record private and show only approved public facts in any report.

| Question | Record before starting |
|---|---|
| Business purpose | What decision will this data support? |
| Site and owner | Public site URL; owner organization; contact responsible for access |
| Basis for permission | Our own site / written authorization / terms expressly permitting the intended use |
| Written authorization | Approver, their authority, approval date, expiry date, permitted purpose |
| Terms | Public terms URL, date checked, exact relevant clause, any separate API or data license |
| Existing agreements | Does our business already have an account or contract with this site? Give relevant agreements to counsel. |
| Robots record | Public robots.txt URL, date and time read, response result, applicable named group and wildcard group, allowed paths, delay instructions |
| API or feed | Available? Public documentation URL, approved access method, fields, limits, and reuse terms |
| Scope | Exact allowed domains, paths, and starting URLs; exclusions; maximum pages per run |
| Data fields | Each field needed; source URL and observation time retained with each row |
| Personal data | Yes / no / uncertain. If yes or uncertain, stop this baseline workflow for review. |
| Reuse rights | Internal analysis / redistribution / publication / AI use; permission for descriptions, images, or other protected material |
| Rate and schedule | Owner's maximum rate, proposed lower rate, runs per day or week, permitted hours, total request budget |
| Identity | Business name, contact address, and truthful `User-Agent` identifier |
| Stop rules | Any challenge, block, login, paywall, withdrawal of permission, or unexpected personal data |
| Retention | Delete raw pages after: ___; delete derived rows after: ___; owner of deletion: ___ |
| Storage and sharing | Who may read results; where approved records are held; AI provider, if any; onward sharing allowed? |
| Review | Business approver, technical operator, lawyer when needed, next review date |

Do not read a disallowed path to find out whether its content would be useful. Resolve conflicting terms, robots instructions, and written permission with the owner first. Silence is not written consent.

Robots.txt is the owner's published request to visiting programs. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) defines the protocol; it is not a law or an access license. It also says its rules are not access authorization. Record both the robots result and the separate basis for permission.

## Permission request email

This is a draft for a person to review and send. Fill the blanks with the actual proposed job.

```text
Subject: Permission to read selected public data from [site]

Hello [name],

I represent [business]. We would like to read [exact pages or paths]
and copy [specific fields] for [purpose]. Is there an official API,
feed, or export you would prefer us to use?

If page collection is acceptable, we propose [frequency], no more
than [rate] and [total requests] per run, identified as [business
identifier and contact]. We will obey your robots.txt instructions,
avoid login areas and personal data, and stop if you block us or
withdraw permission. We will not get around a CAPTCHA or other check.

We will retain [data] for [period], share it with [recipients], and
use it for [internal analysis / other specifically described use].
Please confirm whether this includes permission to [reproduce text,
use images, publish results, or supply data to an AI service, if needed].

Could someone authorized by the site owner confirm the approved
scope, limits, dates, and any additional conditions in writing?

Thank you,
[person, role, business, contact details]
```

## Before you install

- [ ] Complete the target record. Prefer an approved export, API, or feed.
- [ ] Give IT the approved URLs, fields, rate, retention period, output format, and named reviewer.
- [ ] Ask IT to use an isolated Python environment and to record the exact package version before a pilot. The README requires [Python 3.10 or higher](https://github.com/D4Vinci/Scrapling#installation).
- [ ] Read the [BSD-3-Clause license](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE). It permits use and redistribution subject to its notice and endorsement conditions and provides no warranty. It does not grant rights to anyone else's data.
- [ ] Note the README's [disclaimer](https://github.com/D4Vinci/Scrapling#disclaimer), including “educational and research purposes only.” Ask counsel about any uncertainty between that wording and your intended commercial use; neither statement supplies target-site permission.
- [ ] Start with parsing a page your business already owns. Parsing means finding the selected parts of an existing page, like locating the price column in a form.

These commands are copied from the README's Installation and Optional Dependencies sections. They are choices for an approved setup, not a requirement to install every extra.

Parser only:

```bash
pip install scrapling
```

For requests and spiders:

```bash
pip install "scrapling[fetchers]"
```

The basic install does not include the request or spider dependencies. A fetcher is the part that asks a website for a page. A spider manages the list of pages to visit and the rules for extracting their data. Ordinary HTTP requests do not need a browser installation, according to the [README](https://github.com/D4Vinci/Scrapling#optional-dependencies).

Only if an approved job genuinely needs the browser component does the README add:

```bash
scrapling install
```

That command installs browsers and related system and fingerprint dependencies. Browser mode is outside this kit's baseline settings.

For clean Markdown conversion, a plain text format with headings and links, the README lists:

```bash
pip install "scrapling[rag]"
```

For the optional AI connection, the README lists:

```bash
pip install "scrapling[ai]"
```

Hold that last option back until IT approves the AI assistant's permissions and data handling. An MCP server is a program that gives an AI assistant tools to call. Scrapling's [MCP tools](https://github.com/D4Vinci/Scrapling/blob/main/docs/ai/mcp-server.md) include requests, browser interaction, screenshots, JavaScript execution, saved session state, and recorded network responses. They go beyond reading a table. Do not give an assistant the ability to submit forms, reuse signed-in sessions, or select new target sites under this kit.

## Safe settings card for IT

These are conservative proposed settings, checked against the current documentation and source. They are not a complete program, a guarantee of legal compliance, or proof that a website welcomes the job. Use the owner's lower rate whenever it is stricter.

### Spider settings

| Setting | Current source default | Proposed baseline |
|---|---|---|
| `robots_txt_obey` | `False` | `True`, plus manual review of applicable robots groups |
| `concurrent_requests` | `4` | `1` |
| `concurrent_requests_per_domain` | `0`, no per-domain cap | `1` |
| `download_delay` | `0.0` seconds | `2.0` seconds or the longer agreed delay |
| `max_blocked_retries` | `3` | `0` |
| `autothrottle_enabled` | `False` | `False`; keep an explicit approved delay |
| `allowed_domains` | Empty set | Approved domains, with separate exact-host and path checks |

Source: [spider settings and hooks](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py) and [advanced spider documentation](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/advanced.md). The two-second delay is our suggested starting policy, not a published permission threshold. Apply the budget across all workers and include robots requests. Do not run a separate uncounted worker against the same site.

The [engine's domain check](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py) also permits subdomains of an allowed domain. IT must add an exact-host check when permission covers only one hostname, as well as the approved path checks.

`max_blocked_retries=0` stops retrying an individual blocked request. It does **not** stop the remaining crawl. The default `is_blocked()` checks response status codes; it cannot identify every challenge page. Before scheduled use, IT must add a supervisor that recognizes challenges from page content, stops all remaining work for that target, and alerts the named reviewer. A rate-limit response also pauses the whole target until its stated waiting period and the owner's terms have been reviewed. No changing addresses or browser identities to continue.

The [robots implementation](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/robotstxt.py) queries the wildcard `*` group. It treats `4xx` responses as empty rules; other response failures and request exceptions disallow crawling for that run. Therefore, inspect any group matching your business identifier yourself, and stop if the robots request is denied or challenged. Turning on the flag does not resolve every permission question.

### Ordinary request session

The following options belong on the `FetcherSession` constructor. These are configuration fragments for IT to integrate, without a target URL or a command to start collecting:

```python
impersonate=None
stealthy_headers=False
retries=1
follow_redirects=False
verify=True
headers={"User-Agent": "BusinessDataCheck/1.0 (contact: data@example.com)"}
```

Replace the example identity and contact with your business's real details. Both `impersonate=None` and `stealthy_headers=False` matter: the source defaults to a Chrome identity and generated browser headers. `retries=1` means one attempt, including the initial request. Keep certificate verification enabled. Review any redirect destination against the permission record before adding it. These settings come from the [request-session source](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/static.py) and [request documentation](https://github.com/D4Vinci/Scrapling/blob/main/docs/fetching/static.md).

Do not supply login cookies, saved browser state, rotating network services, or third-party CAPTCHA services. Limit the job to read requests; do not use the library's write methods.

### Browser caution

The [v0.5 migration guide](https://github.com/D4Vinci/Scrapling/blob/main/docs/migrating-to-v0.5.md) says `DynamicFetcher` and its sessions were removed; browser work now uses `StealthyFetcher` and its associated sessions. `solve_cloudflare=False` leaves the explicit solver off, but does not turn off the whole browser backend's anti-bot behavior. There is no verified universal “safe mode” switch in this card. Prefer ordinary requests or the approved feed. If your own site needs JavaScript, have IT review a separate browser plan with the site's operator before using it.

## Six small business use cards

All businesses and schedules below are illustrations. Each card requires robots instructions to be obeyed, one request at a time at the agreed low rate, a truthful business identity, no login areas, and no personal data. The collection library carries no license fee under its [BSD license](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE). Labor, hosting, and any supplier or directory data charges are unknown until the particular job is scoped. None of these collection steps requires a paid AI model.

### Own website quality check

- **Illustration:** A hardware store checks its public pages after a release and once daily.
- **Data:** Page URL, response result, displayed price, and presence of the structured facts called schema that help machines understand a page.
- **Permission:** Written internal approval for the store's own domain; use its existing site export or monitoring API where available.
- **Controls:** Limit paths to public product and information pages. Exclude account, checkout, and search forms. Apply all shared controls above.
- **Review and cost:** The store manager checks a flagged price against the approved catalog before changing it. Library license fee: none; staff and hosting costs unknown.

### Supplier catalog with written approval

- **Illustration:** A parts distributor updates a comparison table weekly at the supplier's agreed time.
- **Data:** Product code, variant, price, availability label, and source URL. Copy descriptions or images only if licensed.
- **Permission:** Supplier's written scope and reuse permission. Prefer its official catalog feed or API.
- **Controls:** Approved public paths only, robots obeyed, agreed low rate, business identity, no login or personal data. The [Shopify template](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/platform-templates.md) returns one item per variant; an accessible store endpoint is still subject to permission.
- **Review and cost:** Purchasing checks units, currency, and freshness before ordering. Library license fee: none; feed charges and implementation cost unknown.

### Own business listings

- **Illustration:** A local repair company checks its directory listings monthly.
- **Data:** Business name, business phone, public address, hours, and website URL. Exclude reviewer profiles and personal contact details.
- **Permission:** Each directory's terms must allow this collection; ownership of the business does not confer ownership of the directory. Prefer approved account exports or APIs.
- **Controls:** Record each directory separately. Obey its robots rules and lower rate, identify the business, and avoid login areas.
- **Review and cost:** A person reviews discrepancies and submits corrections through the directory's approved process. Library license fee: none; directory access and staff costs unknown.

### Public government data without an API

- **Illustration:** A small contractor checks an agency's public tender notices each business day.
- **Data:** Notice identifier, project title, deadline, and public source URL. Leave named contacts and other personal details out.
- **Permission:** Check that agency's terms and data license; confirm that no approved API or feed already exists.
- **Controls:** Approved notice pages only, robots obeyed, low rate, business identity, no account areas or personal data.
- **Review and cost:** A person reads the actual notice before bidding. Library license fee: none; operator and hosting costs unknown.

### Competitor public prices within terms

- **Illustration:** A bike shop checks a short approved list of comparable public prices weekly.
- **Data:** Product name, size or variant, price, currency, availability label, and observation time. No customer or seller profiles.
- **Permission:** The site's terms must permit the proposed collection and use. If unclear, request written permission or leave the target out. Prefer any offered licensed feed.
- **Controls:** Obey robots instructions and the agreed rate, identify the business, avoid login areas, and stop on any challenge or block.
- **Review and cost:** The owner checks whether prices include tax, delivery, or membership conditions before comparing them. Library license fee: none; ongoing maintenance cost unknown.

### Own help pages for a support bot

- **Illustration:** A software consultancy refreshes its public help text after an approved documentation change.
- **Data:** Article title, clean text, public URL, and version date. Exclude customer tickets, account details, and embedded personal information.
- **Permission:** Internal authorization and rights to all included content. Prefer a direct documentation export.
- **Controls:** Approved help paths only, robots obeyed, low rate, business identity, no login areas. A person reviews the clean text before it reaches an AI service.
- **Review and cost:** Test answers against the source and provide a route to a person. Library license fee: none; collecting and converting text needs no AI model. The downstream bot's model, hosting, and support costs are unknown until selected.

## Data handling and first-run acceptance

Scrapling can return records for your code to export; [spider documentation](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/advanced.md) also describes optional response storage and restart checkpoints. Treat those stored pages and pending requests as business data. If enabled, they need the same access restrictions and deletion schedule as the final table. Do not retain whole pages merely because a single price was needed.

- [ ] Confirm that only the approved domains and paths were requested, including redirects and robots requests.
- [ ] Compare a small sample of extracted values against the source pages by hand.
- [ ] Distinguish an unavailable field from zero, unavailable stock from a count, and a stale record from a current observation.
- [ ] Verify that challenges, refusals, and unexpected personal data halt the whole target job.
- [ ] Verify that no form is submitted and no account state is loaded.
- [ ] Record observed request count, schedule, output destination, and reviewer.
- [ ] Confirm retention cleanup for raw pages, checkpoints, tables, and copies sent to any AI provider.
- [ ] Recheck permission when the target's terms, scope, or requested fields change.

## Risk review

This is informational, not legal advice. Ask a lawyer licensed in your state about uncertain scope, contracts, personal data, or a stop notice. Have authorization, purpose, paths, data fields, rate, retention, and reuse permission in writing before questionable work starts.

| Risk | Law or contract to review | How to lower it |
|---|---|---|
| Continuing after an access barrier or withdrawal | Federal Computer Fraud and Abuse Act, [18 U.S.C. § 1030](https://www.law.cornell.edu/uscode/text/18/1030) | Stop the target job; do not get around a login, challenge, or block; seek advice before resuming |
| Violating agreed site restrictions | Site terms and other contracts | Preserve the relevant clause and account agreements; obtain written permission when terms do not cover the use |
| Reusing protected text or images | Copyright owner's rights under [17 U.S.C. § 106](https://www.law.cornell.edu/uscode/text/17/106); fair use under [§ 107](https://www.law.cornell.edu/uscode/text/17/107) is context dependent | Collect necessary facts; license expressive material; review intended redistribution and AI use |
| Collecting personal information | Applicable privacy laws, including the [California Consumer Privacy Act](https://oag.ca.gov/privacy/ccpa) where it applies | Exclude personal fields from this baseline; obtain a separate privacy review if needed |
| State computer-access liability | State computer crime laws as well as federal law, including [California Penal Code § 502](https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?sectionNum=502.&lawCode=PEN) and [Texas Penal Code § 33.02](https://tcss.legis.texas.gov/resources/PE/htm/PE.33.htm#33.02) | Have local counsel review the target and method; do not assume a federal decision settles state claims |
| Overloading a site or ignoring its limits | Site terms and potential civil claims | Use the approved rate and total request budget, coordinate workers, and stop on refusal |

The owner's CAPTCHA or block is a clear operational stop signal. Continuing is where exposure increases. The program's ability to continue does not decide whether your business may do so.

## Sources and review dates

Public source documents were read on November 3, 2026. The settings above reflect that source snapshot; recheck the current version before implementation.

- [README, Installation, Optional Dependencies, and Disclaimer](https://github.com/D4Vinci/Scrapling): Python requirement, exact install commands, optional browser dependencies, and disclaimer; reviewed 2026-11-03 at 21:36:55 UTC.
- [BSD-3-Clause license](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE): permitted use, redistribution conditions, endorsement restriction, and no warranty; read 2026-11-03 at 21:34:53 UTC.
- [Request documentation](https://github.com/D4Vinci/Scrapling/blob/main/docs/fetching/static.md): request options and retry counting; read 2026-11-03 at 21:34:56 UTC.
- [v0.5 migration guide](https://github.com/D4Vinci/Scrapling/blob/main/docs/migrating-to-v0.5.md): removal of `DynamicFetcher`, browser backend, explicit solver default, and robots failure changes; read 2026-11-03 at 21:34:58 UTC.
- [Advanced spiders](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/advanced.md): concurrency, delay, optional storage, and hooks; read 2026-11-03 at 21:34:59 UTC.
- [Generic templates](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/generic-templates.md): link following, feeds, and content conversion; read 2026-11-03 at 21:35:01 UTC.
- [Shopify template documentation](https://github.com/D4Vinci/Scrapling/blob/main/docs/spiders/platform-templates.md): public collection endpoints and variant output; read 2026-11-03 at 21:35:02 UTC.
- [MCP guide](https://github.com/D4Vinci/Scrapling/blob/main/docs/ai/mcp-server.md): assistant tools, saved state, and browser actions; read 2026-11-03 at 21:35:04 UTC.
- [Spider source](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/spider.py): defaults and status-based refusal detection; read 2026-11-03 at 21:35:05 UTC. [Engine source](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/engine.py): individual blocked-request handling; read at 21:35:07 UTC.
- [Robots source](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/spiders/robotstxt.py): wildcard group and response-failure behavior; read 2026-11-03 at 21:35:08 UTC.
- [Request-session source](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/static.py): constructor flags, generated headers, and attempts; read 2026-11-03 at 21:35:14 UTC.
- [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), section 1: robots rules are not access authorization; read 2026-11-03 at 21:35:15 UTC.
- [18 U.S.C. § 1030](https://www.law.cornell.edu/uscode/text/18/1030): federal computer-access and damage provisions; read 2026-11-03 at 21:34:58 UTC.
- [17 U.S.C. § 106](https://www.law.cornell.edu/uscode/text/17/106) and [§ 107](https://www.law.cornell.edu/uscode/text/17/107): copyright rights and fair-use factors; read 2026-11-03 at 21:35:12 UTC and 21:35:14 UTC respectively.
- [California Attorney General's CCPA guide](https://oag.ca.gov/privacy/ccpa): covered businesses, rights, and exceptions; read 2026-11-03 at 21:35:06 UTC.
- [California Penal Code § 502](https://leginfo.legislature.ca.gov/faces/codes_displaySection.xhtml?sectionNum=502.&lawCode=PEN): state computer-access and data-copying provisions; browser excerpts reviewed 2026-11-03 at 21:39 UTC.
- [Texas Penal Code § 33.02](https://tcss.legis.texas.gov/resources/PE/htm/PE.33.htm#33.02): access without effective owner consent; read 2026-11-03 at 21:37:01 UTC.

*This kit is informational, not legal advice. Mentions of third parties are nominative fair use. No affiliation or endorsement is implied.*