getting started
Spider settings
Defaults for rendering, crawl limits, etiquette delays, URL rules, and user-agent in the Spider tab.
Where to find them
Open Settings with ⌘J and select Spider. Changes auto-save and apply to the next crawl in this workspace. Max pages can also be set on the start screen before each run; plan limits cap the maximum per scrape.
JavaScript rendering
Controls how the spider handles JavaScript-heavy pages.
| Mode | Behavior | Use when |
|---|---|---|
| Auto-detect | Renders only when a page needs JavaScript | Default — recommended for most sites |
| Always render | Every page runs in a headless browser | SPAs, client-side routing, dynamic content |
| Static HTML only | Fetches HTML without executing JS | Server-rendered CMS, static sites, speed priority |
Crawl limits
| Setting | Default | Range | What it does |
|---|---|---|---|
| Max pages | 1,000 | 1 — plan cap | Stops the crawl after this many URLs |
| Request timeout | 30 sec | 15 — 120 sec | Marks a page as timed out if it does not respond in time |
Etiquette
| Setting | Default | What it does |
|---|---|---|
| Respect robots.txt | On | Honors crawl directives in robots.txt before requesting pages |
| Request delay | 500 ms | Pause between requests to the same crawl (0 — 10,000 ms) |
Keep robots.txt enabled on sites you do not own. Increase request delay for rate-limited or sensitive hosts.
URL rules
Normalize and filter which URLs the spider collects.
| Setting | Default | What it does |
|---|---|---|
| Strip tracking params | On | Removes utm_*, gclid, fbclid, and other analytics params from URLs |
| Max redirects | 10 | Stops following a redirect chain after this many hops (1 — 50) |
Include and exclude patterns
Glob-style patterns filter URLs during the crawl. Empty include list = no restriction.
| Pattern type | Purpose | Examples |
|---|---|---|
| Include only | Crawl URLs matching these patterns | `/blog*`, `*products*`, `/en/*` |
| Exclude | Skip URLs matching these patterns | `/admin*`, `*.pdf`, `/tag/*` |
You can also set include/exclude patterns from Refine scope on the start screen before a crawl.
User-Agent
Identify requests with a preset or custom header.
| Preset | Purpose |
|---|---|
| Default | Silkra spider user-agent |
| Chrome | Full Chrome browser string |
| Googlebot | Googlebot-compatible string |
| Bingbot | Bingbot-compatible string |
Leave the custom field empty to use Default. Edit the field for a fully custom user-agent string.
Max response size
Default 10 MB. Set to 0 for unlimited. Pages larger than the limit are skipped.
Spider scope
Spider scope (domain, subfolder, subdomain, exact) controls link-following during a spider crawl. It is not in the Spider settings tab — set it on the start screen or from crawler controls in the title bar when starting a spider crawl.
| Scope | Follows | Example |
|---|---|---|
| Domain | Entire domain, including subdomains | `example.com` → whole site |
| Subfolder | Starting path and below | `example.com/blog` → `/blog/*` |
| Subdomain | Same subdomain only | `blog.example.com` only |
| Exact | Single URL | No link discovery |
Reset to defaults
Use Reset at the top of the Spider settings panel to restore all spider defaults. Confirm when prompted.
Quick presets
Starting points — adjust per site.
| Scenario | Rendering | Request delay | Notes |
|---|---|---|---|
| Own site, fast | Static HTML | 100 — 300 ms | robots.txt optional if you control the host |
| External site, polite | Auto-detect | 500 — 1000 ms | Keep robots.txt on |
| JavaScript SPA | Always render | 500 ms | Increase timeout for slow apps |