getting started
How to start a crawl
Enter a URL, pick a crawl mode, preview the scope, and run your first workspace crawl in Silkra.
Start screen
After sign-in you land on the start screen. Enter a URL, reopen a saved workspace from the rail, or upload a URL list. The helper chat on the start screen can answer setup questions.
Crawl modes
Silkra auto-detects mode from your URL when possible.
| Mode | Discovery | Best for |
|---|---|---|
| Spider | Follows links from a starting URL | Full site audits, structure analysis |
| Sitemap | Reads URLs from sitemap.xml | Known URL lists, indexed page coverage |
| List | Only URLs you provide (CSV or paste) | Specific pages, competitor URL sets |
Spider vs preview scope
Toggle on the start screen before entering a URL.
| Action | When to use |
|---|---|
| Spider | Start crawling from your URL with scope options below |
| Preview scope | Map folders or sitemaps first, then narrow before crawling |
Scope preview
After entering a URL in preview mode, Silkra maps folders from the sitemap, homepage links, or robots.txt. Click folder pills to include or exclude sections. For sitemap indexes, pick child sitemaps. If no map is found, enter folder paths manually. Unreachable sites show a no-map fallback.
Spider scope
Applies to spider mode only. Subfolder scope auto-applies when you start from a path like example.com/blog. Defaults live in Settings → Spider.
| Scope | Follows | Example |
|---|---|---|
| Domain | All pages on the domain, including subdomains | example.com → entire site |
| Subfolder | Only the starting path and below | example.com/blog → /blog/* |
| Subdomain | Only the same subdomain | blog.example.com → not shop.example.com |
| Exact | Single URL only | No link discovery |
Max pages and URL rules
Set max pages on the start screen before you crawl. Expand Refine scope to add include or exclude URL patterns. Plan limits cap the maximum per run. Spider settings (⌘J → Spider) hold defaults for rendering, politeness, and robots.txt.
URL list uploads
Switch to list mode by uploading a CSV of URLs or pasting a list. Preview shows the URL count before you start. Only listed URLs are crawled.
During the crawl
While crawling, the calm crawl view replaces the table. It shows live progress, recent URLs, and a checklist for scrape and semantic map lanes. You can pause, stop, or cancel from the title bar.
After the crawl
When intelligence finishes, the crawl table fills in. Map and Search unlock in the view switcher. The sidebar opens automatically for new workspace crawls.
Versions and re-crawl
Each crawl run is a version inside the workspace. Switch versions from the title bar version selector. Start a new crawl from the title bar or ⌘N to add another version without losing prior runs.