Send Millions of Web Scraping Requests Asynchronously
Submit batches of up to 50,000 URLs or structured scraping tasks in a single API call. Our dedicated job queue handles retries, residential proxy rotation, and anti-bot challenges in the background — delivering clean JSON via real-time stream or signed Webhooks.
- ✓ Submit up to 50,000 target URLs in a single batch POST request
- ✓ Automatic background retries with zero timeout failures
- ✓ Receive results via signed Webhook callback or real-time SSE stream
- ✓ Pay only for successful 200 OK extracted responses
curl -X POST https://api.crawlomatic.com/v1/jobs \
-H "X-API-Key: cm_live_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"task": "zillow-scraper",
"urls": ["https://www.zillow.com/miami-fl/", "https://www.zillow.com/austin-tx/"],
"country": "us",
"webhookUrl": "https://your-app.com/webhooks/crawlomatic"
}'
{
"status": "queued",
"job_id": "job_9f82a1c4-71b0-42e9-91f2",
"endpoint": "zillow-scraper",
"poll_url": "https://api.crawlomatic.com/v1/jobs/job_9f82a1c4-71b0-42e9-91f2",
"stream_url": "https://api.crawlomatic.com/v1/jobs/job_9f82a1c4-71b0-42e9-91f2/stream"
}
Why Data Teams Choose Crawlomatic for Async Scraper Service
Scrape Millions of Pages Without Timeouts
Decouple request submission from data extraction. Submit massive URL batches and let our distributed worker fleet process them 24/7.
Automatic Anti-Bot & Proxy Retries
Failed attempts automatically rotate sticky residential sessions across 40+ countries and escalate to dynamic browser rendering.
Webhook & Real-Time SSE Delivery
Stream records as they arrive via /v1/jobs/{id}/stream, poll job status, or receive an HMAC-SHA256 signed webhook payload upon completion.
How Crawlomatic Works in 4 Simple Steps
Focus on building your product while our infrastructure handles every layer of web data extraction.
Send URL or Select SDE
Call /v1/html with any URL or choose one of our 33 Verified Structured Data Endpoints.
Smart Geo-Proxy Routing
Requests automatically route through 40M+ country-matched residential proxies with session persistence.
Anti-Bot & JS Rendering
Dynamic JavaScript pages and Cloudflare, Kasada, DataDome, or PerimeterX challenges are solved automatically.
Clean JSON, CSV or Webhook
Receive schema-validated JSON immediately, stream via SSE, or schedule recurring deliveries in DataPipeline.
Power Up Your Infrastructure or Go Low-Code
Every Crawlomatic subscription gives you three flexible ways to collect web data depending on your workflow.
Universal Scraping API
Pass any public URL to /v1/html and receive unblocked raw HTML with automatic proxy rotation and dynamic JS rendering.
- ✓ 40M+ Residential & Mobile Proxies
- ✓ 50+ Country Geotargeting
- ✓ Automatic Anti-Bot Bypass
33 Structured Data Endpoints
Skip HTML parsing completely. Call /v1/scrape/{slug} to get clean, schema-validated JSON from 33 high-value domains.
- ✓ Pre-Built & Maintained Parsers
- ✓ Multi-Page Pagination & Deduplication
- ✓ Real-Time SSE Streaming Support
DataPipeline Visual Scheduler
Automate recurring hourly, daily, or weekly scraping jobs from the Crawlomatic Dashboard and receive CSV, JSON, or Webhook deliveries.
- ✓ Visual Cron Job Builder
- ✓ One-Click CSV & JSON Export
- ✓ Signed Webhook Delivery
Verified Structured Data Endpoints for Async Scraper Service
Extract properties for sale, for rent, and recently sold on Zillow with Zestimates, rent estimates, and broker contacts.
Detect 7,300+ web technologies, CMS platforms, payment gateways, pixels, and analytics tools on any domain.
Extract verified customer reviews, star ratings, review dates, replies, and company TrustScores from Trustpilot.
Extract luxury watch listings, reference numbers, asking prices, dealer locations, and box/papers status from Chrono24.
Extract complete guest reviews, star ratings, stay dates, language tags, and host responses from any Airbnb listing.
Extract active job listings, compensation ranges, remote status, company ratings, and job descriptions from Indeed.
Extract verified business emails, phone numbers, LinkedIn, X/Twitter, Instagram, and team links from any company domain.
Extract Instagram profiles, account status (ACTIVE / REMOVED / RESTRICTED), 12 timeline posts, 12 Reels with video play counts, engagement rates, and optional B2B website contact & tech-stack enrichment — with zero Instagram login.
Aggregate, normalize, and compare live luxury watch listings across Chrono24, eBay, WatchCharts, Bob’s Watches, and Watchfinder with sub-variant dial/bezel/bracelet filtering.
Frequently Asked Questions About Async Scraper Service
How is Async Scraper Service different from the synchronous Scraping API?
Synchronous endpoints return data immediately in the same HTTP connection. Async Scraper Service returns a job_id in under 50ms and processes thousands of URLs in the background, pushing the completed dataset to your webhook or polling URL.
Are failed requests charged against my API credit balance?
No. Crawlomatic only deducts API credits when a job item completes with a verified 200 OK response.
Pricing Plans for Async Scraper Service
Start free with 5,000 API credits. Every plan includes full access to Async Scraper Service and all 33 Verified Structured Data Endpoints.
Hobby
For small projects and personal use.
- ✓ All Structured Data Endpoints Included
- ✓ Universal Web Scraping API (/v1/html)
- ✓ Smart Residential Proxy Rotation
- ✓ Automatic Retries & 24h Cache
Startup
For low-volume production workflows.
- ✓ All Structured Data Endpoints Included
- ✓ Dynamic JavaScript Rendering
- ✓ Automatic Anti-Bot & CAPTCHA Bypass
- ✓ Async Batch Queue (/v1/jobs) & SSE
Business
For production-grade scraping at scale.
- ✓ Global 40+ Country Residential Proxies
- ✓ No-Code DataPipeline Scheduler
- ✓ Advanced Anti-Bot Bypass Included
- ✓ Unlimited Analytics History
Scaling
For scaling multi-source data operations.
- ✓ Global 40+ Country Residential Proxies
- ✓ No-Code DataPipeline + Webhooks
- ✓ Priority High-Concurrency Routing
- ✓ Pay-As-You-Go Overage Protection
Professional
For recurring, high-volume data teams.
- ✓ Priority Engineering Support
- ✓ 1 Free Custom Domain Endpoint / Qtr
- ✓ Dedicated Rendering Capacity
- ✓ 99.9% Uptime SLA Guarantee
Enterprise
Custom volume, SLAs & dedicated clusters.
- ✓ Unlimited Custom Scraper Builds
- ✓ Dedicated Slack & Telegram Channel
- ✓ Direct S3 / Parquet / Postgres Sync
- ✓ Custom Legal & Compliance SLA
Guide to API Credit Multipliers
You are only charged API credits when our gateway returns a successful 200 OK response. Failed requests or automatic retries cost 0 credits.
| Request Type / Endpoint | Routing & Infrastructure | API Credit Cost | Effective Cost (Scaling Plan) |
|---|---|---|---|
Standard Scraping API (/v1/html) & Tech Scanner (/v1/scan) |
Smart Rotating Proxy Pool + Automatic Retries | 1 API Credit | $0.09 / 1,000 requests |
Structured Data Endpoints (/v1/scrape/{slug} — 33 Verified Domains) |
Geo-Targeted Residential Proxy + Structured JSON Parser | 5 API Credits | $0.47 / 1,000 requests |
Protected Anti-Bot JS Rendering (render=true) |
Full Browser JS Rendering + Managed Anti-Bot Bypass | 25 API Credits | $2.35 / 1,000 requests |