ScrapFly Review: A Data Collection API Provider Analysis

Scrapfly: General Information

ScrapFly specializes in data collection APIs for developers. The company behind it is Joam Intelligence, LLC, a U.S.-based business located in Middletown, Delaware. 

On LinkedIn, the company is listed in the 11-50 employee range, with its headquarters officially registered in Paris, France, and founded in 2021. This indicates that ScrapFly is a small, relatively young, and growing business. 

Despite its size, it has managed to diversify its offerings with a set of three different data collection APIs, whereas most competitors typically focus on a single product.

Strict KYC procedures are in place to prevent misuse of its technology, making responsible use a core value of the company.

👍 Pros:

• Built-in integrations with datacenter and residential proxies
• Multiple data collection API products available
• Covers a wide range of use cases
• Strong KYC procedures for ethical usage
• Free trial available
• Extensive integration guides, tutorials, and an academy for learning
• Well-documented APIs
• Visual API playground for testing requests

👎 Cons:

• No support for mobile proxies
• Limited refund policy
• No fully flexible pay-as-you-go plan

Data Collection APIs: Products and Features

In this section, you will discover the different data collection APIs offered by provider, along with their features.

Web Scraping API

A Web Scraper API to retrieve raw HTML from most web pages, including those protected by advanced anti-scraping measures.

Features:

• Scalable scraping: Achieve high scraping success rates with a battle-tested API that scales effortlessly with your needs.
• Customize any part of the request: Get full control over request components, including headers, cookies, data payloads, and even the operating system.
• Automatically bypass scraper blocking: Avoid anti-bot challenges and JavaScript-based obstacles for uninterrupted scraping.
• Automatic proxy rotation: 130M+ proxies from 120+ countries from a vast pool of residential or datacenter proxies.
• Sessions for persistent proxies: Maintain persistent sessions for both residential and datacenter proxies, ensuring uninterrupted data scraping.
• Support for real web browsers: Scrape JavaScript-powered sites, automatically loading all elements using real web browsers with configurable fingerprints.
• Send browser commands: Execute real-time browser commands, enabling interaction with dynamic web content like filling forms, clicking buttons, and scrolling.
• Access browser data: Directly capture background requests, hidden data, and delayed elements from the browser for more comprehensive scraping.
• Switch sessions between browser and non-browser requests: Seamless switching between browser and non-browser sessions for flexible scraping setups.

AI Web Scraping API

This API uses AI models to automate web scraping via LLM prompts.

Features:

• Automatically identify data objects: Extract common data objects like products, reviews, and articles without requiring additional input.
• Predictable outputs with strong schema: Provide structured outputs based on strong schema models, ensuring consistent and reliable results with every scrape call.
• Support for any LLM prompt to query the scraped data: Leverage an LLM engine optimized for scraping, allowing freeform querying of the scraped data.
• Prompt for exact data structures and formats: Request specific output formats such as Markdown, JSON, CSV, or any other structure that suits their needs.
• Same Web Scraping API browser-related features

Extraction API

This API leverages the power of AI to automate data extraction from any document. 

Features:

• Understand your data with LLM prompts: Ask freeform questions or issue specific commands using LLM prompts, providing flexible and intuitive data querying.
• Use LLMs to extract structured data: Leverage LLMs to extract clean, structured data (like JSON or CSV) from any document.
• Support for almost any data format: Support a broad range of formats, including HTML, PDFs, JSON, CSV, and more.
• Auto-extract objects: Automatically extract structured data such as products and reviews with AI-powered precision, using a predictable schema format.
• Quality self-reports: Each extraction includes a report on the coverage and quality of the data extracted.
• Define your own extraction rules: Customize your extraction process by creating templates with XPath/CSS selectors and built-in processors, ensuring you extract exactly what you need. Templates allow for fine-tuning to meet specific data extraction needs.

Screenshot API

An API to capture high-quality screenshots of web pages without interruptions. This solution uses real browsers to ensure accurate rendering.

Features:

• Capture clean screenshots: Block ads, cookie pop-ups, and other on-page distractions, ensuring clean and accurate screenshots.
• Target full page or specific areas: Capture either the entire webpage or specific sections for more precise control.
• Change viewport resolution: Adjust the viewport resolution to fit your device requirements or target display settings.
• Control the screenshot flow: Manage the entire screenshot process, from waiting for elements to load to scrolling and executing JavaScript.
• Use server-side caching to speed up: Cache captured screenshots on ScrapFly servers for easier integration with real-time tools.
• Export to different screenshot formats: Export screenshots in formats such as JPG, PNG, WebP, or GIF.
• Same anti-bot bypass capabilities as Web Scraping API

Pricing

ScrapFly operates on a credit-based system. Each plan gives you access to some credits you can then use for any of its data collection API products.

Below is a summary table of the available pricing plans:

Note that when you subscribe to a plan, you receive a prepaid monthly quota of API credits. Once this quota is used up, you switch to a pay-as-you-go model, where additional usage is charged based on the “Extra API” column in the pricing table.

To better understand API credit usage, consider that a basic scraping request using a datacenter proxy without JavaScript rendering costs 1 credit. Instead, a request with a residential proxy and browser rendering costs 30 credits.

In detail, here is a breakdown of how credits are allocated for the Web Scraping API product:

Overall, ScrapFly’s pricing is competitive and aligned with industry standards for data collection API solutions. A free trial with 1,000 free API credit is available to all newly subscribed users.

The main drawback is the lack of a truly flexible pay-as-you-go option that is not tied to a subscription plan. Also, while the $30/month entry price is lower than many competitors, it may still be too high for individual developers who need fewer than 200,000 credits.

Features

These are the common features provided by all scraping APIs offered by ScrapFly:

• 99.99% uptime
• Support for workflow automation with no-code platforms like Zapier, Make, and N8N
• Support for building LLM and RAG applications with LlamaIndex and LangChain
• Official Python SDK, TypeScript SDK, and Scrapy SDK
• A dedicated web UI platform to configure, control, and monitor all of your ScrapFly projects in real-time

Use Cases

These are all use cases officially supported by the data collection API provider:

• AI Training: Collect the latest images, videos, and user-generated content for AI model training.
• Compliance and Security: Scrape online presence to validate compliance and security measures.
• eCommerce: Extract product details, reviews, and pricing to enhance brand awareness.
• Financial Services: Gather stock market, shipping, and financial data to improve datasets.
• Fraud Detection: Monitor product listings to detect fraud and counterfeit activity.
• Jobs Data: Scrape job listings, salary data, and company reviews for job market insights.
• Lead Generation: Extract online profiles and contact details to improve lead generation strategies.
• Logistics: Gather shipping, tracking, and container price data to optimize deliveries.
• News and Media: Scrape the latest articles, headlines, and media content to stay informed.
• Real Estate: Collect property listings, agent details, and sales history for better decisions.
• SERP and SEO: Scrape search engine results, keywords, and rankings to refine SEO strategies.
• Social Media: Extract profiles, posts, and comments to enhance online presence and lead generation.
• Travel and Hospitality:Scrape hotel listings, reviews, and prices to improve travel and booking services.

In particular, the list of supported sites for web scraping includes Aliexpress, Amazon, BestBuy, Bing, Booking, Crunchbase, Domain.com.au, eBay, Etsy, Fashionphile, Glassdoor, Goat, Google, Homegate, Idealista, Immobilienscout24, Immoscout24, Immowelt, Indeed, Instagram, Leboncoin, Nordstrom, Realestate, Realtor, Reddit, Redfin, Rightmove, Seloger, Similarweb, StockX, Threads, TikTok, TripAdvisor, Trustpilot, Twitter (X), Vestiaire Collective, G2, Walmart, Wellfound, LinkedIn, YellowPages, Yelp, YouTube, Zillow, ZoomInfo, and Zoopla.

User Reviews

ScrapFly’s official site claims that the company is trusted by 30,000+ developers. While it is difficult to verify that number, a more reliable way to gauge a company’s service quality is by checking online reviews.

As is sometimes the case in this industry, ScrapFly appears to have focused its review efforts on a single platform. That must be the reason why G2 and Trustpilot have no reviews. On the contrary, Capterra lists 138 reviews with an impressive 4.9/5 average rating.

The vast majority of reviews are 5-star, which is a sign of very high user satisfaction. The only notable criticism is that the anti-scraping engine seems to have some problems against certain Cloudflare-protected pages.

Ethics and Privacy

ScrapFly complies with GDPR regulations in both its data processing practices through its APIs and its handling of user data. However, there is no mention of CCPA compliance.

To enforce responsible and ethical use of its data collection APIs, all users must go through a comprehensive KYC (Know Your Customer) process. This verification step helps restrict access to authorized business cases that align with ethical market practices.

During registration, users are required to provide detailed information and verify their phone number. This process effectively prevents most bots and malicious actors from accessing the data collection products.

Additionally, ScrapFly’s Terms of Service include a strict list of prohibited use cases to guarantee that the platform is used legally and ethically. Users must also confirm they have obtained the necessary permissions before scraping targeted websites or other data sources.

On the less bright side, subscription fees and extra usage charges are generally non-refundable. Limited refund options are considered:

• If less than 10% of the subscription’s API credits have been used.
• In cases of a forgotten running subscription.

Additional Tools and Services

ScrapFly is a company focused on providing data collection APIs, with no additional products or services beyond web scraping tools. 

Still, it does offer a range of utilities to assist with web scraping, including:

• Base64 Encoder/Decoder: Convert text or files to and from Base64 format.
• URL Prettier:Parse a URL into a structured, easy-to-read format.
• URL Encoder/Decoder:Encode or decode URLs for web compatibility.
• XPath & CSS Selector Tester: Test and refine XPath and CSS selectors for web scraping.
• cURL to Python Converter: Convert cURL commands into equivalent Python code.
• Fingerprint Testing Tools: Dedicated pages to test JA3, HTTP/2, IP Info, canvas, WebGL, audio, screen, and browser fingerprints.

Final Verdict

ScrapFly is a data collection API provider that focuses on offering reliable and trustworthy services, supported by strict KYC procedures. The company is clearly passionate about what it does, as the development team has contributed to several open-source libraries online.

The company offers a wealth of resources through its Academy, blog posts, documentation, and GitHub page. The data collection APIs it provides are varied and cover a wide range of user needs, from traditional scraping tasks to more advanced AI integrations. 

One standout element is the Extract API, which allows scraping data from documents, not just web pages. That sets ScrapFly apart from other providers in the market. There are not many online reviews from users, but these are all highly positive.

The main downsides include the absence of a flexible pay-as-you-go plan and some questionable refund policies.

In summary, ScrapFly is an excellent data collection provider with reasonable pricing, a strong focus on ethics, and support for open-source projects. Its varied product offerings cover most user needs, making it a solid choice for developers.

Final verdict: 3.9/5

FAQs