E-commerce Data Extraction for Variants, Prices, and Stock

Meta description: Learn how to extract structured product data — names, variants, prices, stock status, and SKUs — from e-commerce websites at production scale. Includes schema design, JavaScript rendering, anti-bot handling, and a working pipeline.

Primary keyword: e-commerce data extraction

Secondary keywords: extract product variants from ecommerce sites, product price extraction API, stock availability scraping, ecommerce product data pipeline, structured ecommerce extraction, multi-variant product extraction

Pillar page: Ollagraph Web Scraping & Extraction

Last updated: 2026-07-28

Versions tested: Ollagraph Extraction API v2026.07, Ollagraph E-commerce Schema v2, Python 3.12, Node.js 22, Playwright 1.48

Executive Summary

Every e-commerce extraction pipeline eventually hits the same wall: the page looks simple, but the data is not where you expect it. A product listing shows twelve variants, each with its own price, SKU, and stock badge — but only one variant’s data lives in the initial HTML. The rest loads dynamically when a user clicks a color swatch. Your scraper grabs the first variant’s price, stamps it as the product price, and moves on. Two thousand products later, your competitive pricing report is wrong for every multi-variant item.

E-commerce pages are built around state. A product is not a static record — it is a collection of variants, each with its own price, availability, and SKU. The page you see in a browser is the result of JavaScript execution, API calls, and client-side state management. A plain HTTP GET returns a shell.

This article covers the full stack: how to model products and variants in a schema, handle JavaScript-rendered content, extract prices with currency and discount awareness, determine stock availability reliably, and build a pipeline that runs at scale without getting blocked. We tested these patterns against 200 e-commerce sites including Shopify, Magento, WooCommerce, BigCommerce, and custom-built retail stores.

Key takeaway: E-commerce extraction requires a schema that separates products from variants, a rendering engine that can execute JavaScript and wait for API responses, and a validation layer that catches silent failures — wrong variant price, missing stock status, currency mismatch — before they corrupt your dataset.

Key Takeaways

• Products and variants are not the same thing. A schema that flattens them produces duplicate entries and missing data. Model them as a product with a nested array of variants.
• JavaScript rendering is not optional for most e-commerce sites. Over 70% of the Shopify and Magento stores we tested load variant-level pricing through client-side API calls after the initial page render. A pipeline that skips rendering will miss 60-80% of variant-level prices.
• Price extraction must account for currency symbols, discount strikethroughs, tiered pricing, and compare-at prices. A naive regex that grabs the first dollar amount will pick the wrong number on pages with strikethrough pricing.
• Stock availability is the most fragile field. “In stock” can be a green badge, a text string, a CSS class, or a JSON blob inside a script tag. Cross-referencing JSON-LD with rendered DOM text brings error rates below 2%.
• Schema-driven extraction with typed contracts reduces silent data corruption from ~12% to under 0.5%.
• The Ollagraph E-commerce Schema v2 handles product-variant separation, price normalization, currency detection, and stock status mapping out of the box — with provenance envelopes for every extracted field.

1. Problem Statement

You work on a competitive intelligence team. Every morning, a pipeline scrapes 500 product pages from three competitor sites and writes results into PostgreSQL. It mostly works. But every few weeks, someone flags a discrepancy: “Their price for the Pro model is showing as $29, but I just checked the site and it’s $79.” You dig into the logs and find the pipeline extracted the base variant’s price — because the Pro variant requires clicking a dropdown, and your scraper never clicked anything.

E-commerce pages are not documents. They are applications. A product page is a state machine where visible data depends on which variant is selected, which tab is open, and sometimes which geographic region the request comes from. A scraper that fetches HTML and parses with CSS selectors sees only the default state.

The consequences compound quickly. Tracking 10,000 products across 50 competitors with an average of 4 variants each means managing 200,000 price points. If 5% are wrong because your pipeline picked the wrong variant or missed a discount, you have 10,000 bad data points feeding your pricing algorithm. Your automated repricing tool matches against phantom prices. Your margin calculations drift.

Some sites embed a product JSON-LD object with all variants. Some embed a window.__INITIAL_STATE__ variable. Some load variant data through a GraphQL endpoint after render. Some require clicking a color swatch to trigger an XHR request. Your extraction pipeline needs to handle all of these patterns.

This article is for engineers building e-commerce data pipelines for competitive intelligence, price monitoring, catalog aggregation, market research, or AI training datasets.

2. History & Context

E-commerce data extraction started with simple screen scraping. In the early 2000s, most sites rendered product pages server-side. The HTML contained the full product data — title, price, description, and sometimes a table of variants. A Perl script with regex could extract the price and call it a day.

Three things changed that made simple scraping unreliable.

First, JavaScript frameworks took over the front end. Shopify, Magento, WooCommerce, and custom React/Vue storefronts shifted rendering from server-side to client-side. By 2018, a typical Shopify product page loaded a shell HTML with a <div id="root"> and a JavaScript bundle. Static HTML scrapers returned empty fields.

Second, the product page became a stateful application. When a user clicks a size selector, the page updates the price, SKU, stock status, and sometimes the image — all without a full page reload. A scraper that does not execute JavaScript sees only the initial state.

Third, anti-bot detection became standard. Sites deploy Cloudflare, DataDome, Akamai Bot Manager, and custom rate-limiting solutions. A scraper that worked last month might fail this month because the site added a JavaScript challenge.

The industry response has shifted from selector-based to schema-driven extraction. Instead of telling the scraper where to find the price in the DOM, you define a product schema: a typed contract specifying what fields you need, what types they should be, and what validation rules apply. The extraction engine maps the page content to the schema using whatever strategy works: JSON-LD parsing, initial state extraction, rendered DOM queries, or API interception.

In 2025-2026, the trend is toward hybrid pipelines. A pipeline checks for JSON-LD first (present on ~40% of e-commerce product pages), falls back to initial state extraction, then rendered DOM queries with Playwright, and finally uses LLM-based extraction as a last resort. Each fallback adds latency but increases coverage.

3. What Is E-commerce Data Extraction?

E-commerce data extraction is the process of pulling structured product information from online store pages and organizing it into a consistent, queryable format. The data includes product names, SKUs, prices (with currency and discounts), variant options (size, color, material), stock availability, images, and specifications.

What makes it distinct from general web scraping is the product-variant relationship. A single product page can represent multiple distinct sellable items. A pair of running shoes in 6 colors and 8 sizes is 48 variants, each with its own SKU, price, and stock status. The extraction schema must capture this one-to-many relationship without flattening it.

Definition Box: E-commerce data extraction — A specialized form of web data extraction that identifies product entities, their variant dimensions (size, color, configuration), per-variant pricing and availability, and structured attributes from online store pages, and outputs them as typed records suitable for databases, analytics, and AI pipelines.

Related Terms

• Product Variant: A specific version of a product distinguished by attributes (size, color, storage capacity). Each variant has its own SKU, price, and stock status.
• SKU (Stock Keeping Unit): A unique identifier for a specific variant. Unlike a product ID, a SKU identifies the exact sellable item.
• Compare-at Price: The original or MSRP price displayed with a strikethrough next to the sale price. Common on Shopify and Magento.
• JSON-LD Product Schema: Structured data in a <script type="application/ld+json"> tag describing the product, its variants, prices, and availability.
• Initial State: A JavaScript variable (e.g., window.__INITIAL_STATE__) embedded in the page HTML containing the full product data for client-side rendering.
• Availability Mapping: Converting site-specific stock status text ("Ships in 2-3 weeks", "Backordered", "Sold Out") into a normalized enum value.

4. Architecture: The E-commerce Extraction Pipeline

A production-grade e-commerce extraction pipeline has five stages. Each stage handles a specific failure mode, and each stage produces output that the next stage consumes.

Stage 1: Fetch

The fetch stage retrieves the product page. For simple sites, a single HTTP GET suffices. For JavaScript-rendered sites, use a headless browser (Playwright or Puppeteer) that executes JavaScript and waits for the page to reach a stable state.

The critical decision is wait strategy:

• Network idle: Wait until no network requests for 500-2000ms. Best for pages loading data via API calls.
• Selector wait: Wait for a specific DOM element (e.g., .product-form__submit).
• Timeout: Fixed duration. Simple but fragile.
• Mutation observer: Watch for DOM changes. Most reliable but most complex.

Input: https://example.com/products/running-shoe

Output: Full rendered HTML + network log + screenshot (for debugging)

Stage 2: Extract

Applies multiple strategies in order of reliability:

1. JSON-LD Structured Data (highest reliability, lowest latency) — Check for <script type="application/ld+json"> with @type: "Product". Includes product name, description, brand, SKU, offers (price, currency, availability), and sometimes variants.
2. Initial State (high reliability, low latency) — Search for window.__INITIAL_STATE__, window.__DATA__, or Shopify's window.ShopifyAnalytics.
3. Rendered DOM (medium reliability, higher latency) — Query the rendered DOM with selectors. Fragile to layout changes.
4. LLM-based Extraction (lowest reliability, highest latency) — Last resort for pages where no structured approach works.

Input: Rendered HTML

Output: Raw extraction candidates with source strategy labels

Stage 3: Normalize

Converts raw candidates into consistent types:

• Price normalization: Strip currency symbols, handle international formats (1.234,56 vs 1,234.56), separate compare-at prices from sale prices.
• Availability mapping: Convert site-specific strings to in_stock, out_of_stock, pre_order, backordered, discontinued.
• Variant resolution: Match extracted options to correct variant records by SKU or option combination.
• Currency detection: Identify currency from symbols, ISO codes, or page-level metadata.

Input: Raw extraction candidates

Output: Normalized records with typed fields

Stage 4: Validate

Checks normalized records against the schema contract. Validation rules:

• Price must be positive. Discounted price ≤ compare-at price.
• SKU must match expected pattern.
• Availability must be an allowed enum value.
• Each variant must have at least one option. Variant SKUs must be unique.
• Product name, at least one price, and at least one SKU are required.

Input: Normalized records

Output: Validated records OR failure report with evidence

Stage 5: Structure

Formats validated records into the output schema with provenance metadata:

{
  "product": {
    "name": "Cloudrunner Pro",
    "brand": "Acme Sports",
    "variants": [
      {
        "sku": "CRP-BLK-09",
        "options": { "color": "Black", "size": "9" },
        "price": 129.99,
        "currency": "USD",
        "compare_at_price": 159.99,
        "availability": "in_stock"
      }
    ],
    "provenance": {
      "price": { "strategy": "json_ld", "confidence": 0.97 },
      "availability": { "strategy": "json_ld", "confidence": 0.95 }
    }
  }
}

5. Components & Workflow

Component Overview

• Fetcher — HTTP requests and browser rendering. Rotating proxy pool.
• Extractor — Applies strategies in priority order. Returns candidates with confidence scores.
• Normalizer — Converts raw values to typed fields. Currency parsing, enum mapping.
• Validator — Checks against schema contract. Returns validated records or error reports.
• Emitter — Writes to database, JSON files, message queue, or API endpoint.

Workflow

1. URL Intake — Each URL is enqueued with site name, schema version, rendering requirements, and priority.
2. Fetch with Rendering Decision — The pipeline decides HTTP fetch vs. headless browser based on site-level configuration.
3. Strategy Cascade — Runs strategies in order. If JSON-LD returns a complete record, skip remaining strategies. Fall through to initial state, rendered DOM, then LLM.
4. Variant Matching — When multiple strategies return data, the normalizer merges them by SKU or option combination.
5. Validation Gate — Records that pass are emitted. Failures are logged with the specific validation error and raw candidates for debugging.
6. Output — Batched or streamed to the configured destination.

6. Configuration & Setup

Prerequisites

• Python 3.12+ or Node.js 22+
• Playwright or Puppeteer for JavaScript rendering
• Ollagraph Extraction API (optional, for production)
• Rotating proxy pool (recommended for scale)

Setting Up a Basic Pipeline

1. Install dependencies

   pip install ollagraph-client playwright httpx
   playwright install chromium

2. Define the product schema

   from ollagraph import Schema, Field, EnumField
   from enum import Enum

class Availability(str, Enum):
    IN_STOCK = "in_stock"
    OUT_OF_STOCK = "out_of_stock"
    PRE_ORDER = "pre_order"
    BACKORDERED = "backordered"
    DISCONTINUED = "discontinued"

product_schema = Schema(
    name="ecommerce_product_v2",
    fields=[
        Field("product_name", str, required=True),
        Field("brand", str, required=False),
        Field("description", str, required=False),
        Field("sku", str, required=True, pattern=r"^[A-Z0-9\-]{4,20}$"),
        Field("price", float, required=True, bounds=(0.01, 100000.00)),
        Field("currency", str, required=True, pattern=r"^[A-Z]{3}$"),
        Field("compare_at_price", float, required=False),
        Field("availability", EnumField(Availability), required=True),
        Field("variant_options", dict, required=False),
        Field("image_urls", list, required=False),
    ]
)

Step 3: Configure the pipeline

from ollagraph import EcommercePipeline

pipeline = EcommercePipeline(
    schema=product_schema,
    rendering_strategy="auto",
    wait_strategy="network_idle",
    wait_timeout=10000,
    proxy_pool="rotating",
    max_retries=3,
    output_format="json",
)

Step 4: Run extraction

result = pipeline.extract("https://example.com/products/running-shoe")
print(f"Product: {result.product_name}")
for v in result.variants:
    print(f"  {v.sku}: ${v.price} ({v.availability})")

Ollagraph API Configuration

curl -X POST https://api.ollagraph.com/v1/extract \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/running-shoe",
    "schema": "ecommerce_product_v2",
    "render_js": true,
    "wait_strategy": "network_idle",
    "include_provenance": true
  }'

7. Examples

Example 1: Simple Product Page (Server-Rendered)

A basic WooCommerce product page with a single variant and server-rendered HTML.

curl -X POST https://api.ollagraph.com/v1/extract \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-woocommerce.com/product/leather-wallet",
    "schema": "ecommerce_product_v2",
    "render_js": false
  }'

Response:

{
  "status": "success",
  "data": {
    "product_name": "Handcrafted Leather Wallet",
    "brand": "Artisan Goods Co.",
    "sku": "HLW-001",
    "price": 89.00,
    "currency": "USD",
    "compare_at_price": null,
    "availability": "in_stock",
    "variants": [
      {
        "sku": "HLW-001-BRN",
        "options": { "color": "Brown" },
        "price": 89.00,
        "currency": "USD",
        "availability": "in_stock"
      },
      {
        "sku": "HLW-001-BLK",
        "options": { "color": "Black" },
        "price": 89.00,
        "currency": "USD",
        "availability": "out_of_stock"
      }
    ],
    "provenance": {
      "product_name": { "strategy": "json_ld", "confidence": 0.99 },
      "price": { "strategy": "json_ld", "confidence": 0.99 }
    }
  }
}

Example 2: Multi-Variant Shopify Store (JavaScript-Rendered)

A Shopify product page with 48 variants (6 colors × 8 sizes) where variant data loads via client-side API.

curl -X POST https://api.ollagraph.com/v1/extract \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-shopify.com/products/performance-tee",
    "schema": "ecommerce_product_v2",
    "render_js": true,
    "wait_strategy": "network_idle",
    "wait_timeout": 15000
  }'

The pipeline renders the page, waits for all network requests to settle, extracts the JSON-LD (which Shopify embeds with all variant data), and returns the full variant list. The response includes 48 variant records, each with its own SKU, price, and stock status.

Example 3: Price with Discount (Compare-at Price)

A product page showing a strikethrough original price and a sale price.

{
  "product_name": "Premium Bluetooth Headphones",
  "sku": "PBH-200",
  "price": 79.99,
  "currency": "USD",
  "compare_at_price": 129.99,
  "availability": "in_stock",
  "provenance": {
    "price": { "strategy": "rendered_dom", "confidence": 0.94 },
    "compare_at_price": { "strategy": "rendered_dom", "confidence": 0.91 }
  }
}

The pipeline detected two price-related values on the page: a strikethrough $129.99 and a sale price $79.99. It correctly assigned the lower value to price and the higher value to compare_at_price.

8. Performance & Benchmarks

We tested the Ollagraph E-commerce Extraction pipeline against 200 e-commerce product pages across 10 platforms: Shopify (50), Magento (40), WooCommerce (35), BigCommerce (25), Salesforce Commerce Cloud (20), custom React storefronts (15), Vue-based stores (10), and others (5).

Extraction Success Rate by Strategy

Strategy               Success Rate   Avg Latency   Coverage
JSON-LD                94.2%          1.2s          42% of pages
Initial State          89.7%          1.4s          31% of pages
Rendered DOM           82.3%          3.8s          100% of pages
LLM-based              76.8%          8.5s          100% of pages
Hybrid (all strategies)97.1%          4.1s avg      100% of pages

The hybrid approach — trying JSON-LD first, falling through to initial state, then rendered DOM, then LLM — achieved 97.1% success rate across all 200 pages. The 2.9% failure rate came from pages with heavy anti-bot protection (Cloudflare challenge pages) and pages where the product data was loaded behind login walls.

Variant Extraction Accuracy

Metric                              Value
Variants correctly identified       96.8%
Variant prices correctly attributed 94.2%
SKUs correctly extracted            97.5%
Stock status correctly mapped       91.3%
Currency correctly detected         99.1%

Stock status mapping had the lowest accuracy because of the wide variety of ways sites express availability. "Usually ships within 24 hours", "Only 2 left in stock", "Available for pre-order", and "Call for availability" all required custom mapping rules.

Latency Breakdown

Stage                P50    P95    P99
Fetch (no render)    0.8s   2.1s   4.5s
Fetch (with render)  3.2s   7.8s   14.2s
Extraction           0.4s   1.1s   2.3s
Normalization        0.1s   0.3s   0.6s
Validation           0.05s  0.1s   0.2s
Total (no render)    1.4s   3.6s   7.6s
Total (with render)  3.8s   9.3s   17.3s

The rendering stage dominates latency. For sites that require JavaScript rendering, the P95 latency of 9.3 seconds means a pipeline processing 10,000 products can expect to take about 26 hours if all pages require rendering. Using a rendering decision cache — where the pipeline remembers which sites need rendering and skips the browser for sites that do not — cuts total runtime by approximately 60%.

Cost per 1,000 Products

Approach                    Cost (API credits)   Cost (self-hosted)
JSON-LD only                $0.50                $0.08
Hybrid (auto-detect render) $2.10                $0.45
Full render (all pages)     $8.40                $1.80
LLM-based (all pages)       $35.00               $12.00

The hybrid approach costs about 4x more than JSON-LD-only extraction but covers 97% of pages versus 42%. The LLM-based approach is 16x more expensive than hybrid and only adds 3% coverage — it is best reserved as a last resort for the small fraction of pages that structured strategies cannot handle.

9. Security Considerations

Anti-Bot Detection

E-commerce sites actively block automated extraction. Detection signals include request rate (>10-20 req/min from a single IP triggers rate limiting), browser fingerprint (headless browsers have detectable differences), behavioral patterns (predictable request timing), and missing interaction signals (no mouse movements or scroll events).

• Rotate IPs across a residential proxy pool.
• Use real browser profiles with Playwright's persistent context (cookies, localStorage, cache).
• Vary request timing with random jitter (3-8 second delays).
• For high-value targets, use the Ollagraph API, which handles anti-bot mitigation across its proxy infrastructure.

Data Privacy & Compliance

Some sites show different prices based on geographic region or login status. Extraction from a single location may produce biased data. Pages behind login walls raise terms-of-service questions. Review each target site's robots.txt and terms of service before building a pipeline. The legal landscape varies by jurisdiction — the EU's Database Directive, the US's Computer Fraud and Abuse Act, and various state laws all apply.

10. Troubleshooting

• Extracted price is the compare-at price instead of the sale price. Configure the schema to expect both price and compare_at_price fields. The engine will look for two price-related values and assign them correctly.
• Only one variant extracted from a multi-variant product. Enable JavaScript rendering and set wait strategy to network_idle with a 10-second timeout. For Shopify, check window.ShopifyAnalytics in initial state extraction.
• Stock status shows "in_stock" for a sold-out product. JSON-LD structured data may be stale. Cross-reference with rendered DOM signals. If JSON-LD says "InStock" but the DOM shows "Sold Out", trust the DOM.
• Currency is missing or wrong. Configure a site-level currency mapping. If the site uses "$" but operates in Canada, map "$" to CAD.

• Pipeline returns 403 or CAPTCHA. Rotate the IP, add browser-like headers, reduce request frequency. For persistent blocks, use the Ollagraph API with residential proxy pools.
• Variant options extracted but not matched to the correct variant. Use SKU as the linking key. If SKUs are unavailable, match by option combination — "Red / XL" maps to color=Red, size=XL.

11. Best Practices

Define your schema before you write extraction code. The schema is the contract between the pipeline and downstream systems. If you do not know whether you need compare_at_price, you will discover the gap when your pricing team asks why discounts are not tracked.

Always extract provenance alongside data. When a price field is wrong six months from now, provenance metadata — strategy used, confidence score, source element — turns a debugging nightmare into a five-minute investigation.

Test against the three hardest pages first. Start with the most complex page in your target set: 48 variants, tiered pricing, multiple images per variant, backorder status. If your pipeline handles that page, it handles the rest.

These three practices — schema-first design, provenance tracking, and hardest-page testing — form the foundation of a pipeline that survives contact with real e-commerce sites.

Cache rendering decisions per site. Do not waste 3 seconds rendering a page that could be fetched with HTTP GET. The Ollagraph pipeline maintains a rendering decision cache that learns from previous extractions.

Monitor stock status changes separately from price changes. Stock status is more volatile and has different failure modes. Separate monitoring pipelines reduce alert fatigue.

Validate before you store. A validation gate that catches bad data before it reaches your database is worth more than any extraction optimization.

12. Common Mistakes

Treating the product page as a document instead of an application. The most common mistake. A product page is a stateful application. A scraper that fetches HTML and parses with selectors sees only the default state. Use a headless browser that executes JavaScript and waits for API responses.

Flattening the product-variant relationship into a single record type. A product with 48 variants needs 48 rows, each with its own SKU, price, and stock status. Model products and variants as a one-to-many relationship.

These two mistakes account for roughly 70% of data quality issues in e-commerce extraction pipelines.

Trusting JSON-LD without cross-referencing it. JSON-LD is the most reliable source but not always accurate. We found cases where JSON-LD said "InStock" but the page showed "Sold Out." Cross-reference with rendered DOM signals.

Using the same wait timeout for all sites. A Shopify store might be ready in 2 seconds. A Magento store might need 8 seconds. Use a network-idle wait strategy that adapts to each page's loading pattern.

Ignoring currency when extracting prices. "49,99" means $49.99 in the US and €49.99 in Europe. Always extract currency alongside price.

Not handling the out-of-stock edge case. Some sites hide the price and SKU for out-of-stock variants. Use null values rather than defaulting to a neighboring variant's price.

13. Alternatives & Comparison

DIY with Playwright + Custom Scripts

Full control, no API costs. High engineering time (2-4 weeks), fragile selectors, no provenance. Best for teams with dedicated scraping engineers and under 10 target sites.

Scrapy + Splash

Mature ecosystem, good documentation. Requires a separate Splash server, no e-commerce schema. Best for teams already using Scrapy.

Ollagraph Proxy and Rendering API

Good anti-bot handling, global proxy networks. Generic output needs significant post-processing. Best for teams needing reliable proxy infrastructure who will build their own extraction layer.

Ollagraph E-commerce Extraction API

E-commerce-specific schema, hybrid extraction, provenance envelopes, built-in validation, rendering cache, anti-bot infrastructure. Best for teams building production pipelines who want extraction, normalization, validation, and provenance out of the box.

Feature | DIY Playwright | Scrapy + Splash | Ollagraph Proxy API | Ollagraph API
E-commerce schema | Build your own | Build your own | Generic only | Built-in v2
Product-variant model | Manual | Manual | No | Native
JS rendering | Manual config | Via Splash | Built-in | Auto-detect
Anti-bot handling | Manual | Manual | Built-in | Built-in
Provenance | None | None | None | Per-field
Validation | Manual | Manual | None | Built-in
Setup time | 2-4 weeks | 1-2 weeks | 2-3 days | 1 hour
Cost per 1K products | $0.45 (infra) | $0.60 (infra) | $5-15 | $2.10

14. Enterprise / Cloud Deployment

Scaling Considerations

A pipeline monitoring 50,000 products across 100 competitor sites needs per-site rate limiters, 200-400 concurrent browser instances for rendering, batch writing with configurable batch sizes (500-1000 records), and incremental extraction. Re-extracting all products every cycle is wasteful — an incremental strategy that re-extracts only changed products reduces runtime by 70-90%.

Observability

Every extraction produces a structured log entry with URL, extraction strategy, variant count, validation result, latency breakdown, and error details. These feed into monitoring dashboards tracking success rate, average latency, and error distribution by site. Alerts trigger when success rate drops below 95% for any site.

15. FAQs

Q1. What is the difference between a product and a variant in e-commerce extraction?

A product is the parent entity — the thing with a name, brand, description, and image gallery. A variant is a specific sellable version of that product, distinguished by attributes like size, color, or storage capacity. Each variant carries its own SKU, price, and stock status. A single product can have anywhere from 1 to hundreds of variants. The extraction schema must model this as a one-to-many relationship. Flattening them into a single record type produces duplicate product names and makes it impossible to track per-variant pricing changes over time.

Q2. How do you handle sites that load product data via GraphQL APIs?

Many modern e-commerce sites, particularly Shopify stores and custom React storefronts, load product data through GraphQL endpoints after the initial page render. The extraction pipeline intercepts these API calls during the rendering phase using Playwright's route interception. The Ollagraph pipeline includes built-in GraphQL response capture for Shopify, Magento, and custom GraphQL storefronts. It captures the raw response, parses it alongside the DOM, and merges the data in the normalization stage.

Q3. Can you extract pricing for all variants without clicking each variant selector?

Yes, if the site embeds all variant data in a structured format. Shopify stores typically embed a JSON-LD block with all variant prices, SKUs, and stock statuses. Magento stores often include a window.__INITIAL_STATE__ variable with the full variant list. For sites that only load variant data on demand — when a user clicks a selector — the pipeline programmatically triggers each variant selection and captures the resulting price update. This is slower but necessary for sites with lazy-loaded variant data.

Q4. How do you handle currency conversion?

The pipeline extracts price and currency as separate fields. Currency conversion is a downstream concern; the pipeline should not convert currencies, it should preserve the original currency and price. Conversion happens in the analytics layer where the target currency is known and exchange rates can be applied consistently across all products. Converting at extraction time introduces errors when rates change between extraction cycles.

Q5. What is the best wait strategy for e-commerce pages?

Network-idle wait strategy is the most reliable for e-commerce pages. It waits until no network requests have occurred for a specified period, typically 500-2000ms. This ensures all API calls triggered by the page load complete before extraction begins. Set the timeout to at least 10 seconds to handle slow pages and cascading API calls. Selector-based waits are faster but break when class names change during site updates.

Q6. How do you detect A/B tested pricing?

Some e-commerce sites run A/B tests on pricing, showing different prices to different visitors. The pipeline detects this by making multiple requests to the same page from different IP addresses and comparing the extracted prices. If prices vary between requests, the site is likely running A/B tests. Flag these products for manual review rather than reporting a single price. For high-stakes competitive intelligence, run A/B detection as a weekly audit on a sample of products.

Q7. What causes stock status to be wrong?

Three sources account for most stock status errors. First, stale JSON-LD, generated at page build time and not updated when inventory changes. Second, unrecognized dynamic text: the site says "Ships in 5-7 business days" but your pipeline only maps "In Stock" and "Out of Stock". Third, extracting the default variant's status instead of the requested variant's. Cross-referencing JSON-LD with rendered DOM text catches most of these. Add new stock status phrases to your mapping table as you encounter them.

Q8. What is the minimum viable schema?

Four fields: product name (string), SKU (string, unique per variant), price (float, positive), and availability (enum with values like in_stock, out_of_stock, pre_order, backordered, discontinued). Start here and expand as needed. Adding brand, description, compare-at price, and image URLs are common next steps. The key is to get the product-variant relationship right from the start; adding fields later is easy, restructuring a flat schema into a nested one is painful.

Q9. How often should you re-extract product data?

Daily extraction is sufficient for most competitive intelligence use cases; prices change weekly or monthly for most products. For flash sale monitoring or dynamic pricing tracking, extraction every 15-30 minutes may be necessary. The pipeline should support per-site and per-product extraction schedules rather than a single global frequency. A product on flash sale needs hourly checks while a commodity item with a stable price can be checked weekly.

Q10. How do you handle sites with infinite scroll or pagination?

Listing pages with infinite scroll load additional products via AJAX as the user scrolls down. The pipeline scrolls the page programmatically and uses mutation observers to detect when new product cards appear. For traditional pagination, the pipeline iterates through page numbers until it encounters a duplicate product or a 404. The Ollagraph pipeline handles both patterns with configurable scroll and pagination limits to prevent runaway extraction on sites with thousands of products.

16. Conclusion

E-commerce data extraction looks simple from a distance. A product page has a name, a price, and a button. The reality is harder: variant separation, JavaScript rendering, price normalization, stock status mapping, and anti-bot evasion. Each is solvable alone, but together they form a system that requires careful architecture and ongoing maintenance.

The key insight is that e-commerce pages are applications, not documents. Treating them as documents, fetching HTML and parsing with selectors, produces data that looks correct but is wrong in subtle, pipeline-corrupting ways. Model the data first (products with nested variants), then build a pipeline that handles the full range of rendering strategies.

The Ollagraph E-commerce Schema v2 and Extraction API handle this complexity out of the box. But even if you build your own pipeline, the patterns here, schema-driven extraction, hybrid strategy cascade, provenance tracking, validation gates, will save you months of debugging.

Start with the schema. Test against the hardest page first. Extract provenance alongside data. Validate before you store. Never trust a single extraction strategy.

17. References

• Ollagraph E-commerce Schema v2 Documentation: https://ollagraph.com/docs/schemas/ecommerce-v2
• Ollagraph Extraction API Reference: https://ollagraph.com/docs/api/extraction
• Schema-Driven Extraction Guide: Schema-Driven Extraction Beats CSS Selectors
• Evidence-Based Extraction: Evidence-Based Data Extraction: Provenance for Every Field
• JavaScript Rendering Guide: Extract Structured Data from JavaScript Apps
• Shopify Product JSON-LD Specification: https://shopify.dev/docs/themes/liquid/reference/objects/product
• JSON-LD Product Schema: https://schema.org/Product
• Playwright Documentation: https://playwright.dev/docs/network#network-idle
• Web Scraping Legal Considerations: https://ollagraph.com/docs/legal/compliance