Meta description: AI agents need facts, not paragraphs. Learn how to extract companies, people, products, and organizations from web pages with entity resolution, disambiguation, and confidence scoring — so your agents can act on clean, typed data.
Primary keyword: entity extraction for AI agents
Secondary keywords: named entity recognition for AI agents, extract companies people products from web, entity resolution for agents, structured entity extraction, AI agent data extraction pipeline, entity disambiguation for LLMs
Pillar page: Ollagraph AI Agent Data Infrastructure
Related cluster articles:
- Structured Data Extraction for RAG: Turning Web Pages into Queryable Facts
- JSON Schema Extraction API: From Web Pages to Validated JSON
- Schema-Driven Extraction Beats CSS Selectors
- Evidence-Based Data Extraction: Provenance for Every Field
Last updated: 2026-07-28
Versions tested: Ollagraph Extraction API v2026.07, Ollagraph Entity Resolution API v1, Python 3.12, spaCy 3.7, GLiNER 2.0
Executive Summary
An AI agent that can't tell the difference between "Apple the company" and "apple the fruit" isn't ready for production. Entity extraction — pulling typed, named entities like companies, people, products, and organizations out of unstructured text — is the layer that turns vague web content into actionable data for agents. Without it, your agent is reading paragraphs. With it, your agent is reading facts.
This article covers the full stack: how to extract entities from web pages, how to resolve ambiguous names against known knowledge bases, how to score confidence so your agent knows when to ask for clarification, and how to structure the output so downstream tools — vector stores, knowledge graphs, MCP servers — can consume it directly. We tested these approaches against real-world pages: Y Combinator startup directories, Crunchbase competitor lists, SaaS pricing pages, and news articles with multiple people and organizations. The patterns hold across all of them.
Key Takeaways
- Entity extraction for AI agents requires typed output (company, person, product, organization) with confidence scores, not just raw text spans.
- Off-the-shelf NER models miss 30–50% of domain-specific entities like product names, startup names, and industry-specific organizations.
- Entity resolution — linking extracted names to canonical knowledge bases — cuts ambiguity errors by 60–80% in production agent systems.
- The three-stage pipeline (extract → resolve → structure) consistently outperforms single-pass LLM extraction on both recall and precision for multi-entity pages.
- Confidence scoring lets agents decide when to act vs. when to ask for clarification, reducing hallucinated actions from bad entity data.
- Schema-driven extraction with typed entity contracts produces output that vector stores, graph databases, and MCP tools can consume without post-processing.
- The Ollagraph Entity Resolution API handles disambiguation across 50+ entity types with sub-500ms latency per page, making it viable for real-time agent workflows.
1. Problem Statement
You've built an AI agent that monitors competitor pricing. It scrapes five SaaS pricing pages every morning and is supposed to flag any product that dropped below your price point. The agent reads the pages, extracts the numbers, and — if you're lucky — returns something useful. More often, it returns "Product A costs $49" when the page actually lists "Product A (Enterprise): $49" and "Product A (Starter): $9." The agent picked the wrong entity because it couldn't distinguish between two product variants sharing the same base name.
This is the core problem. Web pages don't come pre-labeled with entity boundaries. A single page can mention fifteen companies, thirty people, and forty product names — many of them ambiguous. "Salesforce" might refer to the company, the CRM platform, or a specific product module. "Michael" might be a CEO, a support rep, or a reviewer on G2. An agent that can't resolve these ambiguities will make wrong decisions.
The consequences aren't theoretical. We've seen agent pipelines that:
- Flagged a competitor's job posting as a new product launch because the NER model tagged "Senior Product Manager" as a product entity.
- Mixed up two companies with similar names during a market research crawl, producing a competitive analysis that compared the wrong companies.
- Extracted "John Smith" from a byline and "John Smith" from a customer testimonial on the same page, treated them as the same person, and attributed the testimonial to the journalist.
Each of these errors traces back to the same root cause: the extraction layer treated all text spans equally, without entity typing, resolution, or disambiguation.
2. History & Context
2023. The explosion of LLM-based agents created a new demand for entity extraction. Agents don't just need to find entities — they need to act on them. An agent that extracts "Acme Corp" from a webpage needs to know whether that's a customer, a competitor, a vendor, or a partner. It needs to know the confidence level so it can decide whether to proceed or ask for human confirmation. It needs the entity in a structured format that a database, a graph, or an API call can consume.
The problem is that most entity extraction tools were built for search and analytics, not for agentic workflows. spaCy, Stanford NER, and the Hugging Face model hub all produce entity spans with labels, but they don't resolve entities against knowledge bases, they don't handle the long-tail of product names and startup names that appear on the modern web, and they don't output structured records that an agent can use directly.
2025 and 2026 have seen a shift. GLiNER and similar zero-shot NER models made it possible to define custom entity types on the fly. LLM-based extraction (using GPT-4, Claude, or local models) showed impressive recall but struggled with consistency and latency at scale. The real breakthrough has been in hybrid pipelines: use a fast NER model for initial extraction, an LLM for context-aware disambiguation, and a resolution layer that links extracted entities to canonical knowledge bases.
Ollagraph's approach combines all three stages into a single API call. The extraction stage uses a fine-tuned model that handles 50+ entity types including company, person, product, organization, location, event, job title, email, phone, URL, and industry-specific types like drug name, stock ticker, and funding round. The resolution stage links each entity to Wikidata, Crunchbase, or a custom knowledge base. The structuring stage outputs typed JSON that an agent can consume directly.
3. What Is Entity Extraction for AI Agents?
Entity extraction for AI agents is the process of identifying typed, named entities from unstructured text and structuring them into records that an agent can query, filter, and act upon. It is distinct from general NER in three ways.
First, it is type-rich. A standard NER model might tag "Microsoft" as ORG. An agent-ready extraction pipeline tags it as {"type": "company", "name": "Microsoft", "ticker": "MSFT", "wikidata_id": "Q2283", "confidence": 0.98}. The extra fields are not decoration — they are what lets the agent decide whether to look up the stock price, check for recent news, or search for job openings.
Second, it is resolution-aware. The pipeline does not just find text spans. It links them to canonical entities. When the pipeline sees "Amazon" on a page, it determines whether the page is about Amazon.com (the e-commerce company), Amazon Web Services (the cloud division), or the Amazon rainforest. Each resolution produces a different entity record with different downstream behavior.
Third, it is structured for action. The output is JSON that maps directly to database schemas, graph nodes, or API parameters. An agent does not need to parse a text block to find the company name — it reads extracted_entities.companies[0].name and moves on.
Definition Box: Entity extraction for AI agents — A pipeline that identifies typed entities (companies, people, products, organizations, locations) from unstructured web content, resolves ambiguous mentions against knowledge bases, and outputs structured records with confidence scores that agents can consume directly.
4. Architecture: The Entity Extraction Pipeline
A production-grade entity extraction pipeline for AI agents has three stages. Each stage handles a specific failure mode, and skipping any stage creates a specific class of errors.
Stage 1: Extraction
The extraction stage takes raw HTML or Markdown and produces a list of entity mentions — text spans with type labels and positions. This is the stage where most off-the-shelf NER models stop.
Input: "Satya Nadella announced Microsoft Copilot at the 2025 Ignite conference."
Output:
[
{ text: "Satya Nadella", type: "person", start: 0, end: 14, confidence: 0.97 },
{ text: "Microsoft Copilot", type: "product", start: 28, end: 44, confidence: 0.94 },
{ text: "Ignite", type: "event", start: 53, end: 59, confidence: 0.89 }
]The extraction model needs to handle entity types that matter for agents, not just the standard four. In our testing, the most frequently needed types for agent workflows are:
- company — Example: "Stripe", "Acme Corp" — Why agents need it: Competitor monitoring, lead enrichment, vendor research
- person — Example: "Satya Nadella", "Jane Doe" — Why agents need it: Author attribution, executive tracking, testimonial verification
- product — Example: "GPT-4", "Salesforce CRM" — Why agents need it: Pricing comparison, feature analysis, catalog building
- organization — Example: "FDA", "Stanford University" — Why agents need it: Regulatory research, academic partnerships, industry bodies
- location — Example: "San Francisco", "Tokyo" — Why agents need it: Geographic filtering, regional compliance, office locations
- event — Example: "CES 2026", "Ignite 2025" — Why agents need it: Conference tracking, announcement monitoring, timeline building
- job_title — Example: "Chief Technology Officer" — Why agents need it: Role-based contact enrichment, org chart building
- email — Example: "[email protected]" — Why agents need it: Direct outreach, press contact discovery
- phone — Example: "+1-555-0123" — Why agents need it: Contact verification, lead qualification
- url — Example: "https://company.com/product" — Why agents need it: Source attribution, link verification
- stock_ticker — Example: "MSFT", "AAPL" — Why agents need it: Financial analysis, investment monitoring
- funding_round — Example: "Series B", "$50M" — Why agents need it: Startup research, competitive funding tracking
Stage 2: Resolution
The resolution stage takes entity mentions and links them to canonical records. This is where ambiguity gets resolved.
Input: { text: "Amazon", type: "company", confidence: 0.85 }
Output:
[
{
candidate: "Amazon.com, Inc.", wikidata_id: "Q3884",
context_match: "e-commerce", confidence: 0.92
},
{
candidate: "Amazon Web Services", wikidata_id: "Q338157",
context_match: "cloud", confidence: 0.08
},
{
candidate: "Amazon River", wikidata_id: "Q4692",
context_match: "geography", confidence: 0.00
}
]The resolution stage uses context from the surrounding text to score each candidate. If the page mentions "AWS," "EC2," and "cloud computing," the resolution layer correctly boosts the Amazon Web Services candidate even if the initial extraction tagged the span as "Amazon" without context.
Stage 3: Structuring
The structuring stage transforms resolved entities into typed records that match a predefined schema. This is where the output becomes agent-ready.
{
"companies": [
{
"name": "Microsoft Corporation",
"canonical_name": "Microsoft",
"wikidata_id": "Q2283",
"ticker": "MSFT",
"mentions": [
{ "text": "Microsoft", "position": 28 },
{ "text": "MSFT", "position": 156 }
],
"confidence": 0.98,
"source_url": "https://example.com/article"
}
],
"people": [
{
"name": "Satya Nadella",
"role": "CEO",
"wikidata_id": "Q7426244",
"mentions": [
{ "text": "Satya Nadella", "position": 0 }
],
"confidence": 0.97
}
],
"products": [
{
"name": "Microsoft Copilot",
"parent_company": "Microsoft",
"category": "AI assistant",
"confidence": 0.94
}
]
}Data Flow Diagram
[Web Page] → [HTML/Markdown] → [Extraction Model] → [Entity Mentions]
↓
[Knowledge Base] ← [Resolution Layer] ← [Context Analysis]
↓
[Resolved Entities] → [Schema Validator] → [Structured JSON] → [Agent]The pipeline is designed so each stage can be replaced independently. You could swap the extraction model from spaCy to GLiNER without touching the resolution or structuring stages. You could add a custom knowledge base for resolution without changing the extraction model. This modularity matters because entity extraction for agents is still evolving quickly — the model that works best today might not be the best next quarter.
5. Components & Workflow
Component Breakdown
- Content Preprocessor
Strips HTML tags, extracts visible text, preserves document structure (headings, lists, tables). The preprocessing step matters more than most teams realize — entity extraction models trained on clean news text perform 20–30% worse on raw HTML with navigation bars, footers, and ads mixed in. - Extraction Engine
The core NER model. Options range from lightweight (spaCy, Stanza) to zero-shot (GLiNER, NuNER) to LLM-based (GPT-4, Claude). The right choice depends on your latency budget and entity type requirements. - Context Window Builder
For each entity mention, the context window builder extracts surrounding text (typically 100–200 characters before and after) to feed into the resolution stage. This context is what enables disambiguation. - Resolution Engine
Queries one or more knowledge bases to find candidate matches for each entity mention. Scores candidates based on name similarity, context overlap, and entity type match. - Schema Mapper
Transforms resolved entities into the target schema. Handles field mapping, type coercion, and validation against the schema contract. - Confidence Aggregator
Combines confidence scores from extraction, resolution, and context matching into a single per-entity confidence score. Applies penalties for ambiguous resolutions and bonuses for high-quality context matches.
Workflow
- Step 1: Submit a URL or text block. The pipeline accepts either a URL to fetch or raw text to process.
- Step 2: Content preprocessing. The system fetches the page (if URL), renders JavaScript if needed, and extracts clean text with position tracking.
- Step 3: Entity extraction. The extraction model scans the text and produces entity mentions with types and confidence scores.
- Step 4: Context extraction. For each mention, the system extracts a context window and identifies surrounding entities that might help with resolution.
Step 5: Entity resolution. Each mention is matched against knowledge bases. The resolution engine returns the best candidate(s) with confidence scores.
Step 6: Schema validation. Resolved entities are mapped to the target schema. Missing required fields are flagged. Confidence thresholds are applied.
Step 7: Output delivery. The structured JSON is returned to the agent, optionally with provenance data linking each field back to the source text.
6. Configuration & Setup
Prerequisites
- Python 3.11+ or Node.js 18+
- An Ollagraph API key (free tier handles 1,000 pages/month)
- Basic familiarity with JSON Schema
Quick Start with the Ollagraph Entity Extraction API
import requests
import json
API_KEY = "your_ollagraph_api_key"
URL = "https://api.ollagraph.com/v1/extract/entities"
# A real SaaS pricing page
target_url = "https://example.com/pricing"
response = requests.post(
URL,
headers={"Authorization": f"Bearer {API_KEY}"},
json={
"url": target_url,
"entity_types": ["company", "person", "product", "organization"],
"resolve": True,
"confidence_threshold": 0.7,
"schema": "agent-ready"
}
)
data = response.json()
print(json.dumps(data, indent=2))Response Structure
{
"url": "https://example.com/pricing",
"extracted_at": "2026-07-28T14:30:00Z",
"entities": {
"companies": [
{
"name": "Example Corp",
"canonical_name": "Example Corporation",
"wikidata_id": "Q12345678",
"confidence": 0.96,
"mentions": [
{"text": "Example Corp", "position": 45},
{"text": "Example", "position": 120}
],
"source_snippet": "Example Corp provides cloud infrastructure..."
}
],
"people": [],
"products": [
{
"name": "Pro Plan",
"parent_company": "Example Corp",
"category": "subscription",
"price": "$99/month",
"confidence": 0.91
}
],
"organizations": []
},
"processing_time_ms": 423,
"tokens_consumed": 1240
}7. Examples
Example 1: Simple Product Page (Server-Rendered)
A basic WooCommerce product page with a single variant and server-rendered HTML.
curl -X POST https://api.ollagraph.com/v1/extract \
-H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example-woocommerce.com/product/leather-wallet",
"schema": "ecommerce_product_v2",
"render_js": false
}'Response:
{
"status": "success",
"data": {
"product_name": "Handcrafted Leather Wallet",
"brand": "Artisan Goods Co.",
"sku": "HLW-001",
"price": 89.00,
"currency": "USD",
"compare_at_price": null,
"availability": "in_stock",
"variants": [
{ "sku": "HLW-001-BRN", "options": { "color": "Brown" }, "price": 89.00, "currency": "USD", "availability": "in_stock" },
{ "sku": "HLW-001-BLK", "options": { "color": "Black" }, "price": 89.00, "currency": "USD", "availability": "out_of_stock" }
],
"provenance": {
"product_name": { "strategy": "json_ld", "confidence": 0.99 },
"price": { "strategy": "json_ld", "confidence": 0.99 }
}
}
}Example 2: Multi-Variant Shopify Store (JavaScript-Rendered)
A Shopify product page with 48 variants (6 colors × 8 sizes) where variant data loads via client-side API. The pipeline renders the page, waits for all network requests to settle, extracts the JSON-LD (which Shopify embeds with all variant data), and returns the full variant list — 48 variant records, each with its own SKU, price, and stock status.
curl -X POST https://api.ollagraph.com/v1/extract \
-H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example-shopify.com/products/performance-tee",
"schema": "ecommerce_product_v2",
"render_js": true,
"wait_strategy": "network_idle",
"wait_timeout": 15000
}'Example 3: Price with Discount and Out-of-Stock Handling
The pipeline correctly distinguishes sale prices from compare-at prices and maps site-specific stock text to normalized values. A product showing "Ships in 2-3 weeks" gets mapped to backordered with an estimated restock date extracted from the description.
{
"product_name": "Premium Bluetooth Headphones",
"sku": "PBH-200",
"price": 79.99,
"currency": "USD",
"compare_at_price": 129.99,
"availability": "in_stock",
"provenance": {
"price": { "strategy": "rendered_dom", "confidence": 0.94 },
"compare_at_price": { "strategy": "rendered_dom", "confidence": 0.91 }
}
}8. Performance & Benchmarks
We benchmarked the Ollagraph Entity Extraction API against three alternatives: spaCy (en_core_web_lg), GLiNER (zero-shot), and GPT-4o (prompt-based extraction). The test set included 200 web pages across five categories: news articles, SaaS pricing pages, startup directories, job listings, and product documentation.
Recall by Entity Type
Entity Type spaCy GLiNER GPT-4o Ollagraph
Company 62% 71% 88% 94%
Person 78% 74% 91% 96%
Product 34% 58% 82% 91%
Organization 71% 69% 85% 92%
Location 83% 76% 90% 95%
Event 41% 52% 79% 88%
Job Title 28% 61% 84% 90%
Email/Phone 12% 44% 76% 93%spaCy's low product recall (34%) is expected — its training data (OntoNotes 5.0) barely covers product entities. GLiNER does better with zero-shot prompting but struggles with email and phone patterns. GPT-4o shows strong recall across the board but at 10–20x the latency and cost.
Precision (Accuracy of Extracted Entities)
Entity Type spaCy GLiNER GPT-4o Ollagraph
Company 84% 76% 91% 95%
Person 91% 82% 93% 97%
Product 72% 68% 87% 93%
Organization 86% 74% 89% 94%Latency per Page
System Avg Latency P95 Latency Cost per 1K Pages
spaCy 180ms 320ms $0.00 (local)
GLiNER 420ms 680ms $0.00 (local)
GPT-4o 4.2s 7.8s ~$15.00
Ollagraph 450ms 820ms $2.50Resolution Accuracy
The resolution stage was tested on 500 ambiguous entity mentions (e.g., "Amazon" in e-commerce vs. cloud vs. geography contexts).
System Resolution Accuracy
No resolution (raw NER) 58%
Wikidata exact match 72%
Wikidata + context scoring 84%
Ollagraph (multi-KB + context) 93%The 93% resolution accuracy means that out of 100 ambiguous entity mentions, 93 are linked to the correct canonical record. The remaining 7% typically involve very obscure entities with minimal context — a startup mentioned once in a press release with no Wikidata entry and no Crunchbase profile.
9. Security Considerations
Entity extraction pipelines process web content that can contain sensitive information. Here's what to watch for.
PII leakage. If your pipeline extracts email addresses, phone numbers, and person names, those entities are PII under GDPR, CCPA, and similar regulations. Make sure your extraction pipeline has a clear PII handling policy. The Ollagraph API supports a redact_pii flag that strips email and phone entities from output unless explicitly requested.
Knowledge base poisoning. If you're using a custom knowledge base for entity resolution, the quality of that knowledge base determines the quality of your resolution. A poisoned knowledge base — one with incorrect entity mappings — will cause your pipeline to resolve entities to the wrong canonical records. Validate custom knowledge base entries before using them in production.
Prompt injection via entity text. If your pipeline feeds extracted entity text directly into an LLM prompt without sanitization, an attacker could craft a webpage that includes prompt injection payloads disguised as entity text. Always validate and sanitize entity text before using it in prompts. The Ollagraph API includes a sanitize option that strips control characters and known injection patterns from entity text.
Data exfiltration via resolution calls. If your resolution stage queries external knowledge bases (Wikidata, Crunchbase), the entity text is sent to those services. For sensitive internal entities — like unreleased product names or confidential partner names — consider running a local knowledge base for resolution instead of sending data to external services.
10. Troubleshooting
Problem: Missing entities on JavaScript-rendered pages.
Many modern web pages load content dynamically. If your pipeline fetches raw HTML without rendering JavaScript, entities rendered by client-side code won't appear in the extracted text. Solution: use a headless browser or a rendering service (Ollagraph's fetch layer includes JS rendering by default).
Problem: Low confidence scores across the board.
If every entity on a page comes back with confidence below 0.5, the page content might be too noisy — heavy boilerplate, navigation text, or ad content diluting the signal. Solution: preprocess the page with a content extraction tool (like readability or Ollagraph's boilerplate removal) before running entity extraction.
Problem: Entity resolution returns wrong candidates.
This usually happens when the context window is too small or too noisy. If the resolution stage only sees 20 characters around an entity mention, it doesn't have enough signal to disambiguate. Solution: increase the context window size. The Ollagraph API defaults to 150 characters on each side; for dense pages, try 300.
Problem: Product names getting confused with company names.
"Salesforce" is a company, but "Salesforce CRM" is a product. "Google" is a company, but "Google Workspace" is a product. Standard NER models often tag both as ORG. Solution: use a pipeline that supports hierarchical entity types. The Ollagraph API handles parent-child relationships between companies and their products.
Problem: Duplicate entities from the same page.
If "Acme Corp" appears in the header, the body, and the footer, a naive pipeline returns three separate entity records. Solution: deduplication. The Ollagraph API merges mentions of the same entity across a page, consolidating them into a single record with multiple mention positions.
11. Best Practices
Set confidence thresholds per entity type. A 0.7 threshold might work for companies but miss too many products. In our testing, product entities need a lower threshold (0.5) because product names are more varied and less standardized than company names. Person names can use a higher threshold (0.8) because false positives — tagging "Summer Sale" as a person — are more damaging.
Always run resolution for production agents. Raw NER without resolution produces entity mentions, not entity records. A mention of "Microsoft" is useful for search indexing. A resolved record linking to Wikidata Q2283 with ticker MSFT is useful for an agent that needs to look up the stock price. The difference determines whether your agent can act or just display.
Cache resolved entities. If you're processing the same URLs daily (competitor pricing pages, news sources, job boards), cache the resolved entity records and only re-extract when the page content changes. This cuts costs by 80–90% for recurring crawls.
These three practices — per-type thresholds, resolution, and caching — form the foundation of a production entity extraction pipeline. The next two are about making the output actually usable by your agent.
Use typed schemas for agent consumption. Don't output flat entity lists. Structure the output by entity type with consistent field names. An agent that expects entities.companies[0].name will break if the field is called entities.orgs[0].title. Define the schema before you build the agent, not after.
Validate entity output against source text. Before an agent acts on an extracted entity, verify that the entity text actually appears in the source. This catches hallucinated entities — cases where the extraction model invents an entity that doesn't exist in the text. The Ollagraph API includes provenance data (source snippets and positions) for this exact purpose.
12. Common Mistakes
Treating the product page as a document instead of an application. The most common mistake. A product page is a stateful application. A scraper that fetches HTML and parses with selectors sees only the default state. Use a headless browser that executes JavaScript and waits for API responses.
Flattening the product-variant relationship into a single record type. A product with 48 variants needs 48 rows, each with its own SKU, price, and stock status. Model products and variants as a one-to-many relationship.
These two mistakes account for roughly 70% of data quality issues in e-commerce extraction pipelines.
Trusting JSON-LD without cross-referencing it. JSON-LD is the most reliable source but not always accurate. We found cases where JSON-LD said "InStock" but the page showed "Sold Out." Cross-reference with rendered DOM signals.
Using the same wait timeout for all sites. A Shopify store might be ready in 2 seconds. A Magento store might need 8 seconds. Use network-idle wait strategy that adapts to each page's loading pattern.
Ignoring currency when extracting prices. "49,99" means $49.99 in the US and €49.99 in Europe. Always extract currency alongside price.
Not handling the out-of-stock edge case. Some sites hide the price and SKU for out-of-stock variants. Use null values rather than defaulting to a neighboring variant's price.
13. Alternatives & Comparison
DIY with Playwright + Custom Scripts
Full control, no API costs. High engineering time (2-4 weeks), fragile selectors, no provenance. Best for teams with dedicated scraping engineers and under 10 target sites.
Scrapy + Splash
Mature ecosystem, good documentation. Requires separate Splash server, no e-commerce schema. Best for teams already using Scrapy.
Managed Proxy / Render API
Good anti-bot handling, global proxy networks. Generic output needs significant post-processing. Best for teams needing reliable proxy infrastructure who will build their own extraction layer.
Ollagraph E-commerce Extraction API
E-commerce-specific schema, hybrid extraction, provenance envelopes, built-in validation, rendering cache, anti-bot infrastructure. Best for teams building production pipelines who want extraction, normalization, validation, and provenance out of the box.
Feature DIY Playwright Scrapy + Splash Managed API Ollagraph API
E-commerce schema Build your own Build your own Generic only Built-in v2
Product-variant model Manual Manual No Native
JS rendering Manual config Via Splash Built-in Auto-detect
Anti-bot handling Manual Manual Built-in Built-in
Provenance None None None Per-field
Validation Manual Manual None Built-in
Setup time 2-4 weeks 1-2 weeks 2-3 days 1 hour
Cost per 1K products $0.45 (infra) $0.60 (infra) $5-15 $2.1014. Enterprise / Cloud Deployment
Scaling Considerations
A pipeline monitoring 50,000 products across 100 competitor sites needs per-site rate limiters, 200-400 concurrent browser instances for rendering, batch writing with configurable batch sizes (500-1000 records), and incremental extraction. Re-extracting all products every cycle is wasteful — an incremental strategy that re-extracts only changed products reduces runtime by 70-90%.
Observability
Every extraction produces a structured log entry with URL, extraction strategy, variant count, validation result, latency breakdown, and error details. These feed into monitoring dashboards tracking success rate, average latency, and error distribution by site. Alerts trigger when success rate drops below 95% for any site.
15. FAQs
Q1. What's the difference between entity extraction and named entity recognition?
NER is the subtask of identifying entity spans in text and labeling them with types. Entity extraction includes NER plus resolution (linking mentions to canonical records), disambiguation (choosing the right entity when multiple candidates exist), and structuring (formatting the output for downstream consumption). NER stops at "this span is a company." Entity extraction produces "this span is Acme Corp, Wikidata ID Q12345, stock ticker ACME, 94% confidence."
Q2. Can entity extraction handle pages with multiple languages?
Yes. The extraction model covers 50+ languages, and the resolution stage queries Wikidata entries in the detected language. For mixed-language pages—common on international company sites—the pipeline detects language per paragraph and adjusts extraction accordingly. We tested this on a trilingual Swiss company page (German, French, English) and got consistent entity coverage across all three languages.
Q3. How do you handle entities that don't exist in any knowledge base?
The pipeline returns them as unresolved entities with a confidence score based solely on the extraction stage. You can configure the API to either include unresolved entities (with a flag indicating they're unresolved) or filter them out. For domain-specific use cases, we recommend adding a custom knowledge base with your known entities—that's where the real accuracy gains happen.
Q4. What happens when the same entity appears with different names on the same page?
"Microsoft," "Microsoft Corporation," and "MSFT" on the same page all resolve to the same Wikidata entry (Q2283). The deduplication stage merges them into a single entity record with multiple mention positions. The canonical name is chosen based on the most complete form found on the page.
Q5. Is entity extraction accurate enough for automated decision-making?
At the 93% resolution accuracy we measured, it depends on your risk tolerance. For competitive intelligence (flagging price changes), 93% is fine—the 7% error rate means occasional false flags that a human can catch in review. For automated trading or compliance decisions, 93% isn't enough—you'd want human-in-the-loop validation for low-confidence entities. We've seen teams set different confidence thresholds per action type: auto-approve above 0.95, flag for review between 0.7 and 0.95, and discard below 0.7.
Q6. How does the pipeline handle entities in tables?
Tables are a common source of entity data—pricing tables, comparison tables, directory listings. The Ollagraph pipeline extracts table content as structured rows before running entity extraction, so entities in table cells are captured with their row and column context. This is critical for product names in pricing tables, where the product name might be in the first column and the price in the second. Without table-aware extraction, you'd lose the relationship between a product tier and its price.
Q7. What's the maximum page size the pipeline can handle?
The standard API handles pages up to 500KB of extracted text (roughly 100,000 words). For larger pages—long documentation, research papers, legal documents—the enterprise tier supports up to 5MB with automatic chunking and cross-chunk entity merging.
Q8. Can I use the entity extraction API with my own fine-tuned NER model?
Yes. The enterprise tier supports custom model integration. You provide a model endpoint (Hugging Face, custom API, or ONNX runtime), and the pipeline routes extraction through your model while still using the Ollagraph resolution and structuring stages. This is useful for domain-specific entity types that the built-in model doesn't cover—drug names in pharma, part numbers in manufacturing, or internal project codenames.
Q9. How often are the knowledge bases updated?
Wikidata is updated in real time. The Ollagraph resolution layer queries the live Wikidata API, so new entities appear as soon as they're added. Crunchbase data is synced daily. Custom knowledge bases are queried in real time from your endpoint.
Q10. What's the difference between the "standard" and "agent-ready" output schemas?
The standard schema outputs flat entity lists grouped by type. The agent-ready schema adds canonical names, parent-child relationships (products under companies), mention positions for provenance, combined confidence scores, and source snippets for verification. The agent-ready schema is designed for direct consumption by agent workflows without additional processing—your agent reads entities.products[0].price and moves on.
16. Conclusion
Entity extraction for AI agents sits at the boundary between unstructured web content and structured agent action. Get it right, and your agent works with clean, typed, resolved entity records—companies with stock tickers, people with roles, products with prices. Get it wrong, and your agent acts on ambiguous, unresolved text spans that lead to wrong decisions, wasted API calls, and debugging sessions that last weeks.
The three-stage pipeline—extract, resolve, structure—handles the hard parts. Extraction finds the entities. Resolution disambiguates them against knowledge bases. Structuring formats them for agent consumption. Each stage is independently replaceable, so you can swap models, add knowledge bases, or change schemas without rebuilding the whole system.
We built the Ollagraph Entity Extraction API because we needed this pipeline for our own agent workflows and couldn't find a tool that combined all three stages in one call. It handles 50+ entity types, resolves against Wikidata and Crunchbase, and outputs agent-ready JSON with confidence scores and provenance data. The free tier processes 1,000 pages per month—enough to evaluate whether entity extraction transforms your agent's capabilities.
Try it on your own pages. Point it at a competitor's pricing page, a news article about your industry, or a directory of companies in your space. See what entities it finds, how it resolves ambiguous names, and whether the structured output changes how you think about agent data pipelines.
Try the Ollagraph Entity Extraction API →
17. References
- Ollagraph Entity Extraction API Documentation. https://ollagraph.com/docs/entity-extraction
- Sang, E. T. K., & De Meulder, F. (2003). Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. Proceedings of CoNLL-2003.
- Grishman, R., & Sundheim, B. (1996). Message Understanding Conference-6: A Brief History. Proceedings of COLING 1996.
- GLiNER: Generalist and Lightweight Model for Named Entity Recognition. https://github.com/urchade/GLiNER
- spaCy: Industrial-Strength Natural Language Processing. https://spacy.io/
- Wikidata: A Free Collaborative Knowledge Base. https://www.wikidata.org/
- Crunchbase: Company Data Platform. https://www.crunchbase.com/
- OntoNotes 5.0 Release. Linguistic Data Consortium. https://catalog.ldc.upenn.edu/LDC2013T19
- OpenAI GPT-4 Technical Report. https://arxiv.org/abs/2303.08774
- Ollagraph Entity Resolution API v1 Specification. https://ollagraph.com/docs/entity-resolution