🎯 Major Features

🕵️ Undetected Browser Support

Break through sophisticated bot detection systems with our new stealth capabilities:

from crawl4ai import AsyncWebCrawler, BrowserConfig

# Enable stealth mode for undetectable crawling
browser_config = BrowserConfig(
    browser_type="undetected",  # Use undetected Chrome
    headless=True,              # Can run headless with stealth
    extra_args=[
        "--disable-blink-features=AutomationControlled",
        "--disable-web-security"
    ]
)

async with AsyncWebCrawler(config=browser_config) as crawler:
    # Successfully bypass Cloudflare, Akamai, and custom bot detection
    result = await crawler.arun("https://protected-site.com")
    print(f"✅ Bypassed protection! Content: {len(result.markdown)} chars")

What it enables:

• Access previously blocked corporate sites and databases
• Gather competitor data from protected sources
• Monitor pricing on e-commerce sites with anti-bot measures
• Collect news and social media content despite protection systems

🎨 Multi-URL Configuration System

Apply different crawling strategies to different URL patterns automatically:

from crawl4ai import CrawlerRunConfig

# Define specialized configs for different content types
configs = [
    # Documentation sites - aggressive caching, include links
    CrawlerRunConfig(
        url_matcher=["*docs*", "*documentation*"],
        cache_mode="write",
        markdown_generator_options={"include_links": True}
    ),
    
    # News/blog sites - fresh content, scroll for lazy loading
    CrawlerRunConfig(
        url_matcher=lambda url: 'blog' in url or 'news' in url,
        cache_mode="bypass",
        js_code="window.scrollTo(0, document.body.scrollHeight/2);"
    ),
    
    # API endpoints - structured extraction
    CrawlerRunConfig(
        url_matcher=["*.json", "*api*"],
        extraction_strategy=LLMExtractionStrategy(
            provider="openai/gpt-4o-mini",
            extraction_type="structured"
        )
    ),
    
    # Default fallback for everything else
    CrawlerRunConfig()
]

# Crawl multiple URLs with perfect configurations
results = await crawler.arun_many([
    "https://docs.python.org/3/",      # → Uses documentation config
    "https://blog.python.org/",        # → Uses blog config  
    "https://api.github.com/users",    # → Uses API config
    "https://example.com/"             # → Uses default config
], config=configs)

Perfect for:

• Mixed content sites (blogs, docs, downloads)
• Multi-domain crawling with different needs per domain
• Eliminating complex conditional logic in extraction code
• Optimizing performance by giving each URL exactly what it needs

🧠 Memory Monitoring & Optimization

Track and optimize memory usage during large-scale operations:

from crawl4ai.memory_utils import MemoryMonitor

# Monitor memory during crawling
monitor = MemoryMonitor()
monitor.start_monitoring()

# Perform memory-intensive operations
results = await crawler.arun_many([
    "https://heavy-js-site.com",
    "https://large-images-site.com", 
    "https://dynamic-content-site.com"
] * 100)  # Large batch

# Get detailed memory report
report = monitor.get_report()
print(f"Peak memory usage: {report['peak_mb']:.1f} MB")
print(f"Memory efficiency: {report['efficiency']:.1f}%")

# Automatic optimization suggestions
if report['peak_mb'] > 1000:  # > 1GB
    print("💡 Consider batch size optimization")
    print("💡 Enable aggressive garbage collection")

Benefits:

• Prevent memory-related crashes in production services
• Right-size server resources based on actual usage patterns
• Identify bottlenecks for performance optimization
• Plan horizontal scaling based on memory requirements

📊 Enhanced Table Extraction

Direct pandas DataFrame conversion from web tables:

result = await crawler.arun("https://site-with-tables.com")

# New streamlined approach
if result.tables:
    print(f"Found {len(result.tables)} tables")
    
    import pandas as pd
    for i, table in enumerate(result.tables):
        # Instant DataFrame conversion
        df = pd.DataFrame(table['data'])
        print(f"Table {i}: {df.shape[0]} rows × {df.shape[1]} columns")
        print(df.head())
        
        # Rich metadata available
        print(f"Source: {table.get('source_xpath', 'Unknown')}")
        print(f"Headers: {table.get('headers', [])}")

# Old way (now deprecated)
# tables_data = result.media.get('tables', [])  # ❌ Don't use this

Improvements:

• Faster transition from web data to analysis-ready DataFrames
• Cleaner integration with data processing pipelines
• Simplified table extraction for automated reporting
• Better table structure preservation

🐳 Docker LLM Provider Flexibility

Switch between LLM providers without rebuilding images:

# Option 1: Direct environment variables
docker run -d \
  -e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
  -e GROQ_API_KEY="your-key" \
  -p 11235:11235 \
  unclecode/crawl4ai:0.7.3

# Option 2: Using .llm.env file (recommended for production)
docker run -d \
  --env-file .llm.env \
  -p 11235:11235 \
  unclecode/crawl4ai:0.7.3

Create .llm.env file:

LLM_PROVIDER=openai/gpt-4o-mini
OPENAI_API_KEY=your-openai-key
GROQ_API_KEY=your-groq-key

Override per request when needed:

# Use cheaper models for simple tasks, premium for complex ones
response = requests.post("http://localhost:11235/crawl", json={
    "url": "https://complex-page.com",
    "extraction_strategy": {
        "type": "llm",
        "provider": "openai/gpt-4"  # Override default
    }
})

🔧 Bug Fixes & Improvements

• URL Matcher Fallback: Resolved edge cases in pattern matching logic
• Memory Management: Fixed memory leaks in long-running sessions
• Sitemap Processing: Improved redirect handling in sitemap fetching
• Table Extraction: Enhanced detection and extraction accuracy
• Error Handling: Better messages and recovery from network failures

📚 Documentation & Architecture

• Architecture Refactoring: Moved 2,450+ lines to backup for cleaner codebase
• Real-World Examples: Added practical use cases with actual URLs
• Migration Guides: Complete transition from result.media to result.tables
• Comprehensive Guides: Full documentation for undetected browsers and multi-config

📦 Installation & Upgrade

PyPI Installation

# Fresh install
pip install crawl4ai==0.7.3

# Upgrade from previous version
pip install --upgrade crawl4ai==0.7.3

Docker Images

# Specific version
docker pull unclecode/crawl4ai:0.7.3

# Latest (points to 0.7.3)
docker pull unclecode/crawl4ai:latest

# Version aliases
docker pull unclecode/crawl4ai:0.7    # Minor version
docker pull unclecode/crawl4ai:0      # Major version

Migration Notes

• result.tables replaces result.media.get('tables')
• Undetected browser requires browser_type="undetected"
• Multi-config uses url_matcher parameter in CrawlerRunConfig

🎉 What's Next?

This release sets the foundation for even more advanced features coming in v0.8:

• AI-powered content understanding
• Advanced crawling strategies
• Enhanced data pipeline integrations
• More stealth and anti-detection capabilities

📝 Complete Documentation

• Full Release Notes - Detailed technical explanations
• Changelog - Complete list of changes
• Documentation - Full API reference and guides
• Discord Community - Get help and share experiences

Live Long and import crawl4ai

Crawl4AI continues to evolve with your needs. This release makes it stealthier, smarter, and more scalable. Try the new undetected browser and multi-config features—they're game changers!

- The Crawl4AI Team

📝 This release draft was composed and edited by human but rewritten and finalized by AI. If you notice any mistakes, please raise an issue.