v0.8.0 – Security Hardening & Deep Crawl Recovery

⚠️ Two critical security vulnerabilities fixed — review breaking changes below.

Breaking Changes

• Hooks disabled by default — set CRAWL4AI_HOOKS_ENABLED=true to re-enable; __import__ removed from allowed builtins
• Docker API blocks file://, javascript:, data: URLs — /execute_js, /screenshot, /pdf, /html endpoints now validate URL scheme (http/https/raw only)

Security Fixes

• RCE via hook imports — removed __import__ from sandbox builtins
• LFI on Docker endpoints — added URL scheme validation (blocks local file access)

Features

• init_scripts for BrowserConfig — inject JavaScript before page load for stealth evasion
• Crash recovery in deep crawl — resume_state and on_state_change callbacks for BFS/DFS/Best-First strategies
• Prefetch mode — two-phase crawling with fast link extraction before full processing
• CDP improvements — WebSocket URL support, proper cleanup, browser connection reuse
• PDF/MHTML/screenshots for raw:/file:// URLs — render cached HTML and export formats
• base_url parameter — proper URL resolution for raw: HTML processing
• process_in_browser parameter — new browser pipeline for file-based content
• HTTP strategy proxy support — non-browser crawler now supports proxy rotation and sticky sessions
• Sitemap seeder TTL cache — cache_ttl_hours and validate_sitemap_lastmod parameters

Fixes

• raw: URL parsing — fixed truncation at # (CSS color codes)
• Caching system — improved cache validation and persistence