v0.0.31 introduces two powerful new robots to the Maxun engine: Crawl and Search. These features allow you to automate data discovery at scale, whether you are mapping entire domains or researching the web via natural language.

🕸️Crawl

Stop manually feeding URLs. Crawl intelligently discovers and extracts entire websites.

• Intelligent Discovery: Uses both Sitemap parsing and Link following to find every relevant page.
• Granular Scope Control: Target exactly what you need with Domain, Subdomain, or Path-specific modes.
• Advanced Filtering: Use Regex patterns to include or exclude specific content (e.g., skip /admin, target /blog/*).
• Depth Control: Define how many levels deep the robot should navigate from your starting URL.

https://github.com/user-attachments/assets/d3e6a2ca-f395-4f86-9871-d287c094e00c

🔍 Search

Turn search engine queries into structured datasets.

• Query Based: Search the web with a search query - same as you would type in a search engine.
• Dual Modes: Use Discover Mode for fast metadata/URL harvesting, or Scrape Mode to automatically visit and extract full content from every search result.
• Recency Filters: Narrow down data by time (Day, Week, Month, Year) to find the freshest content.

https://github.com/user-attachments/assets/9133180c-3fbf-4ceb-be16-d83d7d742e1c

Hybrid Access

Manage these robots through our no-code dashboard or trigger them programmatically via our Developer SDK.

What's Changed

• fix: remove onCancel button from robot config page by @amhsirak in https://github.com/getmaxun/maxun/pull/929
• feat: cleaner alert notifs by @amhsirak in https://github.com/getmaxun/maxun/pull/927
• feat(ui): match theme of standard & outlined alerts by @amhsirak in https://github.com/getmaxun/maxun/pull/928
• feat: !show rows per page for runs & robots by @amhsirak in https://github.com/getmaxun/maxun/pull/930
• fix: show formats selected for scrape by @amhsirak in https://github.com/getmaxun/maxun/pull/931
• chore: pass prompt by @RohitR311 in https://github.com/getmaxun/maxun/pull/932
• feat: auto name lists for ai mode by @RohitR311 in https://github.com/getmaxun/maxun/pull/933
• feat: !wrap robots table w table head by @amhsirak in https://github.com/getmaxun/maxun/pull/934
• feat: display api key creation date by @amhsirak in https://github.com/getmaxun/maxun/pull/935
• feat: run scroll customization by @amhsirak in https://github.com/getmaxun/maxun/pull/936
• feat(core): real time progress update for runs by @amhsirak in https://github.com/getmaxun/maxun/pull/937
• feat: add crawl and search by @RohitR311 in https://github.com/getmaxun/maxun/pull/938
• chore: pre-release v0.0.31 by @amhsirak in https://github.com/getmaxun/maxun/pull/940

Full Changelog: https://github.com/getmaxun/maxun/compare/v0.0.30...v0.0.31