Configure web collectors and inspect collected content

Choose Monitor or Search

Use a Monitor collector when you know the starting URL. Use a Search collector when you want a configured search service to find starting results for a query. A collector stores its definition separately from each run, making it possible to inspect what happened over time.

Before running, choose the correct workspace, an available result-storage provider and at least one enabled source. Search also needs a configured search provider. Collection produces stored source material; it does not automatically approve it, publish a post or add it to every AI agent's knowledge. Review the result before reusing it through Publishing or an explicitly configured knowledge workflow.

Configure the sources and output

1. Create the collector with a useful name, type and result-storage provider.
2. Add a Monitor URL or a Search request, according to the collector type.
3. Choose each source's result format: HTML, Markdown, JSON, Images or CSV.
4. Set its depth and quantity conservatively, enable the source, and save the definition.

Result format belongs to each source, so one collector can contain sources with different outputs. Depth zero processes the starting page or result without deeper crawling. CSV can provide discovered links as tabular output. The stored manifest remains JSON and records output references, warnings and errors regardless of the selected source formats.

Run once and inspect the result

Make the collector active and request a manual Run. A run request returns an execution to follow; it does not mean collection has already completed. Open its history and inspect individual source outcomes, output files and warnings. A completed source may still have useful warnings about skipped material or a content limit.

Running collectors cannot be edited normally. Stop the current execution when you need to interrupt work, then inspect its final state before changing the definition. Inactive collectors cannot run manually or from a trigger. If a result cannot be downloaded, check your result-download permission and the configured storage provider rather than assuming the extraction itself failed.

Schedule repeat collection

Add a trigger with a name, valid cron expression, start time, optional expiry and enabled state. Confirm its next occurrence and keep the collector active. Disabled or expired triggers do not start new runs. Manual and scheduled runs use the same collection pipeline, but scheduled work also competes for the deployment's available collection capacity.

An AutoFailed execution means a scheduled run exceeded the configured parallel capacity before scraping started. It is different from a source failing during extraction. Spread triggers across different times or ask the administrator to review the configured limits. Repeatedly starting overlapping manual runs does not repair an unavailable provider or insufficient storage.

Understand collection limits

The default extraction uses static HTML. Increasing Scroll quantity changes the content budget; it does not guarantee that a browser scrolls through a dynamic page. Browser extraction requires explicit deployment enablement. A page that depends on browser execution can return limited static material and a warning. Depth, downloads, page counts and execution size are bounded, and Monitor crawling stays on the starting host by default.

Current search adapters cover Google Custom Search, Serper and Vertex. Older labels are misleading: Bing currently maps to the Google adapter, and Superdev maps to Serper. Confirm the configured service with your administrator. A Collector-channel source has no execution adapter in the current implementation. Do not treat a catalog option as proof that a source or provider can run; check the actual execution result and configuration.