🛰️ Site Scraper — CLI reference

Run from the project root

All commands are dispatched by index.php (and cron jobs by cronjobs.php). Invocation: php index.php <command> [instance] [arg1] [arg2] [arg3] — the optional instance number (default 1) lets you run workers in parallel and drives the per-worker lock file under temp/ (pass f to bypass the lock). Workers self-exit after about 10 minutes. Click any example to copy it.

Import / seeding

CommandDescriptionExample
import Diff the all_domains pool against sites for a TLD and insert + queue the new domains.
php index.php import
csv-import Import domains from a CSV (or a plain newline list when the column is empty) and queue them. Args: file, column, rows to skip.
php index.php csv-import 1 /tmp/domains.csv domain 0
update-block-list Rebuild blocked_domains from main-domains that have more than 50 sub-sites.
php index.php update-block-list

Queue (write crawl jobs)

CommandDescriptionExample
enqueue Auto top-up: fills the queue from several sources until the configured thresholds are met.
php index.php enqueue
enqueue-new Queue never-crawled sites (status IS NULL) for the configured TLDs.
php index.php enqueue-new
enqueue-older-than Queue sites whose lastCheck is older than the given date. Arg: date.
php index.php enqueue-older-than 1 2025-01-01
enqueue-platform Queue all live sites running a given platform. Arg: platform slug.
php index.php enqueue-platform 1 shopware_6
enqueue-ecommerce Queue all live sites on the known e-commerce shop systems.
php index.php enqueue-ecommerce
enqueue-ecommerce-score Queue live sites with a high e-commerce score but no detected platform yet.
php index.php enqueue-ecommerce-score
enqueue-generator Queue unknown-platform sites whose generator meta starts with a prefix. Arg: prefix.
php index.php enqueue-generator 1 WordPress
enqueue-unknown Queue unknown-platform sites that expose any non-empty generator meta tag.
php index.php enqueue-unknown

Crawl / process

CommandDescriptionExample
crawl Core loop: consume the job queue and scrape each site. Run several instances in parallel (each keeps its own lock).
php index.php crawl 1
single Force-scrape a single host immediately, ignoring the skip/recency checks. Arg: host.
php index.php single 1 example.com

Enrichment

CommandDescriptionExample
imprint Scrape contact details (email / post code / city) from imprint pages of online sites.
php index.php imprint
sw5payment Scrape configured payment methods from Shopware 5 shops.
php index.php sw5payment
ip-regions Fill sites_ips.country from the GeoIP database for high-score online sites.
php index.php ip-regions
task Run a one-off task manager from TaskManagers/Custom/. Arg: class name (see list below).
php index.php task 1 AmazonPayInstallationsEmail
custom Run the hard-wired ShopwareSofortTaskManager.
php index.php custom
custom2 Run the hard-wired ShopwareAmazonPayTaskManager.
php index.php custom2

Stats / maintenance

CommandDescriptionExample
stats Print open-job counts (total + priority) and the number of e-commerce sites.
php index.php stats
cleanup-migrations Merge flapping platform migrations: when a site migrates again within 14 days, collapse the changes into one (first → last) and drop round-trips. Add dry to preview without writing.
php index.php cleanup-migrations 1 dry
test Ad-hoc experimental routine (currently probes Shopware suggest endpoints). Edit before use.
php index.php test

Cron (cronjobs.php)

CommandDescriptionExample
<cron job> Run a class from TaskManagers/Cronjobs/. The first argument is the class name (see list below).
php cronjobs.php AmazonPayInstallationStatus

Available task managers — 20 in TaskManagers/Custom/

AmazonPayInstallationsEmail GambioCv1TaskManager GambioCv2TaskManager GambioShopsTaskManager Magento2AmazonMerchantIdTaskManager PlentyIdTaskManager PrestaAmazonMerchantIdTaskManager RedirectDomains RedirectDomainsTaskManager Shopware5ConfiguratorTaskManager Shopware6AmazonPayTaskManager Shopware6CloudTaskManager Shopware6QrCodePluginTaskManager Shopware6StripeAppTaskManager Shopware6VersionTaskManager ShopwareAmazonPayTaskManager ShopwareDreamrobotTaskManager ShopwareSofortTaskManager ShopwareSubscriptionsTaskManager TextOnHomepageTaskManager

Available cron jobs — TaskManagers/Cronjobs/

AmazonPayInstallationStatus AmazonPaySaveStats

Site status codes — the status column

ValueMeaning
NULLUncrawled — never checked
1Online
-1Crawl error
-2Empty / inactive
-3Redirect (see redirectTo)
-4For sale