Pre-Launch Testing at Scale: Custom Crawler for Multi-Client Deployments

A UK marketing agency with 75 employees was managing enterprise website launches for clients in various retail and B2B sectors. Each project involved complex tech stacks: multiple CDNs, payment gateways, and third-party integrations. Their manual quality checking process was very time consuming. Additionally they faced limitations in validating performance and functionality across different devices and browsers before launching client's websites.

Under constant deadline pressure, quality assurance cycles were often shortened, causing client's website issues to surface after going life.

These post-launch surprises eroded client confidence and pulled the agency’s developers into reactive firefighting instead of forward planning. The team needed a reliable, automated way to validate real-world performance and user journeys before clients did. Ensuring launches were smooth, predictable, and reputation-safe.

We built a self-hosted crawler that runs within their infrastructure. The system uses configuration files, making it adaptable for each client without code changes. Agency's web devs can define crawl patterns, authentication, device profiles, and performance thresholds specific to each project.

The crawler validates checkout flows, tests across Chrome/Firefox/Safari and specific geolocations (client's target markets) and checks web performance areas UK-based agency was interested in. It integrates Lighthouse for performance auditing. Results are delivered as easy-to-follow dashboards and JSON reports, with Slack alerts for critical issues.

Development took 2 months, including testing against three live client projects. The solution runs entirely on-premises, protecting client confidentiality and enabling testing behind VPNs and authentication.

The first three launches using the crawler identified almost 200 critical issues pre-launch including: payment integration failures on mobile Safari, and performance bottlenecks. These would have caused significant customer impact if caught post-launch.

Emergency support calls decreased by approximately 45% as fewer issues reached production. The web dev team shifted from repetitive manual testing to higher-value exploratory work.

Within a month, the agency configured the system for two additional clients. Each setup was completed in just 2–3 days. A dramatic improvement from the five weeks originally estimated by agency's dev team. The crawler became a standard deliverable, differentiating them from competitors still relying on manual testing.