CDN Testing Methodology: Active Probing, P50/P95, and Reproducible Rankings

The goal is not one fastest run; it is a reproducible comparison

Network performance changes with geography, carrier, time, cache state, and routing path. A browser speed test can help diagnose one user’s problem, but it cannot by itself become a provider ranking. A fair benchmark fixes the test object and publishes its probe, timing, outlier, and aggregation rules.

CdnChart’s active-probing methodology follows one principle: the same object and rules are measured continuously from different vantage points, with the statistical boundary made visible. Ranking pages show the aggregate; the methodology explains how raw observations become metrics.

A unified test object is the starting point for fairness

Each provider is measured with versioned standard objects covering small-object, medium-object, and large-object scenarios. Small objects expose connection and first-byte behavior. Medium objects approximate common static assets. Large objects reveal sustained transfer and throughput. A typical configuration may include 500 B, 50 KB, and 1 MB, but the public methodology should always be the source of truth for the active version.

The object definition includes cache policy, expected status, content length, compression, request method, URL, headers, and timeout. Otherwise one provider may be measured on a cache hit while another is measured on an origin fetch. When the object or rule changes, the new version begins its own sample history.

Break one request into phases

Total time matters to a user, but diagnosis requires knowing where time was spent. CdnChart records interpretable phases:

• DNS: time to resolve the hostname;
• TCP: time to establish the transport connection;
• TLS: time to complete the secure handshake;
• TTFB: time from request dispatch to the first byte;
• download: time from the first byte until the object completes;
• total: the full request duration.

Connection reuse, protocol differences, and network libraries can change phase boundaries. The platform therefore publishes the measurement definition instead of treating one browser DevTools timing as equivalent to an active-probe observation.

What latency, throughput, and availability answer

Latency describes how quickly a request reaches its first byte or completion. Throughput describes effective transfer rate during a larger object. Availability describes whether the request completed under the success rules. They are related, but none replaces the others.

A provider can have a fast first byte and average large-file transfer, while another can deliver high throughput but show intermittent connection failures in one region. CdnChart displays raw dimensions separately and marks which dimensions are available for any composite ranking.

Why use P50 and P95

Network latency is usually long-tailed. An arithmetic mean can be pulled upward by a small number of severe spikes and then represent neither a typical request nor a common bad experience.

• P50 is the median and approximates a typical request;
• P95 exposes the tail experienced by the slower but still common requests;
• mean is a supporting statistic, not a sufficient ranking metric by itself.

Percentiles must be read with sample size and time window. A P95 calculated from a very small sample can be dominated by one observation. CdnChart therefore applies sample gates and shows a data-collection state when a dimension is not mature enough to rank.

Probe location and network type change the result

A probe is a real network vantage point, not a decorative map marker. CdnChart organizes probes by region, country or area, city, network type, and health state while preserving the observation time and egress context. Mainland China carrier exits cannot be averaged with overseas edge locations; two carriers in the same country can follow different paths.

Global rankings answer a broad coverage question. Regional rankings answer whether a provider performs well for a particular audience. Provider selection should use the scope that matches the users being served instead of choosing the first name on a global list.

Scheduling and freshness

The scheduler creates tasks with a target, object version, probe constraints, timeout, and idempotency key. Raw observations preserve phase timing, status, response IP, and failure reason. Aggregation then produces 24-hour, 7-day, and 30-day trends. Every displayed metric should include its latest measurement time and sample count.

Failures are not discarded without classification. The pipeline distinguishes target errors, DNS failures, connection timeouts, TLS failures, interrupted downloads, and probe faults. Probe health, recent heartbeats, and duplicate tasks are part of the scheduling and cleaning contract.

From raw observations to a ranking

The aggregation flow is:

1. Validate the target, probe, and object version.
2. Store phase timings, status, response IP, and failure reason.
3. Exclude observations that fail quality rules while retaining the exclusion reason.
4. Compute P50/P95, throughput, and success rate by provider, region, and time window.
5. Apply sample gates, coverage gates, and dimension-availability rules.
6. Publish rankings, trends, and a traceable data timestamp.

A composite score is a versioned decision model, not a law of nature. Weighting, normalization, and missing-dimension behavior belong in the methodology and should be bound to the ranking snapshot. One scoring version should never be presented as a permanent statement about a provider.

Reusing and reviewing the data

Use the fastest CDN ranking for latency, the throughput ranking for large-object delivery, and the CDN comparison for same-scope side-by-side review. Provider pages expose trends and regional cuts; the methodology page exposes probes, samples, and quality gates.

For a migration or major configuration change, preserve the before-and-after window and method version. Compare the same region, object, and approximate network scope. Do not subtract values from different object versions or unrelated regions as if they were one experiment.

Method limitations

Active probing observes controlled external paths; it is not full real-user monitoring. It cannot reproduce every browser cache, connection reuse pattern, device constraint, enterprise proxy, Wi-Fi condition, or customer cache policy. A standardized object also does not represent every production asset.

Use CdnChart for provider screening, regional baselines, route investigations, and post-change validation. Combine it with your own logs, RUM, error rates, and contractual SLA for final business decisions.

Summary

Reproducible CDN testing is not one script. It is a measurement contract covering a unified object, phase-level timing, distributed probes, failure handling, percentile statistics, sample gates, and versioned scoring. CdnChart publishes those rules with the ranking so every number can be explained, compared, and reviewed.