If you are an ISP that measures or plans to measure the Internet traffic into and out of residential subscribers’ homes, do you know for certain that your meter system is accurate—and does it matter if it is not?
Meter accuracy matters to subscribers—especially when usage-based billing (UBB) is implemented. Subscriber skepticism about usage meter accuracy has consequences. Savvy network users with simple measurement tools scrutinize meter accuracy, and such scrutiny often spawns criticism about meter inaccuracy—sometimes forcing an ISP to withdraw a meter. In our experience there is ample reason for concern about meter accuracy, because meters are often incorrect.
Some governments are considering instituting meter accuracy certification requirements to protect consumers. In the US, for example, draft federal legislation now in committee stipulates: “An Internet service provider may not use a data usage monitoring system as part of usage-based billing unless the data usage monitoring system is certified [as accurate].”
Additionally, in its Protecting and Promoting the Open Internet Order the US Federal Communications Commission (FCC) emphasizes the importance of transparency in Internet operations. Specifically, the Order “reaffirms the importance of ensuring transparency, so that consumers are fully informed about the Internet access they are purchasing.”
An ISP’s best course is to ensure that its Internet data usage meter works properly, to be transparent about how the meter functions, and to provide solid evidence of its accuracy. It is a good practice to have an independent third party measure and certify accuracy. This approach is more credible to customers and potential government regulators than an ISP’s unsubstantiated assertion that its meter is accurate.
What We Do
We are the Internet data usage meter accuracy experts! Our business is to measure the accuracy of Internet data usage meters. Wherever you are in the lifecycle of your meter—we can help. We provide a choice of services to improve and validate the accuracy of your meter.
NetForecast has assessed the accuracy of 12 ISPs’ data usage meters over the past nine years. Our customers serve 70% of the US wireline broadband Internet subscribers.
During development and pre-production, we work with ISPs to create a meter specification, and assess the accuracy of the meter system as a whole. We also assess meter subsystem performance to help pinpoint accuracy problems.
Once a meter is slated for production, NetForecast monitors system accuracy relative to the meter specification, and if the system complies with the specification, we certify that fact in a NetForecast-branded public meter accuracy certification report. We also provide continuous, proactive reports regarding meter subsystem accuracy so ISPs can find and correct meter subsystem issues that develop over time.
After meter launch, system elements will change due to upgrades, vendor replacements, network reconfigurations, etc.—and these changes often degrade meter system accuracy. NetForecast continuously monitors the effects of such changes, and provides information and advisory services to help proactively detect and correct resulting accuracy problems in meter subsystems.
How We Do It
NetForecast’s specially-instrumented routers measure data usage at the handoff point between the subscriber’s Internet connection and home LAN. Our cloud-based Usage Mapping (UMapsm) platform aggregates and analyzes the measurement data that provides the basis for meter accuracy reporting. The ISP supplies counter data from various points within its meter system. Accuracy is determined by comparing NetForecast counts with ISP counts. (See Figure 1 for more details.)
NetForecast’s usage meter accuracy validation services are delivered in two ways. Active testing uses traffic generators to create precisely-known quantities of synthetic reference traffic that are transferred on exact schedules over dedicated, “quiet” Internet connections. Passive testing measures Internet traffic generated by real-user volunteers under real-world conditions.
NetForecast recommends a mix of active and passive test sites.
Active reference testing is useful to:
- know precisely how much traffic traversed a circuit and when,
- proactively look for known meter problems,
- monitor system software functionality, and
- assess and ensure meter system accuracy to the customer portal.
Passive traffic measurement is useful to:
- cover a very large service footprint,
- gather information across all times of the day,
- understand the effects of real-user traffic that encompasses all applications and traffic types, and
- discover previously unknown functionality issues affecting meter accuracy.
Figure 1 – Alternative Accuracy Measurement Methods
Where We Do It
An ISP meter is a complex system that integrates data from multiple subsystem elements as described in Figure 2. Your meter may have more or fewer subsystems with different names; however, the general approach is to count usage at a network element within the network near the subscriber’s access line (the network edge). The counter data is then passed through several processing stages, which convert raw counts into meaningful usage data associated with a particular subscriber for a time period. Eventually the usage is stored in a database as “formal” meter records for an account. These records are used for billing and accessed by a web server to present usage information to the subscriber.
Meter accuracy must be verified on an end-to-end basis. NetForecast provides end-to-end validation to the formal meter record, which is what appears on a subscriber’s bill. For active testing we can also validate to the subscriber portal view. Additional subsystems can be monitored as an optional service.
Figure 2 – Multiple Places to Evaluate Accuracy
Validating the accuracy of an Internet data usage meter may sound easy, but it isn’t. It requires in-depth knowledge combined with rigorous planning, process, and documentation to mitigate the following risks. In addition, the results must successfully withstand intense public scrutiny.
Subsystem accuracy ≠ system accuracy: It is tempting to believe that if each individual meter subsystem passes QA inspection, the meter system as a whole is accurate. Although proper subsystem operation is essential, it is insufficient for successful meter operation. In NetForecast’s experience, errors often occur during handoffs between subsystems. Checking subsystems in isolation treats each subsystem as a silo. Each silo may pass QA inspection, but the overall system may be inaccurate. The only way to prove system accuracy is to validate on an end-to-end basis.
True end-to-end validation is complex: End-to-end validation is hard to achieve because consumer CPE is not appropriately instrumented. A validation counter must be on the subscriber side of the access line; therefore, special instrumentation is needed to obtain valid independent usage counts. Once such counts are obtained, in-depth statistical expertise, rigor, and experience are essential to recognize subtle discrepancies and properly document, assess, and track system accuracy.
Poor signal-to-noise in meter data yields coarse error results: A truly useful meter assessment involves meticulous precision. End-to-end test results may appear to show that meter values reflect known test traffic counts; however, this assessment is typically performed at a very coarse granularity, i.e., “the meter tracks traffic within +/-10%.” Such an imprecise comparison precludes meaningful accuracy judgments.
The fundamental error calculation compares two usage data values: the subscriber count and the meter record count. The two data values have a poor signal-to-noise ratio (i.e., high noise levels distort the true signal). The precision of the meter validation results must be at least one order of magnitude better than the meter specification. Assuming an error bound specification of +/-1%, the assessment must operate at better than +/-0.1%.
NetForecast performs sophisticated statistical filtering on the counter data to remove the noise. This results in error results with a high degree of confidence. The UMap system has been validated at a precision of +/-0.003%, which is unsurpassed in the industry.
Meter accuracy is often viewed as a one-time project and not funded or staffed as an ongoing effort: Meter system accuracy fluctuates because meter systems are ever-changing. In NetForecast’s experience, meters do not remain accurate after initial certification. Even minor software updates can adversely affect accuracy. Maintaining meter accuracy requires continuous accuracy validation over the life of the meter system—a labor-intensive effort for technical teams with many other responsibilities.
NetForecast specializes in ongoing meter accuracy validation, and we bring experience and “lessons learned” from many client engagements to each of our ongoing meter accuracy validation projects.
Need More Information?
Contact us here to arrange a call to discuss your needs.