UMap

UMap is a distributed cloud-based service. It gathers data usage and network performance data from UMap home routers, aggregates it, and generates reports.

In addition, we perform user consumption modeling to help an ISP predict evolving bandwidth patterns over its user population.

  1. ISP Data Usage Meter Validation

    Is your data usage meter accurate?

    If you are an ISP that measures or plans to measure the Internet traffic into and out of residential subscribers’ homes, do you know for certain that your meter system is accurate—and does it matter if it is not?

    Meter accuracy matters to subscribers—especially when usage-based billing (UBB) is implemented. Subscriber skepticism about usage meter accuracy has consequences. Savvy network users with simple measurement tools scrutinize meter accuracy, and such scrutiny often spawns criticism about meter inaccuracy—sometimes forcing an ISP to withdraw a meter. In our experience there is ample reason for concern about meter accuracy, because meters are often incorrect.

    Some governments are considering instituting meter accuracy certification requirements to protect consumers. In the US, for example, draft federal legislation now in committee stipulates: “An Internet service provider may not use a data usage monitoring system as part of usage-based billing unless the data usage monitoring system is certified [as accurate].”

    Additionally, in its Protecting and Promoting the Open Internet Order the US Federal Communications Commission (FCC) emphasizes the importance of transparency in Internet operations. Specifically, the Order “reaffirms the importance of ensuring transparency, so that consumers are fully informed about the Internet access they are purchasing.”

    An ISP’s best course is to ensure that its Internet data usage meter works properly, to be transparent about how the meter functions, and to provide solid evidence of its accuracy. It is a good practice to have an independent third party measure and certify accuracy. This approach is more credible to customers and potential government regulators than an ISP’s unsubstantiated assertion that its meter is accurate.

    What We Do

    We are the Internet data usage meter accuracy experts! Our business is to measure the accuracy of Internet data usage meters. Wherever you are in the lifecycle of your meter—we can help. We provide a choice of services to improve and validate the accuracy of your meter.

    NetForecast has assessed the accuracy of 12 ISPs’ data usage meters over the past nine years. Our customers serve 70% of the US wireline broadband Internet subscribers.

    During development and pre-production, we work with ISPs to create a meter specification, and assess the accuracy of the meter system as a whole. We also assess meter subsystem performance to help pinpoint accuracy problems.

    Once a meter is slated for production, NetForecast monitors system accuracy relative to the meter specification, and if the system complies with the specification, we certify that fact in a NetForecast-branded public meter accuracy certification report. We also provide continuous, proactive reports regarding meter subsystem accuracy so ISPs can find and correct meter subsystem issues that develop over time.

    After meter launch, system elements will change due to upgrades, vendor replacements, network reconfigurations, etc.—and these changes often degrade meter system accuracy. NetForecast continuously monitors the effects of such changes, and provides information and advisory services to help proactively detect and correct resulting accuracy problems in meter subsystems.

    How We Do It

    NetForecast’s specially-instrumented routers measure data usage at the handoff point between the subscriber’s Internet connection and home LAN. Our cloud-based Usage Mapping (UMapsm) platform aggregates and analyzes the measurement data that provides the basis for meter accuracy reporting. The ISP supplies counter data from various points within its meter system. Accuracy is determined by comparing NetForecast counts with ISP counts. (See Figure 1 for more details.)

    NetForecast’s usage meter accuracy validation services are delivered in two ways. Active testing uses traffic generators to create precisely-known quantities of synthetic reference traffic that are transferred on exact schedules over dedicated, “quiet” Internet connections. Passive testing measures Internet traffic generated by real-user volunteers under real-world conditions.

    NetForecast recommends a mix of active and passive test sites.

    Active reference testing is useful to:

    • know precisely how much traffic traversed a circuit and when,
    • proactively look for known meter problems,
    • monitor system software functionality, and
    • assess and ensure meter system accuracy to the customer portal.

    Passive traffic measurement is useful to:

    • cover a very large service footprint,
    • gather information across all times of the day,
    • understand the effects of real-user traffic that encompasses all applications and traffic types, and
    • discover previously unknown functionality issues affecting meter accuracy.

    NF_Web_1

    Figure 1 – Alternative Accuracy Measurement Methods

     

    Where We Do It

    An ISP meter is a complex system that integrates data from multiple subsystem elements as described in Figure 2. Your meter may have more or fewer subsystems with different names; however, the general approach is to count usage at a network element within the network near the subscriber’s access line (the network edge). The counter data is then passed through several processing stages, which convert raw counts into meaningful usage data associated with a particular subscriber for a time period. Eventually the usage is stored in a database as “formal” meter records for an account. These records are used for billing and accessed by a web server to present usage information to the subscriber.

    Meter accuracy must be verified on an end-to-end basis. NetForecast provides end-to-end validation to the formal meter record, which is what appears on a subscriber’s bill. For active testing we can also validate to the subscriber portal view. Additional subsystems can be monitored as an optional service.

    NF_Web_2

     Figure 2 – Multiple Places to Evaluate Accuracy

     

    Self-Validation Risks

    Validating the accuracy of an Internet data usage meter may sound easy, but it isn’t. It requires in-depth knowledge combined with rigorous planning, process, and documentation to mitigate the following risks. In addition, the results must successfully withstand intense public scrutiny.

    Subsystem accuracy system accuracy: It is tempting to believe that if each individual meter subsystem passes QA inspection, the meter system as a whole is accurate. Although proper subsystem operation is essential, it is insufficient for successful meter operation. In NetForecast’s experience, errors often occur during handoffs between subsystems. Checking subsystems in isolation treats each subsystem as a silo. Each silo may pass QA inspection, but the overall system may be inaccurate. The only way to prove system accuracy is to validate on an end-to-end basis.

    True end-to-end validation is complex: End-to-end validation is hard to achieve because consumer CPE is not appropriately instrumented. A validation counter must be on the subscriber side of the access line; therefore, special instrumentation is needed to obtain valid independent usage counts. Once such counts are obtained, in-depth statistical expertise, rigor, and experience are essential to recognize subtle discrepancies and properly document, assess, and track system accuracy.

    Poor signal-to-noise in meter data yields coarse error results: A truly useful meter assessment involves meticulous precision. End-to-end test results may appear to show that meter values reflect known test traffic counts; however, this assessment is typically performed at a very coarse granularity, i.e., “the meter tracks traffic within +/-10%.” Such an imprecise comparison precludes meaningful accuracy judgments.

    The fundamental error calculation compares two usage data values: the subscriber count and the meter record count. The two data values have a poor signal-to-noise ratio (i.e., high noise levels distort the true signal). The precision of the meter validation results must be at least one order of magnitude better than the meter specification. Assuming an error bound specification of +/-1%, the assessment must operate at better than +/-0.1%.

    NetForecast performs sophisticated statistical filtering on the counter data to remove the noise. This results in error results with a high degree of confidence. The UMap system has been validated at a precision of +/-0.003%, which is unsurpassed in the industry.

    Meter accuracy is often viewed as a one-time project and not funded or staffed as an ongoing effort: Meter system accuracy fluctuates because meter systems are ever-changing. In NetForecast’s experience, meters do not remain accurate after initial certification. Even minor software updates can adversely affect accuracy. Maintaining meter accuracy requires continuous accuracy validation over the life of the meter system—a labor-intensive effort for technical teams with many other responsibilities.

    NetForecast specializes in ongoing meter accuracy validation, and we bring experience and “lessons learned” from many client engagements to each of our ongoing meter accuracy validation projects.

    Need More Information?

    Contact us here to arrange a call to discuss your needs.

     

  2. NetForecast Usage Meter Accuracy Services

    Do you need meter accuracy assurance?

    NetForecast’s family of data usage meter accuracy services provides comprehensive support throughout the lifecycle of an ISP’s meter.

    Meter Accuracy Certification Audit: NetForecast’s Internet usage meter accuracy certification audit entails running active reference tests of synthetically-generated traffic traversing a “quiet” Internet connection at a few, carefully controlled locations. NetForecast determines if an ISP’s meter system is working as specified to the formal meter database and to the customer portal. If it is, we independently certify and document that fact in a public audit report.

    Active sites are installed on standard circuits that are dedicated to NetForecast tests and measurements. NetForecast testing for each circuit is performed using UMap measurement and reporting software embedded into a customized, fully-featured, wireless home router supplied and supported by NetForecast, and a test PC (UMap probe) running NetForecast software.

    NetForecast obtains subscriber accounts and performs accuracy validation testing using FTP accounts on NetForecast servers on the Internet. We perform both downstream and upstream testing under a variety of conditions. The tests consist of repeatedly transferring files of varying sizes in complex patterns.

    The UMap system collects precise counts of NetForecast-generated reference test patterns, along with detailed timing information for each transfer. We analyze the data and compare our independently-measured counts of synthetic traffic with the counts generated by the ISP’s internal meter system for the same traffic, as well as with counts shown on the customer portal, to determine whether the meter system’s counts match our validated counts within the ISP’s target accuracy range. Certification of meter accuracy is then documented in a public report posted on the NetForecast website. Click here to see examples of our published reports.

    Meter Accuracy Ongoing Validation Audit: NetForecast’s ongoing validation audit service continuously monitors the ISP’s Internet data usage meter to ensure that it remains accurate over time. Meter system elements change due to upgrades, vendor replacements, network reconfigurations, etc., and those changes can adversely affect meter system accuracy. To achieve long-term accuracy, it is essential to continuously and independently validate meter system accuracy.

    Ongoing accuracy validation is achieved by continuously performing passive measurements of real-user traffic in subscriber homes, and active reference tests in which a NetForecast PC and server generate the only traffic on dedicated test lines in subscriber homes. In combination, these two approaches help identify potential meter accuracy issues quickly, and supply data needed to troubleshoot and resolve issues.

    In both cases, NetForecast performs independent traffic measurements, obtains hourly usage meter records from the ISP for each location, analyzes the data, and compares NetForecast’s measurements with the ISP’s records continuously monitor end-to-end accuracy of the ISP’s data usage meter system. NetForecast generates monthly meter accuracy reports, and will notify and assist the ISP if accuracy issues arise. Assuming the data usage meter system works as specified, NetForecast will provide a yearly audit report to confirm that fact.

    Portal Monitoring (for Active Test Sites): This optional service tracks and reports monthly on the accuracy of the meter results presented to a user on the customer portal.

    Weekly Meter Record Check (for Active Test Sites): This optional service provides email reports on meter data sent by the ISP to NetForecast up to four times each month (frequency depends on how often the ISP sends usage data). The reports, which are generally delivered within two business days of receiving the usage data, document the integrity of the meter records received and whether the values generally tracked the reference tests.

    Subsystem Monitoring: In this optional service, NetForecast monitors the accuracy of one or more meter subsystems and provides monthly reports that compare usage documented at the intermediate reporting point(s) to usage measured by the UMap router at the subscriber location (see Figure 3). In addition, the usage recorded by the subsystem is compared to usage recorded in the billing system formal meter records.

    NF_Web_3Figure 3 – Subsystem Monitoring Options

    The ISP provides monthly data from intermediate reporting point(s), and NetForecast analyzes the data and generates reports that help diagnose and pinpoint meter accuracy issues. For example, if the meter is inaccurate in the formal meter record but accurate at subsystem B, then the ISP can deduce that the accuracy problem is between the two subsystems.

    Meter systems span the purview of multiple groups within an ISP, making it challenging to determine which group is best positioned to resolve a problem. Subsystem monitoring helps locate the problem, shortens mean time to repair, and helps ensure that the right resources are deployed to fix a problem.

    Each subsystem tracked requires a separate implementation of this optional service.

    Need More Information?

    Contact us here to arrange a call to discuss your needs.

  3. ISP Network Performance Measurement

    How well does your Internet service perform?

    NetForecast provides ongoing Internet broadband performance measurement services using active tests run via NetForecast-supplied UMapsm probes installed in subscribers’ homes or on dedicated test circuits at ISP-selected locations. The UMap probes measure performance from the subscriber’s perspective. NetForecast generates performance reports on down and up speeds, latency, and packet loss. This data can be used to comply with FCC reporting requirements, as well as to help the ISP’s network operations team identify and diagnose network performance problems.

    The UMap probe is easy and fast to deploy. Subscribers can connect it to any household router or gateway, or ISPs can install it at strategic network locations. NetForecast uses UDP reference tests that are more reliable than alternative speed tests, and our approach is not constrained by TCP bandwidth-latency product. Unlike other testing services, the results are the joint property of the ISP and NetForecast rather than other parties.

    NetForecast’s performance measurement service is enabled by software in small probes (slightly larger than a standard desktop mouse) connected to routers or gateways. The measurement software tests Internet connection performance in both directions, and reports results to NetForecast’s UMap management system which aggregates the data. NetForecast generates quarterly performance reports.

    The service provides reliable data about actual speeds delivered across the network reported as percent of advertised (POA) speed, as well as latency and packet loss. This is accomplished by measuring at strategic locations with sufficient sample counts to provide a statistically-grounded inference that the test results truly reflect general performance across the network. The FCC requires that performance results be reported for 30 days for at least 30 sites for each speed tier that constitutes a minimum of 5 percent of an ISP’s subscribers.

    Benefits of NetForecast’s Approach:

    • UDP reference tests are more reliable that other tests available
    • Performance is not limited by TCP bandwidth-latency product
    • Results are joint property of the ISP and NetForecast (anonymized) rather than other parties.
  4. Data Consumption Modeling

    How do users, devices and apps consume network resources?

    Internet user behavior is in a never-ending state of flux as new network-based applications and activities gain traction and old ones morph or fall by the wayside. This makes it hard for end users to understand how much bandwidth they consume, and how much bandwidth they need–and it makes it challenging for an ISP to understand and predict evolving bandwidth consumption patterns over its entire population of households served.

    NetForecast has amassed a growing library of profiles for networked applications. These profiles provide insight into the behavior of applications individually and in combination. NetForecast formulates use scenarios and creates a traffic model based upon that information and the relevant application profiles. The model predicts individual subscriber and aggregate traffic demands under various “what if” scenarios. The model also predicts performance parameters affecting the user experience under the scenarios.

    Based on information from the model, NetForecast makes capacity planning recommendations. This information is useful to provide educational materials for broadband subscribers about how to estimate their capacity needs, and properly size their network connection.

    Benefits:

    • “What if” modeling can lead to better understanding of what to expect under different scenarios
    • Modeling results help ISPs and enterprises make decisions based on hard data rather than mere guesses
    • Consumption information can enable ISPs to “right size” their networks
    • Consumption information can help consumers to identify the right service for their needs
Convoy