CSRIC Indoor Location Test Bed Report Summary Results

To read the full CSRIC report, click here.

To read about the test procedures and plan click here.

7. SUMMARY OF RESULTS


Only brief summary results by morphology are provided here for accuracy (horizontal and vertical distance errors), yield, TTTF and Uncertainty. Horizontal accuracy results are also summarized by technology vendor. The detailed results of all testing are provided in the attached report entitled: “Indoor Test Report to CSRIC III-WG3.” Those results include individual results per test point per technology and summaries aggregated by building and by morphology. Detailed scatter plots overlaid on Google Earth images are also provided there per technology at each test point. Those provide significant detailed insight that is not captured in the summaries.

7.1 Yield


Yield in general varied with the difficulty of the indoor test environment. This is manifested in the yield results in Table 7.1-1. This variation was particularly true of the location technology that was integrated into the network (AGPS) and thereby required multiple message exchanges to carry out the location determination process per the standard implementation. (The technology from Polaris was processed off line and therefore this observation does not apply to it). The NextNav technology also showed some dependence in yield on the difficulty of the environment, but to a lesser extent, since less interaction with the infrastructure is required in the implementation tested. In all, the values of yield observed were encouraging given the difficulty of the environments in which the test calls were made.

Table 7.1-1 Summary Test Call Yield Results

7. 2 Overall Location Accuracy Summary


A concise overall summary of location accuracy aggregated by morphology is presented in Table 7.2-1 for the three technologies under test. In the following sections, more illustrative summaries of accuracy performance, including comparative CDFs, are provided for each morphology. To provide the reader with added insight gained from the testing, comments are provided on the accuracy observed within each of these morphologies (dense urban, urban, etc.). The results are also summarized for each technology across the 4 distinct morphologies.

Table 7.2-1 Summary Horizontal Accuracy Statistics in All Environments


7.3 Location Accuracy by Environment


7.3.1 Dense Urban Environment

The following are the results aggregated over buildings BD1, BD2, BD3, BD14, BD15 and BD16, all in and around downtown San Francisco.

Figure 7.3-2 Accuracy Percentiles in the Dense Urban Environment

Figure 7.3-2 Accuracy Percentiles in Dense Urban Environment

The results for the dense urban buildings highlighted the challenges that satellite signals have in penetrating to those points that are in the interior of large buildings. Consequently, AGPS fall-back modes, such as AFLT, were experienced frequently. Accuracy degraded as expected when GPS fixes were not attained. While a surprising proportion of hybrid fixes were experienced, even at test points where one would not expect a satellite signal to penetrate, the quality of the hybrid fixes was in general significantly degraded compared to GPS fixes. Hence, in many dense urban test points (and in urban buildings as well) a significant amount of spread in location fixes was observed. This often extended over a number of city blocks. Few very poor fixes are seen in the dense urban case, perhaps because the high cell site densities (and consequently small cell radii) in the dense urban core create a reasonable lower bound on fall-back accuracy.

In contrast to the challenges that GPS signals face in the dense urban setting, RF finger printing experienced its best performance in the dense urban setting. This is probably a combination of a confined environment that could be extensively calibrated and many RF cell sites and handoff boundaries that could be leveraged in creating a good RF fingerprint map of the dense urban center.

The best observed performance in the dense urban setting was that of the dedicated terrestrial (beacon) location system—a new infrastructure that will require investment. With this level of accurate location performance it is actually possible to discern some of the vagaries caused by multipath. Oftentimes, when the test point is floors below the roof and in an outside room with windows, the signal is forced to propagate from the handset out (or by reciprocity in) towards or from a building that is across the street or a few blocks away (if the space between it and the test point is open). The signal then propagates to (or from) the location infrastructure, whether terrestrial beacons or satellites. The result is that location fixes that may be relatively close in absolute distance (e.g., 40 m away) are often located in a building across the street, in a neighboring building, or even across a few blocks from the test point. (See for example, BD3_TP1, BD14_TP3 and BD15_TP4 for NextNav In the attached Indoor Test Report. This phenomenon will also become more obvious in the urban setting.)

7.3.2 Urban Environment
The following are the results aggregated over buildings BD4, BD5, BD17, BD18 in and around Downtown San Francisco, plus BD19 in Downtown San Jose.

As mentioned earlier, the specific test buildings used in the urban morphology were challenging, each in their own way. This is because each building represented more distinctly a building type and setting than the high rises of the dense urban environment.

The baseball stadium by the San Francisco Bay (BD4, AT&T Park) created a situation where AGPS fallback fixes could be very far away due to the very exposed RF propagation outside the structure in which the test points were located. This impacted points that were relatively deep inside the stadium building. The structure of the stadium also appears to have created challenges to RF fingerprinting at some test points.

Figure 7.3-3 Cumulative Horizontal Accuracy in the Urban Environment

Figure 7.3-4 Accuracy Percentiles in the Urban Environment

The convention center created in some cases an environment that was deep indoors but with very strong cellular signal from cell sites inside the building (including a DAS). This situation was captured by two points of different depth (BD5_TP2 and BD5_TP3). This situation resulted in the beacon-based location system performing poorer than in most other test points, since attenuation to different directions in the outside world was particularly strong in those scenarios. AGPS and RF fingerprinting relied on the cell sites inside the structure to create adequate location fixes

The US Court of Appeals Building (BD17) represented a classic older, heavy construction, but also had a very large atrium in its middle. Results varied depending heavily on the degree of distance from windows or the central atrium. Again, the phenomenon of apparent location in a building across the street is seen here (e.g., BD17_TP2 for both NextNav and Qualcomm, which was a test point inside a large court room with windows in the direction of the building across the street). As one would expect, the degradation caused by being away from a window or atrium more significantly impacted the satellite based system than the terrestrial beacon based one. RF fingerprinting fixes appeared to cluster about the larger reflectors in this urban corner of San Francisco, which happened to be mostly across the streets from the target building.

The motel building (BD18) provided a very clear example of relatively good location fixes on the basis of absolute error distance but that are mostly in or around other buildings across the street (e.g., NextNav all four test points in BD18). This phenomenon is primarily caused by the physics of the problem. This case poignantly demonstrates the unique challenge with indoor location: absolute distances (like 50 or 150 m) which may have meant much in assessing outdoor performance mean less for the indoors, since emergency dispatch to the wrong building or even the wrong block could be easily encountered at 50 or 150 m. A location across the street is certainly better than one a few or many blocks away but it may still leave some human expectations unmet. RF fingerprinting for this building generates either fixes around the immediate vicinity of the building or clustered around major reflectors in the general area or along streets, presumably where calibration measurements were gathered.

Finally, the tall condominium building in urban downtown San Jose (BD19) demonstrated the mix of high rise construction causing direct signal attenuation, prominent distant reflectors, plus wide area cell site visibility. All combined to create relatively poor AGPS performance, uneven beacon system performance, and RF fingerprinting performance that degraded with the height of the test point.

All of the above factors related to each of the urban buildings, combined with a generally lower cell site density for fall back (than in dense urban), resulted ultimately in an aggregate urban performance that is slightly worse than the dense urban performance. Still, this overall performance is representative of the challenges of the big city with high structural density, whether it be San Francisco or a city in the Northeast or the Midwest.

7.3.3 Suburban Environment
The following are the results aggregated over buildings BD6, BD7, BD8, BD9, BD10 and BD11 in Santa Clara and Sunnyvale.

The effect of smaller buildings with lighter construction and more spacing between buildings is immediately evident on the quality of the location fixes in the suburban environment. This is most clearly demonstrated in the case of individual houses or small apartment buildings. Outstanding GPS performance, almost as good as outdoors, can be achieved inside single story homes (see BD8). The majority of the GPS fixes fall inside the small home or its small lot. Almost as good a performance is achieved inside the upper floor of relatively small buildings with composite or tile roof material (see BD10_PT1, BD11_PT1). CDF’s that are tightly packed at small error values (well below 50 m) signify this type of outstanding performance. Similarly outstanding performance is achieved on average by the beacon based location technology under similar circumstances. RF fingerprinting appears to suffer from performance degradation compared to more dense morphologies in the city. It is able to identify only the part of the neighborhood where the test calls originated, with spreads over a few to several blocks, and fixes that are frequently clustered or spread along roads where calibration was performed (e.g., BD8, BD9, BD10).

Figure 7.3-5 Cumulative Horizontal Accuracy in the Suburban Environment

Figure 7.3-6 Accuracy Percentiles in the Suburban Environment

The AGPS performance predictably changes as the suburban buildings become bigger and higher. Test points that are not on the top floor have significantly more positioning error and spread about them as fall-back modes are more frequently the solution. The terrestrial beacon-based network continues to perform well in the larger suburban buildings (e.g., BD6, BD7). The phenomenon of positioning at the nearest building is only occasionally seen (basically when the propagation physics force it to happen, which is not common in the suburban environment). One example where this is seen is the parking structure (BD7_PT5) where the location signals are forced to tunnel through the garage entrance and bounce off the side of the adjacent building. Curiously, GPS appears to perform well in this specific scenario, perhaps because the parking structure had only 2 floors. RF finger printing shows some enhancement relative to the smaller suburban buildings, but still shows most of the location fixes along the roads, highways or reflecting buildings.

7.3.4 Rural Environment
The following are the results aggregated over buildings BD12 and BD13 in the rural area north of Hollister, CA.

Figure 7.3-7 Cumulative Horizontal Accuracy in the Rural Environment

Figure 7.3-8 Accuracy Percentiles in the Rural Environment

As mentioned earlier, the buildings chosen for the rural environment where limited by what was accessible in the available time. Both buildings selected were large one story structures with metal roofs. Performance of AGPS reflected the effect of the metal roof and some metal siding in limiting the available number of satellite signals available for trilateration at certain test points. In these cases more hybrid fixes were experienced with a concomitant increase in the spread of the location fixes about the true location (e.g., BD13_TP2 and to a lesser extent BD12_TP1). In easier rural scenarios where metallic surfaces or multiple floors are not present, e.g., in a rural house, the expected performance would be very good similar to that seen in a suburban home, like BD8, or a small structure like BD11.


The performance of the beacon based network was less impacted by the metallic roof (since that roof had more impact on sky visibility rather than on side visibility towards terrestrial beacons). Consequently the performance was somewhat better than for AGPS. The performance of the beacon based network would of course depend on the density of its deployed beacons covering the rural area, which was sufficient in the case of the rural test polygon.


RF finger printing showed reduced performance relative to the suburban environment due to the large spacing between surveyed roads (where calibration is done) and the rural structures as well as the lower density of cell sites. The location fixes are spread along relatively long stretches of the rural roads.


7.4 Location Accuracy Summary by Technology


The following charts provide a quick view summary of the horizontal accuracy percentiles for each technology across the four morphologies.

Figure 7.4-1. Indoor Accuracy by Morphology for NextNav


Figure 7.4-2. Indoor Accuracy by Morphology for Polaris

Figure 7.4-3. Indoor Accuracy by Morphology for Qualcomm

7.5 Vertical Error


Altitude results were only provided by the NextNav technology. The statistics of the vertical distance error are as follows. In this statistical computation the absolute value of the altitude error of each test call is used. More detailed vertical error results are provided in the Indoor Test Report. The nature of vertical accuracy and its variation with environment is captured in the four CDFs shown in Figure 7.5-1. The vertical accuracy appears not to be tightly correlated with the density of the test environment.

Table 7.5-1. Summary Vertical Error by Morphology

Figure 7.5-1. Vertical Accuracy by Morphology for NextNav

7.6 TTFF


The summary results for time to first fix are provided below in Table 7.6-1. By design, the TTFF for NextNav and Polaris were set to be 27 and 24 seconds, respectively. This was considered during the design for the trial to fall reasonably well within the 30 second maximum latency target for the E911 application. Since both of these technologies in their test implementations were not tied to the message exchanges within the wireless network, little variation in the design TTFF was observed. In contrast, the AGPS technology exhibited more significant TTFF variations, driven by delays in the network location message exchanges in the difficult indoor environment. Still the 90th percentiles of TTFF in the more difficult urban and dense urban environments were both 33 seconds, indicating that long location delivery delays were not a significant issue in the indoor test settings. It should be noted that by design, the Qualcomm AGPS technology limits the GPS search time to 16 seconds to allow for the various messaging delays between network entities in a standard implementation (e.g., among MSC, PDE and MPC). Hence such delays beyond 30 seconds have no impact on the accuracy observed.

Table 7.6-1. Summary Indoor Test TTFF by Morphology and Technology

7.7 Reported Uncertainty


Uncertainty is statistically computed by the location algorithms of a given location technology, based on the observed measurements during a 9-1-1or a test call, to estimate the quality of the location fix delivered. The reported uncertainty value is the “radius” centered around the reported position within which the location algorithm “thinks” that the actual (and unknown) location will fall inside X% of the time, where X is the associated confidence level. The uncertainty in deployed location systems may correspond to different confidence values but they have been normalized in the current indoor testing to correspond to 90% confidence. Based on its common definition and use, reported uncertainty is a useful parameter that plays an important role in PSAP dispatch decisions. However, individual reported uncertainty values should not be used to substitute for the normally unknown location errors.

Table 7.7-1 Summary Reported Uncertainty per Technology per Morphology

In the context of indoor location testing, reported uncertainty results provide an indication of how useful that parameter would be when the call is placed indoors. Additionally, in the context of location system testing in general (not only indoors) the results provide an indication of how well a location system under test is performing in a certain environment. Lower accuracy is often (but not always) associated with larger statistical deviation from the 90% target for the reported uncertainty values. The measured uncertainty performance relative to the desired confidence level was reasonably well behaved for two of the three technologies under test. The third technology (RF Fingerprinting)
experienced a lower level of reported uncertainty reliability.


7.8 Vendor feedback on testing


7.8.1 NextNav
NextNav feels that its technology performed generally in line with expectations, considering the intentionally rigorous nature of the testing performed and the very dense concentration of buildings in the Urban Polygons. Overall yield across all morphologies was 96%, and overall TTFF at the 90th percentile was 27 seconds. Horizontal location accuracy across all morphologies for the median , 67th and 90th percentiles were 36m, 51m and 94m respectively.  NextNav vertical performance was also tested across all morphologies, and vertical location accuracy for the median, 67th and 90th percentiles was 2m, 2.9m and 4.8m respectively (compared to an average floor height separation of 3 meters). Across the 74 total test points inside 19 buildings, about a third of the fixes fell inside the target building, and the majority of all fixes fell inside of 40 meters. NextNav expects that its next generation system, which was not available for testing in the testbed, will improve upon these results.

7.8.2 Polaris Wireless
The performance for RFPM in the test-bed was not completely consistent with the design parameters for this technology. In reviewing the data from the test, a few specific issues were identified.

The Polaris Wireless implementation of RFPM technology is based on comparing the RSSI values reported by the handset with off-line predictions of the signal strengths that should be observed in various locations and on using these predicted signal strengths to anticipate where these signals should be decoded by the handset. The representation of the signal environment that is used in our deployed location systems allows only one predicted value per cell per location, but in reality, the signal environment in the upper floors of a high rise building is significantly different from that on the lower floors because of the difference in obstructions to the signal paths. Polaris Wireless is currently working on a multiple signature representation that will overcome these limitations, but it was not deemed appropriate for these trials because it did not represent our currently deployed technology. Before the trial started, we anticipated that our performance would suffer on upper floors, and that indeed proved to be the case. The impact of the challenge noted above can be seen in the table below, which compares the summary test results for Dense Urban with a summary that omits the results for upper floor test points (exact test points omitted are identified in the third column of the table):


In addition to the upper floor challenge. Two of the buildings included DAS (distributed antenna system) cells. DAS environments generally represent a very good environment for RFPM. However, due to the very tight time schedule of this testing, we were unable to work with the carriers to create detailed models for these cells. We believe that with proper models for DAS, our performance for calls involving these cells would have been much better and a truer reflection of the performance we would achieve in an actual deployment.

Finally, the tight timelines of the trial did not allow us to use a network connection, such as would be the case in an actual deployment. Instead, we used data recorded from the handset to do off-line location estimates. For the serving cell, a GSM handset reports the signal strength of the downlink traffic channel. Because the down link power is time-varying, the received traffic channel power cannot be used to estimate position without knowing the instantaneous down link power, a quantity known to the network but not to the handset. A connection to the live network would have provided us this information and resulted in an additional (and strong) RSSI measurement (and improved performance).

7.8.3 Qualcomm
Qualcomm believes that the report fairly reflects the deployed system accuracy, and TTFF. Yield degradation appears to be mostly caused by real-world RF issues rather than issues in the phase 2 location calculation system in that a fraction of the calls across morphologies did not reach the location system at all, but were included in yield calculation. Yield varied by morphology; 85% in dense urban to 99% in rural, corresponding to RF issues in the deployed networks.

TTFF at the 90% level ranged from 26 to 33 seconds; again with the more difficult urban and dense urban having the longer fix times due to network timer issues. Accuracy 67th /90Th percentile ranged from 48.5 m/210.1m (rural) to 226.8 m/449.3 m (urban). Qualcomm provides a hybrid solution combining the strength of ubiquitous nationwide coverage from GPS satellites, combined with ranging to terrestrial cell towers at indoor locations where GPS is blocked. This can be seen in the staircase shape of the CDF graphs in the attached report (e.g. BD3_TP3 & TP4), where each technology contributes to the combined solution. Accuracy is driven by the strength of the observed signals and the geometric diversity of the transmitters. GPS provides the most accurate positioning. For example, Qualcomm_BD13_TP1 had 90.5 % GPS with a 33.5 m / 67th percentile. AFLT (Advance Forward Link Trilateration) uses ranging to multiple cell towers. For example, Qualcomm_BD2_TP3 had 75.0 % AFLT with a 111.5 m / 67th percentile. Mixed Cell Sector is a geometry based overlapping of cell sectors calculation. For example, Qualcomm_BD5_TP2 had 89.0 % MCS with 163.3 m / 67th percentile.

Future improvements to 3G AFLT accuracy are expected when 4G LTE/OTDOA is deployed. OTDOA has been designed with a specific PRS ranging signal to have better than AFLT performance. Wide spread commercial deployments of OTDOA capable LTE networks and OTDOA support on newer model handsets will allow testing and validation of indoor accuracy in the next CSRIC test bed.

© 2016 NextNav LLC. All Rights Reserved.