PCT and IOT in LTE Networks: A Study on Test Cases and Test Results

In order to produce communication devices compliant with the LTE standard, a terminal manufacturer must perform a series of tests before mass production. This work analyzes the test cases and results of LTE Protocol Conformance Test (PCT) and Interoperability Test (IOT) for the user equipment (UE). We conduct 175 PCT test cases from the 3GPP specifications and the results show a PCT pass ratio of about 80%. IOT consists of 9 test cases selected by the manufacturers based on an agreed set of important test cases; only a few Devices under Test (DUTs) fail the IOT. Further analysis reveals that there exists a many-to-many mapping between PCT cases and IOT cases: a total of 42 PCT cases, i.e., 24% of PCT, map to all 9 IOT cases, and a PCT case might also map to several IOT cases. On average, every IOT case maps to 13 different PCT cases. The PCT cases that do not have mapped counterparts in IOT are mostly Layer 2 test cases. This is because most of the IOT test flows use Layer 3 parameters to determine passing or not without changing the default Layer 2 parameters. Passing IOT does not warrant passing PCT since PCT has requirements that are more rigid.


INTRODUCTION
In recent years, due to the fast deployment of fourth generation mobile communication systems, manufacturers are devoted in developing related communication devices, such as dongles, CPEs (Customer Premise Equipment), mobile phones, and so on. In order to satisfy the Long Term Evolution (LTE) standard and increase the compatibility between devices, LTE terminal device makers and chip makers have many testing requirements for their manufactured User Equipment (UE). The tests can be categorized into Protocol Conformance Test (PCT), Interoperability Test (IOT), Over-The-Air Test (OTA Test), and Field Trials, generally corresponding to Stage 1 to Stage 4, respectively. A manufacturer first guarantees every feature of each of its products satisfies the standard regulation with the PCT. The manufacturer then guarantees its products can correctly communicate with the base stations of different carriers and can perform functions such as attach and handover by performing the IOT. Lastly, before mass production, the manufacturer tests whether its products provide good user experience and satisfy the expected efficiency in an Operator IOT Lab under OTA test and Field Trials environment.
The 3rd Generation Partnership Project (3GPP) sets forth the LTE specification; together with the LTE standard, the 3GPP also sets forth a series of specifications for the PCT testing of user equipment [1][2][3][4][5]. The PCT is performed on a base station emulator which emulates a base station, Evolved Node B (eNB), and a core network, Evolved Packet Core (EPC). In order to avoid the interference from other signals that affects the PCT results, the PCT uses the conducted method, which utilizes physical cables to connect the UE's antenna to the base station emulator. The test cases are performed based on the 3GPP specifications. The goal of the PCT is to ensure that a UE can satisfy the specifications, thereby guaranteeing the UE's working functionality. Thus, the PCT exhaustively examines whether the message exchange flows satisfy the expected responses in the positive or negative form to avoid a UE opportunistically accomplishing the expected functionality and passing the test. In addition, the 3GPP specifies a series of PCT standards for every rejected message of each procedure, such as the cause of message rejection, proper handling of the rejection, etc.
IOT is performed on actual base stations instead of emulators by using Over the Air (OTA) method to mimic a real communication environment when testing. Another difference between IOT and PCT is that there is no testing standard specification on IOT. The test cases are specified by individual manufacturer or laboratory based on the 3GPP specification [6,7] or other requirements. The purpose of IOT is to ensure that the manufactured UE can operate correctly with base stations of different manufactures. Some related work focuses and explores in-depth the design of the PCT test cases in different systems [8,9]. In particular, the work [8] studies the BGP4+ protocol using the XML language to implement the PCT. This study describes the design of the BGP4+ protocol conformance test, and also details the XML language implementation methodology. The work [9] designs a Mobile IPv6 protocol conformance test system based on the ISO standard. The study also discusses the test structure of the system as well as the formalization of a test suite.
There are some studies focusing on IOT testing [10][11][12][13] on systems different from LTE system. Based on the design of ATM/B-ISDN signaling protocol, the work [10] utilizes the interoperability test suite derivation algorithm to automatically generate test cases. In [11], Seol et al. also propose automatic test derivation method, specifically designed for the interoperability test of communication protocols. Song and Lee propose a design of an interoperability test system that is based on the IEEE 1451.5-802.11 standard [12]. They also presented a test architecture, an interoperability test system, and a case study based on the IEEE 1451.5 standard in [13].
To the best of our knowledge, there is no research work on the PCT and the IOT testing and analysis for the user equipment (UE) in LTE networks. Thus, in this article, we explore the passing rate and the reasons for the PCT and the IOT failures. We also include the methods for adjusting the Protocol Implementation Conformance Statement/Protocol Implementation eXtra Information for Testing (PICS/PIXIT). During testing, we observe that there exists a many-to-many mapping relationship between the PCT and the IOT. Hence, after detailed analyses of these two test suites, we present and further explore the mapping relationship between the PCT and the IOT. Besides simply studying the PCT and the IOT individually, we seek to clearly distinguish the test goals and test significance by exploring the similarities and differences between these two tests.
The remainder of the article is organized as follows. First, we introduce the test procedure and test methodology of the PCT, including the specification set forth by the 3GPP on the PCT test cases in Section 2. Then, we introduce the test cases of IOT, and present the method and results of mapping the PCT to the IOT and vice versa in Section 3. After that, we present our experiment setting, analyze and discuss our observed testing results in Section 4. Finally, we conclude our article in Section 5.

Test Procedures
The UE Protocol Conformance Test (PCT) is a type of test that is used to determine whether the UE is implemented in accordance to a specification. Besides defining the test cases, the 3GPP LTE PCT specifications, TS 36.523-1 [1], also details every step of every test case. If the device does not pass any step in the process of any test, the test is interrupted and the device is determined to have failed the test case.
Going through the test cases set forth by [1], the test outcome can be one of PASS, FAIL, or INCONCLUSIVE. Once we obtain these results, we can proceed to analyzing the obtained results. However, for each test case, we must first understand its goal, its test methods, and its significance. Thus we introduce the test cases, the methods to resolve INCONCLUSIVE outcomes, and the importance of adjusting the PICS/PIXIT. Figure 1 illustrates the detailed test procedure we designed. Firstly, we initialize the test environment, and then perform each test case specified by [1]. If the outcome of the test is INCONCLUSIVE, we must verify whether the PICS/PIXIT settings or other related test settings are correct. If the settings are correct, we repeatedly perform the test until the outcome is no longer INCONCLUSIVE. If the outcome is FAIL, we then analyze whether the PICS/PIXIT settings caused the failure; if we determine that incorrect PICS/PIXIT settings caused the test failure, we perform the test case again after adjusting the settings. If we determine that the failure is not related to the PICS/PIXIT settings, we then use the corresponding IOT to perform more advanced analysis. If the test outcome is PASS, we then directly use the corresponding IOT to perform more analysis.  Table 1 lists the defined test cases in [1]; the 3GPP has categorized the test cases into several sections and each number shown under the Test Case column is the section number corresponding to that test case. Note that Section 16 only defines the name of the test case, but lacks the test contents. Section 6 mainly specifies testing procedures related to the Idle mode; Section 7 and Section 8 perform relevant tests based on Layer 2 and the Radio Resource Control (RRC) Layer, respectively, of the LTE Layer structure. Section 9 mainly specifies tests related to mobility issues. Section 10 specifies testing establishing, adjusting, and releasing a bearer. Section 11 mainly targets testing texting and telephone services. Section 12 specifies tests related to Multi-Input Multi-Output (MIMO). Section 13 mainly specifies testing procedures that require multi-layer support. Section 14 specifies testing procedures related to the Earthquake and Tsunami Warning System (ETWS). Section 15 specifies tests related to Dual Stack Mobile IPv6 (DSMIPv6). Section 17 specifies tests related to the Multimedia Broadcast Multicast Service (MBMS). Finally, Section 18 specifies test cases related to the Public Warning System (PWS).

Special Test Cases
For some cases, we need the manufacturer to provide relevant information or additional required test devices in order to perform the tests. For example, in test cases 9.2.1.1.1 and 9.2.1.1.2, we need the manufacturer to provide supported AT commands; in test cases 6.1.1.1 and 6.1.2.5, we need two base station emulators. Specifically, the AT commands of some tests must be entered before UE powers on; that is, when testing, we must first use the AT commands to enter flight mode for the DUT, then enter the testing AT commands, and exit flight mode after inputting the AT commands. A specific instance is using the at+cfun command with Qualcomm devices. For tests that require the support of two base station emulators, since our laboratory currently has only one Rohde & Schwarz CMW500 emulator, we skip these tests at this time.

PICS/PIXIT Settings
When we perform PCT, we often encounter issues related to adjusting PICS/PIXIT settings. PICS settings are used to implement the system in order to satisfy the standard. PIXIT is related to and set according to the test standard. Since every UE differs in the set of supported functionalities, we must set relevant parameters with respect to UEs of different manufacturers before testing. If the PICS/PIXIT parameters are set differently from those supported by a UE, the test would result in INCONCLUSIVE or TIMEOUT, and we cannot ensure whether the UE implemented the specified functionality in accordance to the protocol specifications. In order to conduct our tests, we list some important PICS/PIXIT parameters that deserve attention when using different UEs. The parameters listed in Table 2. are basic parameters necessary when establishing connections and can be mainly categorized into parameters used to set LTE frame structure, frequency band, channel bandwidth, UE category, IMEI, IMSI, ciphering algorithm, and integrity protection algorithm. Afterward we will introduce the methods of PICS/PIXIT parameters adjustments to avoid the occurrences of some common testing problems.

Test Procedures
Interoperability Conformance Test (IOT) is a type of tests that determines that, while the device is implemented in accordance to the protocol specifications, whether the device could work together in the operational environment. The test observes whether the messages exchanged between a UE and a base station interface are correct and satisfy the test requirements.
IOT does not have a particular specification, but instead performs the test cases defined by each manufacturer. As such, we perform the test cases that most manufacturers deem important after referencing the test cases of several manufacturers. Table 3   Verify the capability of the LTE system to provide support for streaming applications running on the LTE system. Voice Verify the capability of the LTE system to provide support for VoIP applications running on the LTE system.

Multiple Bearer Transmission
To ensure that the UE is able to request and use multiple bearers successfully.

PCT/IOT Test Case Mapping
Before we study the relationship between the PCT and the IOT, we must establish the correspondence between the PCT and the IOT. We use the IOT message exchange flows, and map the behavior of each or multiple flows to the PCT test cases. Thus, with respect to one particular IOT test case, the entire test would correspond to multiple PCT test cases. However, some behaviors of the IOT overlap with itself; hence some PCT test cases also correspond to multiple IOT cases. Overall, the mapping is manyto-many. Figure 2 shows one mapping example, in which we take the IOT Default/Dedicated Bearer test and map it to the PCT. Establishing a default bearer has two parts: attach and bearer establishment. The steps between establishing a default bearer and establishing a dedicated bearer map to PCT reconfiguration test. The Attach procedure starts with the UE sending an RRCConnectionRequest message to the eNB. The eNB, after receiving the request, replies with the RRCConnectionSetup, and the UE then sends ATTCH_REQUEST in an RRCConnectionSetupComplete message. After that, the eNB replies with the ATTACH_ACCEPT in an RRCConnectionReconfiguration message. The procedure is complete after the UE confirms the default bearer and sends out ATTACH_COMPLETE in a ULInformationTransfer message. The bearer establishment test case mainly ensures that the message communications from when the UE receives ATTACH_ACCEPT in the RRCConnectionReconfiguration message until the UE finishes establishing a default bearer are correct. Finally, in the process of establishing a dedicated bearer after establishing a default bearer, the eNB and the UE can communicate using the RRCConnectionReconfiguration and the RRCConnectionReconfigurationComplete messages; this part of the process can map to the PCT test cases related to RRC connection reconfiguration. Due to the complexity of the actual default/dedicated bearer establishment process, here we use a simplified message exchange flow as shown in Figure 2 as our example. We follow the above criteria to map other IOT test cases, and include our detailed mapping results in Table 4

TEST ENVIRONMENT AND RESULT ANALYSIS 4.1 Test Devices and Environment
In PCT testing, we use a Rohde & Schwarz CMW 500 to generate and emulate base station signals. We use Qualcomm MDM9200 and Innofidei Rapid400M UEs as our devices under test (DUTs), and connect them to the base station using the conducted method. We also connect the DUT to a laptop to monitor the communication messages between the DUT and the CMW500. In contrast to the PCT, our IOT test uses actual base stations, including HUAWEI and Nokia Siemens Networks (NSN). We use the same set of DUTs as those in the PCT; however, the DUTs in the IOT use Over-the-Air (OTA) communications in an ideal channel condition, we thus place each DUT in locations where it receives strong signals from the base station. We also connect the DUT to a laptop to monitor the communications between the DUT and the base station. But when we test for handovers, we change the DUT's location to decrease the signal strength between the DUT and the originally attached eNB, so as to force the DUT handover to the target eNB with stronger received signal power.

Examples of PICS/PIXIT Adjustment
As mentioned before, if a test generates an INCONCLUSIVE outcome, we must resolve the outcome by analyzing the log. In order to ensure the correctness of the test results, resolving INCONCLUSIVE outcomes is an important step in the PCT. Here, we use two examples to demonstrate how to adjust PICS/PIXIT parameter settings to resolve INCONCLUSIVE outcomes.
The first example in Figure 3(a) illustrates the test results generated by incorrect EPS attach type settings. The log shows that the expected received EPS attach type at the CMW500 is 010, while the actual received attach type is 001. To determine the significance behind this log entry, we refer to related specifications to uncover the error and find a way to resolve it. Based on the description of 3GPP specification, TS 36.508 [4], this log entry shows the CMW500 emulator expects to receive an EPS attach type of "combined EPS/IMSI attach", but in reality receives an "EPS attach only" which is provided by the DUT. Hence, we must find and adjust the relevant PICS/PIXIT parameters of the DUT; relevant parameters include px_AttachTypeTested, pc_Attach, and pc_Combined_Attach. By adjusting these parameters and subsequently rerun the test, we can resolve the problem.
The second example in Figure 3(b) shows the generated test outcome when the PDN Type setting is incorrect. The log entry shows that the received type value is expected to be 011, but received 001 instead. Similarly, we refer to the related specification to find the meaning of the type value: a PDN Type of 011 represents that both IPv4 and IPv6 are supported, whereas 001 means that only IPv4 is supported. With this knowledge, we can adjust our test parameters based on this log entry, and finish the test.

PCT Results
With the above method of adjusting the PICS/PIXIT parameter settings, we can proceed to performing the test.

IOT Results
We selected a total of 9 IOT test cases. The DUTs enjoy a high IOT passing rate not only because the IOT test cases tend to test the basic functionalities, but also because both DUTs are almost in mass production phase; our results are shown in Table 6.
Innofidei DUT passed all IOT testing while Qualcomm has one failed test case. When testing the Qualcomm MDM9200 DUT for Default/Dedicated Bearer tests, the DUT could not successfully establish a dedicated bearer. The test record is shown in Figure  4(a) with an ESM cause: semantic errors in packet filter(s). After carefully examining the cause, we found that in the PCT test, CMW500 emulator adopts the packet filter parameters as two packet filters in which the packet filter directions are downlink only and uplink only, respectively. However, in the IOT test, our EPC sets the packet filter parameter as one packet filter with the packet filter direction as bidirectional, as shown in Figure 4(b). Thus, the Qualcomm DUT rejects the dedicated bearer establishment. But such IOT failure does not mean that the DUT also fail the PCT test. The reason is that in the PCT test, the PCT specifications clearly specify the relevant parameter settings; however, in the IOT test, the parameters of the base station and the core network could be set according to the needs of manufacturers or operators. In other words, one of the purposes of the IOT is to reveal portions of the PCT that is unspecified and may affect the interoperability. If we modify the EPC packet filter parameter settings as the one adopted in the PCT, the Qualcomm DUT would also pass the IOT Default/Dedicated Bearer test.

Discussions
Based on the mappings shown in In addition, we observe that IOT still fail even though the corresponding PCT test cases have all passed. The IOT Default/Dedicated Bearer test for the Qualcomm MDM9200 DUT in an example we have mentioned in the IOT results. The reason is that in the PCT test, the PCT specifications clearly specify the relevant parameter settings; however, in the IOT test, the parameters of the base station and the core network could be set according to the needs of manufacturers or operators. The design of the PCT still lacks some scenarios (or some parameter settings) for testing, while a good IOT test case design could reveal these unspecified parts of the PCT.

CONCLUSIONS
To understand the cause of inconsistencies between the outcomes of the PCT and the IOT for a UE, we start by studying in detail the PCT and the IOT test methods, we then analyze the corresponding behavior relationships between the PCT and the IOT test cases. Finally, we use the corresponding behavior relationships and the test outcomes to find that the PCT has very strict requirements for every message, whereas the IOT is more relaxed. For the DUTs that we are able to acquire, the IOT passing rate of each DUT is high; if we can obtain more data, we can further deduce whether passing the PCT is sufficient to passing the IOT. We leave this deduction as a possible future work. Through experiments and analysis, we obtain the following conclusions:  Among the 175 PCT test cases, the passing rates of our DUTs are between 80.70% and 83.33%. If the DUT implementation differs from the protocol specification, a test returns FAIL.  Based on our mapping and outcomes, passing the IOT is not sufficient to show that a DUT would also pass the PCT. This is because the PCT has stricter requirements.


We also observe that a DUT failing the IOT does not necessarily also fail the PCT. This is because even though PCT has stricter requirements, it may still lack some test specifications. A good IOT test case design could reveal the unspecified part of the PCT for the testing completeness.