Fuliang Li, Jiahai Yang, Huijing Zhang, Xingwei Wang＊, Suogang Li, Jianping Wu

1 School of Computer Science and Engineering, Northeastern University, China

2 Institute for Network Sciences and Cyberspace, Tsinghua University, Beijing, 100084, China

3 Department of Computer Science, Beijing University of Posts and Telecommunications, China

4 CERNET National Network Center, Beijing, 100084, China

* The corresponding author, email: wangxw@mail.neu.edu.cn

I. INTRODUCTION

Misconfigurations are the main reasons for network interruption and anomalies [1-4].Furthermore, network devices tend to support more functions and provide more services,which will cause more configuration errors.Therefore, configuration management is still a crucial research topic.

1.1 Configuration analysis

Many studies have been conducted to address configuration properties and diagnose misconfigurations. H. Kim et al. [5] analyzed fiveyear configuration data of router, switch and firewall from two large campus networks.They presented a long-term longitudinal study on the configuration evolution, such as how configurations of each task evolve, which parts of configurations change more frequently, etc.Y. Sung et al. [6] investigated the configuration dynamics of five enterprise VPN customers. N. Feamster et al. [7] conducted a statistical analysis of router configurations to identify BGP misconfigurations. F. Le et al. [8] applied data mining technologies to extract router configuration policies, which would be utilized as test cases to detect configuration errors. G. Xie et al. [9] used the router configuration snapshots to investigate network reachability and further to troubleshoot reachability problems.T. Benson et al. proposed two models to quan-tify configuration complexity [10, 11]. They not only captured the router configuration difficulties, but also unraveled the complexity of configuring network services.

This paper analyzes the distribution of configuration tasks and network failures about weekly reports of an operational IP backbone, and draws a conclusion that rate limiting and launching routes for new customers are most frequently configured.An example of how to provide semi-automated configuration is also described in this paper.

Our work is close to a previous study on configuration evolution analysis [5]. However,there are two differences. 1) Unlike existing methodologies that extract changes by comparing two consecutive versions of the configuration files, we capture the configuration evolution by tracking high-level configuration tasks recorded in the weekly reports. 2) We unveil the beneficial information hidden in the weekly reports of an operational IP backbone network, which can supplement our understanding on configuration evolution of backbone networks distinguishing from campus and enterprise networks.

1.2 Configuration provisioning

Network configuration is a complicated and error-prone job that lays heavy burden on operators. Thus, extensive studies have been conducted to achieve the goal of automated configuration provisioning. D. Caldwell et al.[12] developed a tool to identify configuration policies and templates, which could drive the migration of the network towards automated provisioning. J. Gottlieb et al. [13] put forward a template-driven approach for ISPs to configure connections for new BGP-speaking customers automatically. W. Enck et al. [14]proposed a system calledPRESTOto construct device-native configuration templates. They also showed how to configure the service of VPN with the proposed system. X. Chen, et al. [15, 16] put forward a mechanism for configuration management by taking advantage of the database to abstract specific configurations and adopting declarative language to describe dependencies and restrictions among the network components.

The contributions of this paper include two aspects. 1) We make a first-ever and high-level analysis on configuration evolution of an operational IP backbone network. According to the dataset gathered from January 2006 to January 2013, we conduct a statistics of the configuration tasks and capture the tasks which are configured more frequently. 2) We extract configuration templates by associating high-level configuration tasks with specific configuration commands in the configuration files, which provides a practical basis for automated configuration provisioning.

The reminder of the paper is organized as follows. Section II introduces the data sources.Section III presents the analysis methodology and results. We discuss the implications of the observations in Section IV. We conclude the whole paper in Section V.

II. DATA SOURCES

The data sources are collected from an operational IP backbone network. In this section, we first introduce the studied network, and then describe the data sources in details.

2.1 The Studied network

The studied network is China Education and Research Network (i.e., CERNET) [17]. It is a nationwide IP backbone network and provides Internet access to universities and research institutes of China. CERNET operation team focuses on solving technical challenges and overcoming difficulties in network operation and management. The backbone network nodes (POPs) are scattered over 38 universities located in 36 cities. The PoPs connect with each other via 2.5 Gbps, 10 Gbps or 100 Gbps links, providing Internet access for more than 2000 campus networks at 1 Gbps, 2.5 Gbps or 10 Gbps. The users of the studied network are over 20 million. In addition, it has 28 international lines interconnecting with North America, Europe, and Asia Pacific Rim, etc.The total bandwidth of CERNET connecting to domestic ISPs and international ISPs reaches over 70 Gbps and 30 Gbps respectively.

2.2 Historical data

Weekly reports:Our high-level configuration evolution analysis is based on the weekly reports, which contain abundant and valuable information of operating an IP backbone net-work. Each weekly report mainly consists of two aspects of information. 1) It records the tasks processed by the operators. The tasks include adding a new customer (e.g., a university or a research institute) to make it access the studied network, stopping a customer from accessing the studied network, and rate limiting, etc. 2) It also records the failures of the backbone network. The failures include link failure, device failure, and power failure, etc.We gather the weekly reports of the studied network from January 2006 to January 2013.According to the weekly reports, we can track which configuration behaviors dominate the configuration operations and which configuration tasks are configured most frequently,i.e., we can restore the truth and uncover the network configuration evolution.

Configuration files:The studied network consists of more than 100 backbone routers.Operators backup the configuration files of these routers through a self-developed system.Note that only the changed configuration files and the differences between two consecutive versions of the configuration files are recorded. Once we conduct a wrong configuration operation, we can quickly roll back the router configuration to the most recent valid version.The configuration files are regarded as an auxiliary data source to the weekly reports,because each task can be associated with a certain set of commands that are manually implanted into the configuration files.

III. METHODOLOGIES AND HIGH-LEVEL ANALYSIS

First of all, we describe the methodology of capturing the configuration evolution of the studied network. Secondly, we analyze the configuration evolution in a high-level analysis way. At last, we investigate the network failures, which are indirectly related to network configurations.

3.1 Methodology

We capture the configuration evolution by tracking high-level configuration tasks recorded in the weekly reports. Table I shows the types of configuration tasks that can be tracked from the weekly reports. In addition,we correlate each high-level task to a certain set of commands that are manually implanted into the configuration files, based on which,we associate the tasks with three types of basic configuration operations, i.e., modification,addition and deletion. Our method is more direct and efficient to investigate the evolution of network configurations. However, our method may be a more general and coarsegrained classification method. For example,modifying a routing policy may also involve adding or deleting operations.

3.2 High-level analyses of configuration evolution

1) Distribution of the configuration tasks: As depicted in Fig. 1, rate limiting accounts for 37.04% of the configuration changes. Task of rate limiting refers to increasing or decreasing the access bandwidth for the customers.As more and more emerging applications and services with highe network performance demand, customers often have the requirements of bandwidth upgrading. In addition, to maximize their profits, customers will access the Internet in a multi-homing way, so they may decrease the bandwidth of the existing connected ISPs, which can save money for accessing a new ISP to satisfy their special needs. Operators need to conduct rate-limiting configurations to satisfy different requirements of the customers. Configurations of adding customer account for 36.78% of the configura-tion changes. Adding customer (i.e., launching routes for new customers) mainly refers to making a new customer access the studied network. Benefiting from the rapid development of the studied network, more and more customers choose it to access the Internet either for the high speed or for the abundant resources.

Table I types of the configuration behaviors

Suspending customer refers to temporarily preventing a customer without paying in time from accessing the network, while rebooting customer refers to re-activating a customer to access the studied network. These two types of configuration tasks account for 15.73% of the configuration changes. Configurations of routing related account for about 7.81% of the configuration changes. The routing related tasks include adjusting the parameters ofOSPFandBGPfor load balancing, as well as adding new physical links, which will cause the network topology to change. By comparison, routing related configurations are more complicated and error-prone. Tasks of canceling customer account for 2.64% of the configuration changes. With the increasing customers who are willing to access the studied network, and fewer customers who are revoked the privilege to access it, we can expect that the scale of the studied network is still expanding.

Fig. 1 Statistics of the configuration tasks over the seven years

2) Changes of the configuration commands:We roughly correlate each high-level task with a basic configuration operation as shown in Table I. Then we correlate each high-level task with specific commands executed in the configuration files. For example, operation of adding customer will add about eight configuration command lines.

We calculate the number of command lines for the three types of operations. As shown in Table II, operators conduct the adding operations in most cases, which is accordance with the conclusion we could get from Fig. 1.Adding operation mainly refers to the configurations of adding customers. Due to the rapid development of the studied network, more and more customers want to get the permission to access it. With the growth of the customers,the increasing traffic may lead congestion to some links. Operators need to add new links or upgrade the existing links among the backbone routers, as well as adjust the routing policies. Adding new links will invoke a chain of routing adjustments to make the links evenly used. Due to its complexity, operators should pay attention to the adding operations in order to decrease the possibility of misconfigurations.

Modifying operations also account for a considerable part of the basic operations.Modifying operation mainly refers to the configurations of routing related and rate limiting.Routing related tasks play a quite crucial role in a large-scale network operation. Operators need to make a better choice for both inter-domain and intra-domain routing in order to balance the cost and the performance, and gain the maximum profit at the same time. With the growth of the users, as well as the demand on performance improvement and commercial profits maximization, customers of the studied network also need operators to conduct many rate limiting operations. We are delighted to witness the rapid development of the studied network, but we also have to admit that the development also brings many configuration operations that need to be handled timely and successfully. Therefore, we should be aware of the importance of the modifying operations,which are complicated to deal with.

3) Longitudinal analysis of the configuration tasks:Fig. 2 shows how each configuration task evolves over the seven years. We find that the configuration tasks present different patterns during these years. Taking rate limiting as an example, we can see that there are no rate limiting tasks before the year of 2010.However, it presents a sharp increase after that year. We can infer that due to the rapid development, the customer networks need to change the access bandwidth to improve the network performance or realize multi-homing. We also find that some configuration tasks present a downward trend, such as adding customer and routing related. For one thing, the studied network devotes to be an education and research network. Therefore, customers are mainly universities, research institutes and government departments. These types of customers tend to be in a stable scale and new customers become less and less. However, since there are more tasks of adding customers than canceling customers, the size of the studied network is still increasing. For another thing, once an ISP develops to a mature phase, the backbone network will run in a stable state, so configuration tasks related to routing adjustments decrease.Note that operators did not record the tasks of routing related in weekly reports since 2010.

We also find that the configuration tasks concentrate in a few months of each year.Therefore, operators need to try their best to avoid misconfigurations during these months.In addition, adding customer and rate limiting are configured most frequently. Operators can correlate a set of certain command lines in the configuration files to these tasks in order to construct configuration templates. The templates can be transformed into executable scripts (including syntax ofexpect[18]),which can configure the task on the corresponding devices automatically.

3.3 High-level analysis of network failures

We separate configuration behaviors into direct configuration behaviors and indirect configuration behaviors. The former refers to the specific configuration tasks depicted in Table I, while the latter refers to the network failures that may need pre-configurations to reduce the influence caused by them, such as link failure,routing failure and device failure, etc. If we can configure redundant links for some links,which may be congested or broken down, the network can reroute the traffic to other linksor devices automatically. This will greatly enhance the network reliability. Ideally, backups for all the links and devices are necessary.However, due to the high cost and complicated network environment, it is difficult to realize.A tradeoff is to understand the network failures and configure redundant links for those links with high failure frequency.

Table II Change analysis of configuration commands over the seven years

Fig. 2 Longitudinal analysis of the configuration tasks over the seven years

1) Distribution of the network failures: We firstly analyze the failure types and failure ratio of each type of failure. We do not show the number of each failure for privacy reasons. As shown in Table III, network failures are mainly classified into four types. The reasons for these failures are identified and recorded by the operators in the Network Operation Center(NOC) or operators in the regional POPs. We find that link failures account for 82.9% of the network failures. Link failure will cause interruptions of data transmission. However, these links may carry the traffic for some real-time services, which should be guaranteed with a reliable data transmission. Actually, operators often configure redundant links to enhance the fault tolerance of the links. However, we cannot configure redundant links for all the links limited by the network environment and high cost. Thus, it is better to configure redundant links in a selective way, i.e., configuring redundant links for some links which present high frequency of failures.

Routing failure ranks following the link failure. Many undesirable outcomes may stem from routing failure, such as network congestion and routing oscillation, etc. Operators need to resort to reasonable configuration operations to minimize the influence. For example, when a link is congested, some traffic of this link can be rerouted to an underutilized link automatically. Power failure ranks thethird place among these failures. Power failure is mainly caused by some planned factors, so power failures are easy to manage. Operators can take some reasonable measures such as employingUPS, which can effectively solve such emergencies once a power failure happens as planned. Device failure ranks the fourth place. To prevent device failures, operators need to deploy and configure redundant devices for the devices that are often broken down. Once a primary device is out of order,the redundant device will take over the responsibility automatically. Other failures refer to some uncertain failures. These factors may be caused by hostile attacks. Operators should consider security policies on data plane when configuring routers, such as ingress filtering,egress filtering, rate limiting and black hole,etc.

2) Detailed analysis of the link failures: All the network failures finally performs like link failures. For a fault-tolerant network, once a link failure happens, redundant links for this link will be activated. In this section, we conduct a detailed analysis of link failures, which is helpful to make intelligent routing policies configured on the routers. As depicted in Fig.3, each point represents a failure happened in the link during the past seven years. If the points of a link are very dense, we can infer that the link encounters a high rate of failure.Operators should configure redundant links for the links that are easy to break down in order to enhance the reliability of data transmission.We also find that the number of the links that encounter link failures seems to show an increase during the past years. Due to the rapid development of customer networks and the growth of new customers accessing the studied network, operators add some new links to offload the traffic from the heavily utilized links. Operators also need to configure redundant links for the new links with high failure frequency.

IV. IMPLICATIONS OF OBSERVATIONS

We discuss the implications of network man-agement and configuration provisioning according to our observations.

4.1 Network manageme nt

As stated in Table III, link failures are the main reasons for network failures. The failures are either caused by planned operations or by unexpected behaviors. Link failures increase the complexity of the management of a largescale network. Some backbone links may carry the traffic for important or real-time services, such as financial transactions, video conferences and voice-over-IP, etc. In order to guarantee the reliability of data transmission,operators can configure redundant links for all the links. However, this approach is unwise and not encouraged. As depicted in Fig. 3, we can find the links with high rate of failures.As an alternative, operators can configure the redundant links for these links to achieve the goal of fault tolerance. In addition, routing failure also trigger many network failures,which will bring in significant influence to the data transmission. For example, network congestion can be caused by routing failures.Operators should monitor the status of each link in real time. Once a link is congested,redundant link configured in advance will offload the traffic from the overloaded link automatically. In summary, pre-configurations are necessary to guarantee the reliable data transmission.

4.2 Configuration provisioning

As shown in Fig. 1 and Fig. 2, we find that tasks usually configured by the operators are rate limiting and adding customer. Operations of adding customer involve about eight configuration command lines, while operations of rate limiting only need to modify one command line. According to the types of configuration tasks gathered from analyzing the configuration changes, and combining with correlating each task to a certain set of command lines in the configuration files, our goal of providing semi-automated configuration for an operational backbone network is close to the reality. We take the tasks of rate limiting and adding customer as examples to show how to create the configuration template.

Fig. 3 Detailed analysis of the link failures over the seven years

1) Rate limiting

Given a customer (C1) who wants to increase the access bandwidth, we illustrate the processes oftemplate creation under the environment of Cisco IOS Release 12.0(32)SY16.

Step-1: According to the customer name,i.e.,C1, we extract corresponding configurations ofrate limitingofC1from the configuration file. Related Configurations ofrate limitingofC1is shown in Fig. 4(a). In our studied network, operators name theclass-mapforrate limitingin a consistent way. For example,they uselimit-C1to denoteclass-mapofC1.

Step-2: As shown in Fig. 4(a), 300000000 represents that the access bandwidth ofC1is 300Mbit/s. The number of 9375000 is calculated by 300000000 dividing 32. Now,C1wants to increase the access bandwidth to 800Mbit/s.We should modify the underlined values in the line 7 of Fig. 4(a). The bandwidth is needed to be adjusted to 800000000. Correspondingly,the following value is also calculated by dividing 32, i.e., 800000000/32=25000000. Based on the above analysis, we create the configuration template forC1to increase the bandwidth.The configuration template is shown in Fig.4(b).

Step-3: In order to reduce the workload and the possibility of misconfigurations, we transform the configuration template into an automated script, which can connect to the corresponding router and execute the configuration commands automatically. The automated execution benefits from the functionality ofexpect[18]. The script is shown in Fig. 4(c).Note that all the characters underlined can be regarded as variables, the values of which can be got from a text file through ashellprogram.

2) Adding customer

Given a customer (C2) who wants to access the studied network, we also illustrate the processes oftemplate creation under the environment of Cisco IOS Release 12.0(32)SY16.We assume thatC2connects to the interface(GigabitEthernet 0/1) ofR1.

Step-1: This step is mainly about how to configure thestatic route. We first choose an address block forC2. Assume that the address block isa1.a2.a3.a4with the mask number ofb1.b2.b3.b4.With the access permission ofR1,we check whether the interface of GigabitEthernet 0/1 is up or not. The query command isshow interfaces gigabitethernet 0/1. The query results can illustrate the state of the interface. Here, we assume the interface of GigabitEthernet 0/1 is not enabled. We should enable the interface first. The address of the interface isc1.c2.c3.c4and the mask number is255.255.255.252. The address of the corresponding interface of GigabitEthernet 0/1 isc1.c2.c3.c5. Note thatc1.c2.c3.c4andc1.c2.c3.c5are adjacent addresses in the same subnet. We then create astatic routeforC2. The command isip route a1.a2.a3.a4 b1.b2.b3.b4 c1.c2.c3.c5. Thestatic routecan tell the packets coming from other interfaces ofR1how to go to C2.

Step-2: Assume that the access bandwidth forC2is 500Mbit/s, we then show the processes of configuringrate limiting. We check theaccess list(s)that have been defined inR1.The query command isshow ip access-list.According to the query results, we can get theaccess-list numberthat can be used. For example, if the maximumaccess-list numberis 2043, we can select theaccess-list numberof 2044 forC2. Commands of configuringrate limitingis similar to the commands shown in Fig. 4(a). We should calculate the underlined values in the line 7 of Fig. 4(a). The values forC2are 500000000, 15625000, 15625000 in sequence.

Step-3: Based on the above analysis, we create the configuration template forC2to access the studied network with the bandwidth of 500Mbit/s. The configuration template is shown in Fig. 4(d). We define some basic attributes of the interface from line 2 to line 6.Thestatic routeforC2is configured in line 8.We configurerate limitingforC2from line 9 to line 18. We also transform the configuration template into an automated script with the syntax ofexpect. The feature of the script is similar to the one shown in Fig. 4(c), so we do not show it any more.

Fig. 4 Example of configuration template creation provisioning

Configuration templates depicted in Fig.4(b) and Fig. 4(d) are commonly used. If we want to apply these templates to other customers, we just need to determine and replace all the values of the underlined variables. With the configuration templates constructed manually, we can create corresponding scripts,which can provide automated configuration.

VI. CONCLUSION AND FUTURE REMARKS

In this paper, we captured the configuration evolution of an IP backbone network in a high-level analysis way, i.e., correlating each high-level task recorded in the weekly report to specific command lines executed in the configuration files. Results revealed that rate limiting and adding customers are configured most frequently. In addition, we conducted an analysis of network failures. We found that link failures are the main reasons for network failures. We suggested that we should configure redundant links for the links with high rate of failures, which would enhance the reliability of data transmission. More importantly,we showed how to provide semi-automated configuration by the cases of rate limiting and adding customers.

We note that network failures are more complex than general configuration tasks to deal with. Further investigation of network failures, especially the relationships between configurations and failures, still needs to be conducted.

ACKNOWLEDGEMENTS

This work is supported by the National Natural Science Foundation of China under Grant Nos. 61602105 and 61572123; China Postdoctoral Science Foundation under Grant Nos.2016M601323; the Fundamental Research Funds for the Central Universities Project under Grant No. N150403007; CERNET Innovation Project under Grant No. NGII20160126.

[1] R. Mahajan, D. Wetherall, and T. Anderson, “Understanding BGP misconfiguration”,ACM SIGCOMM Computer Communication Review, vol.32, no. 4, pp. 3-16, 2002.

[2] Z. Kerravala, “Configuration management delivers business resiliency”,The Yankee Group,2002.

[3] A. Markopoulou, G. Iannaccone, S. Bhattacharyya, C. N. Chuah, C. Diot, “Characterization of Failures in an IP Backbone”, in proceeding ofTwenty-third AnnualJoint Conference of the IEEE Computer and Communications Societies(INFOCOM), 2004, pp. 2307-2317.

[4] D. Oppenheimer, A. Ganapathi, D. Patterson,“Why do Internet services fail, and what can be done about it”, InUSENIX Symposium on Internet Technologies and Systems(USITS), 2003, pp.1-15.

[5] H. Kim, T. Benson, A. Akella, and N. Feamster,“The evolution of network configuration: a tale of two campuses”, InInternet Measurement Conference(IMC), 2011, pp. 499-514.

[6] Y. Sung, S. Rao, S. Sen, and S. Leggett, “Extracting Network-Wide Correlated Changes from Longitudinal Configuration Data”, InInternational Conference on Passive and Active Network Measurement(PAM), 2009, pp. 111-121.

[7] N. Feamster and H. Balakrishnan, “Detecting BGP Configuration Faults with Static Analysis”,InProceedings of the 2Nd Conference on Symposium on Networked Systems Design & Implementation(NSDI), 2005, pp. 43-56.

[8] F. Le, S. Lee, T. Wong, H. Kim, and D. Newcomb,“Minerals: using data mining to detect router misconfigurations”, InProceedings of the 2006 SIGCOMM workshop on Mining network data,2006, pp.293–298.

[9] G. Xie, J. Zhan, D. Maltz, H. Zhang, A. Greenberg,G. Hjalmtysson. and J. Rexford, “On static reachability analysis of IP networks”, InProceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies(INFOCOM), 2005, pp.2170-2183,.

[10] T. Benson, A. Akella, and D. Maltz, “Unraveling Complexity in Network Management”, InProceedings of the 6th USENIX Symposium on Networked Systems Design & Implementation(NSDI), 2009, pp. 335-348.

[11] T. Benson, A. Akella, and A. Shaikh, “Demystifying configuration challenges and trade-offs in network-based ISP services”,ACM SIGCOMM Computer Communication Review, vol. 41, no. 4,pp. 302-313, 2011.

[12] D. Caldwell, A. Gilbert, J. Gottlieb, A. Greenberg,G. Hjalmtysson, and J. Rexford, “The cutting edge of IP router configuration”,ACM SIGCOMM Computer Communication Review, vol.34, no. 1, pp. 21-26, 2003.

[13] J. Gottlieb, A. Greenberg, J. Rexford, and J.Wang, “Automated Provisioning of BGP Customers”,IEEE Network, vol. 17, no. 6, pp. 44-55,2003.

[14] W. Enck, T. Moyer, P. McDaniel, S. Sen, P. Sebos,S. Spoerel, A. Greenberg, Y. Sung, S. Rao, and W.Aiello, “Configuration management at massive scale: system design and experience”,IEEE Journal on Selected Areas in Communications,vol.27, no. 3, pp.323–335, 2009.

[15] X. Chen, Y. Mao, ZM. Mao, and J. Van der Merwe, “Declarative configuration management for complex and dynamic networks”, InProceedings of the 6th International Conference on Emerging networking experiments and technologies(CoN-ext), 2010, pp. 6.

[16] X. Chen, Y. Mao, ZM. Mao, J. Van der Merwe,“DECOR: DEClarative network management and OpeRation”,ACM SIGCOMM Computer Communication Review, vol. 40, no. 1, pp. 61-66, 2010.

[17] CERNET - China Education and Research Network. http://www.edu.cn/english/.

[18] Expect. http://expect.sourceforge.net/.