High-quality BGP peering is one of the key factors of the excellent connectivity of our global network.
Currently, we are working with over 6,000 partners. Thanks to this, since 2020, we are among the top 10 largest networks in the world by number of direct peer-to-peer connections. This is the result of the coordinated actions of our engineering teams to develop our global network. Today, the G-Core Labs network covers all continents except Antarctica.
BGP peering is a direct exchange of internet traffic between autonomous systems (AS), bypassing higher communication operators (uplinks).
It helps the participants of an internet exchange point (IX) to shorten the data transfer routes between networks, reduce the cost of traffic, and, in most cases, reduce the round-trip time (RTT). It is easier to connect to several participants via an IX—instead of a separate connection to each one, one connection to the IX, where all participants are present, is enough.
Each IX participant benefits from saving on channels and purchasing traffic from uplinks. It is also beneficial for the end user, as network latency and RTT are reduced. Thus, the more IXes we have, the better the internet connectivity, and the more available it is.
The first IXes began to appear in 1994 in large European cities, such as London (LINX), Frankfurt (DE-CIX), Amsterdam (AMS-IX), and Moscow (MSK-IX). Currently, there are more than 500 IXes around the world.
We select data centers by the number of Tier I and Tier II operators and local IXes.
Public peering: We choose the leading internet providers in each region and link up with them in the largest IXes.
Private peering: We have more than 6,000 peering partners around the world. This allows us to keep the traffic locality in different regions, reducing content delivery delays between networks.
BGP peering is not just our presence in a particular IX. It requires an integrated approach and a coordinated effort by all teams.
Being an IX participant is not enough, since many eyeball operators have a closed peering policy (for example, Selective or Restrictive). This means that the operator doesn’t share their networks on the route server, a network service that simplifies peer-to-peer interaction between IX participants and reduces the number of individually managed peer-to-peer sessions.
The establishment of peering with such operators is the result of long negotiations between peering managers, as a result of which the BGP peering itself is created and the traffic exchange takes place directly.
Automation of the configuration process also plays a significant role here. When the number of IXes exceeds 15 and the total number of peers exceeds 1,000, it is no longer possible to manage it manually (i.e., with the help of engineers). That is why we have developed a complete automated cycle for configuring and managing BGP sessions without human intervention. The only thing left for the network engineer to do is to approve the peering, and the automation does the rest.
The approaches to peering automation from the Big Four global operators—Google, Microsoft, AWS, and CloudFlare—simply did not suit us.
Google and Microsoft use the Web Peering Portal, which is not easy to access. And, in the case of Microsoft, you need to have a minimum subscription to Azure.
Once you get access, you need to fill in your information (as an operator) and go through verification—only then you can create peering requests. For each request, a separate task is usually created where it is also required to fill in information and enter (select) parameters for the BGP session.
Typically, it can take up to 10 minutes to create a single request. Now, let’s imagine that we need to make 200 such requests to establish all the sessions we need all over the world, and this is only for one operator. You can easily calculate how much time you’ll need for this task. And that doesn’t include the time needed to respond to the emails that may contain additional questions. It’s much easier to send one email and receive a reply with the pre-set configuration within 24 hours.
Automation: We have developed our own automation system from scratch and are constantly improving it. After all, it is important not only to configure the BGP session but also to support further operations. For example, according to our statistics, we establish over 40 sessions and delete about 20 sessions every day. Within a month, that’s about 1,200 new sessions and between 600 and 800 deleted.
The human factor: It was impossible to manage the configuration manually, as one of the main problems would be the human factor. No matter how good a network engineer is, on such a global scale, anyone can make mistakes that can lead to service downtime.
All you need to do is send a request to email@example.com with “Peering” in the subject and an autonomous system number (ASN) different from ours, which is 199524.
Often, our potential peering partners submit peering requests themselves. Next, these requests are sent to the orchestrator, and the system analyzes their feasibility. If the request meets our criteria, the system automatically configures the corresponding sessions and sends the configuration to the participant.
We prefer to set up peering in all IXes where we have matches with the operator that requested the peering. This approach saves time for our peering partner, allowing us to achieve maximum efficiency. As the network grew, this approach proved to be the most convenient for both us and our partners.
There are also situations when we make a peering request with a potential partner (usually a large operator).
We analyze data that is collected from our edge routers via NetFlow/JFlow. With this data, we know everything about our traffic down to the last byte.
We use telemetry from our services like CDN and other products. The main metrics that we take into account from the network are packet loss and RTT to the end user.
We are already present in 102 IXes around the world.
This became possible thanks to our powerful network infrastructure, without which peering would be impossible. After all, to enable an IX, you need a local presence, and this already implies the presence of infrastructure, network equipment, and data transmission channels.
Latin America, Africa, and Asia are now a priority. This is a potentially very large and promising market that we’re working on and will continue to work on.
G-Core Labs is a fast-growing content operator with an extensive peering network and open peering policy. We invite all interested participants to peer with us for free via IXes.