“Mirantis delivered excellent customized solutions, training, and support so we can be successful in a high load gaming environment”
The online multiplayer gaming market is big business! Analysts estimated 2015 worldwide revenue of $65 billion and predict 12 percent annual growth through 2019 from diverse geographies and demographics. Movie theaters, in comparison, drew only $38 billion in 2015 worldwide revenue.
Many of the largest online multiplayer games, such as Wargaming’s World of Tanks, run on infrastructure pioneered by G-Core. G-Core has designed this gaming IT infrastructure to meet the most demanding player expectations around the world. Its 11 locations across Russia and the Commonwealth of Independent States, each with 40 Gbps of bandwidth, host 100 million players per week and ensure industry-leading latency, availability, and cost metrics. This allows Wargaming and other valued customers to focus on game development instead of infrastructure issues.
With 4000 employees and 150 million users, Wargaming leads the market with award-winning titles and innovative technology. Free-to-play games and continuous feature improvements have propelled its growth since 2011.
Every minute of every day, millions of players from different countries connect to Wargaming titles on G-Core’s infrastructure to face others in battle – to interact, have fun, and win. This deep collaboration drives the two companies’ passion and pursuit of excellence.
Like any hosting company facing tremendous growth, G-Core needed to reevaluate its IT process and infrastructure. The fast paced gaming market required clients to continuously improve games and quickly launch new features, and G-Core needed to make sure they were in a position to do that.
Industry leading customer, Wargaming, had led the market over the past five years, adding successful titles such as World of Warships, and extending World of Tanks access to consoles and mobile devices. Yet, sustaining this innovation required improved infrastructure and increased commitment to Agile IT.
G-Core’s primary goal was to provide this capability to Wargaming, but it struggled to cost effectively maintain infrastructure performance and scale. Its game hosting spanned thousands of servers in 20 data centers and had even set a world record of 1,140,000 peak concurrent users. As platform expenses rose, however, G-Core needed improved resource utilization and cost margins.
Complicating matters, to shorten release cycles, gaming developers required self-service access to resources. Provisioning VM and bare metal servers, however, often involved release managers opening multiple tickets that took many days and administrators to resolve. The activity was complex, labor intensive, and subject to human error. To help make Wargaming successful, G-Core had to help simplify the process of maintaining continuous integration and deployment of high quality games.
To lower IT capital and operating expenses, G-Core needed to decrease virtualization system costs and improve infrastructure utilization. Server virtualization rates were below 50 percent and most workloads ran on bare metal. Meanwhile, customers’ successful games and free-to-play trends were driving massive online player growth. G-Core simply couldn’t afford the virtualization software fees, capacity utilization, and strain on administrators.
“Every time our client had a new project, they had to forecast the expected load and order new equipment,” says Andre Reitenbach, General Manager, G-Core. “If the forecast was not correct, the servers were underutilized, sometimes at just 30 percent. The gaming business has its own specific utilization requirements and being able to provide an agile capacity is important.”
To optimize costs and meet fluctuating game player demand across titles, regions, and time intervals without impacting performance or scale, G-Core needed to increase platform flexibility. Some games start slow and later explode with demand while others begin strong and unexpectedly even out. This meant simplifying and standardizing complex servers, storage, and networking designs to more easily monitor and scale resources.
Clients such as Wargaming fostered the DevOps culture and were comfortable with open source software through the use of CentOS Linux, Puppet, and Fabric. But few had embraced the private cloud. However, with OpenStack’s platform gaining adoption by large finance and entertainment firms with high load and streaming services, G-core was ready to evaluate cloud architectures for Wargaming’s needs.
G-Core and Wargaming’s game development and IT operations teams could likely benefit from OpenStack’s flexibility, cost, and time-to-market gains. But they couldn’t afford any impact on performance or scale.
In mid-2015, G-Core teamed with Wargaming to conduct an evaluation of cloud platforms. A move to private cloud infrastructure-as-a-service (IaaS) would mean a major transformation off existing virtualization systems and bare metal infrastructure. Thus, the companies’ engineers approached the idea carefully.
Wargaming development teams are divided into two groups: those focused on gaming engines and those focused on non-gaming workloads such as e-commerce, account management, and discussion forums. Furthermore, all work streams are split into four stages – development, load testing, beta testing, and production, the last two being customer facing. This matrixed framework would allow for a phased cloud transition.
“After multiple Wargaming development and G-Core operations teams conducted an independent evaluation of cloud solutions and vendors, we unanimously agreed that OpenStack’s capabilities and flexibility met our requirements,” says Reitenbach. The companies also evaluated other vendors’ cloud solutions but the offerings did not meet the needs.
Using DevOps methodologies, Wargaming developers had already grouped applications that ran together for specific projects into OpenVZ containers that were deployed using YAML-based files. This meant G-Core’s transition to OpenStack required custom integrations into this existing framework.
“While OpenStack satisfied Wargaming and other clients’ general requirements, we needed a partner that would help build and customize the solution for our specific needs,” says Reitenbach.
G-Core and Wargaming first turned to Mirantis because of its status as the top contributor to the OpenStack community. Upon meeting the team, however, the gaming partners were quickly impressed by Mirantis’ high degree of experience and professionalism in custom design and development.
“Mirantis attracted our attention from their community participation. More importantly though, they listened to us and thoroughly explained each option,” says Reitenbach. The teams also valued Mirantis’ local and global presence.
The initial transition to the cloud prudently began with two internal facing work stream stages. The conversion was also limited to a single game, and only for non-gaming workloads. This allowed G-Core, Wargaming, and Mirantis to proceed with lower risk.
G-Core hired Mirantis to conduct a three-day workshop and create an Architectural Design Assessment document that specified the requirements for a minimum viable product (MVP) of the private cloud. G-Core and Wargaming also selected Mirantis to build Fuel plug-in custom integrations with Active Directory for authentication, NFS for VM data storage, and Puppet master for a CMDB solution.
Mirantis also wrote a plug-in to enable multiple IP addresses, and thus multiple containers, per VM. This allowed G-Core and Wargaming to benefit from state-of-the-art container technology.
Thereafter, G-Core, Wargaming, and Mirantis built the first two clouds based on the Mirantis OpenStack reference architecture and the four custom Fuel plug-ins. The installation includes core OpenStack services including Cinder, Glance, Keystone, Neutron, and Nova. Additionally, Murano is used for provisioning of virtual machines.
After a few requested adjustments, the teams deployed two additional clouds in just 10 days for the customer facing work streams – beta testing and production.
Next, Mirantis conducted extensive training on OpenStack and Fuel, which helped G-Core and Wargaming gain strong OpenStack platform management skills. As a result, the two companies were ready to transition the first workloads to the new infrastructure.
“Agile IT benefits also include reduced software deployment cycles and increased innovation where new features can be quickly released, tested, and rolled back, if necessary, before reaching production. The complete shift to the cloud is expected to free up to 50 percent of our server capacity, which will allow us to reduce clients’ costs or offer it for other needs”
In early 2016, G-Core transitioned two non-gaming workloads for Wargaming to two OpenStack-based IaaS clouds. The two companies are now in the process of doing the same for the gaming engine workloads. Furthermore, all Wargaming projects recently received access to development workstream clouds.
Wargaming’s deployment now stands at 110 nodes across four G-Core data centers and is already yielding strong benefits. Provisioning of IT resources is simplified and automated. “Developers can now get VMs within one hour,” says Reitenbach.
The companies’ developer efficiency and creativity have also increased as engineers not only spend less time submitting tickets and waiting for resources, but they also beneft from new test and deployment tools. Now, more time and effort can be focused on innovation and adapting to new market needs.
“Agile IT benefits also include reduced software deployment cycles and increased innovation where new features can be quickly released, tested, and rolled back, if necessary, before reaching production,” said Andre Reitenbach. “The complete shift to the cloud is expected to free up to 50 percent of our server capacity, which will allow us to reduce clients’ costs or offer it for other needs”.
Furthermore, expensive virtualization system licenses are no longer needed as new cloud servers leverage KVM and Ubuntu. And within the IaaS OpenStack deployment, server utilization is significantly higher and capital expenditures have declined as expected.
Operating expenses have also decreased as labor consuming infrastructure administration has been replaced by automated processes, many facilitated by the deployment repeatability from custom Fuel plug-ins.
G-Core and Wargaming’s next goal is the mass migration of all games and work streams to the OpenStack cloud. This means a huge transition off virtualization systems and bare metal servers to a highly scalable and available Mirantis OpenStack solution that can handle the high load of award-winning games.
In addition to the mass migration, G-Core and Wargaming plan to add the following OpenStack-based services: billing and chargebacks, database, DNS, Hadoop, load balancing, and platform-as-a-service. And G-Core plans to add OpenStack solutions to its broader suite of managed services including hosting, CDN, peer networking, and DDoS protection.
Finally, G-Core and Wargaming have been very pleased with Mirantis services to-date and have recently contracted the OpenStack leader to provide 24x7 Enterprise Support.
“Mirantis is so much more than just OpenStack distribution,” says Reitenbach. “Mirantis delivered excellent customized solutions, training, and support so we can be successful in a high load gaming environment.”