The Good, The Bad, and the League: 6/13 - 6/26

*Preamble* _Your semi-weekly dose of server problem-os, NA League news, and other!_ We’re moving GBL to a twice monthly cadence, which should be more sustainable with our team’s workload. 9.13 rolled out, Qiyana will be launched shortly, Arcade has a few new faces, and TFT rolls out worldwide. Mordekaiser’s rework is pretty bug free, two factor auth is coming, and summer is here! **Follow-Ups:** * **Mac voice still not working properly** Bug assigned to a specialist. (Core bug is still being worked on). **Ongoing** **Server Stuff: ** * **Automated Comp Mode Errors (6/17, ~11 minutes)** Automated monitoring flags an alert that the automatic comp mode is failing to run, or running into errors. Before an investigation can start, the alert deactivated, as connectivity to the system self restored. * **League websites not available (6/19, ~39 minutes)** Landing page stops loading properly. NOC escalates to appropriate teams, and one team reverts a large policy change that rolled out. That, along with purging the web caches, resets the websites to default working order. * **Watch.lolesports.com stuck (6/20, ~34 minutes)** NOC is notified that there’s an issue with the esports website. Investigation shows console errors, and a recent deploy is suspect. The deploy is rolled back, and the website starts to work again. * **Merch Store Timeout (6/21, ~152 minutes)** Rioters report to the NOC that the Merch store is having issues. NOC investigates, finding the Merch store website is having issues on both internal networks and external viewing. Root cause was a problem associated with AWS, and rebooting the associated AWS instances fixed the issue. * **DNS issues affecting services (6/21, ~33 minutes)** NOC notices that players are unable to log into PBE (Thanks TFT). The NOC throttles down the login rate, getting logins to a healthy state and players are able to log in steadily and without issue again. * **Multiple services having issues (6/24, ~100 minutes)** NOC sees issues with 3rd party software and services. Continued investigation shows issues spreading to other services and websites. NOC can’t fix the 3rd party problem, so when the third party service fixes their problem, the associated services start recovering. **Game Stuff: ** * **Packet Loss Like Symptoms (6/14, ~50 minutes)** Packet loss like symptoms are noticed on a specific game server. After briefly checking logs, NOC drains games from the game server, and performs a full cold reboot. After restart, diagnostics show no problems, so the game server is added back into the pool to be used. * **Avast update cause reconnect issues (6/14, ~50 minutes)** NOC notices and is notified that players are running into reconnect issues. Investigation leads to AVG and Avast having problems. Avast rolls out an official update that should fix the problem. Players updating Avast and AVG should see their reconnect issue fixed. **Player Support Issues:** Morde arrived, Qiyana is incoming, TFT has arrived as well! Hopefully none of them break the servers too much. High queue volume on TFT was a thing, and patching issues may be slightly higher than normal because we rolled out 9.13!
Share
Report as:
Offensive Spam Harassment Incorrect Board
Cancel