+31 31 0
Published 6 years ago by thesavagemonk with 11 Comments

Life as a community game server operator

Whether it's a banned player or just someone with a bad attitude, DDoSing is a common fact of life for game server operators.


Join the Discussion

  • Auto Tier
  • All
  • 1
  • 2
  • 3
Post Comment
  • potateHoe

    Speaking as someone who isn't very knowledgeable about networking, how does one exactly mitigate a DDOS? Do you just tell them to shoo?

    • retrowave

      With great difficulty. I run a small website that got DDoS'd a little while ago. Not only did they take down my site, but everyone else on my provider's network.

      Being such a small provider with no capacity to handle such a large DDoS, their solution was to kick me off their network entirely. Effectively they gave me 6 hours to backup my stuff and hit the road.

      For mitigating websites, services such as CloudFlare work with limited success. They have their servers sit "in front" of your own, sort of acting like a proxy or gateway. Traffic is inspected and only legitmate users are let through.

      Higher capacity networks, such as those owned by ISPs, can employ more complex methods of distinguishing 'real' from 'fake' traffic, and passing that through. However, a DDoSer can very quickly change their attack method to adapt, so it becomes a really annoying game of cat and mouse.

      Honestly, the best method is to just add more capacity. Throw in more servers to adapt. Load balance the end-point that's getting hit. Re-route to higher capacity networks... The list goes on and is very environment-specific. Adding capacity requires time and money and is only really vialbe to companies that can afford it.

      For thesavagemonk's game servers above – it looks to be an automated system set up by the hosting provider. Probably a combination of the methods I described above.

      • thesavagemonk

        We actually switched server hosts due in part to issues like those you describe. On our old host, their "solution" to DDoSing was to "null-route" our IPs, meaning any traffic directed to our game servers, website, VOIP server, etc. would be directed to nothing for 8 hours. As you can imagine, being DDoS'd was absolutely devastating to us.

        We pay a relatively high amount of money for the dedicated servers we use now, but it's absolutely worth it. Many of the events listed in the picture above weren't even noticeable to us. For those that were, it was usually no more than a minute or two of slowness before the automated system fixed it. I had my friend go into detail in this comment about how exactly they handle DDoS mitigation.

        • retrowave

          That's awesome! It's a really good feeling when you find a reliable ISP. Not having to deal with these issues is why you pay them for the service.

          A null-route is not a goddamn solution grumble grumble. Excellent info from your friend. I definitely learnt something today!

      • potateHoe

        Thank you! That sounds really hectic to have to deal with because of the cat and mousing. It also makes it more remarkable when you hear about Blizzard's or Microsoft's servers getting DDOSed to the point of failing. Pretty neat stuff :)

    • thesavagemonk (edited 6 years ago)

      I'm going to copy and paste something my friend wrote about it (his wife does this kinda stuff for a living) since I think he can explain it better than I can:

      Methods differ from place to place, and I don't know of any provider that will tell you how their automatic mitigation systems work (for pretty obvious reasons). As far as I know though, PhoenixNAP (our host) has specific DDoS mitigation hardware in place in at least one level of their infrastructure, and all routers also have their own mitigation rules. They detect sudden large bursts in traffic, analyze it against various rules to see if it's suspicious (ex: 200 new connections all spamming similar abnormal requests), and then drop the packets if they are. The alerts in the picture are all automatic and just serve as a notice to us.

      PNAP in particular has DDoS protection at four different levels.

      1) Their ISPs (multiple 10 Gbit networks, such as Cogent, Level 3, Inteliquent, etc.). I imagine the rules at the ISP level are less strict, but stop most common attacks before they even hit the datacenter. They only utilize each ISP to 50% of their capacity, so if an attack is actually getting through, they can handle an extra ~5Gbit of traffic per ISP without anyone feeling a thing.

      2) The edge routers. These are where all connections to the datacenter come in. The rules at this level can be set by PNAP staff, and can be slightly more strict than the ISP rules. The rules are a closely guarded secret, and anything from this level and beyond at the datacenters require an armed guard escort , access cards, and retinal scans.

      3) Aggregate policers. These are rules that are set to each protocol, such as burst limits, bandwidth, priority, etc. At this point, the amount of traffic alone can trigger a mitigation. Ex: Our server typically sees 20-60mbit of consistent traffic. If multiple people are downloading maps it could get as high as 200-300mbit (for /very/ brief periods). But most the attacks result in 1-4gbit's of traffic, making them clearly outliers and easy to filter out. Think of this level as the "trend watching" layer.

      4) Top of rack. Every server rack has its own mitigation in place as well. I imagine these rules can be quite a bit more strict since fewer servers at a time can be affected by their rules. I honestly have no idea what goes on at this level though.

      And on top of all that, we have our own custom firewall rules (literally pages of them made by hand for our specific needs). If the traffic isn't so high or suspicious that it is actually getting routed to our server, for the most part our rules will either rate limit, or completely block traffic by IP based off suspicious activity. At one point we had someone open a support ticket from a proxy asking why he couldn't access our website directly after port scanning us. That's an example of one of our custom rules working its magic and automatically blocking suspicious activity.

      • potateHoe

        And all this happens within a matter of seconds? Geez, that's pretty sweet. It kinda makes me want to go and take extra classes of networking.

    • ThermalShock

      I'm a bit behind the times on server hosting. But most server hosts have DDOS mitigation built into their services. They don't have much choice these days since any script kiddie can load up LOIC. Or after some googling figure out how to set up a botnet on their schools computers.

  • MrY

    Damn, October must've been an intense month

  • ducky (edited 6 years ago)

    I feel like I've been fairly lucky so far. I only run a single smaller TF2 server which occasionally gets some really rude players joining, but I've yet to actually anger someone enough for them to consider retaliating against me in the form of a DDOS.

    Then again, I never really reveal myself as the admin/owner since I hate it when people start treating you differently from other players just because you're an admin. So as far as most players there knows I'm just one of them and I prefer to keep it that way for my sanity and for theirs.

  • HatTheTurtle

    Jesus, that looks like a nightmare to handle. If you don't mind me asking, how do you keep active users? I've been part of a community that used to have 3 servers, but due to lack of players, we've dropped down to 1, and it hasn't seen regular use for probably over a year.

Here are some other snaps you may like...