Bogus IP Header Length Packet Sniff

[Latex]

I have been having issues since this last patch. Failing to zone, timing out on login, once logged in server list isn't populated, etc. It's incredibly frustrating. P99 has run smoothly for the past year until this last patch for me.

[Ruien]

I had the same issue with TT, after about 30s in zone ping spiked up to 4k then eventually d/ced me. I was however able to log back in eventually and make my way to the seb orb and zone in.

Yes, but with bad luck you can die on the way. You don't even know you're dead until you log back in and behold your own nakedness at the character selection screen.

Something about TT is really broken, and I have no idea what it is [You must be logged in to view images. Log in or Register.]

That makes sense on the MTR analysis. I know I'm seeing packet loss, I'm not questioning that. I just think the server is severing the connection too quickly, and the client is missing the message that the connection was severed so it just keeps trying. What's especially frustrating is circumstances where the character client never loads into a zone (just sits at the previous zone screen with the busy cursor), however when you manually close the client and attempt to log back in, it says you still have a character in zone. So what's happening there? Why isn't the client being sent messages telling it that it zoned?

UDP is a connectionless protocol. When EQ was originally designed, UDP was the optimal choice, because it follows the logic that if you've missed a packet, you don't really care exactly where a PC or mob was, you just want the latest data as to where it is now. EQ was fairly aggressive in terms of what it was trying to do over limited bandwidth and the UDP model is particularly well-suited for the old days of dial-up connections. Specifically, it works well for limited bandwidth (recall that 6kB/sec was normal back then), and most of the time you didn't have much packet loss. Packet loss kills UDP. TCP at least knows when it missed something.

So, to answer your question, those messages are sent, and when a packet is sent, the sender implicitly assumes that it will be received. When the packet is lost, neither side knows what's going on. The game does do some work to try to survive packet loss but it's pretty inefficient to do so. To get a feel for how much of an issue this is, suppose that zoning requires the successful receipt and transmission of 10 packets, otherwise you freeze, and you're seeing 8% packet loss. You have a 57% chance of freezing on zone and crashing out with the server still wondering what's going on (saying you have a character in game). If it requires 20 packets, then that rises to an 81% chance. (57% = 1-0.92^10, 81% = 1-0.92^20)

So, as a result, you really need less than 1% packet loss to play EQ. TCP-based games will still perform sanely (if only more slowly), with higher packet loss.

It would be far more inefficient to try to work around this problem by implementing echo-check validation than it would be to re-implement the game in TCP. So, the underlying structure of the game necessitates the type of maddening problems that packet loss brings. I think it sucks, too. I also live in Asia and would much prefer TCP, but that's never going to happen with EQ and certainly not with p99 (which is locked to the Titanium Client even if by some miracle Live was migrated to TCP).

Probably the only way to pinpoint the issue is to setup a local server, and monitor the connection to that local server to find out what it's supposed to look like, and use something like burpsuite to kill certain packets and find that one that is causing the problems. (Not even sure it would be capable of performing quickly enough to allow a game session to occur though, just theorizing)

I doubt there are any packets that are causing problems. My guess is that the problem is caused by packets you're not receiving, but should have, and the infrastructure of EQ doesn't recover from every packet-loss situation gracefully. It can also result from packets that you send but the server never receives, and not even Wireshark can tell you which packets those are.

The only reasonable approach is to attack the packet loss, so I think the real way to solve your problem is to use a VPN from a server that both (1) you have a good connection to, and (2) has a good connection to p99.

So, go rent some linux VPSes (i'd suggest ubuntu, and note that most hosts have free trial periods specifically for free testing like this). Log in and ping the p99 servers from each. Find one or two that are good (that is, you have a solid connection to that server and it has a solid connection to p99), and install OpenVPN (ubuntu makes this easy). Now, use your normal VPN software that you've been using anyway but instead connect to the OpenVPN instance that you set up instead of renting VPN access from some large VPN provider. By choosing the VPS server yourself, you can hand-pick a viable location that meets the criteria you need.

Ramnode offers VPSes at $2/month and I believe that the basic 256MB RAM server is generally good enough for OpenVPN.

Finally, this is somewhat speculation, because I personally don't use a VPN for anything at all. I actually use Shadowsocks for HTTP traffic and don't use a VPN for EQ, because my direct connection is good enough (as my MTR traffic report demonstrated). But this might be the right solution for you.

[Ruien]

Originally Posted by Latex [You must be logged in to view images. Log in or Register.]

I have been having issues since this last patch. Failing to zone, timing out on login, once logged in server list isn't populated, etc. It's incredibly frustrating. P99 has run smoothly for the past year until this last patch for me.

I'm pretty sure that the latest P99 patch is 100% orthogonal to the connection to the login server. The only possible way that you could see issues such as a more frequent blank server select screen would be something related to the network. You might want to try the tests demonstrated earlier in this thread to see if you, too, are seeing packet loss issues. Given all the new issues I've seen pop up on the forums recently, it's very possible that some important router somewhere took a dump, some ISP pairings changed, some international connection piplines became saturated, or something of that nature. The timing of the latest patch is probably an unfortunate coincidence but unrelated if you're seeing issues related to the login server.

The zoning code on p99 did change, resulting in differences in the way that the zoning sequence looks in Wireshark. In theory it should only help, but might have unexplored implications. So, I'm not ruling out the possibility of problems being caused by that change, only saying that if you're seeing an increase in the frequency of issues connecting to the login server (even before connecting to p99) then the evidence suggests the underlying issue lies elsewhere.

[mdavis522]

Since this patch thing, i have crashed so many times ive lost count. I never had crashed before. And its always while zoning. So thats my exp. Id love to look at this dbg.txt thing. Where do i find it?

[towbes]

Originally Posted by Ruien [You must be logged in to view images. Log in or Register.]

I'm pretty sure that the latest P99 patch is 100% orthogonal to the connection to the login server. The only possible way that you could see issues such as a more frequent blank server select screen would be something related to the network. You might want to try the tests demonstrated earlier in this thread to see if you, too, are seeing packet loss issues. Given all the new issues I've seen pop up on the forums recently, it's very possible that some important router somewhere took a dump, some ISP pairings changed, some international connection piplines became saturated, or something of that nature. The timing of the latest patch is probably an unfortunate coincidence but unrelated if you're seeing issues related to the login server.

The zoning code on p99 did change, resulting in differences in the way that the zoning sequence looks in Wireshark. In theory it should only help, but might have unexplored implications. So, I'm not ruling out the possibility of problems being caused by that change, only saying that if you're seeing an increase in the frequency of issues connecting to the login server (even before connecting to p99) then the evidence suggests the underlying issue lies elsewhere.

I'm not exactly sure what you mean by orthogonal, but when you watch wireshark as you login, you're sending packets to the same IP as the gameserver when you are logged into P1999. 66.55.145.2 The code is all related:

https://github.com/EQEmu/Server

[Ruien]

The IP of the gameserver vs loginserver (same in this case) isn't relevant, rather what parts of the code are used. Connecting to the loginserver to show the server selection screen shouldn't (I bravely assume!) use the zoning code.

From my understanding, according to the patch notes, it was the zoning code that was updated on p99, so if the problem includes an adverse effect on the loginserver, then it suggests the zoning code changes (that is, the latest p99 patch) aren't to blame.

I could be wrong and more changes were imported from github than just the zoning code, but I'm trusting that the patch notes provided by the devs are accurate in this respect. If indeed code was changed that substantially affects all of the areas that Latex mentioned, then it could be due to the patch and not just network problems.

Contrast this with the report by mdavis522, which could very understandably be a result of the changes to the zoning code.

In the past few days, the connection to this server has gotten progressively worse on my end. I'm having all of the problems you guys are reporting now, and I'm pretty sure it's due to network issues at least for me.

[Casey VII]

Went from a stable 36MS no crashing, to this patch where I can't even play the fucking game at all. Awesome.

[Stormfists]

Is there any "official" response to this thread? <3

[Caridry]

Originally Posted by Casey VII [You must be logged in to view images. Log in or Register.]

Went from a stable 36MS no crashing, to this patch where I can't even play the fucking game at all. Awesome.

My ping SIGNIFICANTLY jumped as well since last patch...I used to be consistent at about 18ms-22ms, now I'm 50-60 steady... Constant zone crashing, etc.[You must be logged in to view images. Log in or Register.]

[dafier]

6Kb a sec? LOL! You mean 2Kb? 3.5 or 4 was the norm on a 56k baud connection. 28.8k was around 2Kb. I am talking average btw....

I have seen connection issues of late as well, but not that bad.