Zoning Issues

Discussion in 'General' started by Akkadius, Sep 8, 2019.

  1. Akkadius

    Akkadius Administrator Staff Member

    Messages:
    221
    If you have been experiencing zone issues around not being able to login and the server saying something to the extent of "The Server is Not Responding" while attempting to zone, your data goes in this thread.

    We rebooted 9/7/2019 with some tweaks and things appeared to have cleared, but I'm not convinced we're completely in the clear

    Thanks everyone
     
  2. Popper

    Popper Orc Centurion

    Messages:
    60
    Issue repeated again Sun evening 11:30 est

    Zoned 51 from PoK into Shadowrest and they all crashed. Couldn't get back in, The server is not responding, then disconnected back to server select.
     
  3. Gaellespie

    Gaellespie Orc Centurion

    Messages:
    56
    Yes this is happening again. Zoning from MPG to Adaptation raid trial and cannot get back in. This seemed to start around 10:30 pm central.
     
  4. Gaellespie

    Gaellespie Orc Centurion

    Messages:
    56
    By about 10:45 was able to log in again
     
  5. Noktis

    Noktis Orc Pawn

    Messages:
    16
    I was doing great all day yesterday but today I zoned 30 crew pok - wos - mpg fine then foresight trial zone crashed on zone in cant log back in gets as far as character select then kicks me.
     
  6. Tigwoodelf

    Tigwoodelf Orc Pawn

    Messages:
    1
    Today had zoning issues going from Nexis to Nethebian Lair, Kissed can not be logged in and i Have to exit out of EQ altogether. this was at 9pm PST. Sep 8 2019.
     
  7. YirranPEQ

    YirranPEQ Orc Pawn

    Messages:
    39
    Had a toon parked in Akheva Ruins and had logged him in a couple times yesterday with no problem. This am when I tried to log him in he just hung at the char select till he timed out and would just return to the server select screen.

    I disabled autologin and hit the return home button. He hung there also and returned to server select again. So i selected the server and when he got back to the char select he now showed as being in Surefall Glade but still hung and returned to the server select screen.
    I logged him out, moved him to Nexus using peqtgc site, logged him in and ran him back to Akheva and he zoned in fine.
     
  8. clippy

    clippy A Griffin

    Messages:
    229
    Was playing well last evening (september 9th), but trying to log in now - all 36 toons goes to char select and then "you have been disconnected". All parked in rss at the moment.

    Some example toons that got stuck: Spela, Bandy, Stor, Trams, Blastar

    EDIT: 15 minutes later i logged in successfully. rss was empty except from my toons.
     
    Last edited: Sep 10, 2019
  9. Audite

    Audite Orc Pawn

    Messages:
    7
    I experinced the same both yesterday (9th and today 10th sept) - Guild Lobby, Guild Hall, Gulf of Gunthak

    Same descriptions as above, unable to get past character screen, crashing on zoning in
     
  10. Uleat

    Uleat Orc Warlord

    Messages:
    182
    Can you guys post your client's dbg.txt file contents for the login session that experiences this?

    One for 'Enter World' (character select-to-zone) and one for Zoning (zone-to-zone) activities.

    Just need one per user activity..not the entire army's :)
     
  11. Gaellespie

    Gaellespie Orc Centurion

    Messages:
    56
    Code:
    [Wed Sep 11 22:09:27 2019]00415:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:27 2019]00416:Stopping world display.
    [Wed Sep 11 22:09:27 2019]00415:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:27 2019]00416:Stopping world display.
    [Wed Sep 11 22:09:27 2019]00417:Attempt to send message 18205 on a void connection.
    [Wed Sep 11 22:09:27 2019]00417:Attempt to send message 18205 on a void connection.
    [Wed Sep 11 22:09:27 2019]00418:Beginning EnterZone block.
    
    [Wed Sep 11 22:09:27 2019]00419:Calling WorldAuthenticate.  Countdown is 4
    
    [Wed Sep 11 22:09:27 2019]00420:Networking: using port [62348].
    [Wed Sep 11 22:09:27 2019]00421:Networking: Connection Established [1]
    [Wed Sep 11 22:09:27 2019]00422:WorldAuthenticate: Initiating Login.
    [Wed Sep 11 22:09:27 2019]00418:Beginning EnterZone block.
    
    [Wed Sep 11 22:09:27 2019]00419:Calling WorldAuthenticate.  Countdown is 4
    
    [Wed Sep 11 22:09:27 2019]00420:Networking: using port [62350].
    [Wed Sep 11 22:09:27 2019]00421:Networking: Connection Established [1]
    [Wed Sep 11 22:09:27 2019]00423:WorldAuthenticate.  I got a message of type 0x7ceb (31979).
    [Wed Sep 11 22:09:27 2019]00424:WorldRPServer message: server name PEQTGC
    
    [Wed Sep 11 22:09:27 2019]00425:WorldAuthenticate.  I got a message of type 0x7499 (29849).
    [Wed Sep 11 22:09:27 2019]00422:WorldAuthenticate: Initiating Login.
    [Wed Sep 11 22:09:27 2019]00426:WorldAuthenticate.  I got a message of type 0x578f (22415).
    [Wed Sep 11 22:09:27 2019]00427:WorldAuthenticate.  I got a message of type 0x6259 (25177).
    [Wed Sep 11 22:09:27 2019]00428:WorldAuthenticate.  Access granted.
    
    [Wed Sep 11 22:09:27 2019]00429:Check 1x. 0x47c6922f
    
    [Wed Sep 11 22:09:27 2019]00423:WorldAuthenticate.  I got a message of type 0x7ceb (31979).
    [Wed Sep 11 22:09:27 2019]00424:WorldRPServer message: server name PEQTGC
    
    [Wed Sep 11 22:09:27 2019]00425:WorldAuthenticate.  I got a message of type 0x7499 (29849).
    [Wed Sep 11 22:09:27 2019]00426:WorldAuthenticate.  I got a message of type 0x578f (22415).
    [Wed Sep 11 22:09:27 2019]00427:WorldAuthenticate.  I got a message of type 0x6259 (25177).
    [Wed Sep 11 22:09:27 2019]00428:WorldAuthenticate.  Access granted.
    
    [Wed Sep 11 22:09:27 2019]00429:Check 1x. 0x47c6922f
    
    [Wed Sep 11 22:09:28 2019]00430:Waiting to connect to zone... Character Lelu
    
    [Wed Sep 11 22:09:28 2019]00431:Zone Connect -- 0 -- Received MSG_ZONE_ADDRESS
    [Wed Sep 11 22:09:28 2019]00432:Zone addr [142.44.176.140:20509] received...
    [Wed Sep 11 22:09:28 2019]00414:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:28 2019]00415:Stopping world display.
    [Wed Sep 11 22:09:28 2019]00430:Waiting to connect to zone... Character Theostar
    
    [Wed Sep 11 22:09:28 2019]00433:ZONING
    [Wed Sep 11 22:09:28 2019]00434:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:28 2019]00435:Networking: using port [62354].
    [Wed Sep 11 22:09:28 2019]00436:Networking: Connection Established [1]
    [Wed Sep 11 22:09:28 2019]00416:Attempt to send message 18205 on a void connection.
    [Wed Sep 11 22:09:28 2019]00431:Zone Connect -- 0 -- Received MSG_ZONE_ADDRESS
    [Wed Sep 11 22:09:28 2019]00432:Zone addr [142.44.176.140:20509] received...
    [Wed Sep 11 22:09:28 2019]00417:Beginning EnterZone block.
    
    [Wed Sep 11 22:09:28 2019]00418:Calling WorldAuthenticate.  Countdown is 4
    
    [Wed Sep 11 22:09:28 2019]00419:Networking: using port [62355].
    [Wed Sep 11 22:09:28 2019]00420:Networking: Connection Established [1]
    [Wed Sep 11 22:09:29 2019]00421:WorldAuthenticate: Initiating Login.
    [Wed Sep 11 22:09:29 2019]00433:ZONING
    [Wed Sep 11 22:09:29 2019]00434:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:29 2019]00435:Networking: using port [62356].
    [Wed Sep 11 22:09:29 2019]00436:Networking: Connection Established [1]
    [Wed Sep 11 22:09:29 2019]00422:WorldAuthenticate.  I got a message of type 0x7ceb (31979).
    [Wed Sep 11 22:09:29 2019]00423:WorldRPServer message: server name PEQTGC
    
    [Wed Sep 11 22:09:29 2019]00424:WorldAuthenticate.  I got a message of type 0x7499 (29849).
    [Wed Sep 11 22:09:29 2019]00425:WorldAuthenticate.  I got a message of type 0x578f (22415).
    [Wed Sep 11 22:09:29 2019]00426:WorldAuthenticate.  I got a message of type 0x6259 (25177).
    [Wed Sep 11 22:09:29 2019]00427:WorldAuthenticate.  Access granted.
    
    [Wed Sep 11 22:09:29 2019]00428:Check 1x. 0x47c6922f
    
    [Wed Sep 11 22:09:29 2019]00429:Waiting to connect to zone... Character Theosaur
    
    [Wed Sep 11 22:09:29 2019]00430:Zone Connect -- 0 -- Received MSG_ZONE_ADDRESS
    [Wed Sep 11 22:09:29 2019]00431:Zone addr [142.44.176.140:20509] received...
    [Wed Sep 11 22:09:29 2019]00432:ZONING
    [Wed Sep 11 22:09:30 2019]00433:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:09:30 2019]00434:Networking: using port [62359].
    [Wed Sep 11 22:09:30 2019]00435:Networking: Connection Established [1]
    [Wed Sep 11 22:10:28 2019]00437:Could not connect to 142.44.176.140:7504 [client:DisconnectReasonConnectFail,server:DisconnectReasonNone].  Negotiation count: 597.
    
    [Wed Sep 11 22:10:28 2019]00438:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:10:28 2019]00439:Failed to connect to zoneserver (142.44.176.140, port 7504), result = 1, for MSG_ZONE_ADDRESS.
    
    [Wed Sep 11 22:10:29 2019]00437:Could not connect to 142.44.176.140:7504 [client:DisconnectReasonConnectFail,server:DisconnectReasonNone].  Negotiation count: 596.
    
    [Wed Sep 11 22:10:29 2019]00438:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:10:29 2019]00439:Failed to connect to zoneserver (142.44.176.140, port 7504), result = 1, for MSG_ZONE_ADDRESS.
    
    [Wed Sep 11 22:10:30 2019]00436:Could not connect to 142.44.176.140:7504 [client:DisconnectReasonConnectFail,server:DisconnectReasonNone].  Negotiation count: 597.
    
    [Wed Sep 11 22:10:30 2019]00437:Networking: Connection Closed [0] with 0 pending bytes.
    [Wed Sep 11 22:10:30 2019]00438:Failed to connect to zoneserver (142.44.176.140, port 7504), result = 1, for MSG_ZONE_ADDRESS.
     

    Attached Files:

  12. Akkadius

    Akkadius Administrator Staff Member

    Messages:
    221
    Thanks for the dbg.txt, that actually nailed the problem for me

    This will be fixed next reboot cycle, some folks wanted to know the technicals so here it is

    "So we've been testing a branch of massive changes mainly around the Loginserver, to most people it looks like no net new functionality but there's a lot of reasons for the changes. There's a lot of other peripheral changes that were also introduced into this branch to optimize on performance, server stability and resource utilization. We use dynamic zone processes and have a 1 hour shutdown timer on the processes themselves.

    Once that timer is up it will be relinquished back into a dynamic pool where it can then be assigned to another zone automatically. For the longest time and rammifications of being an open source project, we do not deallocate memory on several objects as these zones get booted into another. So for example if I booted wallofslaughter, loaded maps, navmesh, quests, characters into memory, then went to sleep and then became blackburrow, we don't cleanup everything properly. What ultimately happens is by end of day in PEQ we get some zones that bloat from a normal 100MB - 500-800MB. So while this is not necessarily a memory leak because we've patched a majority of them over the years, this is more of a cleanup problem

    The quick fix was to simply kill the process when a dynamic goes into shutdown 60 minutes after players had left that zone. This frees up memory back into the OS and we've been running at a very low amount of memory everyday in comparison with these changes. Normally when zone processes get booted our EQEmu standard is to automatically increment from port ranges 7000 - 7500, however, remnance of very old (Cavedude days) zone config had our higher port range at 9100. Since we are now hard killing processes, our world process is handing out incrementally new ports far faster than we ever had before and we are seeing ports get assigned above 7500 which we would normally never see in the daily reboot lifecycle of PEQ

    This is a problem when our anti DDOS mitigation appliance has only whitelisted UDP traffic from 7000-7500, when zones are being booted outside the upper bounds of the 7500 range, the appliance will see the unknown UDP traffic and actually blacklist the connection for a time and eventually will let the client connect again. I could add more ports but really we should be staying within the 7000-7500 range which is standard specification for EQEmu today

    This is also why we've not been seeing the problem until later in the evening because by the evening we started burning through more ports. The DDOS mitigation won't always blacklist every UDP connection and it depends on the behavior of your connection between zones that are whitelisted versus blacklisted"
     
    • Like Like x 5
  13. Hive

    Hive Froglok Krup Watcher

    Messages:
    340
    Akkadius ftw!
     
  14. newdlik2

    newdlik2 Orc Pawn

    Messages:
    8
    Still having issues. Kildwen has been having this problem for months. Was a little better last week. Today I couldnt log into guild lobby, took about 10 tries to eventually get in. Zoned to WoS and crashed to login screen. Tried many times but could not get him to zone back into the world.
     
  15. Akkadius

    Akkadius Administrator Staff Member

    Messages:
    221
    I'm fairly certain that your issue is most likely separate from those listed in this thread, but if you could please enclose in code tags in thread, your logs/dbg.txt file exactly when this occurs. When you start another client it can wipe this file
     
  16. newdlik2

    newdlik2 Orc Pawn

    Messages:
    8
    Here is my dgb.txt from this afternoon.

    I just tried to log in unsuccessfully, logged in another character, then tried to log in Kildwen again with no success.

    Thanks for taking a look
     

    Attached Files:

    • dbg.txt
      File size:
      75.1 KB
      Views:
      4
  17. Uleat

    Uleat Orc Warlord

    Messages:
    182
    Code:
    [Wed Sep 25 15:18:58 2019]00188:MSG_TIME_STAMP received. (Items inc).
    [Wed Sep 25 15:19:28 2019]00189:Networking: connection terminated [client:DisconnectReasonOtherSideTerminated,server:DisconnectReasonNone]
    Akkadius, this is occurring at the bulk inventory send.
     
  18. Akkadius

    Akkadius Administrator Staff Member

    Messages:
    221
    Yep I mentioned this to you in Discord before you posted. We need to get a solution for this, saying that players "have too many items" is not really a feasible solution
     
  19. Uleat

    Uleat Orc Warlord

    Messages:
    182
    I'm just slow... :eek:lder man, remember? :p