High Latency - Centurylink ISP - Main Thread 2/11/2020

Having the same problem - GPON Symmetrical Gigabit Internet running on enterprise hardware in Flagstaff.

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                          setup.ubnt.com -    0 | 1903 | 1903 |    0 |    0 |    2 |    0 |
|            phn4-dsl-gw05.phn4.qwest.net -    0 | 1904 | 1904 |    6 |    9 |   70 |    7 |
|             71-32-113-33.phn4.qwest.net -    0 | 1903 | 1903 |    6 |    8 |   92 |    7 |
|              los-brdr-01.inet.qwest.net -    0 | 1903 | 1903 |   15 |   16 |   53 |   17 |
|                           63.146.27.102 -    1 | 1900 | 1899 |   17 |   19 |   25 |   20 |
|                    cr1.la2ca.ip.att.net -    0 | 1903 | 1903 |   16 |   20 |   26 |   20 |
|                  la2ca408me3.ip.att.net -    0 | 1903 | 1903 |   16 |   26 |  247 |   17 |
|               12-122-254-234.attens.net -   94 |  400 |   25 |    0 |   22 |  109 |   19 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |  380 |    0 |    0 |    0 |    0 |    0 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like

This would never be solved by an onsite visit. This is a packet loss and jitter problem on the backbone services between CL’s network edge and Blizzard. We need them to get the backbone provider and CL talking to fix this. I had a thread on this issue about 3 years back that got to 30 pages before some support people at Blizzard slapped CL and Level3 around until it was fixed.

3 Likes

Same problem, last four nights getting lag spikes 400-900 ms. CL as well.

2 Likes

Hey, everyone! I just wanted to update here, we are tracking for the CenturyLink issue resurfacing after about a month since the latency issue subsided, thanks to all of your reports!

Since the initial issue has been calm for the last month or so, we will need new tests to check on the connection issues that have been occurring for the last week.

To help us track this issue, please include a WinMTR test while running the game for 10-15 minutes. Copy and paste the text file created and paste it between two ~~~ like so:

~~~
WinMTR Here
~~~

Based on a few of the tests here, I personally have some concerns that this might be occurring on the Level 3 node or there’s a configuration issue somewhere earlier on in the peers. We’ll need more tests to compare and confirm what’s happening to have this looked into by the appropriate folks.

I encourage to reach out to your ISP as well, so that this makes communication regarding the issue a bit easier for both sides a bit easier. Your reports are appreciated along with any additional information that you can provide :slight_smile: Thanks again, everyone!

3 Likes

Thank you for helping us out, Caterpepi. You have been way more helpful than the service at CL. Appreciate the update! Here is my winMTR from tonight:

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              modem.Home -    1 | 1223 | 1218 |    0 |    2 |   90 |    1 |
|            phn4-dsl-gw02.phn4.qwest.net -    1 | 1223 | 1218 |   21 |   27 |  241 |   22 |
|                phn4-agw1.inet.qwest.net -    1 | 1215 | 1208 |   21 |   28 |  318 |   22 |
|                             4.68.38.185 -   22 |  676 |  534 |    0 |   34 |  414 |   31 |
|                            4.69.215.133 -   99 |  251 |    3 |    0 |   43 |   50 |   39 |
|                              4.7.26.166 -   17 |  758 |  636 |   39 |   46 |  176 |   40 |
|              ae1-br01-csla1.as57976.net -   16 |  767 |  647 |   39 |   44 |  155 |   42 |
|       xe-0-0-1-3-br01-eqla1.as57976.net -   14 |  800 |  689 |   39 |   47 |  415 |   40 |
|              be1-pe01-eqla1.as57976.net -   18 |  732 |  604 |   39 |   42 |  107 |   42 |
|                           24.105.30.129 -   18 |  739 |  612 |   40 |   44 |  421 |   41 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like

CenturyLink - Located in Peoria, AZ and my MS is usually at ~28, however, it is now between 120-400ms for the past 3-4 days. I typically see high spikes later in the day, particular at night. Game has become borderline unplayable.

|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

|                              modem.Home -    0 | 1547 | 1547 |    0 |    0 |    7 |    0 |

|            phn4-dsl-gw06.phn4.qwest.net -    0 | 1547 | 1547 |    8 |   11 |   70 |   36 |

|                phn4-agw1.inet.qwest.net -    0 | 1547 | 1547 |    8 |   10 |   81 |    9 |

|                             4.68.38.185 -   24 |  805 |  619 |   17 |   18 |   32 |   18 |

|                            4.69.215.133 -  100 |  311 |    2 |    0 |   33 |   34 |   33 |

|                              4.7.26.166 -   16 |  949 |  800 |   26 |   30 |  154 |   27 |

|              ae1-br01-csla1.as57976.net -   15 |  969 |  824 |   31 |   33 |  142 |   31 |

|       xe-0-0-1-3-br01-eqla1.as57976.net -   17 |  941 |  789 |   31 |   39 | 1498 |   84 |

|        et-0-0-2-br01-swlv10.as57976.net -   14 | 1012 |  878 |   31 |   35 |  148 |   31 |

|        et-0-0-0-pe02-swlv10.as57976.net -   17 |  933 |  779 |   31 |   34 |  150 |   93 |

|       las-swlv10-ia-bons-02.as57976.net -   16 |  949 |  799 |   31 |   31 |   44 |   32 |

|                           137.221.105.2 -   17 |  936 |  784 |   31 |   31 |   34 |   32 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like

I tried to open a ticket and then insisted on sending someone and because I’m not “under contract” if they don’t find anything they were going to charge $75 for the trip. They won’t find anything because as you said Caterpepi this problem is on L3 and not on my end. This is the same thing we had a few years ago and it took the Blizzard team yelling at L3 and CL to get this fixed. The difference this time is that CL owns L3 as of a year or so ago. PLS HALP!

1 Like

My usual latency is 18ms home/world. Last couple of days it’s been terrible as well. My ISP is CenturyLink also. I ran the WinMTR for about 20 minutes. I have a CL tech coming to the house tomorrow. They tried telling me everything was fine but after my persistence, out of the blue tech support found there was some issue in the fiber causing distortion…

|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

|                              modem.Home -    0 |  969 |  969 |    0 |    0 |    5 |    0 |

|            phn4-dsl-gw01.phn4.qwest.net -    0 |  969 |  969 |    2 |    5 |   59 |    3 |

|              71-32-113-1.phn4.qwest.net -    0 |  969 |  969 |    2 |    5 |   49 |    4 |

|                             4.68.38.185 -   24 |  503 |  386 |    0 |   12 |   21 |   14 |

|                            4.69.215.133 -   99 |  197 |    3 |   28 |   28 |   29 |   28 |

|                              4.7.26.166 -   10 |  706 |  640 |   27 |   31 |  199 |   29 |

|              ae1-br01-csla1.as57976.net -    9 |  716 |  652 |   32 |   44 | 1759 |   34 |

|       xe-0-0-1-3-br01-eqla1.as57976.net -    8 |  731 |  674 |   32 |   76 | 3123 |   34 |

|        et-0-0-2-br01-swlv10.as57976.net -    8 |  735 |  680 |   32 |  129 | 3739 |   34 |

|        et-0-0-0-pe04-swlv10.as57976.net -    8 |  742 |  685 |   32 |   36 |  158 |   33 |

|       las-swlv10-ia-bons-04.as57976.net -    9 |  722 |  660 |   32 |   33 |   41 |   34 |

|                           137.221.105.2 -    9 |  714 |  650 |   32 |   33 |   43 |   34 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like

@invisipwner - not sure why exactly your first winmtr’s and the earlier ones itt didn’t show anything, but your newest tracert does show an issue on your fourth hop owned by level3, as do the last 3 posted in here. good luck getting it fixed.

1 Like

Thank you, everyone! I’m getting each of these tests over on our tracking. Keep the WinMTR’s coming if you are running into this issue! Based on the tests so far, the issues looks very similar to what was happening about a month ago.

By chance, is this happening during certain hours only or is the latency consistent throughout the entire day? The last time, this was only happening during the evening when the traffic is at its peak.

I still need to check through everything here, but I’ll update here with more details when available. Appreciate you all and the tests provided thus far :smiley:

1 Like

My latency issues escalate in the late afternoon and continue throughout the evening. My morning tests show latency around 25 home/world. Evenings I lag so bad I can’t do any group content. I can get over 10 seconds or more of lag/freeze.

1 Like

Pretty much same as Mudpie’s response. While I have seen an over all latency increase, it becomes horrible in the late afternoon and into the night.

1 Like

It can vary from early/late afternoon into the early/late evening. On a couple occasions (but not as common) I have encountered this issue in the late morning.

1 Like

Hey folks,

Thanks for the continued reports. That confirms that it’s similar to the issue that Centurylink resolved last month. This appears to be a peering issue around this area in the tests you all are posting:

Note the high packet loss increase (the % column - in this case it jumps to 24%, but it’s different in each test) around this area. Data starts getting weird shortly before you all transfer into our peer ISP. Please continue posting reports and winMTR tests when the problem happens.

You should also report the issue to Centurylink to get this resolved more quickly. They will likely need to reach out to whoever owns that part of the internet to get the peering issue fixed.

1 Like
|------------------------------------------------------------------------------------------|

|                                      WinMTR statistics                                   |

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |

|------------------------------------------------|------|------|------|------|------|------|

|                             192.168.0.1 -    0 | 1088 | 1088 |    0 |    0 |    1 |    0 |

|            phn4-dsl-gw03.phn4.qwest.net -    1 | 1080 | 1078 |    6 |    9 |   63 |    7 |

|                phn4-agw1.inet.qwest.net -    1 | 1081 | 1079 |    7 |    9 |   59 |   12 |

|                             4.68.38.185 -   20 |  611 |  491 |    0 |   16 |   29 |   17 |

|                            4.69.215.133 -   99 |  221 |    3 |    0 |   87 |  212 |   24 |

|                              4.7.26.166 -    9 |  810 |  740 |   25 |   29 |  207 |   48 |

|              ae1-br01-csla1.as57976.net -    8 |  826 |  760 |   25 |   27 |  132 |   26 |

|       xe-0-0-1-3-br01-eqla1.as57976.net -    9 |  818 |  750 |   31 |   37 |  148 |   32 |

|              be1-pe01-eqla1.as57976.net -    8 |  835 |  771 |   25 |   25 |   72 |   25 |

|                           24.105.30.129 -    9 |  822 |  755 |   32 |   32 |   92 |   33 |

|________________________________________________|______|______|______|______|______|______|

   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                              modem.Home -    3 | 3995 | 3887 |    0 |    3 |  132 |    1 |
|            phn4-dsl-gw02.phn4.qwest.net -    3 | 3984 | 3873 |    0 |   31 |  604 |   24 |
|                phn4-agw1.inet.qwest.net -    3 | 3988 | 3878 |    0 |   31 |  389 |   22 |
|                             4.68.38.185 -   29 | 1838 | 1322 |    0 |   38 |  288 |   32 |
|                            4.69.215.133 -  100 |  886 |    1 |    0 |   42 |   42 |   42 |
|                              4.7.26.166 -   18 | 2602 | 2145 |    0 |   51 |  462 |   41 |
|              ae1-br01-csla1.as57976.net -   15 | 2769 | 2355 |    0 |   49 |  265 |   41 |
|       xe-0-0-1-3-br01-eqla1.as57976.net -   15 | 2771 | 2359 |    0 |   57 | 2563 |   40 |
|              be1-pe01-eqla1.as57976.net -   16 | 2706 | 2277 |    0 |   47 |  258 |   41 |
|                           24.105.30.129 -   18 | 2631 | 2183 |    0 |   48 |  310 |   47 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
1 Like

I can’t tell you how much i would love to report the issue but no matter how many times i call in or go through web support they just want to send a technician out. CL tech support is following a playbook and completely unable to identify problems when they don’t fall within that playbook.

My guess is, sadly, Blizzard has more avenues to alert their backbone team to a problem than CL’s customers do.

2 Likes

Just confirming called 3 times now. They won’t escalate without rolling a truck to my house. This will not help. I’m contacting a local team lead at CL directly for health.

1 Like

I called in again and go the ominous “i’m sending a report to the internal team” and then they didn’t ask for the routers that are experiencing issues. I gave them the CL/ATT routers that seem to be unhappy but i’m skeptical anything actually got sent.

1 Like

I have a case open for this with CL and I think I have the right information in the hands of someone who can get it to their network team to start looking into the peering problem with L3. They own L3 now as of some time in 2018 I believe. Will post back with more information as I get it. I also reached out for the same issue through the corporate support team through work. We don’t play games there but I can replicate the same packet loss to the same target host, so I can complain, so I did. That is a multi-million dollar per year customer complaining through corporate channels about the packet loss, so maybe, just maybe we can get some action!

2 Likes