Latency Issues? (Solved: QMI > MBIM)

[Falias]

Hi all -

I have a wg3526 set up with an em7565 on an unlimited verizon prepaid sim. The modem gets around the same signal strength and upload/download speeds (~15-20/75-90) as my android phone sitting next to it, but the latency is significantly higher with the modem... They are both connecting to the same tower. (They both switch between bands quite often, but this issue happens over all of the ids.)

I tried matching the phone's default ttl of 52 (and also 53, 64, and 117) with no effect. (I can see that it's working on the ping response, but it doesn't decrease the latency.)

Any ideas?
Thanks

From the router:
327 packets transmitted, 327 packets received, 0% packet loss
round-trip min/avg/max = 43.217/65.820/98.320 ms

From the phone's verizon connection (running concurrently):
30/40/175

(Amusingly this short run had the two closer than the one I did overnight last night... that one was like 36 average v 71. This run still shows it though - the minimum ping with the router is higher than the cell phone's average.)

[xdavidx]

Just to understand the test conditions... When you are testing with the router, what is connected to it and how is it connected?

When you are testing with the phone, are you running something on the phone, or are you connecting another device to the phone? If so, what is connected and how?

[Falias]

The router has a computer attached by ethernet cable which I'm ssh'ing from - the actual ping is being run on the router itself though.

Nothing is connected to the phone; the ping is being run in Termux.
https://play.google.com/store/apps/deta ... com.termux

Thanks

[xdavidx]

Well, that certainly reduces the variables.

Thoughts/ideas:

- Try running traceroute on each connection to see if the packets are taking the same path (well, the same IP path, there could still be a different path under IP, I guess).

- It is possible that connection state could be handled differently, with the router modem making and breaking connections for each transmission.

- It is possible the router modem is going down to a pre LTE connection for small amounts of data. There are commands to force it to only use LTE as a test.

- It is possible that something else is sending data from the PC (or wifi, if other devices are attached to the wifi radios in the router) when you are pinging. You can use windows firewall to exclude everything except your ssh program. It is a bit of a pain to set up. Or you could disconnect the ethernet and try wifi from the phone, just as a test, even though wifi itself might add latency. Just keep the phone right next to the router. The phone has background transmission possibilities too.

- How many pings are you doing with each?

- Maybe specify the packet size on each to be sure it is the same.

[serverside]

HAH I thought I was the only one. Something is going on with Verizon. 9+ months with your basically same setup, prepaid unlimited Verizon. Used to get 60-90ms ping on discord and games. Now this last week, it's now hitting minimum 150ms ping in discord and 200ms in games. I've reset the modem contacted the reseller to reset the data, I've checked different antennas etc. If I connect via my phone it does a good bit better.

I think something is going on with Verizon, what region or state are you in? I'm in mid TN.

This is super frustrating as I work remote and this causes a lot of latency issues with my job.

I'm thinking about doing a NEMA box build and moving it to the top of my property or attaching it to a tall tree, but I don't know how much this will help.

This wouldn't help anything would it?
https://wirelessjoint.com/viewtopic.php?f=15&t=330

[Falias]

Thanks for the ideas...

1. Will check tracerts tonight and update.

2. I had noticed that they both dropped ca without a connection, so I tried with a background download at ~2mbps. It had no effect... but that was also downloading from a different server than I was pinging. I'm not really sure what a good way to isolate/test this would be - any thoughts (even on the same server it would be tracked as a different connection...)?

3. I have both selrat=6 and bands restricted to lte. (I've actually watched the bands/tower ids on both the phone and router while doing the pings... They aren't always on the same one at the same time, but they both swap between the same 3 quite often - during the test window even.)

4. The computer is running linux; I don't have any internet related operations in the background except for a connectivity check and ntp. (As an aside: I was absolutely floored when I had to use windows 10 at one point and it was not only downloading in the background, but that download was invisible to both the task manager and resmon... not having any control over your computer is an alarming norm.)

5. The one that I quoted was only a few hundred, I ran one overnight when I first found the problem though... so that would have been ~30,000.

6. I had also run the command from the phone with wifi on and off... so the ping program/parameters were identical... the results were consistent with the other tests - going through the wifi, the latency was around double.

@serverside
I'm disheartened that you're having the issue too... that makes it less likely that it's something fixable... though maybe it's temporary if you've just started having it. (Hopefully!)
I'm in central VA...
I have decent reception, so I don't think that's it.
No, I tried the TTL changes - it has no effect for me.

Thanks again

[xdavidx]

2. I had noticed that they both dropped ca without a connection, so I tried with a background download at ~2mbps. It had no effect... but that was also downloading from a different server than I was pinging. I'm not really sure what a good way to isolate/test this would be - any thoughts (even on the same server it would be tracked as a different connection...)?

Yeah, they'll drop the CA, but should keep the connection to the primary band. However, trying with and without, from a testing perspective was worthwhile, so nothing wrong with you doing that. I would think that pinging on the primary band, without CA, would reduce variables. If the phone is somehow maintaining CA the whole time and splitting the ping "load" across bands, I suppose that could lead to a difference in the phone vs the router.

Do you have some way to enable/disable CA and/or lock/block bands on the phone?

3. I have both selrat=6 and bands restricted to lte. (I've actually watched the bands/tower ids on both the phone and router while doing the pings... They aren't always on the same one at the same time, but they both swap between the same 3 quite often - during the test window even.)

If the tower ids change, then that would be a reasonably big difference, not only because the data is flowing through a different path, but the characteristics of the connection between one tower and another could add to latency. And switching back and forth in the middle would obviously require breaking and making connections, hence adding to latency.

This is my best guess for what might be going on, but it is only a guess. The phone might jump around between towers less often. Additionally, if there is interference between the towers, maybe the phone handles retransmissions differently than the router.

As for the bands, maybe lock to a single band to reduce that to a single variable. That assumes you can lock it on the phone too. But even if you can't, you can see if any single band on the router gives you better latency than the others.

4. The computer is running linux; I don't have any internet related operations in the background except for a connectivity check and ntp. (As an aside: I was absolutely floored when I had to use windows 10 at one point and it was not only downloading in the background, but that download was invisible to both the task manager and resmon... not having any control over your computer is an alarming norm.)

Linux helps with that variable. I agree that the lack of control is horrible. It took me a good bit of time to figure out how to use the firewall to disable everything, except what I wanted enabled, and how to switch back and forth easily.

Hmm...I seem to see the Windows Update network activity in task manager and resource monitor on my machines, both in Windows 7 and Windows 10. Not sure why it wouldn't show on your end, but it is Windows, of course.

How did you see the data activity?

5. The one that I quoted was only a few hundred, I ran one overnight when I first found the problem though... so that would have been ~30,000.

Should be enough. Just to be positive, maybe run them again on each system to be sure it is consistently slower on the router.

And try running both devices at the same time and in a mutually exclusive way, just in case there is some sort of interference going on. Longshot.

6. I had also run the command from the phone with wifi on and off... so the ping program/parameters were identical... the results were consistent with the other tests - going through the wifi, the latency was around double.

In the other stats you supplied, average was about 66 ms vs 40 ms. Did the wifi increase that difference even more? I would have thought a couple milliseconds, but not 10's of ms.

Just as a test, you could try pinging on the linux machine through the router and you could try setting up the phone as a hotspot and pinging from another device through that connection. It doesn't seem like it is ping program related, but maybe something will jump out.

You could also try setting the ping send buffer size to ensure it is consistent and you can try increasing that to see if it has more of an effect on one device versus the other. Again, I'm just brainstorming.

If it is something on the Verizon side of things, the question would be why does it only affect the router and not the phone? If there is some sort of throttling going on (tied to the sim in the router), I guess it could affect latency too. You said the TTL changes didn't have an effect, however, and your speeds are great.

Not sure if your phone sim works in the router and vice versa, but that is a possible test.

It is only a guess, but my best guess is that the connections jumping are leading to differences. That doesn't really explain the mininum latency being different between the two. It is hard to know if that min was a fluke for the phone and the max was a fluke for the phone and it is running 99% in the 40 ms range and that the router is also mainly in the 40 ms range, but had a lot of 90+ ms samples, due to jumping around between towers, and that skewed the average.

Maybe one handles interference better than the other.

A directional antenna is another thing that could be used to test. Of course, that's an expensive test. Alternatively, you could try bringing the router to a single tower and see what that shows for latency of the router and phone each having a non-changing connection to a tower and a very strong signal and less interference. You'd have to run it off an inverter in the car.

[serverside]

@serverside
I'm disheartened that you're having the issue too... that makes it less likely that it's something fixable... though maybe it's temporary if you've just started having it. (Hopefully!)
I'm in central VA...
I have decent reception, so I don't think that's it.
No, I tried the TTL changes - it has no effect for me.

Thanks again

Hmm, with us being in different states, I doubt it's Verizon. What I'm worried about is that the bands we are connected to are now congested. What I noticed yesterday was that when I forced the connection to band 4, which is super weak, it connected and I had great ping, but my upload/download was trash. I use the same service for my phone, and today I was thinking of putting my phones Sim into the router to make sure that I'm not being throttled. Being de-prioritized shouldn't have an effect on latency, but just speed for U/D, so I don't think it's this. If it is, I'll cancel this account and spin up a new one on the same plan.

I also have 2 yagi's coming vs my Proxicast 2x2 panel, which I think because it's a panel isn't able to get a strong enough signal. I am thinking of setting them up at 45 degrees and seeing if I can get CA connected. As CA connects randomly but is 99% of the time no connected.

Also, how do you do the TTL changes? I wanted to try this but I couldn't find on my version of GoldenOrb in the Network -> Firewall, any TTL settings.

[Falias]

Okay... multiple problems with trying to check the tracerts... nothing insurmountable though.

First, on the modem, all ip4 tracerts show a total of 1 hop... I assume verizon is 4>6 natting it immediately and then I just see nothing. While that doesn't happen on the phone, it doesn't appear to be the issue (the latency issues happen with ipv6 too).
(An interesting note here: they are both getting a public 100.x.x.x ip4, but the phone is in a /27 whereas the router is in a /32... as far as I'm aware, there's no point to having a subnet with a single host since it wouldn't be able to talk to anything... which I guess is why it's natted immediately... seems odd.)

The two connections do use different routes for 2001:4860:4860::8888, so I found a server that was in the common path early on at 2001:10:10::1 ... which shows the same pattern. (Path is router/phone > gw > hop that doesn't respond > 2001:10:10::1)

It also turns out the ping6 command in android (at least on my phone) uses a restricted syscall... so can't ping v6 addresses with the phone unless I root... which is annoying. I'm using the hotspot for the ping tests below - it makes the connection significantly worse, but even so, it's still clearly faster than the router's connection.

Histograms:

Figure_2.png

Figure_1.png

These were run concurrently from the computer with both the router and phone on LTE13 without CA (I bandlocked the router and disabled CA, and then I just watched the phone and ended the ping command as soon as it changed.) I ran the pings every 0.2 seconds to get more samples before the phone switched... I ended up with ~2300 samples on each over ~7.5 minutes.

Like I said, using the phone through the hotspot makes it far worse - running on the phone directly gives a much more normal distribution... just a few in 40-45 and almost none above that.

Other clarification: I'm always connected to the same physical tower on both, there are just different cellids for the different bands. I'm in the mountains and I actually wasn't able to connect to other towers even when trying with the directional antenna.

[Falias]

Also, how do you do the TTL changes? I wanted to try this but I couldn't find on my version of GoldenOrb in the Network -> Firewall, any TTL settings.

That tab just adds the following two rules to "Custom Rules" (make sure it's the correct interface)
iptables -t mangle -I POSTROUTING -o wwan0 -j TTL --ttl-set 52
iptables -t mangle -I PREROUTING -i wwan0 -j TTL --ttl-set 52

Verizon CA (at least for me) is only active during during a download/upload... so you may actually be using it without realizing it.

I would recommend not canceling the plan... it isn't offered anymore; we only have it grandfathered as long as the line stays active.

In regards to you trying a new antenna; these are my stats (while getting 101/31 on a speed test):
LTE band: B66 LTE bw: 20 MHz
LTE SSC1 state:INACTIVE LTE SSC1 band: B13
LTE SSC1 bw : 10 MHz
PCC RxM RSSI: -79 PCC RxM RSRP: -100
PCC RxD RSSI: -80 PCC RxD RSRP: -102
SCC1 RxM RSSI: -65 SCC1 RxM RSRP: -84
SCC1 RxD RSSI: -65 SCC1 RxD RSRP: -84
RSRQ (dB): -7.3
SINR (dB): 20.0
So I don't think that getting a better antenna would resolve it.

[serverside]

should I try it out as 117? Vs 52?

iptables -t mangle -I POSTROUTING -o wwan0 -j TTL --ttl-set 52
iptables -t mangle -I PREROUTING -i wwan0 -j TTL --ttl-set 52

What app are you using on your phone to test this?

Here is my connection info, I really think B13 is just congested, hence why I'm grabbing the two yagis to see if I can get Band 4 which doesn't seem to be congested, hence the drop to low 30-50ms when forcing it, just it's not stable.

AT!GSTATUS?
!GSTATUS:
Current Time: 8223 Temperature: 49
Reset Counter: 1 Mode: ONLINE
System mode: LTE PS state: Attached
LTE band: B13 LTE bw: 10 MHz
LTE Rx chan: 5230 LTE Tx chan: 23230
LTE CA state: INACTIVE LTE Scell band:B4
LTE Scell bw:20 MHz LTE Scell chan:2050
EMM state: Registered Normal Service
RRC state: RRC Connected
IMS reg state: No Srv
PCC RxM RSSI: -73 RSRP (dBm): -102
PCC RxD RSSI: -68 RSRP (dBm): -98
SCC RxM RSSI: -100 RSRP (dBm): -121
SCC RxD RSSI: -99 RSRP (dBm): -121
Tx Power: 0 TAC: A104 (41220)
RSRQ (dB): -11.5 Cell ID: 02768701 (41322241)
SINR (dB): 9.4

[xdavidx]

Seeing the distributions helps.

Other clarification: I'm always connected to the same physical tower on both, there are just different cellids for the different bands. I'm in the mountains and I actually wasn't able to connect to other towers even when trying with the directional antenna.

Different cell IDs are still a difference though. You are probably located at an angle from the tower transmitters where you are close to the intersection of the sectors they cover. If you take the rig up to the tower, you can connect to it from different sides to see if you can maintain one CID and see what effect it has on ping.

So there is some sort of difference that you are seeing in the hops and subnets. Are you able to swap sims?

Are the APNs the same for phone and router?

[xdavidx]

Oh, and your SINR you posted in the other message is very nice! I wish I could get an SINR like that at home. How far away from the tower are you?

[serverside]

So update on my issue, I configured my PIA account onto the WG3526 and made sure it was all working. My ping is still terrible, but I'm no longer have 120kbs download speed. Ping is still in the 150 range. So looks like the pageplus unlimited is now a throttled source, even after my plan reset.

Ugg....is there any other verizon plans that aren't throttled?

[Falias]

Ugg....is there any other verizon plans that aren't throttled?

I haven't done any of this, so I could be wrong on some of it, but I looked into it when I was setting my prepaid account up last month...
From my understanding, that post you linked with the ttl stuff allows you to use a tablet line without it being considered a hotspot... so it would get full data rate on plans that limit hotspot usage.
Verizon postpaid unlimited accounts (starting at $75/mo for 1 line) can add a tablet line for $20/mo which is also unlimited for non-hotspot data (thus the point of changing the ttls to make it look like the modem is a tablet). I'm also not sure if the video throttling is always 10mbps on a tablet line, or it varies based on the parent account level (since the unlimited cell phone plans have different video rates listed).

Note: the cheapest tablets I saw that use sim cards are ~$35 on ebay (Verizon Ellipsis 8)... I'm pretty sure you would need to get one in order to actually have verizon send you a sim and then activate it before switching it to the router.

Regarding phone testing - I'm using an app called "NetMonster" for seeing which towers/bands are in use, then I'm using normal gnu/linux tools for everything else in Termux.


@xdavidx
Thanks again for the troubleshooting.

I don't have an inverter, but I can rig up a battery for it directly if I need to. I think I can actually just do that test with the cell phone though - if I can connect to the different radios on the tower with the cell phone, and one of them does have a higher latency, then that would be the answer... I'll need to get some dark clothes and a mask before I can go snoop around the tower though... I probably won't have time till this weekend.

I don't want to swap sims since the plan it's on isn't a cell phone plan, and it's no longer offered. I'm afraid that when I plug it in, it'll either switch it to the old unlimited cell phone plan (which is throttled after 23?GB) or to the current plan of the same price, which isn't unlimited...

Both have the "VZWINTERNET" apn... the modem also adds a "vzwadmin" in the second slot once it connects. (The phone only shows the one in android's settings.)

I'm 1.3mi from the tower... it's nice... that may actually be reason enough to rig up the battery. The antenna is just sitting on my floor atm... I'd go through the effort of putting it outside/on the roof if it's actually going to make a difference, but that's a lot of work/cost if it isn't... unless there's an easier way to test if I'm limited by backhaul instead of signal.

[serverside]

I haven't done any of this, so I could be wrong on some of it, but I looked into it when I was setting my prepaid account up last month...
From my understanding, that post you linked with the ttl stuff allows you to use a tablet line without it being considered a hotspot... so it would get full data rate on plans that limit hotspot usage.
Verizon postpaid unlimited accounts (starting at $75/mo for 1 line) can add a tablet line for $20/mo which is also unlimited for non-hotspot data (thus the point of changing the ttls to make it look like the modem is a tablet). I'm also not sure if the video throttling is always 10mbps on a tablet line, or it varies based on the parent account level (since the unlimited cell phone plans have different video rates listed).

Note: the cheapest tablets I saw that use sim cards are ~$35 on ebay (Verizon Ellipsis 8)... I'm pretty sure you would need to get one in order to actually have verizon send you a sim and then activate it before switching it to the router.

Regarding phone testing - I'm using an app called "NetMonster" for seeing which towers/bands are in use, then I'm using normal gnu/linux tools for everything else in Termux.

Where are you seeing these plans? I'm not able to find anything like this on the verizon site, was it a limited deal? I'm assuming it requires a contract yeah?

I'm 1.3mi from the tower... it's nice... that may actually be reason enough to rig up the battery. The antenna is just sitting on my floor atm... I'd go through the effort of putting it outside/on the roof if it's actually going to make a difference, but that's a lot of work/cost if it isn't... unless there's an easier way to test if I'm limited by backhaul instead of signal.

Just to give you the numbers, inside, my connection was 1/2 to 1/4 of it was with the antenna outside and aimed properly. You can always grab a tri-pod for cheap and a piece of 1 1/2" Schedule 80 PVC conduit and rig up a mobile antenna that way.

[xdavidx]

@xdavidx
Thanks again for the troubleshooting.

I don't have an inverter, but I can rig up a battery for it directly if I need to. I think I can actually just do that test with the cell phone though - if I can connect to the different radios on the tower with the cell phone, and one of them does have a higher latency, then that would be the answer... I'll need to get some dark clothes and a mask before I can go snoop around the tower though... I probably won't have time till this weekend.

I don't want to swap sims since the plan it's on isn't a cell phone plan, and it's no longer offered. I'm afraid that when I plug it in, it'll either switch it to the old unlimited cell phone plan (which is throttled after 23?GB) or to the current plan of the same price, which isn't unlimited...

Both have the "VZWINTERNET" apn... the modem also adds a "vzwadmin" in the second slot once it connects. (The phone only shows the one in android's settings.)

I'm 1.3mi from the tower... it's nice... that may actually be reason enough to rig up the battery. The antenna is just sitting on my floor atm... I'd go through the effort of putting it outside/on the roof if it's actually going to make a difference, but that's a lot of work/cost if it isn't... unless there's an easier way to test if I'm limited by backhaul instead of signal.

A battery will work, assuming the router takes 12 volts. Regarding the cell phone test, it might be fast on both radios on the tower, but you won't know if the router would have been faster on one vs the other, when up close. Up close vs far away is one of the variables that might be affecting things for the router.

I hear you about the mask, etc. I had a guy in a big white van pull up and ask me if I was a cop. When he found out I wasn't, he was less nervous and proceeded to try to sell me frozen meat packs!

Sitting a 1 to 3 blocks from the tower is better than *right* under it for getting a good signal.

I understand about risking the sim being contaminated. Verizon seems pickier than the other companies.

Is there any way you can override the vzwadmin in the second slot of the modem? If I am understanding right, the first text box is just the name of the record in the modem and the second text box is the true APN name. If what I am saying is true, then you'd want VZWINTERNET in the second text box in the modem so that you match what you are using in the phone.

1.3 miles is nice and close! Remind me again, what type of antenna are you using?

I would be surprised if the latency is due to a backhaul bandwidth limitation, especially since the phone isn't having the problem. If you are talking about speeds, however, it is possible. Best guess is that the RF side of things is generally more of a limitation than the backhaul, but I'm sure there are towers out there where backhaul isn't what it should be.

Doing a temporary test outside, with an extension cord, might help you see what it does for your speeds. You are already getting over 100 Mbps down, however, so that's pretty good. What time of day was that taken?

[swwifty]

is it possible you are getting buffer bloat? That can cause high latency during speed tests and such.

I'd suggest testing here to see if that is the issue: http://www.dslreports.com/speedtest

[xdavidx]

@Falias: What do you see in: MODEM > CONNECTION PROFILE > GENERAL > APN ? (screenshot)

And what do you get if you run this command:

Code: Select all

at+cgdcont?

[Falias]

Okay... it's either a hardware issue with my router or a bug in the firmware...

The following 3 tests are using a sprint sim:
1) 7455 in usb enclosure attached to the computer directly - this shows normal/low latency
2) 7455 in usb enclosure attached to the wg3526 - this shows high latency
3) 7565 internal in the wg3526 - this shows high latency as well

Figure_3.png

Figure_4.png

Figure_5.png

I'm using the following firmware:
md5sum openwrt-wg3526-GO2019-03-06-upgrade.bin
a6fdc1bf5284a2d9689d5ff8c78ce3b7 openwrt-wg3526-GO2019-03-06-upgrade.bin

It occurs even after reflashing (and not keeping settings) with the only changes after flashing being:
1) setting the password/ssh key/ssh settings
2) setting the APN under connection profiles

@swwifty No. Except where otherwise noted, all of these tests are being done with no active connections (other than the pings and possibly ntp + connectivity checks (loading a 1.4kB page)).

Another unrelated issue:
Shortly after I plug the usb modem in, the internal modem loses internet connectivity. (It shows as connected, but is then unable to even ping on that interface...)

Could someone move this over to the router forum? (Since it's pretty clear at this point that it isn't actually a modem issue.)

Edit:
Just found 3.10 on https://www.ofmodemsandmen.com/firmware.html... same issue...
I can't get the modem to connect on the x86 image in a VM or running directly...
I also tried the Rasberry Pi image on my Pi Zero W, but it doesn't even boot for me... does it actually work on a Pi 1?

1678eee0e0ad59493e83dc4b91f4a986 openwrt-RaspberryPi-GO2019-03-10.img
a6fdc1bf5284a2d9689d5ff8c78ce3b7 openwrt-wg3526-GO2019-03-06-upgrade.bin
6dbffcb70771af5aacdeac40da64abf1 openwrt-wg3526-GO2019-03-10-upgrade.bin

[xdavidx]

Maybe something to ask the GoldenOrb people about?

Good job troubleshooting with a different provider and hardware.

I always thought my pings were 30-40, since speedtest shows that typically. I just ran ping on the router and the numbers seemed highly variable. Also ran them on Windows (firewall restricted) over ethernet and got this:

Code: Select all

C:\Users\End-User>ping -n 20 google.com

Pinging google.com [172.217.8.174] with 32 bytes of data:
Reply from 172.217.8.174: bytes=32 time=42ms TTL=52
Reply from 172.217.8.174: bytes=32 time=77ms TTL=52
Reply from 172.217.8.174: bytes=32 time=69ms TTL=52
Reply from 172.217.8.174: bytes=32 time=61ms TTL=52
Reply from 172.217.8.174: bytes=32 time=52ms TTL=52
Reply from 172.217.8.174: bytes=32 time=43ms TTL=52
Reply from 172.217.8.174: bytes=32 time=73ms TTL=52
Reply from 172.217.8.174: bytes=32 time=64ms TTL=52
Reply from 172.217.8.174: bytes=32 time=54ms TTL=52
Reply from 172.217.8.174: bytes=32 time=47ms TTL=52
Reply from 172.217.8.174: bytes=32 time=35ms TTL=52
Reply from 172.217.8.174: bytes=32 time=61ms TTL=52
Reply from 172.217.8.174: bytes=32 time=57ms TTL=52
Reply from 172.217.8.174: bytes=32 time=46ms TTL=52
Reply from 172.217.8.174: bytes=32 time=75ms TTL=52
Reply from 172.217.8.174: bytes=32 time=64ms TTL=52
Reply from 172.217.8.174: bytes=32 time=54ms TTL=52
Reply from 172.217.8.174: bytes=32 time=35ms TTL=52
Reply from 172.217.8.174: bytes=32 time=38ms TTL=52
Reply from 172.217.8.174: bytes=32 time=67ms TTL=52

Ping statistics for 172.217.8.174:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 35ms, Maximum = 77ms, Average = 55ms

I'll have to try the sim in a phone sometime to see how it compares. My SINR isn't as good as yours, so it could just be that. Or, it could be that the WE1326 suffers the same issue.

What were you pinging?

[JimHelms]

Okay... it's either a hardware issue with my router or a bug in the firmware...

I remember someone doing various tests on the WE826 using the router's OEM OpenWRT and GoldenOrb and found the latency to be better with the stock OpenWRT firmware. Of course, the OEM OpenWRT was running the modem in PPP mode which, in theory, could invalidate the results.

Could someone move this over to the router forum? (Since it's pretty clear at this point that it isn't actually a modem issue.)

It is much easier to copy and paste this post into the router section with a proper title.

[serverside]

Maybe something to ask the GoldenOrb people about?

Good job troubleshooting with a different provider and hardware.

I always thought my pings were 30-40, since speedtest shows that typically. I just ran ping on the router and the numbers seemed highly variable. Also ran them on Windows (firewall restricted) over ethernet and got this:

Code: Select all

C:\Users\End-User>ping -n 20 google.com

Pinging google.com [172.217.8.174] with 32 bytes of data:
Reply from 172.217.8.174: bytes=32 time=42ms TTL=52
Reply from 172.217.8.174: bytes=32 time=77ms TTL=52
Reply from 172.217.8.174: bytes=32 time=69ms TTL=52
Reply from 172.217.8.174: bytes=32 time=61ms TTL=52
Reply from 172.217.8.174: bytes=32 time=52ms TTL=52
Reply from 172.217.8.174: bytes=32 time=43ms TTL=52
Reply from 172.217.8.174: bytes=32 time=73ms TTL=52
Reply from 172.217.8.174: bytes=32 time=64ms TTL=52
Reply from 172.217.8.174: bytes=32 time=54ms TTL=52
Reply from 172.217.8.174: bytes=32 time=47ms TTL=52
Reply from 172.217.8.174: bytes=32 time=35ms TTL=52
Reply from 172.217.8.174: bytes=32 time=61ms TTL=52
Reply from 172.217.8.174: bytes=32 time=57ms TTL=52
Reply from 172.217.8.174: bytes=32 time=46ms TTL=52
Reply from 172.217.8.174: bytes=32 time=75ms TTL=52
Reply from 172.217.8.174: bytes=32 time=64ms TTL=52
Reply from 172.217.8.174: bytes=32 time=54ms TTL=52
Reply from 172.217.8.174: bytes=32 time=35ms TTL=52
Reply from 172.217.8.174: bytes=32 time=38ms TTL=52
Reply from 172.217.8.174: bytes=32 time=67ms TTL=52

Ping statistics for 172.217.8.174:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 35ms, Maximum = 77ms, Average = 55ms

I'll have to try the sim in a phone sometime to see how it compares. My SINR isn't as good as yours, so it could just be that. Or, it could be that the WE1326 suffers the same issue.

What were you pinging?

So, yesterday I went and did the same thing, but I'm using builds from last year (2018) for both my WE826 and WG3526. Both trying with ATT (Mobley Sim) and Verizon Sims. 8-10 SINR, both producing a wide area of ping. WE826 is running the EM7455 and the WG3526 running the MC7455.

80-250ms. It's really really bad. This is pinging servers for discord, google, and using Fast.com

Verizon tower and ATT towers are in different directions, both around 2 miles from each other using either a proxicast 2x2 mimo Or the 700-2700mhz Yagi's @ 18" on 45 degree angles.

Either Verizon and ATT now know about the EM/MC7455 modems and are heavily deprioritizing them or the firmware on the routers is having issues now after around 1ish years.

Really odd why it's happening now.

[JimHelms]

Have you tested changing the DNS servers?

[Falias]

Figure_6.png

(Ping under load - there have been people streaming most of the time that it's been running.)

Thanks everyone for the help!
I figured out a solution - use qmi.
(That would have been a really simple thing to just randomly try before going through all the effort of actually tracking it down... but I got there in the end )

I ended up building openwrt myself to compare the computer and router directly...
My desktop's NetworkManager was using qmi over mbim automatically.

On a positive note for the future: there are pull requests against openwrt to include feeds for modemmanager and a luci interface for it. I used those in my build and the configuration seemed nicer to me than rooter's current stack. (I had some issues with rooter where I set the APNs in the profile, and then wondered why it wasn't connecting, only to check at+cgd... and see that it hadn't propogated. Manually setting it fixes it then, but that just adds another variable while troubleshooting. It also has simpler config files.)

Have you tested changing the DNS servers?

I was pinging hosts directly (so avoiding DNS completely).

[swwifty]

(Ping under load - there have been people streaming most of the time that it's been running.)

Thanks everyone for the help!
I figured out a solution - use qmi.
(That would have been a really simple thing to just randomly try before going through all the effort of actually tracking it down... but I got there in the end )

I ended up building openwrt myself to compare the computer and router directly...
My desktop's NetworkManager was using qmi over mbim automatically.

On a positive note for the future: there are pull requests against openwrt to include feeds for modemmanager and a luci interface for it. I used those in my build and the configuration seemed nicer to me than rooter's current stack. (I had some issues with rooter where I set the APNs in the profile, and then wondered why it wasn't connecting, only to check at+cgd... and see that it hadn't propogated. Manually setting it fixes it then, but that just adds another variable while troubleshooting. It also has simpler config files.)





I was pinging hosts directly (so avoiding DNS completely).

Off Topic, but do you have any links to info on how to setup modem manager with stock openwrt? I want to replace GoldenOrb with openwrt, so I can have ip pass through.

[serverside]

Have you tested changing the DNS servers?

I run through opendns or Google dns servers. Plus my DL stays way higher when running through PIA vpn

[xdavidx]

Thanks everyone for the help!
I figured out a solution - use qmi.
(That would have been a really simple thing to just randomly try before going through all the effort of actually tracking it down... but I got there in the end )

Funny how that 20/20 hindsight works.

Congrats on finding the solution! I'm away from home, but will check how mine is configured when I get back on Sunday.

[Falias]

Off Topic, but do you have any links to info on how to setup modem manager with stock openwrt? I want to replace GoldenOrb with openwrt, so I can have ip pass through.

I just built it using these three pages:
https://openwrt.org/docs/guide-develope ... uildsystem
https://openwrt.org/toh/hwdata/zbt/zbt_wg3526_16m
https://gitlab.freedesktop.org/mobile-b ... nd-openwrt
*Convenient list of dependencies for compiling mm here too - adjust for your distro obviously: https://github.com/openwrt/packages/pul ... -499324223
I just added all of the kernel modules to the build that had names at all similar...

Alternately, if you just want to wait a bit, I think it should be available from the package manager on a normal install after these two pull requests go through:
GUI: https://github.com/openwrt/luci/pull/2683
MM: https://github.com/openwrt/packages/pull/9137

Also, you may need to use mmcli directly to do passthrough. I don't remember seeing it in the configs. (I wasn't looking for it though.)

[swwifty]

Thanks for the info. I'm going to set it up soon, and do a full write up on how to do this.

[xdavidx]

Either Verizon and ATT now know about the EM/MC7455 modems and are heavily deprioritizing them or the firmware on the routers is having issues now after around 1ish years.

Really odd why it's happening now.

Are you able to bring the equipment to the towers to see if that changes anything? Do you have a phone that shows better latency? Are you able to try QMI like Falias?

[xdavidx]

Thanks everyone for the help!
I figured out a solution - use qmi.
(That would have been a really simple thing to just randomly try before going through all the effort of actually tracking it down... but I got there in the end )

I verified that MBIM has higher latency than QMI with a WE1326 router and EM7565 modem running GoldenOrb. The following pings were run from a Windows computer connected via ethernet to the router.

20 pings at a time with QMI:

Minimum = 32ms, Maximum = 94ms, Average = 56ms
Minimum = 27ms, Maximum = 72ms, Average = 52ms
Minimum = 36ms, Maximum = 74ms, Average = 54ms

Switched to MBIM composition:

Minimum = 59ms, Maximum = 122ms, Average = 81ms
Minimum = 59ms, Maximum = 122ms, Average = 81ms

Switched back to QMI:

Minimum = 28ms, Maximum = 70ms, Average = 47ms
Minimum = 36ms, Maximum = 91ms, Average = 57ms

[serverside]

Are you able to bring the equipment to the towers to see if that changes anything? Do you have a phone that shows better latency? Are you able to try QMI like Falias?

So I thought the idea was to run these modems in MBIM vs QMI as the towers see them as generic?

There was a huge reason for MBIM if I recall, if we can run in QMI now and not worry about it then I'll give it a go.

I verified that MBIM has higher latency than QMI with a WE1326 router and EM7565 modem running GoldenOrb. The following pings were run from a Windows computer connected via ethernet to the router.

20 pings at a time with QMI:

Minimum = 32ms, Maximum = 94ms, Average = 56ms
Minimum = 27ms, Maximum = 72ms, Average = 52ms
Minimum = 36ms, Maximum = 74ms, Average = 54ms

Switched to MBIM composition:

Minimum = 59ms, Maximum = 122ms, Average = 81ms
Minimum = 59ms, Maximum = 122ms, Average = 81ms

Switched back to QMI:

Minimum = 28ms, Maximum = 70ms, Average = 47ms
Minimum = 36ms, Maximum = 91ms, Average = 57ms

How hard is it to switch from MBIM to QMI on the rooter software? Is there a thread or something I can follow for doing this from ROOTer/GoldenOrb/LEDE for a MC7455?

[JimHelms]

To place a MC7455 is QMI composition:

Code: Select all

at!entercnd="A710"
at!usbcomp=1,1,10d
AT!reset

[serverside]

To place a MC7455 is QMI composition:

Code: Select all

at!entercnd="A710"
at!usbcomp=1,1,10d
AT!reset

This will set it permanently to QMI until I change it back to MBIM?

Also, How do I change it back if say it doesn't let me connect?

Also what's the reason for us changing these to MBIM if we could have used QMI in the first place?

Sorry for all the questions, I thought I understood MBIM vs QMI.

[JimHelms]

This will set it permanently to QMI until I change it back to MBIM?

Also, How do I change it back if say it doesn't let me connect?

By using the AT Commands to place it back into MBIM mode. See the link below.

Also what's the reason for us changing these to MBIM if we could have used QMI in the first place?

I assume because you did not view this TUTORIAL:

[xdavidx]

The change does survive reboots. To get your pre-changed values, run these commands, one at a time:

at!entercnd="A710"
at!usbcomp?

usbcomp is a password protected command, hence the need to supply the default password of "A710" first. They are protecting you from shooting yourself in the foot. If you set it to the wrong values for your model of modem, you can get the modem into a bad state where it isn't accessible through normal means. There is a thread on here that describes how to recover, but getting in that state is something to be avoided.

The query command above will show you the current values for the 3 parameters used with the set command. If you see 1, 1, 100d (or any 1000+ value, like 100a), then you are in MBIM mode. If you see 10d (or any 100+ value), then you are in QMI mode. You might also see a 500+ value. I am not sure of the reason for 10d vs 50d.

To see just the QMI vs MBIM text string value, you can go to the modem -> network status page in the GoldenOrb UI and it will be at the bottom of the first block of values.

As Jim provided, you need the reset command after doing a set operation, for it to take effect. The modem will disconnect and reconnect. You can watch the status in the network status screen.

I am not sure of the exact history of the two compositions. I thought it was a matter of compatibility with certain versions of router firmware.

For anyone who doesn't have a 7455 modem, be sure you know what the proper values are before you use usbcomp to do a set operation, or you'll be in for a bumpy ride. For the 7565, for example, the second parameter is a 3, not a 1 (1, 3, 10d and 1, 3, 100d, for QMI and MBIM, respectively).

[JimHelms]

I am not sure of the reason for 10d vs 50d.

The 50d enables the Modem to utilize dual SIM Cards where the 10d is limited to a single SIM card.

I am not sure of the exact history of the two compositions. I thought it was a matter of compatibility with certain versions of router firmware.

Mobile broadband modem control protocols.pdf

[serverside]

By using the AT Commands to place it back into MBIM mode. See the link below.

I assume because you did not view this TUTORIAL:

Naa I built this system like 3+ years ago, when MBIM was what everyone said to use. If I'm reading correctly MBIM does IPV6 and QMI only IPV4. So MBIM would be the better option then no?

Why would QMI handle having better ping than MBIM if it's just IPV4/6 vs just IPV4 capability?

The change does survive reboots. To get your pre-changed values, run these commands, one at a time:

at!entercnd="A710"
at!usbcomp?

usbcomp is a password protected command, hence the need to supply the default password of "A710" first. They are protecting you from shooting yourself in the foot. If you set it to the wrong values for your model of modem, you can get the modem into a bad state where it isn't accessible through normal means. There is a thread on here that describes how to recover, but getting in that state is something to be avoided.

The query command above will show you the current values for the 3 parameters used with the set command. If you see 1, 1, 100d (or any 1000+ value, like 100a), then you are in MBIM mode. If you see 10d (or any 100+ value), then you are in QMI mode. You might also see a 500+ value. I am not sure of the reason for 10d vs 50d.

To see just the QMI vs MBIM text string value, you can go to the modem -> network status page in the GoldenOrb UI and it will be at the bottom of the first block of values.

As Jim provided, you need the reset command after doing a set operation, for it to take effect. The modem will disconnect and reconnect. You can watch the status in the network status screen.

I am not sure of the exact history of the two compositions. I thought it was a matter of compatibility with certain versions of router firmware.

For anyone who doesn't have a 7455 modem, be sure you know what the proper values are before you use usbcomp to do a set operation, or you'll be in for a bumpy ride. For the 7565, for example, the second parameter is a 3, not a 1 (1, 3, 10d and 1, 3, 100d, for QMI and MBIM, respectively).

the commands I'm pretty familiar with, I set this thing up like 3+ years ago now and have just been running it in MBIM mode and back then I don't think you could change the modem from QMI to MBIM inside of the LEDE/Rooter GUI, so it was done on a PC.

[xdavidx]

The 50d enables the Modem to utilize dual SIM Cards where the 10d is limited to a single SIM card.

Ah, I see. The Sierra command reference didn't discuss RMNET0 vs RMNET1 in detail and that they were tied to single vs dual SIMs. Does that mean you can't do dual SIMs with MBIM, or do you need to use 150D to enable that?

Mobile broadband modem control protocols.pdf

I meant I didn't know the history of why someone may have been told to use MBIM instead of QMI, other than firmware compatibility. Your tutorial has a good reason, with IPV6 not being supported over QMI. I guess that's why IPv6 wasn't working for me when I was testing it the other day. Do you know if AT&T supports IPv6 on their iPad plan? I don't use IPv6 currently for anything. It is just for general knowledge.

That was a fun read though. Thanks for sending it. It is interesting to see the path things have taken since the dial up modem days. When I tell my friends that I'm back to using AT commands, they can't believe they still exist. I tell them that there are just a few more commands these days and it isn't as simple as ATDT.

[serverside]

Ah, I see. The Sierra command reference didn't discuss RMNET0 vs RMNET1 in detail and that they were tied to single vs dual SIMs. Does that mean you can't do dual SIMs with MBIM, or do you need to use 150D to enable that?



I meant I didn't know the history of why someone may have been told to use MBIM instead of QMI, other than firmware compatibility. Your tutorial has a good reason, with IPV6 not being supported over QMI. I guess that's why IPv6 wasn't working for me when I was testing it the other day. Do you know if AT&T supports IPv6 on their iPad plan? I don't use IPv6 currently for anything. It is just for general knowledge.

That was a fun read though. Thanks for sending it. It is interesting to see the path things have taken since the dial up modem days. When I tell my friends that I'm back to using AT commands, they can't believe they still exist. I tell them that there are just a few more commands these days and it isn't as simple as ATDT.

My question is why would QMI work potentially better than MBIM, as it seems MBIM is superior being able to handle IPV6 vs just IPV4.

I haven't tested QMI yet, but will after work.

[JimHelms]

Ah, I see. The Sierra command reference didn't discuss RMNET0 vs RMNET1 in detail and that they were tied to single vs dual SIMs. Does that mean you can't do dual SIMs with MBIM, or do you need to use 150D to enable that?

This I am not sure of. I do know that the Cradlpoint dual SIM Card routers all use QMI.

I meant I didn't know the history of why someone may have been told to use MBIM instead of QMI, other than firmware compatibility.

Yes, until the latest stable GoldenOrb build or the newer WiFiX versions thereof, it would not accept QMI. The modem had to be configured in MBIM prior to installing the modem of GoldenOrb would not communicate with the modem.

[serverside]

This I am not sure of. I do know that the Cradlpoint dual SIM Card routers all use QMI.




Yes, until the latest stable GoldenOrb build or the newer WiFiX versions thereof, it would not accept QMI. The modem had to be configured in MBIM prior to installing the modem of GoldenOrb would not communicate with the modem.

This is my build version, do you know if it will work on QMI?

Firmware Version GoldenOrb_2018-07-24 ( OpenWrt 18.06.0-rc2 r7141-e4d0ee5af5 )
Kernel Version 4.14.54

[xdavidx]

My question is why would QMI work potentially better than MBIM, as it seems MBIM is superior being able to handle IPV6 vs just IPV4.

Good question, but probably a difficult answer. The variables at play are the modem firmware, the drivers, and the router firmware. Something in there is adding to the latency. I doubt it is a problem with the MBIM protocol itself.

As for support of IPv6, again, I doubt that is a protocol level issue. It is probably just a matter of firmware support for handling IPv6 properly using one protocol vs the other.

These are just educated guesses.

[xdavidx]

Yes, until the latest stable GoldenOrb build or the newer WiFiX versions thereof, it would not accept QMI. The modem had to be configured in MBIM prior to installing the modem of GoldenOrb would not communicate with the modem.

I vaguely remember reading some old posts and thinking that setting up my system was going to be more difficult than it was. When it came time to set it up, I used a newer tutorial and it was easy (no need to mess with USB compositions or hook it up to a PC, etc.). That's the challenge with things that change over time. The old information is no longer relevant, but it isn't always easy to know what is old and what is new.

And then there is serverside's type of situation, where we're discussing this in a present day context, but he might be using older firmware that has to be treated in the past context.

[serverside]

Good question, but probably a difficult answer. The variables at play are the modem firmware, the drivers, and the router firmware. Something in there is adding to the latency. I doubt it is a problem with the MBIM protocol itself.

As for support of IPv6, again, I doubt that is a protocol level issue. It is probably just a matter of firmware support for handling IPv6 properly using one protocol vs the other.

These are just educated guesses.

Hmm, once I find out if I need to update firmware on my rooter or not I'm going to test this, I really don't want to have to update firmware again just to test, as I've got my VPN configured, firewall rules done, full VLAN and routes, a good bit of work has gone into this.

But it is very odd as MBIM seems to be the new protocol vs QMI which I would assume the old. Maybe cell carriers just haven't caught up yet.

I vaguely remember reading some old posts and thinking that setting up my system was going to be more difficult than it was. When it came time to set it up, I used a newer tutorial and it was easy (no need to mess with USB compositions or hook it up to a PC, etc.). That's the challenge with things that change over time. The old information is no longer relevant, but it isn't always easy to know what is old and what is new.

And then there is serverside's type of situation, where we're discussing this in a present day context, but he might be using older firmware that has to be treated in the past context.

Yep, it sucked when I first set this all up, there was a ton of stuff you had to do. Now it seems, you buy a WG router, install LEDE/Rooter, install the QMI MCXXXX card, put in your sim, put in the APN for your sim, and you're done.

[xdavidx]

Hmm, once I find out if I need to update firmware on my rooter or not I'm going to test this, I really don't want to have to update firmware again just to test, as I've got my VPN configured, firewall rules done, full VLAN and routes, a good bit of work has gone into this.

I hear you.

Maybe another reason to have a separate router to handle all those things and only have the LTE router handling LTE related things. However, when I've been faced with similar situations in the past with other routers, I've taken screenshots of all my settings so that I could reset stuff relatively easily. Modern routers and firmware seem to do fairly well with built in save/load settings functions, even when the firmware has changed, but it seems GoldenOrb is not one of those and it is recommended to not have it save settings on an upgrade. I'm not sure if the same applies to saving the settings externally and reloading them after upgrading the firmware. Jim?

Yep, it sucked when I first set this all up, there was a ton of stuff you had to do. Now it seems, you buy a WG router, install LEDE/Rooter, install the QMI MCXXXX card, put in your sim, put in the APN for your sim, and you're done.

Yep, that was pretty much my experience. I did have an issue where something got in a bad state and I had to do a reset on the router. I was presented with a chinese language screen and a button. I pushed the button and it asked for a file. I fed it the GoldenOrb firmware file that I had used to upgrade the firmware through the normal UI previously, and I was back in action.

But I didn't have to touch the modem firmware. I did that for the first time last week, just so I knew how it worked. I did it over the air, with the AirVantage system, not by hooking it up to a PC.

[Falias]

IPv6 works fine with QMI on both Sprint and Verizon.

Tested on:
1) GO 3-10
2) OpenWRT w/MM
3) Arch w/MM
Verizon was on the EM7565 (FW12) for all 3. Sprint was on the EM7455 (FW23) for all 3.

An interesting note is that ipv4 might actually be handled differently on verizon - I had mentioned previously that I had a single host subnet and that I couldn't trace anything with ipv4... That is not the case now - I'm getting a public ip with a /30 and can trace normally. The only change on my side since then has been using QMI - but it could be a tower/network change too...

[serverside]

IPv6 works fine with QMI on both Sprint and Verizon.

Tested on:
1) GO 3-10
2) OpenWRT w/MM
3) Arch w/MM
Verizon was on the EM7565 (FW12) for all 3. Sprint was on the EM7455 (FW23) for all 3.

An interesting note is that ipv4 might actually be handled differently on verizon - I had mentioned previously that I had a single host subnet and that I couldn't trace anything with ipv4... That is not the case now - I'm getting a public ip with a /30 and can trace normally. The only change on my side since then has been using QMI - but it could be a tower/network change too...

Cool, So now I need to know if my April build will handle QMI for my MC7455. Anyone know when the first build allowed for it to work?

[serverside]

Cool, So now I need to know if my April build will handle QMI for my MC7455. Anyone know when the first build allowed for it to work?

Anyone know the answer to this? When did the Rooter builds start working with QMI?

[swwifty]

Anyone know the answer to this? When did the Rooter builds start working with QMI?

I don't know exactly, but its been a few releases back.

I noticed that too randomly when trying a newer version of GoldenOrb.

[serverside]

I don't know exactly, but its been a few releases back.

I noticed that too randomly when trying a newer version of GoldenOrb.

So just tried it, and it does work on my build version. I'm connected and....it does give me lower latency, still going through a VPN, BUT it's where I used to be, 30-80ms is what I'm used to and it's been 200+ now with the QMI mode I'm down to like 100-150ms, and spiking sometimes to 180-200. Better than nothing, but either verizon is throttling my connection OR the tower I've been enjoying is over loaded at this point, which sucks.

[xdavidx]

Are all your latency tests through a VPN?

[serverside]

Are all your latency tests through a VPN?

Yes, they have to be otherwise my connection is like 10-50kbs on Fast.com and drops packets and lags, it's completely unusable. I'm on a verizon MVNO, but I've never had this issue before.

[serverside]

Yes, they have to be otherwise my connection is like 10-50kbs on Fast.com and drops packets and lags, it's completely unusable. I'm on a verizon MVNO, but I've never had this issue before.

So just tried playing a game and my ping is just as bad as before, might be a little worse since it seems to drop the connection randomly...this is getting REALLY frustrating.

[JimHelms]

Sierra Wireless released a new EM7565 Generic firmware today as well as for the EM7511.

Release notes:

SWI9X50C-CRN-R01.14.pdf

[xdavidx]

Yes, they have to be otherwise my connection is like 10-50kbs on Fast.com and drops packets and lags, it's completely unusable. I'm on a verizon MVNO, but I've never had this issue before.

Well, I understand that you have Verizon issues, but my suggestion is to reduce variables if you want to figure out the cause of the slower pings. Fast.com is a unique situation and that will tell you how Netflix will behave, but to determine if the VPN is adding to the latency, I'd recommend trying some tests without the VPN.

- Try the speedtest app for Windows, on an ethernet connected PC for speed tests.
- Use the ping option in the GoldenOrb UI to google.com and also use ping from a command prompt in the Windows attached PC. The PC will allow you to run more than a few at a time with the -n switch.

If Verizon is snooping on the packets and destroys everything without it going through a VPN (meaning the above non-Fast.com tests are worse *without* the VPN than they are with the VPN), then my recommendation would be to try another VPN temporarily, just as a test. There are some free VPN providers out there that you could use for testing, even if they might not be sufficient for permanent use. They might still be blocked or crippled by fast.com/netflix, so leave that test out of the equation. I'm only talking about testing for speed and latency issues with and without the VPN and with another VPN, to a non-fast.com endpoint.

Obviously, there are a ton of variables with all this, but I'm just taking an educated guess that something might have changed in your VPN. If two cellular providers both showed the same symptoms, and neither had issues before, then it would seem to not be on their end. The other variables I can think of are:

- Antenna got pointed differently (not sure if you are using omni antennas or directional). I thought I read where you said the RSRP was strong or the SINR was good or something, so that would seem to rule out antenna issues. Maybe take a look at the RSRP values and SINR value again to be sure they are still good.

- Hardware issues.

- Wifi issues if you are testing that way.

- Firmware issues (router or modem). Re-flashing both could be done, but I think you said you had a lot of router settings you didn't want to lose.

I wouldn't bet a lot on it, but my bet is still with something in the VPN changing, assuming all the other variables remained static between when it was good and when it went bad.

Does the VPN client run on your PC or in the router? And are you sure your VPN endpoint is still the same and didn't get switched to Switzerland or somewhere?

[xdavidx]

Sierra Wireless released a new EM7565 Generic firmware today as well as for the EM7511.

Release notes:

SWI9X50C-CRN-R01.14.pdf

Interesting. It isn't available on AirVantage yet. Do you know how much lag time there is between firmwares being made available for download and being made available on AirVantage?

[xdavidx]

@serverside: One thing I noticed in the new firmware release notes does pertain to Verizon:

"QTI9X50-1131 Default NONCE value is incorrect for Verizon Requirements VZW / OMADM"

I would think some sort of wrong hash value would make something not work at all, but who knows -- maybe it just makes things not work as well. You did say another provider (Sprint or T-Mobile?) starting acting differently too, so this probably has nothing to do with your issue, but figured I'd point it out just the same.

[xdavidx]

@serverside: Try setting the "use small packets" setting in your PIA VPN client. I looked back over your posts and saw that you mentioned that is your VPN provider.

https://www.privateinternetaccess.com/h ... eally-high

I don't see how it could help the ping command, but it might help with some of the regular data transfers you are doing.

One other thing I ran across is to revert a driver in Windows back to an earlier version (TAP Driver): https://www.privateinternetaccess.com/h ... ng-windows

Seems like they were having issues as recently as 3-4 months ago: https://www.reddit.com/r/PrivateInterne ... h_latency/

[swwifty]

I've noticed that Verizon rate limits to various speed test sites. I believe they do this to help safe airtime on their spectrum. I'm guessing because a lot of people were doing speed tests which use a lot of unnecessary bandwith.

[xdavidx]

I've noticed that Verizon rate limits to various speed test sites. I believe they do this to help safe airtime on their spectrum. I'm guessing because a lot of people were doing speed tests which use a lot of unnecessary bandwith.

Are they limiting it to a specific rate?

[swwifty]

Are they limiting it to a specific rate?

I don't know, but the common speed test sites are very slow on download, but show a normal upload speed.

It became apparent after a while, because normal browsing / web traffic on Verizon's network was fast, but speed tests showed like 200kb/sec.

I did find some sites that weren't rate limited that offer speed tests, and it became apparent then.

[xdavidx]

It became apparent after a while, because normal browsing / web traffic on Verizon's network was fast, but speed tests showed like 200kb/sec.

Heck, an ISDN user would think that was heaven.

I guess times have changed a little since the '90s.

[serverside]

@serverside: Try setting the "use small packets" setting in your PIA VPN client. I looked back over your posts and saw that you mentioned that is your VPN provider.

https://www.privateinternetaccess.com/h ... eally-high

I don't see how it could help the ping command, but it might help with some of the regular data transfers you are doing.

One other thing I ran across is to revert a driver in Windows back to an earlier version (TAP Driver): https://www.privateinternetaccess.com/h ... ng-windows

Seems like they were having issues as recently as 3-4 months ago: https://www.reddit.com/r/PrivateInterne ... h_latency/

Looking into these, but I can't find a small packet option for the VPN setup on Rooter, tried advanced as well, no option. Still trying to figure that one out.

Doubt it's windows as it's same issue on android and linux :/

[serverside]

Well, I understand that you have Verizon issues, but my suggestion is to reduce variables if you want to figure out the cause of the slower pings. Fast.com is a unique situation and that will tell you how Netflix will behave, but to determine if the VPN is adding to the latency, I'd recommend trying some tests without the VPN.

This is with Fast.com, speedtest app android (ookla), ping to 1.1.1.1 and 8.8.8.8. If I don't use a VPN, everything is hosed. Packets get dropped like mad and the ping is through the roof.

- Try the speedtest app for Windows, on an ethernet connected PC for speed tests.
- Use the ping option in the GoldenOrb UI to google.com and also use ping from a command prompt in the Windows attached PC. The PC will allow you to run more than a few at a time with the -n switch.

Never used the speedtest app for windows, just fast.com and ookla speedtest.net. I use ping from cmd and the router, both are just as bad without the VPN.

If Verizon is snooping on the packets and destroys everything without it going through a VPN (meaning the above non-Fast.com tests are worse *without* the VPN than they are with the VPN), then my recommendation would be to try another VPN temporarily, just as a test. There are some free VPN providers out there that you could use for testing, even if they might not be sufficient for permanent use. They might still be blocked or crippled by fast.com/netflix, so leave that test out of the equation. I'm only talking about testing for speed and latency issues with and without the VPN and with another VPN, to a non-fast.com endpoint.

My next test was to jump onto a closer server from PIA, right now I'm using US east, and I live in TN. So going to give US Atlanta a go.

I've tried this on both my phones sim and the dedicated sim for my MC7455 and on the EM7455, using both a proxicast mimo 2x2 panel and 2 yagis at 45 degrees at 18" appart. Same thing, although I get really good speeds without a VPN from ATT in the location I tested. So right now I'm looking to build a Nema box buildout and put up my antenna's, just trying to figure out power right now (don't exactly want to run a 100' extension cord).

Obviously, there are a ton of variables with all this, but I'm just taking an educated guess that something might have changed in your VPN. If two cellular providers both showed the same symptoms, and neither had issues before, then it would seem to not be on their end. The other variables I can think of are:

I appriciate any help and brainstorming, cause I've been driven crazy over this. It was completely fine then one day I wake up my speeds are trash, so I contact the MVNO, like I normally do and have them reset the data, they do, and it didn't fix anything. (which it normally does).

- Antenna got pointed differently (not sure if you are using omni antennas or directional). I thought I read where you said the RSRP was strong or the SINR was good or something, so that would seem to rule out antenna issues. Maybe take a look at the RSRP values and SINR value again to be sure they are still good.

Antenna is right outside my window, and it hasn't changed. I am always looking at RSRP and SINR values, like modem debuging is open all the time lol.

- Hardware issues.

- Wifi issues if you are testing that way.

I don't think it is hardware cause it's same on two different antennas, and two different routers and modems, and two different sim cards on same plan on same carrier (MVNO).

- Firmware issues (router or modem). Re-flashing both could be done, but I think you said you had a lot of router settings you didn't want to lose.

Yea, I really don't want to be doing that, but at this point I'm about to Screenshot everything and rebuild on a new rooter version. ><

I wouldn't bet a lot on it, but my bet is still with something in the VPN changing, assuming all the other variables remained static between when it was good and when it went bad.

Does the VPN client run on your PC or in the router? And are you sure your VPN endpoint is still the same and didn't get switched to Switzerland or somewhere?

So I never used the VPN until this happened. When it happened I tested to see my speeds on the VPN client running the PC only and saw my speeds normal again, so I put the VPN on the router and I get proper DL/UL but ping is just boned.

VPN is US-East.

[xdavidx]

It seems like you have multiple issues.

1) Without VPN, something weird is going on with Verizon. It seems AT&T is okay. Maybe switching to AT&T is your easiest option.

2) With VPN, latency is poor. This is to be expected with VPNs. It is just the nature of the beast. You can try different VPN servers and you can try different VPN providers and you can try running it with PIA's Windos client again, where you have more control over settings. The provider's client software will always provide more advanced options than a generic module in a router and might be fine tuned for their server end.

The other problem with VPNs is that they tax the CPU of a router quite a bit. And if you also have custom routes and filtering rules in the router, your max throughput decreases significantly. I have some test numbers I'll find and post to show how drastic it is.


Regarding (1), what do you get if you run:

at!impref?

and

at!image?

Also for (1), try as many speed test websites as you can find to see if ookla and fast.com are being throttled like swwifty found. Try pinging a lot of different endpoints too. Again, this is non-VPN.

Lastly, can you point to different Verizon towers and test what non-VPN is like with them? Maybe you have a good connection to the tower, but the tower or something upstream from it is in a bad state. I know vpn works, but maybe the bad state only exhibits the behavior wih unencrypted traffic.

[xdavidx]

So right now I'm looking to build a Nema box buildout and put up my antenna's, just trying to figure out power right now (don't exactly want to run a 100' extension cord).

I'm in the same boat. I've been doing my preliminary stuff with an extension cord. The EM7565 seems to be too power hungry to run with power over ethernet. So I'm going to have to run wire up to the roof. However, in order to do it right, a GFCI should be used. I also have a cell booster that I plan on locating outside the house as well. So rather than wiring directly to the enclosure, I think I'll end up putting a GFCI outlet on the roof and plug the router and my cell booster into that. I just need to find one that is fairly enclosed to protect it, and the AC/DC adapters, from the elements.

[xdavidx]

@serverside: Here are those tests that show how much routing rules and encryption affect the throughput capabilities of the MediaTek MT7621A processor, running at 880MHz. That's the processor used in my WE1326 router and your WG3526 router.

Your WE826 uses the MediaTek MT7620A, running at 580MHz.

The 7621 is dual core and the 7620 is single core. I would expect the results below to be a lot slower on the 7620. Besides the clock speed and core differences, the 7621 also has a hardware engine for encryption that the 7620 doesn't have. Whether that is used by the firmware is another thing and whether the hardware encryption is faster or slower than the CPU doing it is another thing. But offloading encryption to the hardware (if the firmware does that) does open the CPU up to do other things, like routing and filtering.

mt7621a_880mhz_ethernet_tests.png

mt7621a_880mhz_ipsec_tests.png

You can see where, with small packets, enough bridging/routing rules could start to become a bottleneck. Encryption, all by itself, could be a *huge* bottleneck. Add the two together and it could be a lot worse. Even medium sized packets could see a processing limitation with both routing rules and encryption combined.

*If* CPU is partly to blame for some of your issues, then daisy chaining another router could help to offload some of that processing. Testing the VPN client on the PC will show if CPU/VPN software features (in the router firmware) are playing a role in your issues. This is all regarding number (2) in my other post. Number (1) is a different issue with Verizon.

You'll never get lower latency with a VPN than without, on any system, however. I mean that in the context of a carrier that isn't doing weird things when you aren't using a VPN, of course.

[serverside]

It seems like you have multiple issues.

1) Without VPN, something weird is going on with Verizon. It seems AT&T is okay. Maybe switching to AT&T is your easiest option.

Yeah, today I turned off the VPN, and now am getting normal connection, ping is still high but download isn't 20kbs anymore. I'm trying to come up with a plan on the ATT deal to get my antenna where it needs to be with the modem.

2) With VPN, latency is poor. This is to be expected with VPNs. It is just the nature of the beast. You can try different VPN servers and you can try different VPN providers and you can try running it with PIA's Windos client again, where you have more control over settings. The provider's client software will always provide more advanced options than a generic module in a router and might be fine tuned for their server end.

Trying this out as well today, to see if the small packets help any. Haven't had a chance to test though.

The other problem with VPNs is that they tax the CPU of a router quite a bit. And if you also have custom routes and filtering rules in the router, your max throughput decreases significantly. I have some test numbers I'll find and post to show how drastic it is.

Hmm, need to check this, as without it my load on the router is like %2 peak over a 30 second span.

Regarding (1), what do you get if you run:

at!impref?

at!impref?
!IMPREF:
preferred fw version: 02.08.02.00
preferred carrier name: GENERIC
preferred config name: GENERIC_002.007_001
current fw version: 02.08.02.00
current carrier name: GENERIC
current config name: GENERIC_002.007_001

OK

and

at!image?

at!image?
TYPE SLOT STATUS LRU FAILURES UNIQUE_ID BUILD_ID
FW 1 GOOD 2 0 0 ?_? 02.08.02.00_?
FW 2 GOOD 1 0 0 ?_? 02.05.07.00_?
FW 3 GOOD 1 0 0 ?_? 02.14.03.02_?
FW 4 EMPTY 0 0 0
Max FW images: 4
Active FW image is at slot 1

TYPE SLOT STATUS LRU FAILURES UNIQUE_ID BUILD_ID
PRI FF GOOD 0 0 0 002.009_001 02.08.02.00_ATT
PRI FF GOOD 0 0 0 002.007_001 02.08.02.00_GENERIC
PRI FF GOOD 0 0 0 002.012_000 02.14.03.02_SPRINT
PRI FF GOOD 0 0 0 002.008_003 02.05.07.00_VERIZON
Max PRI images: 50


OK

Also for (1), try as many speed test websites as you can find to see if ookla and fast.com are being throttled like swwifty found. Try pinging a lot of different endpoints too. Again, this is non-VPN.

Will do.

Lastly, can you point to different Verizon towers and test what non-VPN is like with them? Maybe you have a good connection to the tower, but the tower or something upstream from it is in a bad state. I know vpn works, but maybe the bad state only exhibits the behavior wih unencrypted traffic.

Unfortunately the next tower is blinded by trees and hills. I don't think I'll ever be at level with it to get a good signal. This is the signal with the proxicast and tower I can hit now:

AT!GSTATUS?
!GSTATUS:
Current Time: 70751 Temperature: 54
Reset Counter: 1 Mode: ONLINE
System mode: LTE PS state: Attached
LTE band: B13 LTE bw: 10 MHz
LTE Rx chan: 5230 LTE Tx chan: 23230
LTE CA state: INACTIVE LTE Scell band:B4
LTE Scell bw:20 MHz LTE Scell chan:2050
EMM state: Registered Normal Service
RRC state: RRC Connected
IMS reg state: No Srv
PCC RxM RSSI: -72 RSRP (dBm): -101
PCC RxD RSSI: -71 RSRP (dBm): -98
SCC RxM RSSI: -102 RSRP (dBm): -123
SCC RxD RSSI: -99 RSRP (dBm): -123
Tx Power: 23 TAC: A104 (41220)
RSRQ (dB): -10.0 Cell ID: 02768701 (41322241)
SINR (dB): 10.2
OK

[xdavidx]

I thought maybe something might be messed up where you were on SPRINT or AT&T's firmware. GENERIC should be fine. Not sure if VERIZON would help you at all.

Where are you getting the 2% number? I've only seen a load number in the GoldenOrb UI. I haven't found a percentage number yet.

So without VPN and with QMI, your ping is still high? What do the low/high/average look like for google.com? I guess you are going to try a bunch of ping targets, so that should shed more light on it if you save the results.

At least the speeds went back up. Maybe just a temporary glitch on Verizon's end.

[xdavidx]

And yeah, if that is the *best* Verizon tower you can get and the antennas are aimed well, then another tower probably won't get you better speeds. Of course, there are a lot of variables with bandwidth and bands. I suppose for a ping test, it might still be an okay test, just to see if latency is higher with one tower vs another, even if speeds are worse. You might only be able to get Band 13 for testing.

[swwifty]

I'm in the same boat. I've been doing my preliminary stuff with an extension cord. The EM7565 seems to be too power hungry to run with power over ethernet. So I'm going to have to run wire up to the roof. However, in order to do it right, a GFCI should be used. I also have a cell booster that I plan on locating outside the house as well. So rather than wiring directly to the enclosure, I think I'll end up putting a GFCI outlet on the roof and plug the router and my cell booster into that. I just need to find one that is fairly enclosed to protect it, and the AC/DC adapters, from the elements.

Can you elaborate on your issues powering the EM7565 via POE? I was planning on doing this soon

[xdavidx]

Can you elaborate on your issues powering the EM7565 via POE? I was planning on doing this soon

I could be 100% wrong. Jim would probably have better guidance. I'm just going on theory, not on actual experience.

When I first got my equipment, I had no doubt that I would have the router inside and use LMR400 cables for the 50 feet or so from the roof to my basement. I bought a WE1326 router, since it had a gigabit ethernet port, so that I wouldn't hit a 100 Mbps bottleneck. That router doesn't allow for POE, but it didn't matter for me at the time.

Fast forward to a few weeks ago when I started getting more serious about putting together the gear needed for setting up my system. I realized that LMR400, over 50 feet, was going to lose enough signal that I'd probably want to go with LMR600. That was going to be pretty costly. I couldn't justify spending that much just to have the router inside.

I'd *really* like to have the router inside, because I'd like to have physical access to it in the case of any issues where I need to reset it. I'd also like the ability to temporarily take it with me to towers to get stats on bands and speeds, etc. There is no way in heck I can get on my roof in the winter, so if something bad happens, I'll be out of luck. Even in good conditions, my roof is too steep for me to be climbing around on it without a support of some type. I'm currently using an old garden hose tied off on the other side of the house, but I can't keep that up there indefinitely.

The end solution is that I'll have to put the router up on the roof. I can't use POE with the current router. I started looking at other routers that support POE. I also had in the back of my mind that I would need about 2.5 amps for peak usage, as I've read about other people having disconnect issues with lower current adapters. That could just be with certain combinations of modems and routers, but I didn't want to get all the equipment only to find out I didn't have enough power.

thewirelesshaven.com sells these power adapters for people who have those issues: https://thewirelesshaven.com/shop/power-adapter/1 ... ter-2-1mm/

It is also mentioned on the WE826-T2 page: https://thewirelesshaven.com/shop/routers/lte-rou ... te-router/

Power Adapter Upgrade:
Our WE826-T2 routers are special ordered to include a 2 Amp Power Adapter after determining that 1 Amp is insufficient when using the Sierra Wireless MC7455 modem under load–resulting in disconnection issues.

We strongly recommend upgrading to our top quality 12V 2.5A (30W) power adapter when using the Sierra Wireless EM7565 CAT12 modem. One must not underestimate the importance of providing adequate power to the router and modem.

With POE, the protocol standard version makes a difference regarding how much power can be transmitted. There is a maximum power transmitted, but the power available to the device is lower, due to losses in the power supply in the device, and transmission losses.

*If* we truly need 2.5 amps at 12 volts to prevent disconnects when the modem is at a full 3CA and the router is consuming at max power, then we need 30 watts. The original POE, 802.3af (same as 802.3at Type 1) supplies 12.95 watts to the device. I don't believe the power supply losses are included in that. 802.at Type 2 supplies 25.5 watts to the device. You have to go up to the relatively recent 802.3bt, Type 3 or Type 4, to get 51 watts or 71 watts, respectively, to the device.

Now, if we can get by with only 2 amps instead of 2.5, then we only need 24 watts. 802.at *might* be enough. So if you can find a router that supports 802.at POE, you *might* have enough power. I imagine it would depend on whether you are using the wifi radios too or have those shut off, how much of a load you are putting on the router CPU, etc.

As an alternative strategy, I though of getting an 802.3bt injector (or a non-compliant injector that supplies higher wattage than an af/at injector can) and then employing a POE splitter at the device end. This would split the power and the data lines out, such that you'd use the regular DC power input for powering the device and only getting the data lines at the ethernet port of the device. I didn't realize that thewirelesshaven.com sells such things until just now. I had looked at the prices for some of the devices through other sources and realized that I'd be losing much of the cost savings by going with POE vs longer antenna cables.

I just looked at thewirelesshaven.com's products. They are significantly less expensive than what I had looked at in my brief online searches. So you might be able to use that approach and ensure that you'll have enough current available to the device.

Here is the splitter from the store, built into a bulkhead connector: https://thewirelesshaven.com/shop/power-adapter/p ... mm-jumper/

Rated Voltage: DC12~57V;
Rated Current: 5A Max. ;

You'd have to ensure the router can handle the output voltage of the injector and you are using an injector that, when line losses are taken into account, can still deliver the 30 watts *at the device* (assuming we need that much). The ratings on the injectors will be at the source, not at the device. Alternatively, buy a splitter that is also a voltage converter, ensuring you get 12 volts out of the splitter. Alternatively, use the The Wireless Haven splitter, but add an extra DC-to-DC step down converter to provide a consistent 12 volts to the device.

I also see that there is a unique beastie for sale on the store: https://thewirelesshaven.com/shop/routers/lte-routers/wg3526-p/

ZBT doesn't list that one on their website, so I'm not sure if that is a special build for The Wireless Haven or The Wireless Haven modified the board or how that thing was born. It doesn't list 802.3bt, yet the specs for the power input for POE seem to match that protocol version. It is either that, or some special non-standard arrangement where there is a built in splitter on the other side of the WAN port of the router.

In my case, I also need to power a T-mobile booster device. https://support.t-mobile.com/docs/DOC-14947 I'm using an original version of that booster currently, with the window unit in an upstairs window. It barely gets signal, specially when it is wet outside. When I get a newer version (constantly back-ordered), I'll put it in an outdoor enclosure to get a better signal strength. It needs 12 volt power too, and there is no way to use POE, other than with an injector and splitter arrangement and not using the data portion.

It makes more sense for me to just run power to the location so that I can power the two devices. I thought I'd be adding a GFCI circuit from my subpanel circuit breaker that is in my equipment room, with a GFCI outlet outside near ground level, running wires from the load side up to the roof, through conduit, and a non-GFCI outlet on/near the roof. I planned on having a large, weatherproof outlet cover that the AC/DC adapters for the two devices would fit inside of.

It dawned on me that I do have power in the attic on that part of the house and I could probably tie into that circuit. I could put a GFCI on the inside of the attic and a non-GFCI on the outside, as originally planned. The only problem is if the GFCI trips, then I have to get up in the attic to reset it, vs just resetting it near ground level. The attic isn't easily accessible. I'm still debating which way to go with it. Hopefully the GFCI won't ever trip and I won't have to get up in the attic to reset it. The trade off is that possible issue vs the cost of the additional wire, conduit, conduit straps, LB box, etc.

As I was typing this, I thought of a 3rd solution. I could get DC extensions for the AC/DC adapters and plug them into an outlet in the attic. With the outlet being in the attic and only 12 volts output, a GFCI probably isn't even required by code. Although it wouldn't take much to add such an outlet. This would remove the need for outdoor outlets, waterproof covers, outdoor 120 volt wiring, conduit, etc.

I was leaning toward my first option (GFCI near ground level and running wire up to the roof), just to have the ability to reset the GFCI more easily. Now I'll have to think more about option 3, just using DC extensions from inside the attic to the roof. I'll have to consider wire length and DC voltage drop.

Just thought of another option. With the attic in play, I could put the router in the attic and use a shorter length of antenna cable that wouldn't break the bank and wouldn't lose that much signal. The only problem with this approach is the attic getting very hot and overheating the router. Our attics have horrible ventilation. Fixing that has been on my to-do list for years. It could easily get to 130 degrees up there. Operating temperature on the router is 104 F. I *could* put a temperature controlled fan on it, but that's a moving part that could fail and I might as well ventilate my attic at that point. This doesn't help with my cellular booster and I've also already bought the enclosures.

With your Pi setup, maybe you can get by with normal POE without a splitter. Does the Pi support 802.3af or 802.3at and how much power does the device draw without adding on the modem and USB enclosure?

[swwifty]

With your Pi setup, maybe you can get by with normal POE without a splitter. Does the Pi support 802.3af or 802.3at and how much power does the device draw without adding on the modem and USB enclosure?

I currently power my setup with 802.3af. It doesn't look like the PI supports 802.3at from what I can find.

I have a splitter currently on my setup, and haven't noticed any connection issues that appear related to power issues (although i could be wrong, but i've always assumed my disconnects and resets via connection monitoring were due to signal issues).

I guess I'll find out soon enough. My PI currently sits at 3-5 watts usage via PoE with a MC7455 in the USB enclosure.