Poor upload - Several EM7455s/Ubuntu

[lpr600]

Looking for some help from the experts here with an issue that neither we nor the vendor can diagnose.

In the last several weeks we started to deploy edge/industrial computers that have the EM7455 modem embedded in them. We have deployed 4 so far in the last week and the performance results are miserable, particularly with upload speeds - which of course is what is important for our application.

The background:
Devices are i7-10700 processors, Ubuntu 18.04.3. I think they are using MBIM, but I'm not 100% sure how to tell.
[Antennas are Proxicast low profile, MIMO, ANT-122-S02 (https://www.amazon.com/Proxicast-Profil ... B07DDC9WV5)
Antennas are mounted at about 8', on a grounded aluminum cabinet, OUTSIDE.

To measure speed we used speedtest-cli from the commandline, OpenSignal's speed test on our phones, and speedtest.net tethered to our laptops. In each case the results between the 3 were not drastically different, so we ruled out variability of the testing sites. Phone is an iphone 6S so nothing fancy, for sure. Additionally, our phone testing is with Visible which should be the same as the VZW modems, but throwing that our there for full transparency.

Upload performance of these devices has been miserable. At the first site we could not get anything more than 0.28Mbps. What was strange was that on the device (Ubuntu) using speedtest-cli it would always report 0.28, no matter how many times we tried and no matter what different remote test sites it used. This happened again at the 3rd site, but this time it was 0.17Mbps, never higher, always the same. It was so slow we couldn't even SSH into it from our office so we pulled it to bring it back.

These are not remote, rural locations either - we're in the middle of a city. Results on the OpenSignal app and tethered are NEVER less that 5Mbps/5Mbps at any location, often download speeds are 20Mbps+ and upload speeds approach 10Mbps. And that's with the phone on hip - not 8 feet in the air on a metal cabinet with a good ground plane and a mimo antenna!

We tried reaching out to the vendor of the industrial computers and they were super helpful - "Send it back to us, we'll check it." We did a lot of research here but didn't come up with any smoking guns, so we're looking for any sage advice to help us figure this out. There must be something wrong with the configuration of these devices. Firmware in all of the devices is the same and appears to be the latest from Sierra Wireless.

Cellmapper shows sites with Bands 2, 5, 13, 66 surrounding us, all within a mile or less and no significant terrain obstruction.

Below is all the information from one of the sites.

Would appreciate any advice.

Code: Select all

AT!GSTATUS?
Current Time:  66375            Temperature: 18
Reset Counter: 1                Mode:        ONLINE         
System mode:   LTE              PS state:    Attached     
LTE band:      B2               LTE bw:      5 MHz   
LTE Rx chan:   975              LTE Tx chan: 18975
LTE CA state:  INACTIVE         LTE Scell band:B13    
LTE Scell bw:10 MHz             LTE Scell chan:5230
EMM state:     Registered       Normal Service 
RRC state:     RRC Connected  
IMS reg state: Full Srv         IMS mode:    Normal       

PCC RxM RSSI:  -64              RSRP (dBm):  -94
PCC RxD RSSI:  -67              RSRP (dBm):  -97
SCC RxM RSSI:  -79              RSRP (dBm):  -98
SCC RxD RSSI:  -77              RSRP (dBm):  -97
Tx Power:      --               TAC:         6701 (26369)
RSRQ (dB):     -9.0             Cell ID:     018B1518 (25892120)
SINR (dB):     15.8

AT!LTECA?
Hardware:
LTEB1: B8, 
LTEB2: B2, B5, B12, B13, B29, 
LTEB3: B7, B20, 
LTEB4: B4, B5, B12, B13, B29, 
LTEB5: B2, B4, B30, 
LTEB7: B3, B7, B20, 
LTEB8: B1, 
LTEB12: B2, B4, B30, 
LTEB13: B2, B4, 
LTEB20: B3, B7, 
LTEB25: 
LTEB26: 
LTEB29: 
LTEB30: B5, B12, 
LTEB41: B41, 

Permitted Bands:
LTEB2: B2, B5, B13, 
LTEB4: B4, B5, B13, 
LTEB5: B2, B4, 
LTEB13: B2, B4, 

Prune_ca_combos:
2A-13A-0           2A-13A-1           13A-2A-0           13A-2A-1           
4A-13A-0           13A-4A-0           

!LTEINFO: 
Serving:   EARFCN MCC MNC   TAC      CID Bd D U SNR PCI  RSRQ   RSRP   RSSI RXLV
              975 311 480 26369 018B1518  2 2 2   4 433 -10.1  -94.7  -70.1 --

IntraFreq:                                          PCI  RSRQ   RSRP   RSSI RXLV
                                                    433 -10.1  -94.7  -70.1 --
                                                    207 -14.5 -102.1  -78.1 --
                                                    434 -15.4 -102.6  -78.0 --

InterFreq: EARFCN ThresholdLow ThresholdHi Priority PCI  RSRQ   RSRP   RSSI RXLV
             5230            0           0        0 433 -11.6  -97.4  -77.0   0

WCDMA:     UARFCN ThreshL ThreshH Prio PSC   RSCP  ECN0 RXLV

AT!BAND=?
Index, Name,                        GW Band Mask     L Band Mask      TDS Band Mask
00, All bands                     0002000007C00000 000000000000101A 0000000000000000
01, Europe 3G                     0002000000400000 0000000000000000 0000000000000000
02, North America 3G              0000000004800000 0000000000000000 0000000000000000
06, Europe                        0002000000400000 0000000000000000 0000000000000000
07, North America                 0000000004800000 000000000000101A 0000000000000000
08, WCDMA ALL                     0002000007C00000 0000000000000000 0000000000000000
09, LTE ALL                       0000000000000000 000000000000101A 0000000000000000

                                                   0000000000001000 - B13    
                                                   0000000000000010 - B5     
                                                   0000000000000008 - B4     
                                                   0000000000000002 - B2     
                                  0002000000000000 - B8  (900)
                                  0000000004000000 - B5  (850)
                                  0000000002000000 - B4 (1700)
                                  0000000001000000 - B3 (1700)
                                  0000000000800000 - B2 (1900)
                                  0000000000400000 - B1 (2100)

AT!BAND?
Index, Name,                        GW Band Mask     L Band Mask      TDS Band Mask
00, All bands,                      0002000007C00000 000000000000101A 0000000000000000

AT+CSQ
+CSQ: 20,99

AT+RSRP?
+RSRP: 433,975,"-094.40",207,975,"-102.50",434,975,"-102.80",433,5230,"-097.50"

AT+RSRQ?
+RSRQ: 433,975,"-09.10",207,975,"-14.60",434,975,"-15.50",433,5230,"-11.30"

!IMPREF: 
 preferred fw version:    02.33.03.00
 preferred carrier name:  VERIZON
 preferred config name:   VERIZON_002.079_001
 preferred subpri index:  000
 current fw version:      02.33.03.00
 current carrier name:    VERIZON
 current config name:     VERIZON_002.079_001
 current subpri index:    000


 mmcli -m 0

 --------------------------------
 General  |            dbus path: /org/freedesktop/ModemManager1/Modem/0
          |            device id: 5df8cc516462f6b618fb04cf260d80648377ddac
 --------------------------------
 Hardware |         manufacturer: Sierra Wireless, Incorporated
          |                model: Sierra Wireless EM7455 Qualcomm® Snapdragon™ X7 LTE-A
          |             revision: SWI9X30C_02.33.03.00
          |         h/w revision: EM7455
          |            supported: gsm-umts, lte
          |              current: gsm-umts, lte
          |         equipment id: 
 --------------------------------
 System   |               device: /sys/devices/pci0000:00/0000:00:14.0/usb1/1-12
          |              drivers: qcserial, cdc_mbim
          |               plugin: Sierra
          |         primary port: cdc-wdm0
          |                ports: ttyUSB0 (qcdm), ttyUSB2 (at), cdc-wdm0 (mbim), 
          |                       wwp0s20f0u12i12 (net)
 --------------------------------
 Numbers  |                  own: +
 --------------------------------
 Status   |       unlock retries: sim-pin2 (3)
          |                state: connected
          |          power state: on
          |          access tech: lte
          |       signal quality: 38% (cached)
 --------------------------------
 Modes    |            supported: allowed: 3g; preferred: none
          |                       allowed: 4g; preferred: none
          |                       allowed: 3g, 4g; preferred: 3g
          |                       allowed: 3g, 4g; preferred: 4g
          |              current: allowed: 3g, 4g; preferred: 4g
 --------------------------------
 Bands    |            supported: utran-1, utran-3, utran-4, utran-5, utran-8, utran-2, 
          |                       eutran-2, eutran-4, eutran-5, eutran-13
          |              current: utran-1, utran-3, utran-4, utran-5, utran-8, utran-2, 
          |                       eutran-2, eutran-4, eutran-5, eutran-13
 --------------------------------
 IP       |            supported: ipv4, ipv6, ipv4v6
 --------------------------------
 3GPP     |                 imei: 
          |        enabled locks: fixed-dialing
          |          operator id: 311480
          |        operator name: Verizon Wireless
          |         registration: home
 --------------------------------
 3GPP EPS | ue mode of operation: csps-2
 --------------------------------
 SIM      |            dbus path: /org/freedesktop/ModemManager1/SIM/0
 --------------------------------
 Bearer   |            dbus path: /org/freedesktop/ModemManager1/Bearer/1

[Didneywhorl]

Welcome!


You can search the forum here for the proper command to find the protocol mode of an em/mc7455. Our tutorials section has a lot. But yes it looks like its using mbim. I would maybe try qmi and see if that works better. The MBIM and QMI are simply communication protocols between the router and modem. Sometimes one works better than the other though. QMI tends to have lower latency, but MBIM can support higher transmission speeds, but only noticeable on the top end.

The upload on a 7455 was never spectacular for me. The iphone6 has a MUCH better modem and antenna array, but we should hope you would get better than that for upload. Honestly if you need upload, then I would look at the 7411 as it has proven to uplink aggregate 2x. But I digress.

[MattB29]

Since you are experiencing the same thing over several identical setups I would guess the problem is with Visible. Verizon is no friend to sim cards used in anything but approved phones and hotspots.

[lpr600]

Since you are experiencing the same thing over several identical setups I would guess the problem is with Visible. Verizon is no friend to sim cards used in anything but approved phones and hotspots.

Sorry, Visible is my mobile provider. The cards are on an "unlimited government" plan with Verizon. That same agency has a boatload of Sierra Wireless vehicle modems and they have no complaints at all, so I'm not convinced that it's VZW limiting the upload speed based on the device. I really think there is something with the way the modem is configured, but we don't know enough to really track it down. I find it very difficult to believe that being less than a mile from a site, with essentially flat terrain and a MIMO antenna externally mounted with a textbook groundplane can't deliver more than 0.3Mbps.

[punkguy00017]

U tried changing ttl settings?

[lpr600]

U tried changing ttl settings?

I know what TTL is, but I don't think that applies in this case. I've looked around here and that seems to be something related to using the device as a hotspot/router on a mobile phone plan? We're not doing that - these are legitimately activated SIM cards with Verizon Wireless on a "government unlimited/public safety" data plan. They know they are being used for data only, and that they are in modems. The agency has 100 or more modems in vehicles on the same plan.

I am really inclined to think its something to do with the modem, programming, or some combination, but I don't know where to start to prove/disprove any of these theories. The best speed we've seen with these devices, ANYWHERE, is ~2.5Mbps. Never anything higher than that, and most times at their actual install locations speeds NEVER exceed about 0.3Mbps. At one location - in the middle of the City - the modem won't even connect most of the time. When it does, for just a few minutes, we get repeated speedtest-cli upload speeds of 0.17Mbps, every single time, it never changes, like some parameter or something is capping it.

[punkguy00017]

Ok I understand I had visible and I enabled ttl to 65 then went into custom rules and put in the following

#start TTL rules
iptables -t mangle -I POSTROUTING -j TTL --ttl-set 64
iptables -t mangle -I PREROUTING -j TTL --ttl-set 64
ip6tables -t mangle -A POSTROUTING -o wwan0 -j HL --hl-set 65
ip6tables -t mangle -I PREROUTING -i wwan0 -j HL --hl-set 65
#end TTL rules


Doing this sped up my download/upload.
Give it a try can't hurt right?

[Dr-BroadBand]

Captain obvious here, can you swap the SIM with a working modem.

[mtl26637]

Additionally, our phone testing is with Visible which should be the same as the VZW modems, but throwing that our there for full transparency.

Not quite sure if your testing setup (phone setup) contains the visible sim or if the new equipment but either way SIM provisioning (provider) will matter. Even though Visible uses verizon's network doesn't mean it is same. Not an expert but pretty sure the connection to the 'physical' cell tower is the only thing common. Once the packets get to the tower they are routed from there depending on the SIM cards "APN" settings and further throttled/deprioritized. Visible SIM vs. Verizon SIM will not be same speeds. Vzw gonna give their customers better service than MVNO provider for sure. I would doubt it is a problem with the equipment but rather with the provisioning of the SIM / modem combo. The TTL settings can affect whether you will get full connection speeds but that is not the point I suppose since that contains a bit of trickery. You might try any other SIM and/or even change TTL just to try to track down the problem. I would put my money on the SIM/provider being the culprit.

edit: Reading your last post, it does seem the vzw sim is in the new equipment. You still might try the visible sim or a different provier's sim in the equipment just to narrow it down. Sounds like VZW doesn't like the IMEI of your new setups. Even if they say its not on their end of things, depending on the situation, I wouldn't put too much faith in that based on past experiences with tech support. Not until you get hours deep in tech support and to the real techs anyway.

[lpr600]

Well, after a great, great deal of testing and experimentation, we have found the cause: Ubuntu 18.04.

We put one of the SIMs in a brand new Sierra Wireless LX40 and it was 20Mbps down/20Mbps up. Back in the Ubuntu device it was 20Mbps down, 3Mbps up. We tried different modems, different devices, antennas, and could never get the Ubuntu device above 3Mbps.

While we waited for a Windows 10 bootable USB to be created we upgraded Ubuntu 18.04 to 20.04 and what do you know? 20 down, 20 up.

Just to make sure it wasn't a fluke we put 18.04 back on - 20 down, 3 up.

So there's something with Ubuntu 18.04. What it is, I don't know, but we have full rate now which is a relief!

[Didneywhorl]

Crazy!

Thanks for the follow up, that is good to know for sure.

[kperkin1]

I'm running ubuntu 20.04 LTS, with Sierra Wireless EM7455, and having slow speeds. Any suggestions would be appreciated.
$ mmcli -m 0 === shows this below...
--------------------------------
Status | lock: sim-pin2
| unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (3), sim-puk2 (10)
| state: connected
| power state: on
| access tech: lte
| signal quality: 68% (recent)

root@PLIOT03:/usr/local/bin# speedtest-cli --single --simple
Ping: 244.059 ms
Download: 1.08 Mbit/s
Upload: 0.05 Mbit/s