MC7354 inside GX450 keeps loosing service

[axel1]

Hi all,

I've been testing out an AT&T 15GB hotspot plan on my Sierra GX450 (a vehicular router with an integrated MC7354).

I've got it up and running on the AT&T network, but I keep loosing service after an inconsistent amount of time. I'm running on a temporary antenna here on my desk, and signal levels are fine, but about five times in the past 6 hours I've lost connectivity with a "no service" status on the modem. I'm not moving the radio or otherwise changing signal, so I'm not sure what's going on. As far as I can tell, the only way to recover from this state is a hard reboot of the router.

Here's what my status page looks like when everything is good:


And here's when things have died:


As you can see there's almost no change in the signal levels, so something non-RF is going on here I think. MC7354 is on the latest AT&T firmware so I don't think that's the issue.

I'd love to know any good troubleshooting steps to figure out what's going on. I'm guessing it's likely a configuration issue somewhere with this new service. I was previously using this same router on US Mobile and never saw this issue. All I had to do for the switch to AT&T was update the APN to "Broadband" and things started working, so I can't think of what else could be going on.

Help is greatly appreciated!

[axel1]

Here's the output from the debug information tool in the router's web gui (with redactions) when the router gets stuck in the "no service" state:

Code: Select all

ATI
Manufacturer: Sierra Wireless, Incorporated
Model: MC7354
Revision: SWI9X15C_05.05.58.00 r27038 carmd-fwbuild1 2015/03/04 21:30:23
MEID: 35xxx
ESN: 128xxxxxxxx, 804xxxxx
IMEI: 359xxxxxxxxxxxxxxx
IMEI SV: 18
FSN: J86xxxxxxxxxxxx
+GCAP: +CGSM


OK
AT!GSTATUS?
!GSTATUS: 
Current Time:  1461Temperature: 37
Bootup Time:   0Mode:        ONLINE         
System mode:   LTE        PS state:    Attached     
LTE band:      B4     LTE bw:      10 MHz  
LTE Rx chan:   2000LTE Tx chan: 20000
EMM state:     Registered     Normal Service 
RRC state:     RRC Idle       
IMS reg state: No Srv  

RSSI (dBm):    -53Tx Power:    0
RSRP (dBm):    -81TAC:         486C (18540)
RSRQ (dB):     -10Cell ID:     0478E217 (75031063)
SINR (dB):     10.8


OK

Not sure if that helps with the troubleshooting at all. I'm guessing there's some AT commands I can run to get some more info as well.

[Didneywhorl]

RRC State: Idle usually means the network is letting you login but is blocking your data, especially since you have a PS State: Attached.

Your SINR (Signal to noise ratio) isn't great. You'll want that to be closer to 20. https://thewirelesshaven.com/cellular-signal-guide/

Oddly though, all other stats are great. RSRP, RSRQ, RSSI.

If rebooting the modem fixes it it may be an ATT issue. Have you tried testing your connection over a simple VPN or proxy?

https://ngrok.com/

If you can use a proxy or VPN and it doesn't happen, then that is another sign it's ATT causing your issues, not your router.

[axel1]

I also noticed SINR being lower compared to the other signal levels. I figure there may be a bit of local interference in the area. I'm also not testing this on the world's greatest antenna, but it should be perfectly serviceable for a downtown location with sites all over the place. (Laird DBA series)

I've noticed now after letting the router sit overnight that the connection does eventually come back after ~ 15 minutes or so. But then it'll do the same thing a little later. Very strange.

Know of any extra logging or commands I can check out to better see what's going on? I think I can access the MC7354's AT terminal via the GX450's web GUI.

[Didneywhorl]

AT!GSTATUS? is really your best command for overall info. That and any system log the router may have.

If you have a lot of cell towers with the same bands in your area, that will explain the noise. It's called cellular band cross-talk. Your modem is hooking up to one tower and the other towers are broadcasting in the same frequency range and causing noise at your radios receiver.

[axel1]

So here's that command again when I'm in the no service state. This time it shows as RRC Connected but IMS as No Srv.

Code: Select all

AT!GSTATUS?
!GSTATUS: 
Current Time:  3094Temperature: 39
Bootup Time:   0Mode:        ONLINE         
System mode:   LTE        PS state:    Attached     
LTE band:      B4     LTE bw:      10 MHz  
LTE Rx chan:   2000LTE Tx chan: 20000
EMM state:     Registered     Normal Service 
RRC state:     RRC Connected  
IMS reg state: No Srv  

RSSI (dBm):    -50Tx Power:    0
RSRP (dBm):    -77TAC:         486C (18540)
RSRQ (dB):     -11Cell ID:     0478E217 (75031063)
SINR (dB):      5.4


OK

I've enabled some extra logging on the GX450 so we'll see if anything stands out from those the next time it happens.

[axel1]

So here's the log from the most recent connection drop.

Here's the two entries prior to the connection failing. Just good signal reports and nothing abnormal.

Code: Select all

Jun 22 16:05:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-51, RSRQ=-13, RSRP=-75, SINR=5.8
Jun 22 16:20:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-50, RSRQ=-12, RSRP=-75, SINR=10.1

Then the connection drops at 16:23Z. The ping test warning is the first thing I see.

Code: Select all

Jun 22 16:23:04 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:23:04 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - No Service
Jun 22 16:23:04 notice ALEOS_LINKMGMT_linkmon: New Link Status - No Service
Jun 22 16:23:10 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:23:10 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:23:10 UTC
Jun 22 16:23:11 alert ALEOS_CONNECTMGMT: IPV4 Disconnect Handled
Jun 22 16:23:21 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:23:21 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:23:21 UTC
Jun 22 16:28:03 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 5 minutes ]
Jun 22 16:33:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 10 minutes ]
Jun 22 16:34:44 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:35:58 notice ALEOS_LINKMGMT_linkmon: Current: RSSI=-50, RSRQ=-12, RSRP=-75, SINR=9.5
Jun 22 16:38:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 15 minutes ]
Jun 22 16:43:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 20 minutes ]
Jun 22 16:46:25 warning ALEOS_LINKMGMT_servicemon:  Ping Test failed to reach 1.1.1.1, Cellular is not connected
Jun 22 16:47:10 warning ALEOS_EVENTS_dpRptGen: No GNSS Fix. Quality Indicator is 0 and 0 Satellites in Use. 0 seconds passed without fix.
Jun 22 16:47:10 warning ALEOS_EVENTS_dpRptGen: monitorFix : Lost fix at: Tue, 22 Jun 2021 16:47:10 UTC
Jun 22 16:48:04 notice ALEOS_LINKMGMT_linkmon: Network Watchdog - No reboot, Network Watchdog disabled by application [ 25 minutes ]

The fact that I'm also getting a GNSS failure at the same time (I do have a GPS antenna connected and outside) makes me wonder if the module is faulting out somehow. Not sure how I would test that further though.

And here's the log sequence once things come back up:

Code: Select all

Jun 22 16:56:25 notice ALEOS_LINKMGMT_linkmon: Radio Connection State - Connected
Jun 22 16:56:25 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Link Down
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: *** Network Connection successful. WAN IP [Cellular] : 10.243.xx.xxx
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: *** IP UP 10.243.xx.xxx : wwan0
Jun 22 16:56:33 notice ALEOS_CONNECTMGMT: Setting WWAN MTU size to 1430
Jun 22 16:56:34 notice ALEOS_CONNECTMGMT: GDNS is restarting dnsmasq
Jun 22 16:56:45 notice ALEOS_LINKMGMT_linkmon: New Link Status - Network Ready
Jun 22 16:56:45 warning ALEOS_SERIAL_HMC: not in tcp mode current mode 0 
Jun 22 16:56:50 alert ALEOS_LAN: MTU Source Auto - Set DHCP option 26 MTU: 1430
Jun 22 16:56:50 alert ALEOS_LAN: MTU Source Auto - Ignore Opt 26 MTU size 1500

[Didneywhorl]

Your SINR levels fluctuate a LOT. All the other stats seem to be pretty steady.

I'm not sure on the GPS stuff.

[axel1]

Part of the SINR fluctuation is likely band changes. I see it regularly switch between bands 2, 4, and 17, among others. However I don't think that's the issue with my connection drops.

I'm not sure what else to test at this point. AT&T support will be useless I'm sure. Perhaps the Sierra Wireless forums might be able to shed some light.

[mtl26637]

Dont know much about your specific modem, however, isn't it usually preferred to use 'generic' firmware rather than 'carrier' firmware? I'll let someone else chime in on that but wanted to throw it out there in case it was overlooked. You would think that the carrier firmware would be the way to go but that isn't always the case.

[axel1]

I'm giving the generic firmware a shot now. We'll see if it makes any difference.

[axel1]

No dice. Generic firmware installed and the same issue just happened again. Same exact log entries as before too.

GPS does seem to die at the same time as cell service. Maybe the entire modem module is experiencing some kind of fault? No clue how I'd even begin troubleshooting that.

[mtl26637]

Couldn't be that easy right. Unfortunately the logs don't help much in showing the problem. Possible lack of supply power causing it to randomly drop out? Hopefully someone else will jump in with some additional info to try too.

[axel1]

So I started plotting ping data last night, just to get a feel for how the modem was behaving. The following graph is ping times from the last ~12 hours, where the big triangles are loss of connection (average ping time slowly increasing as no pings make it through)



Notice:

• How the losses of connection are identical in duration and time.
• How after 2:30 this morning they stopped

Very strange behavior. I'll be watching it again today just to see if things are magically fixed or if the issue still persists.

[Didneywhorl]

If the device is officially registered to the network, then the official firmware is usually best, but not always. It may be worth trying, and also making sure the firmware is up to date. With the 7354, I believe it only holds one image at a time. If you need the latest, I have them on hard drive.

[axel1]

I've got the latest firmware I believe, both the ATT & the generic ones. It actually has enough space to store two images at a time which is handy. SWI9X15C_05.05.58.00 is what I've got, I downloaded it directly from Sierra's site.

Also more drops today, seems that maybe I just got lucky last night. Or it's a cell issue related to congestion/usage on the specific cell the radio is talking to.

[axel1]

Well I now have a Sierra MP70 on the way thanks to a lowball eBay offer, so I'll have a second modem (also with an MC7354) to compare with once that arrives. Still, it's going to be a few weeks before it shows up so I'd like to try and figure out what the heck is going on here in the meantime.

[Adm1jtg]

I havent read through all the posts here but have you tried locking to just one band? If it is a congestion issue some bands get more congested then others.

[axel1]

Unfortunately I can't find a way into the AT terminal for the modem itself to test that. The GX450 has band selections but they're more general for regions and modes as opposed to specific bands.

Not a bad theory. Seems like the radio is almost always on Band 4 so trying a different one could yield different results.

[axel1]

So again over the past day, lots of connection drops throughout the day and then absolutely zero connection drops between midnight and 9AM. Then they start up again. Same thing happened the day before.



At this point I think it must be some kind of local issue with the cell sites near me. I can't think of any reason it would work fine all night and then start right back up with the hourly drops.

[Didneywhorl]

I think your right. I wonder if the tower is freaking out under congestion load.