wg1608/quec em12 - randomly powers down

[user232]

Been using my wg1608 GoldenOrb_2023-07-26 ( OpenWrt 22.03-SNAPSHOT )/em12 for a couple years and recently been losing connection randomly once or twice a day. It seems the router stays powered on (can still log into rooter etc) but the modem itself powers down. I checked the debug area right after it powered down and the following was shown:

Code: Select all

POWERED DOWN
M","SM"
TIMEOUT ERROR
Wed Jan 17 13:24:07 2024 : /dev/ttyUSB2 Can't open device /dev/ttyUSB2.
  
Wed Jan 17 13:24:07 2024 : /dev/ttyUSB2 Can't open device /dev/ttyUSB2.
  

It eventually reloads the modem after a couple minutes and reconnects to the internet with no issue. The router/modem isn't overheating etc.
Anyone experience this before or have any idea how I could further diagnose the issue?

[Didneywhorl]

Might need a reflash and factory reset, maybe update in firmware, but it might just be failing .. maybe.

[user232]

Cheers, been reading more of the posts including a couple you replied to and I got some ideas on what data I can gather that may be helpful.
Will be monitoring the kernel logs and others today and report back if I find any patterns

[user232]

Might need a reflash and factory reset, maybe update in firmware, but it might just be failing .. maybe.

I checked the kernel after it just recently powered down again, I notice that it throws a similar error each time. Does this excerpt provide any clues or could it just be too many possibilities and warrant a complete replacement? (Copied it right after the em12 powered down and also included some of the kernel above it (unsure how much to include)

p.s. the 0Ω resistor is in place on the pcb's watchdog jumper

Code: Select all

[   16.787243] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   16.787416] ieee80211 phy1: copying sband (band 1) due to VHT EXT NSS BW flag
[   16.931977] PPP generic driver version 2.4.2
[   16.939360] NET: Registered protocol family 24
[   16.948836] usbcore: registered new interface driver qmi_wwan
[   16.959128] usbcore: registered new interface driver rndis_host
[   16.985901] usbcore: registered new interface driver sierra
[   16.991866] usbserial: USB Serial support registered for Sierra USB modem
[   17.003058] usbcore: registered new interface driver sierra_net
[   17.009400] mt7615e 0000:02:00.0: HW/SW Version: 0x65326363, Build Time: 2005101654debug
[   17.009400] 
[   17.022381] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[   17.030318] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[   17.078831] cdc_mbim 2-1.4:1.8: cdc-wdm0: USB WDM device
[   17.085943] cdc_mbim 2-1.4:1.8 wwan0: register 'cdc_mbim' at usb-1e1c0000.xhci-1.4, CDC MBIM, c6:df:85:2f:5c:fc
[   17.096507] usbcore: registered new interface driver cdc_mbim
[   17.114878] usbcore: registered new interface driver option
[   17.120718] usbserial: USB Serial support registered for GSM modem (1-port)
[   17.128492] option 2-1.4:1.0: GSM modem (1-port) converter detected
[   17.135160] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB0
[   17.142895] option 2-1.4:1.1: GSM modem (1-port) converter detected
[   17.149874] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB1
[   17.157761] option 2-1.4:1.2: GSM modem (1-port) converter detected
[   17.164699] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB2
[   17.172338] option 2-1.4:1.3: GSM modem (1-port) converter detected
[   17.179547] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB3
[   17.195441] usbcore: registered new interface driver qcserial
[   17.201647] usbserial: USB Serial support registered for Qualcomm USB modem
[   17.226445] usbcore: registered new interface driver rt2800usb
[   17.233334] kmodloader: done loading kernel modules from /etc/modules.d/*
[   17.478069] mt7615e 0000:02:00.0: N9 Firmware Version: 7663mp1827, Build Time: 20200512032642
[   17.486599] mt7615e 0000:02:00.0: Region number: 0x4
[   17.491599] mt7615e 0000:02:00.0: Parsing tailer Region: 0
[   17.500742] mt7615e 0000:02:00.0: Region 0, override_addr = 0x00118000
[   17.507368] mt7615e 0000:02:00.0: Parsing tailer Region: 1
[   17.513902] mt7615e 0000:02:00.0: Parsing tailer Region: 2
[   17.519847] mt7615e 0000:02:00.0: Parsing tailer Region: 3
[   17.525837] mt7615e 0000:02:00.0: override_addr = 0x00118000, option = 3
[   32.015613] mtk_soc_eth 1e100000.ethernet eth0: Link is Down
[   32.037636] mtk_soc_eth 1e100000.ethernet eth0: configuring for fixed/rgmii link mode
[   32.045997] mtk_soc_eth 1e100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   32.050600] mt7530 mdio-bus:1f lan1: configuring for phy/gmii link mode
[   32.061708] 8021q: adding VLAN 0 to HW filter on device lan1
[   32.070629] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   32.078620] br-lan: port 1(lan1) entered blocking state
[   32.083940] br-lan: port 1(lan1) entered disabled state
[   32.090510] device lan1 entered promiscuous mode
[   32.095174] device eth0 entered promiscuous mode
[   32.132458] mt7530 mdio-bus:1f lan2: configuring for phy/gmii link mode
[   32.140138] 8021q: adding VLAN 0 to HW filter on device lan2
[   32.150895] br-lan: port 2(lan2) entered blocking state
[   32.156195] br-lan: port 2(lan2) entered disabled state
[   32.164541] device lan2 entered promiscuous mode
[   32.186045] mt7530 mdio-bus:1f lan3: configuring for phy/gmii link mode
[   32.193584] 8021q: adding VLAN 0 to HW filter on device lan3
[   32.204202] br-lan: port 3(lan3) entered blocking state
[   32.209636] br-lan: port 3(lan3) entered disabled state
[   32.217926] device lan3 entered promiscuous mode
[   32.239063] mt7530 mdio-bus:1f lan4: configuring for phy/gmii link mode
[   32.246330] 8021q: adding VLAN 0 to HW filter on device lan4
[   32.256796] br-lan: port 4(lan4) entered blocking state
[   32.262197] br-lan: port 4(lan4) entered disabled state
[   32.269801] device lan4 entered promiscuous mode
[   32.294634] mtk_soc_eth 1e100000.ethernet wan: PHY [mdio-bus:04] driver [MediaTek MT7530 PHY] (irq=POLL)
[   32.304250] mtk_soc_eth 1e100000.ethernet wan: configuring for phy/rgmii-rxid link mode
[   34.140405] mt7530 mdio-bus:1f lan1: Link is Up - 100Mbps/Full - flow control off
[   34.148031] br-lan: port 1(lan1) entered blocking state
[   34.153244] br-lan: port 1(lan1) entered forwarding state
[   34.160203] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   34.900919] br-lan: port 5(wlan0) entered blocking state
[   34.906326] br-lan: port 5(wlan0) entered disabled state
[   34.912606] device wlan0 entered promiscuous mode
[   34.917947] br-lan: port 5(wlan0) entered blocking state
[   34.923329] br-lan: port 5(wlan0) entered forwarding state
[   35.157875] br-lan: port 5(wlan0) entered disabled state
[   42.550101] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   42.556980] br-lan: port 5(wlan0) entered blocking state
[   42.562755] br-lan: port 5(wlan0) entered forwarding state
[  126.254588] 8021q: adding VLAN 0 to HW filter on device wwan0
[ 1184.302503] mt7530 mdio-bus:1f lan1: Link is Down
[ 1184.307422] br-lan: port 1(lan1) entered disabled state
[ 1187.368775] mt7530 mdio-bus:1f lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[ 1187.376394] br-lan: port 1(lan1) entered blocking state
[ 1187.381611] br-lan: port 1(lan1) entered forwarding state
[10262.095307] usb 2-1.4: USB disconnect, device number 3
[10262.100953] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
[10262.109715] option 2-1.4:1.0: device disconnected
[10262.116078] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
[10262.124756] option 2-1.4:1.1: device disconnected
[10262.130578] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
[10262.139184] option 2-1.4:1.2: device disconnected
[10262.145296] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
[10262.153880] option 2-1.4:1.3: device disconnected
[10262.159322] cdc_mbim 2-1.4:1.8 wwan0: unregister 'cdc_mbim' usb-1e1c0000.xhci-1.4, CDC MBIM
[10262.171143] xhci-mtk 1e1c0000.xhci: WARN Cannot submit Set TR Deq Ptr
[10262.177600] xhci-mtk 1e1c0000.xhci: A Set TR Deq Ptr command is pending.
[10269.673101] usb 2-1.4: new SuperSpeed Gen 1 USB device number 4 using xhci-mtk
[10269.703851] usb 2-1.4: config 1 has an invalid interface number: 8 but max is 5
[10269.711138] usb 2-1.4: config 1 has an invalid interface number: 9 but max is 5
[10269.718465] usb 2-1.4: config 1 has an invalid interface number: 9 but max is 5
[10269.725925] usb 2-1.4: config 1 has no interface number 4
[10269.731299] usb 2-1.4: config 1 has no interface number 5
[10269.742151] option 2-1.4:1.0: GSM modem (1-port) converter detected
[10269.749399] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB0
[10269.757925] option 2-1.4:1.1: GSM modem (1-port) converter detected
[10269.765259] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB1
[10269.773786] option 2-1.4:1.2: GSM modem (1-port) converter detected
[10269.780823] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB2
[10269.789482] option 2-1.4:1.3: GSM modem (1-port) converter detected
[10269.796692] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB3
[10269.824508] cdc_mbim 2-1.4:1.8: cdc-wdm0: USB WDM device
[10269.831965] cdc_mbim 2-1.4:1.8 wwan0: register 'cdc_mbim' at usb-1e1c0000.xhci-1.4, CDC MBIM, c6:df:85:2f:5c:fc
[10364.659790] 8021q: adding VLAN 0 to HW filter on device wwan0
[34894.792850] mt7530 mdio-bus:1f lan1: Link is Down
[34894.798461] br-lan: port 1(lan1) entered disabled state
[60192.854609] mt7530 mdio-bus:1f lan1: Link is Up - 1Gbps/Full - flow control rx/tx
[60192.862189] br-lan: port 1(lan1) entered blocking state
[60192.867469] br-lan: port 1(lan1) entered forwarding state
[73238.953937] neighbour: arp_cache: neighbor table overflow!
[79985.783336] usb 2-1.4: USB disconnect, device number 4
[79985.789056] option1 ttyUSB0: GSM modem (1-port) converter now disconnected from ttyUSB0
[79985.797505] option 2-1.4:1.0: device disconnected
[79985.803482] option1 ttyUSB1: GSM modem (1-port) converter now disconnected from ttyUSB1
[79985.812048] option 2-1.4:1.1: device disconnected
[79985.818881] option1 ttyUSB2: GSM modem (1-port) converter now disconnected from ttyUSB2
[79985.827314] option 2-1.4:1.2: device disconnected
[79985.833145] option1 ttyUSB3: GSM modem (1-port) converter now disconnected from ttyUSB3
[79985.841637] option 2-1.4:1.3: device disconnected
[79985.847185] cdc_mbim 2-1.4:1.8 wwan0: unregister 'cdc_mbim' usb-1e1c0000.xhci-1.4, CDC MBIM
[79985.858627] xhci-mtk 1e1c0000.xhci: WARN Cannot submit Set TR Deq Ptr
[79985.865084] xhci-mtk 1e1c0000.xhci: A Set TR Deq Ptr command is pending.
[79985.872229] xhci-mtk 1e1c0000.xhci: WARN Cannot submit Set TR Deq Ptr
[79985.878958] xhci-mtk 1e1c0000.xhci: A Set TR Deq Ptr command is pending.
[79985.886619] xhci-mtk 1e1c0000.xhci: WARN Cannot submit Set TR Deq Ptr
[79985.890501] do_page_fault(): sending SIGSEGV to gcom for invalid write access to 0041c000
[79985.893061] xhci-mtk 1e1c0000.xhci: A Set TR Deq Ptr command is pending.
[79985.901374] epc = 00403735 in
[79985.907875] xhci-mtk 1e1c0000.xhci: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state.
[79985.907884]  comgt[400000+6000]
[79985.923345] ra  = 004036e9 in comgt[400000+6000]
[79993.088527] usb 2-1.4: new SuperSpeed Gen 1 USB device number 5 using xhci-mtk
[79993.119120] usb 2-1.4: config 1 has an invalid interface number: 8 but max is 5
[79993.126435] usb 2-1.4: config 1 has an invalid interface number: 9 but max is 5
[79993.134192] usb 2-1.4: config 1 has an invalid interface number: 9 but max is 5
[79993.141945] usb 2-1.4: config 1 has no interface number 4
[79993.147381] usb 2-1.4: config 1 has no interface number 5
[79993.160220] option 2-1.4:1.0: GSM modem (1-port) converter detected
[79993.167192] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB0
[79993.175635] option 2-1.4:1.1: GSM modem (1-port) converter detected
[79993.183132] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB1
[79993.191977] option 2-1.4:1.2: GSM modem (1-port) converter detected
[79993.199508] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB2
[79993.207951] option 2-1.4:1.3: GSM modem (1-port) converter detected
[79993.215213] usb 2-1.4: GSM modem (1-port) converter now attached to ttyUSB3
[79993.249624] cdc_mbim 2-1.4:1.8: cdc-wdm0: USB WDM device
[79993.257012] cdc_mbim 2-1.4:1.8 wwan0: register 'cdc_mbim' at usb-1e1c0000.xhci-1.4, CDC MBIM, c6:df:85:2f:5c:fc

[Didneywhorl]

Hard to say. It's not really jumping out at me.

[user232]

Thank you for your help either way @Didneywhorl , always appreciated.
I am not too well versed on what parts of the kernel/sys logs I should share and didn't want to bog you down with the entire logs in case they weren't helpful.

Tried updating to the latest rooter pre-release and sadly the issue still persists.
I haven't tried updating the firmware on the em12g, a couple years ago I seen some people on the quec forums raising concerns/issues about the fw back then and so I figured I shouldn't try messing with it. I still am running EM12GPAR01A20M4G_01.001.01.001. Maybe it is time to break out the sled and try flashing the latest FW as a last resort.

Will update in the future if I figure anything out in the mean time.

[Didneywhorl]

Have you also replaced the power supply with a new one that has at least 2.5A of supply?

[user232]

Have you also replaced the power supply with a new one that has at least 2.5A of supply?

Yessir, I actually never even used the stock adapter that came with the wg1608 thanks to seeing your recommendations on old posts a couple years ago.
Was using a 12v 5a for the past 2-3 years and today I bought a new adapter after reading a suggestion from a gentleman on the whirlpool rooter thread.
Sadly the problem persists even with the new 12v 8a, overkill but atleast this one runs cooler.
I am still in the process of trying to figure this out and will update if I find anything in the mean time.

[roddel007]

do you have heatsink installed?

[user232]

Yes ~2 years ago I removed the lid of the 1608 and put a finned cpu heatsink on the em12g's metal shield held in place with white thermal compound. It reduced the temperature by some but I doubt its nesecary in the environment I have it in.
I do not think the resetting is due to temperature, I monitor the temps and they are low. It is also slightly colder than usual atm where I live.

pic of board/hs: https://i.imgur.com/NxaFmLL.jpeg

[BillA]

Yes ~2 years ago I removed the lid of the 1608 and put a finned cpu heatsink on the em12g's metal shield held in place with white thermal compound. It reduced the temperature by some but I doubt its nesecary in the environment I have it in.
I do not think the resetting is due to temperature, I monitor the temps and they are low. It is also slightly colder than usual atm where I live.

pic of board/hs: https://i.imgur.com/NxaFmLL.jpeg

Your issue may be caused by three factors.

1. Router software (which you already reflashed so that's out)
2. Router hardware failure
3. Modem hardware failure

I would flash and reseat the modem to see if that helps.
If still restarting, then the quickest way to narrow down between the router and modem hardware, is by installing another modem (doesn't have to the same make/model).
If it restarts, then it's likely a router failure.
If it doesn't restart, then it's likely the Quectel EM12 modem failure.

Hope that helps narrowing down the issue.

[user232]

Good advice @BillA . I've not flashed the em12 before since it's been problem free. I will see if I can do it with qfirehose and/or try to find m2 sled I bought years back.

Unfortunately I don't have any other router besides the 1608 nor do I have another module. Looks like most options of buying one for test purposes costs enough to warrant just buying a more modern quectel module.

I've been out of the loop in regards to what is new and what works, will do some research.
Will update with what happens. Thanks again fellas.

[BillA]

Good advice @BillA . I've not flashed the em12 before since it's been problem free. I will see if I can do it with qfirehose and/or try to find m2 sled I bought years back.

Unfortunately I don't have any other router besides the 1608 nor do I have another module. Looks like most options of buying one for test purposes costs enough to warrant just buying a more modern quectel module.

I've been out of the loop in regards to what is new and what works, will do some research.
Will update with what happens. Thanks again fellas.

Ha! You're not alone out of the loop, I've taken off a couple of years too, and just catching up with the latest. heh
See if you can maybe borrow a spare modem or router for testing purposes from someone near you, it would narrow down the issue quickly.
At this point it's a toss-up, could be the modem or the router, but hopefully not both.

[roddel007]

I'm done with zbt routers after years. for a 5G module the processor aint powerful enough. I switch to openwrt x86 mini pc with 5G enclosure. Works great and i switch to ecm mode rather than qmi or mbim been stable.

[user232]

Update: Seems like the issue is resolved. Here is what I did:
1) Asked on quectel forums for the latest firmware and got a circa 2023 one EM12GPAR01A21M4G_01.203.01.203.zip (newer than the ones on that github page I think)
2) used QFirehose to flash the em12 while it was installed in my wg1608. I had to attempt this like 4-5 times until it completed, it reset itself during flashing attempts.
3) I then did a factory reset on the wg1608 and re-installed the rooter firmware again
------
It has been 10+ hours without the modem auto resetting which makes me think it has been resolved. If it occurs again I will update.
I cannot be 100% sure if the em12-g fw upgrade is what did it because I am not 100% sure if I did a factory reset when installing the rooter firmware in the past, I think I may have just installed it using the update method on the rooter ui (without pressing the reset button on the router and reverting it to oem chinese fw)

Lastly, I wanted to mention that before doing the above I emailed quectel support and asked them about quectel forum post I read where a technician mentioned to someone about using quectel software to be able to read quectel module error logs. They asked for my company info etc and I told them I was a consumer using this in a router for internet (I assumed they would ignore me after telling them this) I was surprised when one of their technicians emailed me back and was incredibly helpful giving me software to use and telling me how to use it/to ask him if I had any issues.

Thanks again for all the help and advice fellas, will update if anything changes.
Cheers.

[BillA]

Update: Seems like the issue is resolved. Here is what I did:

Thanks again for all the help and advice fellas, will update if anything changes.
Cheers.

I'm glad that it seems to be working, at least for now.
For random modem/router issues it's always a good idea to start with a fresh baseline/factory reset state.
If it's still failing, replacing the power supply first then the modem with a known good one, will usually tell you whether it's a power, modem, or router issue.

If I may ask, where did you get the QFirehose flasher for the EM12G?
Don't remember being part of the regular firmware package, but I may be wrong.

[user232]

If I may ask, where did you get the QFirehose flasher for the EM12G?

Yessir it comes already bundled/installed with rooter and I used the fw that quectel emailed me. I then I used this method with a usb drive and putty to execute QFirehose command on my 1608 16mb variant.

modemsandmen .pdf guide on qfirehose: https://drive.google.com/file/d/1kofvBR ... sp=sharing

Was incredibly easy, only hiccup was my em12g would reset itself 10-20s into executing QFirehose so it required a few attempts.