Router Locking Up

[Oblivious]

I've been having ongoing issues with my network, which seems to be related to the invisagig or the cell network. If you scroll to the last few posts in the following forum thread it appears that the modem is handing out DHCP leases when the modem loses it signal causing my LAN devices to lose connective to other LAN devices:
https://forums.pfsense.org/topic/186210 ... 9060365703

Note that I have 2 WAN, the primary WAN setup through a wireless ISP and the secondary using the invisagig. Even when the primary WAN connection is working, whenever my network starts having connectivety issues it appears that the gateway IP of the invisagig changes to 192.0.0.1. Again, then this happens LAN devices (like my PC) lose connectivy to other devices on the LAN, not just interenet. Please provide any suggestions you may have to check down this issue.

[hazarjast]

I've been having ongoing issues with my network, which seems to be related to the invisagig or the cell network. If you scroll to the last few posts in the following forum thread it appears that the modem is handing out DHCP leases when the modem loses it signal causing my LAN devices to lose connective to other LAN devices:
https://forums.pfsense.org/topic/186210 ... 9060365703

Note that I have 2 WAN, the primary WAN setup through a wireless ISP and the secondary using the invisagig. Even when the primary WAN connection is working, whenever my network starts having connectivety issues it appears that the gateway IP of the invisagig changes to 192.0.0.1. Again, then this happens LAN devices (like my PC) lose connectivy to other devices on the LAN, not just interenet. Please provide any suggestions you may have to check down this issue.

Hello Oblivious,
For fastest support turnaround we always recommend using the official InvisaGig Support contact form here: https://invisagig.com/support/. That being said, we are happy to provide support through this forum especially if it will help other InvisaGig PFSense users in the process.

TL;DR
My gut here is that the gateway pinger under pfsense is wreaking havoc if/when your IG is going between 192.168.225.x and 192.0.0.x. due to its default IPPT configuration. There are changes you can make to improve the situation with IPPT or simply disable it if you don't mind the additional NAT layer. All of this explained in detail if you read on

In addressing some of your issues it will first be good to shed some light on why you are seeing a 192.0.0.x address on your WAN interface even though the default UI access IP is 192.168.225.1. The InvisaGig provides IP Passthrough (IPPT) mode to the device connected to its Ethernet port by default and it uses ARP to find the connected device's MAC address which is to be assigned the IPPT address. This means that when the modem successfully connects to the carrier it bridges the IPv4 assigned by the carrier to the Ethernet interface of the InvisaGig.

If your cellular carrier network carrier uses IPv6 for the connection and you do not have a plan which specifically provides a statically assigned IPv4 address, the modem will connect to the carrier over IPv6 but perform IPv4 translation. This translation is referred to as dual stack lite, customer-side translation (CLAT) and will utilize an IPv4 address in the range defined by RFC 6333 which is typically 192.0.0.1 at the modem gateway and 192.0.0.2 assigned to the IG-connected IPPT device. In the USA, this scenario is most typical of T-Mobile consumer cellular plans and/or business plans without the static IP add-on so I am guessing this is your carrier. This is especially the case when connecting over 5G Standalone (5G SA) as that is IPv6 only. Under 5G Non-standalone (5G NSA) and LTE it is possible you may still receive a native, legacy IPv4 address from the carrier grade NAT (CGNAT) DHCP pool instead of a 192.0.0.x address.

Most devices and/or routers connected to the InvisaGig with the default IPPT functionality enabled will not have an issue with 192.0.0.x addressing on their WAN but from time to time we do see some enterprise equipment which has an issue with this address space and/or issues with IPPT bridge creation / MAC autodetection of its WAN port by the IG when reconnecting to the carrier. Depending on the root cause in the case of your PFSense router/firewall there may be a couple of settings which need changing. The first I would suggest would be to set the IPPT MAC address of the PFSense WAN port manually.

You can set the MAC address for IPPT manually by selecting 'Local IP & Multiple Modem Setup' from the IG's main menu. From there select 'Single Unit' (assuming this is your only IG unit) and entering 'N' when asked if you wish to use a custom IP address. You will then enter 'Y' to specify a MAC address and enter the MAC address of the WAN port in the next screen. Alternatively you could simply disable IPPT at this last prompt by leaving the MAC address blank prior hitting 'Enter'. If you disable IPPT then your WAN will then always be assigned an IP in the subnet of your IG (by default, 192.168.225.x).

Reading through your Netgate post, I see you set a VIP for accessing the IG UI when receiving the IPPT address of 192.0.0.x when connected to the carrier. This should not be necessary and it is possible it may be causing some issue. If PFSense is not caching the route to 192.168.225.x after it switches to 192.0.0.x then you can simply add a static route to 192.168.225.1 using the appropriate gateway interface (WAN) that the IG is connected to. Ensure you have the appropriate firewall rule to allow LAN clients to access this subnet as well. For the WAN interface the IG is connected to, you may also need to uncheck the option for blocking private networks since the 192.168.225.1 is in the RFC 1918 space.

You should not need to refuse DHCP leases from either of these ranges as in the IPPT process it is normal for DHCP to assign 192.168.225.x until the modem connects to the carrier over IPv6 at which point DHCP assigns the 192.0.0.x CLAT address. 192.168.225.1 will be the address you should use to access the IG UI regardless of IPPT enablement. I run OPNSense now myself but when I did run PFSense the gateway pinger service typically caused me a headaches with DHCP enabled WAN interfaces. Especially since you're using the IG as failover it might be helpful to change your monitor IP for the pinger to something always 'up' on the internet like 1.1.1.1 or 8.8.8.8. That way your pinger won't go nuts and try to failover to the other gateway if you keep IPPT enabled on the IG and it changes between 192.168.225.x and 192.0.0.x if it is forced to reattach to the tower by the carrier due to outage/maintenance/etc.

As your setup has some additional complexity with VLANs and content filtering based on ASN (pfBlockerNG) I cannot say whether the above suggestions will completely resolve all of your issues, but I believe they should help get your IG WAN behaving in a consistent manner which should create a stable platform from which you can better troubleshoot any remaining issues.

[Oblivious]

Thank you for the detailed post.

For clarity, "The first I would suggest would be to set the IPPT MAC address of the PFSense WAN port manually" I assume means copy the IPPT MAC from the IG ui, then paste it into the "MAC Address" setting of the pfSense WAN interface configuration? (The subtext of that setting says its to used the "spoof" the MAC address of the interface)

Since pfSense supports IPv6, does it make more sense to enable the IPv6 configuration?

Yes, I'm using Google Fi which is T-Mobile

[BillA]

For the WAN interface the IG is connected to, you may also need to uncheck the option for blocking private networks since the 192.168.225.1 is in the RFC 1918 space.

Your post prompted me to read the RFC 1918/Private Network Addressing Wiki, which made me realize that the private address range 10.x.x.x is much easier to say and write then 192.168.x.x. Heh I'll be converting my LAN gateway to the nicer 10.0.0.1

[hazarjast]

Thank you for the detailed post.

For clarity, "The first I would suggest would be to set the IPPT MAC address of the PFSense WAN port manually" I assume means copy the IPPT MAC from the IG ui, then paste it into the "MAC Address" setting of the pfSense WAN interface configuration? (The subtext of that setting says its to used the "spoof" the MAC address of the interface)

Since pfSense supports IPv6, does it make more sense to enable the IPv6 configuration?

Yes, I'm using Google Fi which is T-Mobile

No problem!

No, there should not be any entering of IG MAC address in pfSense; just the oppositive in fact. Apologies that this was not clear. The MAC of the pfSense WAN interface NIC needs to be entered into the IG configuration using the steps indicated. The MAC to use will be the one for the physical NIC port you have configured as the IG WAN under pfSense. You will need to verify that is the correct MAC by checking it in the interface assignment under pfSense or using the pfSense CLI.

No, I would not configure IPv6. If you end up using a home internet product like T-Mobile Home Internet (TMHI) and some sort of IPv6 passthrough, then LAN devices may possibly be able to receive IPv6 from the carrier but in most cases the WAN just receives a /64 from the carrier which would require NAT6 (likely the case with an MVNO like Google Fi). Even if you have TMHI or equivalent and pfSense can support IPv6 passthrough, you will find that the IPv6 addresses have very short address leases from the carrier which makes them not so useful.

Personally, I use a phone SIM and stick with IPv4 letting the modem handle all IPv6 connection/translation on the carrier link.

[Oblivious]

Just letting you know that so far my network has been stable. Thank you for the help!

[hazarjast]

Just letting you know that so far my network has been stable. Thank you for the help!

Fantastic to hear!

When you have a moment, and are confident the issue has been resolved, could you please confirm which specific suggestions you implemented in your configuration (ex. set pfSense WAN MAC in the IG configuration, unchecked the block RFC1918 networks option on the pfSense WAN, changed the gateway monitor IP for pinger, etc.). Confirming the specific configuration changes you made will be a great help to others who find this thread and would also allow us to create a 'pfSense Best Practice' KBA to assist other InvisaGig owners who may face similar issues. TYIA!

[Oblivious]

Sure. I'm going to give it a few days, at least through the weekend. Previously I've thought I'd solved it only to have the issue start back up a few days later. This time though, I'm a lot more optimistic.

[Oblivious]

Another clarification question regarding the IPPT function: Does the IG actually have a MAC, or is more like its relaying the pfSense interface MAC to the carrier (so the IG is more or less transparent)?

The reason I ask, is that prior to my previous question about this I did just copy the IPPT MAC reported by IG and paste it into MAC Address configuration of pfSense. After reading your response, I wasn't sure what the difference was between spoofing the MAC of the IG in the router or setting the pfSense WAN MAC in the IG as you recommended; it all appeared to be working so I let it alone... Well my network went wonky again this morning, so I manually set the IPPT MAC as you suggested. So far, so good.

In other words, I think what you're saying in your explanation is that if the modem losses & regains connection to the carrier it may not re-detect the router interface WAN MAC & I guess that somehow causes pfSense DHCP to break in turn causing my LAN devices to eventually lose comms with each other.

[hazarjast]

Another clarification question regarding the IPPT function: Does the IG actually have a MAC, or is more like its relaying the pfSense interface MAC to the carrier (so the IG is more or less transparent)?

The reason I ask, is that prior to my previous question about this I did just copy the IPPT MAC reported by IG and paste it into MAC Address configuration of pfSense. After reading your response, I wasn't sure what the difference was between spoofing the MAC of the IG in the router or setting the pfSense WAN MAC in the IG as you recommended; it all appeared to be working so I let it alone... Well my network went wonky again this morning, so I manually set the IPPT MAC as you suggested. So far, so good.

In other words, I think what you're saying in your explanation is that if the modem losses & regains connection to the carrier it may not re-detect the router interface WAN MAC & I guess that somehow causes pfSense DHCP to break in turn causing my LAN devices to eventually lose comms with each other.

Yes, all physical network interfaces have a MAC address including the InvisaGig. However, entering that into your WAN config in pfsense sense will only cause problems because the IP Passthrough is looking for a different MAC with which to bind the carrier IP to. The pfsense WAN MAC should be entered into the IG configuration as you have now done.

If the modem loses and regains connection to the carrier you will have set the IPPT MAC binding manually eliminating any oddities with detection via ARP.

[Tech_Junky]

The way I'm reading this is you have 2 ISP WWAN connections being fed into the PF box....

IMEI / MAC will be different for each ISP

I use TMO and 192.0.0.1 is their standard IP you would see on the "WAN" side. It's some sort of IPv6 xlate / tunnel they issue to the connection.
-- I started out with their TMHI option and recently for the past few months built my own modem setup and converted to a phone SIM for $38.50/mo all in unlimited data + 40GB/mo hotspot as I also use the sim in my phone as needed

So, if you have a "dual WAN" you need to feed them into the PF box individually and probably tag them or change the metric to prefer one as primary and one as backup...

Code: Select all

ip r
default via 192.0.0.1 dev wwan0 proto static metric 700 
192.0.0.0/27 dev wwan0 proto kernel scope link src 192.0.0.2 metric 700 
192.168.0.0/24 dev br0 proto kernel scope link src 192.168.0.1 

Not to bog down on subnetting but the /27 is the limited scope provided from the modem/ISP. Now, the question would be what's your other modem providing from the ISP? If they're both sending you the same info that's where the tagging / metric will help. I'm not sure if all modems issue this same IP/subnet as I've only played around with this one so far.

Another option would be to use a dual sim / USB setup or PCIE card that would allow a combined modem/sim setup for a bit more control.

[Oblivious]

Thanks. I'm not quite following your post though. I think you're saying that I need to priortize the routes to each WAN. There are 2 WANs, my primary which is provided by a wireless ISP & the secondary which uses the IG with Google FI (TMobile MVNO). The primary ISP doesn't have a modem, its a Ubiquiti Air device that connects directly to my router wan port (with a POE injector to power the antenna. there might be a modem in the antenna, i don't know but there is nothing I can access) https://uisp.com/us/wireless

In pfSense, you create Gateway Groups which can then be set-up as a balanced connection or failover. I'm set up for failover (ISP goes down, fail over to IG.)

Unfortunately, this problem has occurred twice sense my last post. I've updated my post on the netgate forums with my latest observation. I don't think the issue is with either WAN device, but the status monitoring is a symptom of something preventing all IPs (LAN & WAN) from being found.
https://forum.netgate.com/topic/186210/ ... traffic/25

[hazarjast]

Thanks. I'm not quite following your post though. I think you're saying that I need to priortize the routes to each WAN. There are 2 WANs, my primary which is provided by a wireless ISP & the secondary which uses the IG with Google FI (TMobile MVNO). The primary ISP doesn't have a modem, its a Ubiquiti Air device that connects directly to my router wan port (with a POE injector to power the antenna. there might be a modem in the antenna, i don't know but there is nothing I can access) https://uisp.com/us/wireless

In pfSense, you create Gateway Groups which can then be set-up as a balanced connection or failover. I'm set up for failover (ISP goes down, fail over to IG.)

Unfortunately, this problem has occurred twice sense my last post. I've updated my post on the netgate forums with my latest observation. I don't think the issue is with either WAN device, but the status monitoring is a symptom of something preventing all IPs (LAN & WAN) from being found.
https://forum.netgate.com/topic/186210/ ... traffic/25

Sorry to hear the issue has not been resolved yet. I agree it does not sound like the WAN devices are at fault but were you able to check the monitor IP for your gateways? In some cases the gateway IP is not pingable for various reasons such as IPPT enablement on the WAN device. Especially for cellular devices I have found I need to change the monitor IP for the gateway. As a troubleshooting step you could temporarily disable gateway monitoring as well to see if the issue occurs with it deactivated or not.

[Oblivious]

IG gateway has been using 8.8.8.8 as monitoring IP. I have tried other DNS (like, 1.1.1.1 & 9.9.9.9) & disabled montoring also, but I honestly can't remember if that was before or after making the other changes you suggested. I just disabled monitoring & we'll see what happens.... crossing fingers.

[hazarjast]

IG gateway has been using 8.8.8.8 as monitoring IP. I have tried other DNS (like, 1.1.1.1 & 9.9.9.9) & disabled montoring also, but I honestly can't remember if that was before or after making the other changes you suggested. I just disabled monitoring & we'll see what happens.... crossing fingers.

Today I actually tested setting the gateway monitor IP for my OPNSense IG WAN to Google/Cloudflare/Quad9 and I was not seeing the best metrics during peak times due to ICMP traffic shaping by T-Mobile (RTT was especially erratic at times). Taking that data point into consideration, if disabling monitoring improved the situation for you but you would still like to have gateway monitoring enabled I would suggest setting the monitor IP to the IG config UI IP (192.168.225.1, by default). This should accomplish two things. The first being successful gateway monitoring but second it should force the persistence of a route table entry for the IG config UI IP even when the IP changes on carrier connect when using the default IP Passthrough (IPPT). That way you can always get to the config interface along with the improved monitoring of the gateway.

[Oblivious]

OK. I haven't had any issues since disabling monitoring on the IG gateway but not ready to declare success just yet. Over on the netgate forums, they are thinking it may be a sporadic hardware failure in my router. Yesterday I left it running memtest for several hours, no errors. If it fails again, I intend to run some ping tests to certain LAN & WAN IPs and host names to verify what works/fails. I've manually set the IPs on a couple LAN devices (PC, Laptop, etc) so I (assume I) should not lose comms unless there's an issue with the cabling or switch.

With your suggestion, will monitoring be able to tell if the cell-to-carrier connection is offline? I'm not sure it matters if the IG gateway, as backup connection, is actually online or off because at that point I'm out of fail-over options anyway. But, it may rule-out certain specific configurations (routing low bandwidth devices primarily thru the cell WAN) & the possibility of using TMHI as a primary ISP (something I've been considering.)

[hazarjast]

OK. I haven't had any issues since disabling monitoring on the IG gateway but not ready to declare success just yet. Over on the netgate forums, they are thinking it may be a sporadic hardware failure in my router. Yesterday I left it running memtest for several hours, no errors. If it fails again, I intend to run some ping tests to certain LAN & WAN IPs and host names to verify what works/fails. I've manually set the IPs on a couple LAN devices (PC, Laptop, etc) so I (assume I) should not lose comms unless there's an issue with the cabling or switch.

With your suggestion, will monitoring be able to tell if the cell-to-carrier connection is offline? I'm not sure it matters if the IG gateway, as backup connection, is actually online or off because at that point I'm out of fail-over options anyway. But, it may rule-out certain specific configurations (routing low bandwidth devices primarily thru the cell WAN) & the possibility of using TMHI as a primary ISP (something I've been considering.)

Great to read things have been more stable, hopefully that will continue! For my suggestion on setting the gateway monitor IP to the UI IP address of the IG, no this won't help dpinger to tell if the cell-to-carrier connection is offline but I would still not recommend using a public DNS IP due to the other issues I mentioned (carrier shaping/deprioritization of ICMP and UDP traffic). Fortunately with the InvisaGig, monitoring the modem to carrier connection is not a task that you need the router/firewall to perform. If you are updated to IG v1.0.10 it already has Connection WatchDog which covers monitoring connection to the carrier. You can simply enable this feature if you haven't already.

As for using T-Mobile as your primary WAN, FWIW they have been my *only* WAN connection for 8 years now (before they even offered Home Internet I had long been using a SIM out of one of my phones for this purpose). I have upgraded my modem hardware a few times over the years to improve reliability and throughput but it has served our family well enough that we haven't needed to sign back up with Comcast/Xfinity (who had outages all the time in our neighborhood). I am currently using the InvisaGig installed in a Quadlink (https://store.invisagig.com/products/in ... ull-system) as WAN under OPNSense and this setup has been great for primary WAN connectivity. Of course YMMV depending on how close your tower is, if you have LoS to it, etc.

[Didneywhorl]

Side idea if general local to internet connectivity monitoring is wanted, you could setup a service like "Uptime KUMA" and make it monitor out to a website doing an https check with keyword lookup so that you aren't using icmp protocols, but are using a bit more involved/advanced check metric. and it can send alerts to TONS of services if you want to be alerted, which also can act as a 3rd party logging as well.

[Oblivious]

So unfortunately the issue came back. Over at the netgate forums:
https://forum.netgate.com/topic/186210/ ... traffic/51

Ah yes DS-Lite. I see that so infrequently I always forget about it. So 192.0.0.2 is a legitimate IP address in that case.

I agree the switch from local DHCP to bridged is what I always suspected is the issue. The problem happens is the modem reboots or it;s link is lost and it starts handing out IPs in it's own subnet. When the link is re-established pfSense will not necessarily try to pull a new lease so fails until it renews.

This is quite common in cable modems and the usu workaround is to deny leases from the cable modem IP so it just keeps trying until the upstream link comes back up.

If you are seeing that it will be logged. The dhclient logs in the dhcp log will show the local modem address that's serving leases.

So I take that to mean I need to reject leases from 192.0.0.1

[Tech_Junky]

192.0.0.1

That's your WAN off the WWAN side.

More specifically the Gateway as your modem has 192.0.0.2 and an IPV6 address assigned when it connects to the network.

[hazarjast]

So unfortunately the issue came back. Over at the netgate forums:
https://forum.netgate.com/topic/186210/ ... traffic/51

Ah yes DS-Lite. I see that so infrequently I always forget about it. So 192.0.0.2 is a legitimate IP address in that case.

I agree the switch from local DHCP to bridged is what I always suspected is the issue. The problem happens is the modem reboots or it;s link is lost and it starts handing out IPs in it's own subnet. When the link is re-established pfSense will not necessarily try to pull a new lease so fails until it renews.

This is quite common in cable modems and the usu workaround is to deny leases from the cable modem IP so it just keeps trying until the upstream link comes back up.

If you are seeing that it will be logged. The dhclient logs in the dhcp log will show the local modem address that's serving leases.

So I take that to mean I need to reject leases from 192.0.0.1

Sorry to hear that. Just so I am up to speed: do you still have the IPPT config on the IG statically set to the correct MAC address of your pfSense WAN and have your monitor IP for the gateway set to 192.168.225.1? If so then you shouldn't have to mess with any lease rejects. What the Netgate poster says about the address/subnet change is correct but that is only a momentary change when the IG modem gets disconnected from the carrier tower, upon reconnect it will receive the 192.0.0.2 address as expected. On my OPNSense I do not reject any leases and can see in the System log if there is a tower disconnect where the IP change occurs but on reconnect it goes back to the expected IPPT address. This all takes place within seconds automatically without intervention so it is still not clear to me why pfSense would be behaving differently in this scenario. Which version of pfSense are you running?

You can always deactivate IPPT for now as detailed in my initial reply (viewtopic.php?p=28641#p28641):

You can set the MAC address for IPPT manually by selecting 'Local IP & Multiple Modem Setup' from the IG's main menu. From there select 'Single Unit' (assuming this is your only IG unit) and entering 'N' when asked if you wish to use a custom IP address. You will then enter 'Y' to specify a MAC address and enter the MAC address of the WAN port in the next screen. Alternatively you could simply disable IPPT at this last prompt by leaving the MAC address blank prior hitting 'Enter'. If you disable IPPT then your WAN will then always be assigned an IP in the subnet of your IG (by default, 192.168.225.x).

That would at least eliminate a variable in your troubleshooting.

[Oblivious]

As you suggested around Feb 28, I manually configured the IPPT that day. And, later, I did try changing the monitoring IP to ping the IG ip address. Just for clarity, I changed that IP from 192.168.225.1 to 192.168.5.1. I don't know if it matters, but when I set-up a static route or VIP, I am able to access the admin portal login screen for my wireless ISP at 192.168.225.1. I'm guessing they leave it accessible for the techs to be able access on service calls. Seemed like it might be a routing issue if I left the IG at the same IP.

The current config state as of yesterday:
- Not rejecting leases
- Disabled IPPT per your previous suggestion & the latest netgate post.
- I do have a static route set-up for 192.168.5.1
- Gateway monitor is set to ping 192.168.5.1

Since disabling IPPT pass thru, I have not had any issues but I'm not ready to hold my breath yet either. What do you see in your logs when the disconnect happens? I'm not sure what to look for

[hazarjast]

As you suggested around Feb 28, I manually configured the IPPT that day. And, later, I did try changing the monitoring IP to ping the IG ip address. Just for clarity, I changed that IP from 192.168.225.1 to 192.168.5.1. I don't know if it matters, but when I set-up a static route or VIP, I am able to access the admin portal login screen for my wireless ISP at 192.168.225.1. I'm guessing they leave it accessible for the techs to be able access on service calls. Seemed like it might be a routing issue if I left the IG at the same IP.

The current config state as of yesterday:
- Not rejecting leases
- Disabled IPPT per your previous suggestion & the latest netgate post.
- I do have a static route set-up for 192.168.5.1
- Gateway monitor is set to ping 192.168.5.1

Since disabling IPPT pass thru, I have not had any issues but I'm not ready to hold my breath yet either. What do you see in your logs when the disconnect happens? I'm not sure what to look for

Ah, apologies for my assumption you were still using the default UI IP address. Sounds like you have it setup well to eliminate the IPPT variable. Below are my OPNSense 'System > General' logs for when the IG has to reconnect with IPPT enabled. All of the events for this span less than 20 seconds. In case you are wondering the unknown 0x78 DHCP option has to do with SIP servers and isn't relevant to OPNSense (this is a commonly occurring message that can be ignored).

Code: Select all

2024-03-14T23:55:57	opnsense[4490]	/usr/local/etc/rc.linkup: ROUTING: keeping current default gateway '192.0.0.1'	 
2024-03-14T23:55:57	opnsense[4490]	/usr/local/etc/rc.linkup: ROUTING: setting IPv4 default route to 192.0.0.1	 
2024-03-14T23:55:57	opnsense[4490]	/usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to wan	 
2024-03-14T23:55:57	opnsense[4490]	/usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : webgui_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : vxlan_configure_interface())	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : unbound_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : openssh_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : opendns_configure_do())	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : ntpd_configure_do())	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (execute task : dyndns_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure newwanip (,wan)	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure vpn (execute task : openvpn_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure vpn (execute task : ipsec_configure_do(,wan))	 
2024-03-14T23:55:57	opnsense[37551]	plugins_configure vpn (,wan)	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: Adding static route for monitor 192.168.225.1 via 192.0.0.1	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: Removing static route for monitor 192.168.225.1 via 192.0.0.1	 
2024-03-14T23:55:56	opnsense[37551]	plugins_configure monitor (execute task : dpinger_configure_do())	 
2024-03-14T23:55:56	opnsense[37551]	plugins_configure monitor ()	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '192.0.0.1'	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to 192.0.0.1	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'	 
2024-03-14T23:55:56	opnsense[37551]	plugins_configure hosts (execute task : unbound_hosts_generate())	 
2024-03-14T23:55:56	opnsense[37551]	plugins_configure hosts (execute task : dnsmasq_hosts_generate())	 
2024-03-14T23:55:56	opnsense[37551]	plugins_configure hosts ()	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: On (IP address: 192.0.0.2) (interface: WAN[wan]) (real interface: igb0).	 
2024-03-14T23:55:56	opnsense[37551]	/usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'igb0'	 
2024-03-14T23:55:56	dhclient[57544]	Creating resolv.conf	 
2024-03-14T23:55:56	dhclient[5466]	route add default 192.0.0.1	 
2024-03-14T23:55:56	dhclient[70696]	New Routers (igb0): 192.0.0.1	 
2024-03-14T23:55:56	dhclient[45125]	New Broadcast Address (igb0): 192.0.0.31	 
2024-03-14T23:55:56	dhclient[29088]	New Subnet Mask (igb0): 255.255.255.224	 
2024-03-14T23:55:56	dhclient[22377]	New IP Address (igb0): 192.0.0.2	 
2024-03-14T23:55:56	dhclient[50078]	unknown dhcp option value 0x78	 
2024-03-14T23:55:55	opnsense[4490]	/usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan	 
2024-03-14T23:55:55	opnsense[4490]	/usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan	 
2024-03-14T23:55:53	opnsense[98228]	/usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure dns (execute task : unbound_configure_do())	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure dns (execute task : dnsmasq_configure_do())	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure dns ()	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure dhcp (execute task : dhcpd_dhcp_configure())	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure dhcp ()	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure ipsec (execute task : ipsec_configure_do(,wan))	 
2024-03-14T23:55:50	opnsense[25505]	plugins_configure ipsec (,wan)	 
2024-03-14T23:55:50	opnsense[25505]	/usr/local/etc/rc.linkup: ROUTING: skipping IPv4 default route	 
2024-03-14T23:55:50	opnsense[25505]	/usr/local/etc/rc.linkup: ROUTING: IPv4 default gateway set to opt1	 
2024-03-14T23:55:50	opnsense[25505]	/usr/local/etc/rc.linkup: ROUTING: entering configure using 'wan'	 
2024-03-14T23:55:50	opnsense[25505]	/usr/local/etc/rc.linkup: The command '/sbin/dhclient -c '/var/etc/dhclient_wan.conf' -p '/var/run/dhclient.igb0.pid' 'igb0'' returned exit code '11', the output was 'DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPNAK from 192.168.225.1 DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 2 DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 4 DHCPOFFER from 192.168.225.1 unknown dhcp option value 0x78 igb0 link state up -> down DHCPREQUEST on igb0 to 255.255.255.255 port 67 DHCPREQUEST on igb0 to 255.255.255.255 port 67 igb0 link state down -> up DHCPDISCOVER on igb0 to 255.255.255.255 port 67 interval 2 DHCPOFFER from 192.168.225.1 unknown dhcp option value 0x78'	 
2024-03-14T23:55:50	dhclient[41712]	exiting.	 
2024-03-14T23:55:50	dhclient[41712]	connection closed	 
2024-03-14T23:55:49	dhclient[41712]	unknown dhcp option value 0x78	 
2024-03-14T23:55:48	dhclient[41712]	send_packet: Network is down	 
2024-03-14T23:55:47	dhclient[41712]	send_packet: Network is down	 
2024-03-14T23:55:45	dhclient[41712]	unknown dhcp option value 0x78	 
2024-03-14T23:55:42	opnsense[25505]	/usr/local/etc/rc.linkup: HOTPLUG: Configuring interface wan	 
2024-03-14T23:55:42	opnsense[25505]	/usr/local/etc/rc.linkup: DEVD Ethernet attached event for wan	 
2024-03-14T23:55:39	opnsense[64096]	/usr/local/etc/rc.linkup: Clearing states for stale wan route on igb0	 
2024-03-14T23:55:39	dhclient[51156]	exiting.	 
2024-03-14T23:55:39	dhclient[51156]	connection closed	 
2024-03-14T23:55:39	opnsense[64096]	/usr/local/etc/rc.linkup: DEVD Ethernet detached event for wan

[Oblivious]

An update: I had to go out of town, so I wrote a sh script in pfSense router to ping LAN IPs of several devices that should be online. If no response, reboot. If OK, then ping internet servers. If no reponse, cycle the WAN gateway on/off. The evening before I was leaving, the network went down, the script executed reboot but the router locked up during shutdown so had to physically power cycle. I then quickly wrote a Node Red flow to ping the router, and if no response power cycle the router using a zigbee (hue) plug.

Almost two weeks later the script has cycled the WAN gateway a couple times but otherwise the LAN & router have been working fine. Its not had to be power cycled once since writing the Node Red flow. Go figure.