WAN Bonding with ROOter and OpenMPTCProuter

[bnhf2]

I am having doubts about the ability to power the pi+modem off PoE, or at least with the switch I got, so I might just need to bite the bullet and do some wiring regardless.

I think you're on the right track with the network architecture you have in mind, with the cellular routers in the barn, and the OMR router in the house. I'm fairly confident the 3B+ is the problem. I used the Raspberry Pi myself originally with ROOter, and had great success with the 3B. The 3B+ though was never as solid, particularly with VLANs in use. That, and the lack of a stable OpenWRT for the 4B, resulted in me moving to the MikroTik RBM33G.

I'd highly recommend the RBM33G, as it has a speedy processor (for a router product), 2x mPCIe slots, a USB3 port, real gigabit Ethernet, an m.2 SSD slot, a wide range of power voltages including passive PoE, and a price point similar to the RPi. For me the 24V PoE was a huge plus, as I already use Ubiquiti for all of my WiFi.

If you're in an urban area, having your cellular routers in close proximity to the OMR router is usually fine. I'm guessing that since you have a barn, you're rural, and as such want to be able to carefully locate your cellular routers and pair them with high performance antennas. Stick with the network of gateways approach in this scenario, and it will serve you well for years to come.

[ciarlill]

I think you're on the right track with the network architecture you have in mind, with the cellular routers in the barn, and the OMR router in the house. I'm fairly confident the 3B+ is the problem. I used the Raspberry Pi myself originally with ROOter, and had great success with the 3B. The 3B+ though was never as solid, particularly with VLANs in use. That, and the lack of a stable OpenWRT for the 4B, resulted in me moving to the MikroTik RBM33G.

I'd highly recommend the RBM33G, as it has a speedy processor (for a router product), 2x mPCIe slots, a USB3 port, real gigabit Ethernet, an m.2 SSD slot, a wide range of power voltages including passive PoE, and a price point similar to the RPi. For me the 24V PoE was a huge plus, as I already use Ubiquiti for all of my WiFi.

If you're in an urban area, having your cellular routers in close proximity to the OMR router is usually fine. I'm guessing that since you have a barn, you're rural, and as such want to be able to carefully locate your cellular routers and pair them with high performance antennas. Stick with the network of gateways approach in this scenario, and it will serve you well for years to come.

Oh wow that microtik device looks awesome. Do you plug the modems into it directly instead of over USB then? Sucks that I wasted so much time and money on raspberry pis, usb enclosures, sd cards, adapters, etc but if these are as good as the look, definitely worth another $100 after everything I've already invested.

[bnhf2]

Oh wow that microtik device looks awesome. Do you plug the modems into it directly instead of over USB then? Sucks that I wasted so much time and money on raspberry pis, usb enclosures, sd cards, adapters, etc but if these are as good as the look, definitely worth another $100 after everything I've already invested.

Yes, you can use an mPCIe modem directly (both Sierra and Quectel modems need to be configured to use USB2 "lines" only -- which is easily done by AT command), or if you have an m.2 modem in an m.2 to mPCIe adapter it's plug-and-play. I just ordered some RBM33G units from Streakwave Wireless, and they shipped the next day.

Baltic Networks sells an enclosure called the "Maxxwave UBTik 433 MikroTik Access Point (Enclosure Only)" that's pretty slick, though I frequently build my own outdoor boxes using Bud Industries boxes. There's also a pretty good 3D printer indoor enclosure design floating around.

Flashing the RBM33G with ROOter is easy once you get the hang of it, using tftp and the reset button. It's a really sweet board for ROOter, and addresses all of the downfalls of the RPi as far as I'm concerned. Other than setting mPCIe modems for USB2 lines only, the other thing to know is that the current "stable release" of ROOter doesn't have the board's Ethernet ports organized as it should, so you might want to use the 2019-03-10 build, which I'll link for you here:

https://www.dropbox.com/s/edc036nkcvsp0 ... -03-10.zip

[bnhf2]

Oh, and the RBM11G is a nice (and cheaper) board too -- it's the 33G with a single Ethernet port and single mPCIe slot.

The RPi 3B+ makes an awesome Kodi (OSMC) box, Pi-hole DNS server or low-power Linux (Raspbian) file server by the way -- I've got at least half-a-dozen in daily use on my LAN!

[ciarlill]

Yes, you can use an mPCIe modem directly (both Sierra and Quectel modems need to be configured to use USB2 "lines" only -- which is easily done by AT command), or if you have an m.2 modem in an m.2 to mPCIe adapter it's plug-and-play. I just ordered some RBM33G units from Streakwave Wireless, and they shipped the next day.

Baltic Networks sells an enclosure called the "Maxxwave UBTik 433 MikroTik Access Point (Enclosure Only)" that's pretty slick, though I frequently build my own outdoor boxes using Bud Industries boxes. There's also a pretty good 3D printer indoor enclosure design floating around.

Flashing the RBM33G with ROOter is easy once you get the hang of it, using tftp and the reset button. It's a really sweet board for ROOter, and addresses all of the downfalls of the RPi as far as I'm concerned. Other than setting mPCIe modems for USB2 lines only, the other thing to know is that the current "stable release" of ROOter doesn't have the board's Ethernet ports organized as it should, so you might want to use the 2019-03-10 build, which I'll link for you here:

https://www.dropbox.com/s/edc036nkcvsp0 ... -03-10.zip

So you cannot use an m.2 modem directly in the available m2 slot?

The other nice thing is it looks like I can ditch the PoE splitters I was using too.

I already have 2 nice NEMA encloures I can put them in, only concern is securing the antenna pigtails in some way since I won't have any bulkhead to secure the SMA side to inside the enclosure, but I'm sure I can rig up a little plate.

It's been a while since I flashed anything using tftp - i think way back flashing XBMC on an original Xbox or trying to unbrick one - but I can dust off those skills. Last question: are you still using your same 4 port PoE injector? Any idea how many watts your devices are drawing? This PoE pass-through switch is only rated for a total of 18W, so I think I will need to go another direction. Either the Ubiquiti passthrough one or just run 2 PoE lines from the garage and do the injection there. I've seen a lot of conflicting things about total power requirements on here.

[ciarlill]

For flashing you don't need a USB -> Serial cable? I was looking at this guide here: https://www.ttl.one/2018/08/installing- ... bm11g.html

[bnhf2]

So you cannot use an m.2 modem directly in the available m2 slot?

The other nice thing is it looks like I can ditch the PoE splitters I was using too.

I already have 2 nice NEMA encloures I can put them in, only concern is securing the antenna pigtails in some way since I won't have any bulkhead to secure the SMA side to inside the enclosure, but I'm sure I can rig up a little plate.

It's been a while since I flashed anything using tftp - i think way back flashing XBMC on an original Xbox or trying to unbrick one - but I can dust off those skills. Last question: are you still using your same 4 port PoE injector? Any idea how many watts your devices are drawing? This PoE pass-through switch is only rated for a total of 18W, so I think I will need to go another direction. Either the Ubiquiti passthrough one or just run 2 PoE lines from the garage and do the injection there. I've seen a lot of conflicting things about total power requirements on here.

Unfortunately, the m.2 slot has lines for storage devices only.

I've had 2 RBM33G units and a Ubiquiti NanoStation AC running off of a single Ubiquiti NanoSwitch (outdoor 4-port PoE switch). That NanoSwitch was powered by a single port on a PoE Texas Gigabit injector (GPOE-4B) -- with no problems. If you migrate to the world of 24V passive PoE, MikroTik, Ubiquiti and PoE Texas are the brands to know and use. If you need to use the NanoSwitch for anything non-PoE, the GPOE-1B can serve as a PoE blocker that supports gigabit Ethernet.

[bnhf2]

This guide is good for flashing the RBM33G of RBM11G, just skip ignore anything to do with puTTY and a serial cable -- use the reset button to put the unit in netboot mode instead:

https://www.ttl.one/2018/08/installing- ... bm11g.html

Getting the board in netboot mode is described here, under "Buttons and jumpers":

https://help.mikrotik.com/docs/display/UM/RBM33G

If you do more than one board be sure to delete the DHCP address issued by tftp64, from the list of issued addresses or it won't kick out another. There will be a couple of Windows 10 "new network" side blades that you'll need to "accept" during the process.

[ciarlill]

Last question (for now): When using the M.2 to mPCIe adapter, do I need one with a SIM slot or can I still use the SIM slot on the board itself?

[bnhf2]

Last question (for now): When using the M.2 to mPCIe adapter, do I need one with a SIM slot or can I still use the SIM slot on the board itself?

I recommend the adapters without a SIM slot, but I've occasionally had to use the others due to availability issues. In those cases, the slot on the board has still functioned -- and is highly preferred for convenience.

[ciarlill]

I recommend the adapters without a SIM slot, but I've occasionally had to use the others due to availability issues. In those cases, the slot on the board has still functioned -- and is highly preferred for convenience.

I received my RBM33G's and flashed one of them. Installed m.2 to PCIe adapter and my EM12-G modem, did the normal modem setup setting APN and TTL. But the interfaces seem to not have all the necessary protocol extensions? I am using the 2019-03-10 ROOter build you linked. I flashed the "factory" version via tftpd and then flashed the "upgrade" version through Luci.



When I click on "Install protocol extensions" I see this:



Did you have to install those? I'll need to get this thing online via another gateway to do so I think, but wanted to check in before I mess with it anymore.

Thanks

[docderwood]

FYI,

New version posted: https://www.openmptcprouter.com

[bnhf2]

I received my RBM33G's and flashed one of them. Installed m.2 to PCIe adapter and my EM12-G modem, did the normal modem setup setting APN and TTL. But the interfaces seem to not have all the necessary protocol extensions? I am using the 2019-03-10 ROOter build you linked. I flashed the "factory" version via tftpd and then flashed the "upgrade" version through Luci.



When I click on "Install protocol extensions" I see this:



Did you have to install those? I'll need to get this thing online via another gateway to do so I think, but wanted to check in before I mess with it anymore.

Thanks

I'm thinking support for the EM12-G has come more recently than last year's ROOter, so you'll want to upgrade to the latest build. I was steering you away from that version due some minor issues with the network switch configuration.

Go ahead and upgrade to 2020-03-10, but be aware you'll need to use the middle Ethernet port until you've edited special.sh and made necessary GUI changes to Network - Switch. Power needs to come from the barrel port, or the left most PoE port. Once you've fixed the switch issue, the left port will become a LAN port.

I just walked somebody through this in the ROOter thread, so head over there and check my most recent posts, including this one:

https://whrl.pl/Rf7eGD

WinSCP is the best tool for accessing the ROOter file system, and editing /usr/lib/rooter/special.sh. Be sure to use the SCP protocol.

[ciarlill]

I'm thinking support for the EM12-G has come more recently than last year's ROOter, so you'll want to upgrade to the latest build. I was steering you away from that version due some minor issues with the network switch configuration.

Go ahead and upgrade to 2020-03-10, but be aware you'll need to use the middle Ethernet port until you've edited special.sh and made necessary GUI changes to Network - Switch. Power needs to come from the barrel port, or the left most PoE port. Once you've fixed the switch issue, the left port will become a LAN port.

I just walked somebody through this in the ROOter thread, so head over there and check my most recent posts, including this one:

https://whrl.pl/Rf7eGD

WinSCP is the best tool for accessing the ROOter file system, and editing /usr/lib/rooter/special.sh. Be sure to use the SCP protocol.

Thank you! It's finally all working on the bench. Just hit 60Mbps down! Hopefully can do even better once these are hooked up to the good antennas out at the barn.

The current network diagram looks a little something like this. I am really starting to enjoy experimenting with VLANs now that they work like they should (looking at you RPi 3 B+). I have only added one additional besides those used for the gateways to isolate some IoT and streaming devices. Ubiquiti controller makes it very easy to setup isolated SSIDs on the VLAN with bandwidth limiting as well.



Still need to integrate the nanoswitch and finish putting all the hardware into it's enclosures. Once that's done I'll be doing my own write up/post so I can give back and help others as much as you have helped me. I really appreciate it!

[RussWestrem]

Ok. I jumped on the wagon here and setup openMPTCP using Vultr and a linksys wrt3200. I have a cba820/em7565 sprint on wan1 and a netgear nighthawk att on wan2. Everything seems to be working great but now im trying to get all my streaming services, online gaming, and plex/ombi servers working right. My kid is complaining about roblox, mine craft and a few other multiplayer games not loading despite having open nats. This is on pc and xbox. Any suggestions? Also I can connect to my ombi server locally but from the outside it just gets stuck on loading. Http://lsp.hopto.org:5000 is my ombi login page.

[ciarlill]

Ok. I jumped on the wagon here and setup openMPTCP using Vultr and a linksys wrt3200. I have a cba820/em7565 sprint on wan1 and a netgear nighthawk att on wan2. Everything seems to be working great but now im trying to get all my streaming services, online gaming, and plex/ombi servers working right. My kid is complaining about roblox, mine craft and a few other multiplayer games not loading despite having open nats. This is on pc and xbox. Any suggestions? Also I can connect to my ombi server locally but from the outside it just gets stuck on loading. Http://lsp.hopto.org:5000 is my ombi login page.

I'm sure already found it, but this was helpful to me: https://github.com/Ysurac/openmptcprout ... forwarding I forwarded a few ports for internal web services this way, and had no issue connecting to them, but they are just basic web servers. I don't think anything requiring UPnP will work over a VPN connection like this, so you will likely need to manually port forward the required game ports.

[bnhf2]

Ok. I jumped on the wagon here and setup openMPTCP using Vultr and a linksys wrt3200. I have a cba820/em7565 sprint on wan1 and a netgear nighthawk att on wan2. Everything seems to be working great but now im trying to get all my streaming services, online gaming, and plex/ombi servers working right. My kid is complaining about roblox, mine craft and a few other multiplayer games not loading despite having open nats. This is on pc and xbox. Any suggestions? Also I can connect to my ombi server locally but from the outside it just gets stuck on loading. Http://lsp.hopto.org:5000 is my ombi login page.

Hey Russ! Glad to have you on the OMR user team -- it's really sweet once you've got everything configured.

You're going to want to forward all ports from the VPS to your OMR router using the tick box under System -OpenMPTCProuter - Advanced settings. Then forward Plex, Ombi and whatever as you would normally. Plex often shows the red exclamation next to Remote Access, but it works nonetheless.

Any streaming services that object to being used via a data center or through a VPN will need to be bypassed. Services -OMR Bypass - Source LAN or IP address (use the default interface setting), will send any given IP on your LAN through one of your WANs directly rather than the through the tunnel -- this always works. There's also a wide ranging section for bypssing based on Protocols and services. This often works, but isn't as foolproof as my previous recommendation.

Best of luck, and let us know how you get on.

[bnhf2]

Thank you! It's finally all working on the bench. Just hit 60Mbps down! Hopefully can do even better once these are hooked up to the good antennas out at the barn.

The current network diagram looks a little something like this. I am really starting to enjoy experimenting with VLANs now that they work like they should (looking at you RPi 3 B+). I have only added one additional besides those used for the gateways to isolate some IoT and streaming devices. Ubiquiti controller makes it very easy to setup isolated SSIDs on the VLAN with bandwidth limiting as well.



Still need to integrate the nanoswitch and finish putting all the hardware into it's enclosures. Once that's done I'll be doing my own write up/post so I can give back and help others as much as you have helped me. I really appreciate it!

That is some really nice work! That network of gateways approach using VLANs is one of the favorite things I've done to my network in the last few years (along with OMR of course). It's great to be able to take a gateway down and just have everything continue to hum along.

Due to the current craziness here in the US, there are three other RVers currently camped on my property in the Colorado mountains (soon to be five total rigs), all using my Internet -- so up-time has taken on a whole new meaning. All of my friends are Internet mad as far as I can tell. Are there any Luddites left?

[RussWestrem]

Hey Russ! Glad to have you on the OMR user team -- it's really sweet once you've got everything configured.

You're going to want to forward all ports from the VPS to your OMR router using the tick box under System -OpenMPTCProuter - Advanced settings. Then forward Plex, Ombi and whatever as you would normally. Plex often shows the red exclamation next to Remote Access, but it works nonetheless.

Any streaming services that object to being used via a data center or through a VPN will need to be bypassed. Services -OMR Bypass - Source LAN or IP address (use the default interface setting), will send any given IP on your LAN through one of your WANs directly rather than the through the tunnel -- this always works. There's also a wide ranging section for bypssing based on Protocols and services. This often works, but isn't as foolproof as my previous recommendation.

Best of luck, and let us know how you get on.

Seems that ombi doesn't like a proxy server. Something about websockets which is all confusing to me but luckily they have a beta V4 that works through a proxy so that is fixed.

I'm still working on roblox for my kid. It works fine when I OMR bypass the lan ip of the pc but I really need this pc to have the public ip as it is my media server as well with plex, ombi, and web servers running in the background. I've tried to figure out all the domains roblox uses and have bypassed them but games still get stuck on connecting to the individual game servers. Again I'm not really savvy with networking. I'm down to just needing to fix minecraft and roblox servers so he can join games.

EDIT: I found the port range that roblox uses and was able to just bypass 49152-65535.

[RussWestrem]

After some thought I think I'm going to run the openmptcp VPS part on a old laptop I don't use anymore and stick it at my mother in laws place where she has fiber in town. That will solve all my VPN blocking streaming services and also even less ping. I guess all I need to do is give it a static ip and then put it in the dmz on her router and it should be the same setup.

[njolin]

This seems like a really neat setup, thanks for the writeup! After paging through all the posts, I don't know if I saw it or not so I thought I'd ask:

Is it possible to have OMR manage 2 LTE modems on the routerboard rbm33g so it's an all-in-one solution?

[bnhf2]

Things have changed for us here in the far reaches of the Colorado mountains -- we now have fiber!

It's been a few weeks now, and I'm still liking having OMR in the mix, despite our new found Internet riches. I've always had two WiFi networks setup, one that uses OMR and another that doesn't. House guests, and others with limited appreciation of things like LAN wide ad-blocking, can use the Internet un-affected by having traffic routed through a data center, using a VPN, ad-blocking and a number of other things I prefer.

I've relegated my cellular links to a backup role, as the fiber gives us plenty of performance. Not sure if this will be the long term solution, but for now it's working great. So, if you find yourself with a single link that meets your need-for-speed in the future, just wanted to let you all know that you don't necessarily need to dump OMR.

screenshot-192.168.100.1-2020.08.25-18_25_08.png

screenshot-www.speedtest.net-2020.08.25-15_19_26.png

[RussWestrem]

Things have changed for us here in the far reaches of the Colorado mountains -- we now have fiber!

Bnhf2, you seem to really understand vlans and routing. I'm having an issue which I could really use your assistance. I currently am running it on the wrt3200 with their precompiled image. Wifi is very flaky on the wrt3200 and always has been with openwrt for some reason. It auto sets the switch to where the wan is lan and the 4 lans are wans. This works great like this. On to my issue. I compiled for my netgear nighthawk r7800 router and it loads fine but only lan 1 works for a lan and can't get any wan ports working. The network configuration looks completely different than the wrt3200 network configuration. I tried swapping the configuration but just killed all ports on the r7800. I'm not sure how to set the vlans and wans on the switch in the router.

I can give you access to the router if that helps you see the configurations.

[bnhf2]

I currently am running it on the wrt3200 with their precompiled image. Wifi is very flaky on the wrt3200 and always has been with openwrt for some reason. It auto sets the switch to where the wan is lan and the 4 lans are wans. This works great like this. On to my issue. I compiled for my netgear nighthawk r7800 router and it loads fine but only lan 1 works for a lan and can't get any wan ports working. The network configuration looks completely different than the wrt3200 network configuration. I tried swapping the configuration but just killed all ports on the r7800. I'm not sure how to set the vlans and wans on the switch in the router.

There are three ways I've seen Ethernet ports handled on versions of OpenWRT:

The first is when there is support for an integrated switch (driver required), and in this case there will be a menu item under Network - Switch. All ports are managed in a VLAN like fashion where VLAN 1 is the LAN, VLAN 2 is the WAN, and additional user defined VLANs (generally "tagged", so that the VLAN ID is required for that interface). An example (from ROOter), looks like this:

screenshot-192.168.9.1-2020.08.28-12_10_37.png

The second case is when each individual Ethernet port has it's own physical interface. My Qotom industrial PC, that I use as my OMR router, is like this. The ports are eth0, eth1, eth2, and eth3. If I want to assign a VLAN to eth1, I would create a custom interface of eth1.3 for example.

And the final variant would be hardware that supports software VLANs like the Raspberry Pi. This also uses the concept of adding "." followed by the VLAN ID to create a VLAN, but everything is handled in software as there's no support for this built-in to the hardware.

The /etc/config/network for my ROOter based example above, looks like this (Actual MAC addresses replaced with xx:xx:xx:xx:xx:xx):

Code: Select all

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd02:31f0:662f::/48'

config interface 'lan'
	option type 'bridge'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ifname 'eth0.1 tap0 tap-server'
	option ipaddr '192.168.90.1'

config device 'lan_eth0_1_dev'
	option name 'eth0.1'
	option macaddr 'xx:xx:xx:xx:xx:xx'

config interface 'wan'
	option ifname 'eth0.2'
	option proto 'dhcp'
	option metric '1'

config device 'wan_eth0_2_dev'
	option name 'eth0.2'
	option macaddr 'xx:xx:xx:xx:xx:xx'

config interface 'wan6'
	option ifname 'eth0.2'
	option proto 'dhcpv6'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '0 1 6t'
	option vid '1'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '2 6t'
	option vid '2'

config interface 'VPN'
	option proto 'none'
	option ifname 'tun0'

config interface 'VPNS'
	option proto 'none'
	option ifname 'tun-server'

config interface 'TAP'
	option proto 'none'
	option ifname 'tap0'
	option auto '1'

config interface 'TAPS'
	option proto 'none'
	option ifname 'tap-server'
	option auto '1'

config interface 'wwan'
	option proto 'dhcp'
	option metric '2'

config switch_vlan
	option device 'switch0'
	option vlan '3'
	option ports '0t 1t 6t'
	option vid '9'

config interface 'VLAN9'
	option proto 'static'
	option ifname 'eth0.9'
	option ipaddr '192.168.9.1'
	option netmask '255.255.255.0'

config interface 'wan2'
	option proto 'dhcp'
	option metric '20'
	option ifname 'wan2'

config interface 'wan1'
	option proto 'mbim'
	option device '/dev/cdc-wdm0'
	option metric '10'
	option currmodem '1'

That same file on my OMR box (no in-built switch) looks like this:

Code: Select all

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'
	option multipath 'off'
	option macaddr '00:00:00:00:00:00'
	option metric '1'

config globals 'globals'
	option ula_prefix 'fd90:ba49:898f::/48'
	option multipath 'enable'
	option mptcp_path_manager 'fullmesh'
	option congestion 'bbr'
	option mptcp_checksum '0'
	option mptcp_debug '0'
	option mptcp_syn_retries '1'
	option mptcp_fullmesh_num_subflows '1'
	option mptcp_fullmesh_create_on_err '1'
	option mptcp_ndiffports_num_subflows '1'
	option mptcp_scheduler 'default'

config interface 'lan'
	option proto 'static'
	option ipaddr '192.168.100.1'
	option netmask '255.255.255.0'
	option delegate '0'
	option multipath 'off'
	option ip4table 'lan'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option metric '2'
	option type 'bridge'
	option ifname 'eth0'

config rule 'lan_rule'
	option lookup 'lan'
	option priority '100'

config interface 'wan1'
	option proto 'static'
	option ip4table 'wan'
	option defaultroute '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option metric '3'
	option peerdns '0'
	option ipv6 '0'
	option ifname 'eth1.3'
	option ipaddr '192.168.3.30'
	option netmask '255.255.255.0'
	option gateway '192.168.3.1'
	option label 'AT&T #1'
	option multipath 'off'

config interface 'wan2'
	option proto 'static'
	option ip4table 'wan'
	option defaultroute '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option metric '4'
	option peerdns '0'
	option ipv6 '0'
	option ifname 'eth1.4'
	option ipaddr '192.168.4.40'
	option netmask '255.255.255.0'
	option gateway '192.168.4.1'
	option label 'Viaero'
	option multipath 'backup'

config interface 'wan3'
	option proto 'static'
	option ip4table 'wan'
	option defaultroute '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option peerdns '0'
	option ipv6 '0'
	option ifname 'eth1.6'
	option ipaddr '192.168.6.60'
	option netmask '255.255.255.0'
	option gateway '192.168.6.1'
	option multipath 'off'
	option metric '5'

config interface 'wan4'
	option proto 'static'
	option ip4table 'wan'
	option defaultroute '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option peerdns '0'
	option ipv6 '0'
	option ipaddr '192.168.7.70'
	option netmask '255.255.255.0'
	option gateway '192.168.7.1'
	option ifname 'eth1.7'
	option metric '6'
	option label 'NanoStation Loco M5'
	option multipath 'off'

config interface 'wan5'
	option ifname 'eth4'
	option ip4table 'wan'
	option defaultroute '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option peerdns '0'
	option proto 'dhcp'
	option ipv6 '0'
	option metric '7'
	option label 'Verizon MiFi 8800L'
	option multipath 'off'

config interface 'wan6'
	option ifname 'eth1.8'
	option defaultroute '0'
	option peerdns '0'
	option proto 'static'
	option label 'Bullet M2HP'
	option ipaddr '192.168.8.80'
	option netmask '255.255.255.0'
	option gateway '192.168.8.1'
	option ipv6 '0'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option multipath 'off'
	option metric '8'

config interface 'wan7'
	option ip4table 'wan'
	option ipv6 '0'
	option proto 'static'
	option netmask '255.255.255.0'
	option ifname 'eth1'
	option metric '9'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option defaultroute '0'
	option peerdns '0'
	option label 'Fiber'
	option ipaddr '192.168.1.10'
	option gateway '192.168.1.1'
	option multipath 'master'

config interface 'wan8'
	option ifname 'eth1.9'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '192.168.9.90'
	option gateway '192.168.9.1'
	option defaultroute '0'
	option peerdns '0'
	option ipv6 '0'
	option metric '10'
	option macaddr 'xx:xx:xx:xx:xx:xx'
	option label 'AT&T #2'
	option multipath 'backup'

config interface 'omrvpn'
	option ifname 'tun0'
	option ip4table 'vpn'
	option multipath 'off'
	option leasetime '12h'
	option type 'tunnel'
	option txqueuelen '1000'
	option ipv6 '0'
	option proto 'none'
	option metric '11'

config interface 'omr6in4'
	option proto '6in4'
	option ip4table 'vpn'
	option multipath 'off'
	option gateway 'fe80::a00:1'
	option ip6addr 'fe80::a00:2'
	option auto '0'
	option metric '12'
	option ipaddr '10.255.251.2'
	option peeraddr '10.255.251.1'

[RussWestrem]

There are three ways I've seen Ethernet ports handled on versions of OpenWRT:

The first is when there is support for an integrated switch (driver required), and in this case there will be a menu item under Network - Switch.

[/code]

Ok. That's what this r7800 is doing. So I assume I just need to adjust the switch to use, say the wan port, as the single lan and the other 4 lan ports my physical wans. Or, to make things more simple I should just hook the lan into my switch and use macvlan for the other wan ports on any switch in the house. Does this sound correct? The .sh that I'm compiling with from openmptcp obviously isn't setting up the physical switch like they do with the wrt3200 and I'm not sure if it's as easy as just changing the settings on the switch or not.

[tlkelley]

Are you guys suggeting that MikroTik RBM33G be openmptcrouter? I ask as that was what I thought and I do not see a download image for it?

[RussWestrem]

You can compile the firmware and see.

[bnhf2]

Are you guys suggeting that MikroTik RBM33G be openmptcrouter? I ask as that was what I thought and I do not see a download image for it?

We're not suggesting that -- and it's not powerful enough. Everything we've been talking about related to the RBM33G, is for downstream routers -- on the WAN side of an OpenMPTCProuter device. An RBM33G running ROOter makes an excellent gateway, to combine with other gateways, and then aggregated into a single high-speed link with OMR.

Choose one of the few supported consumer routers with enough processing power to do OMR right, use a Raspberry Pi, or an industrial PC (those spec'd for pfSense are a good choice).

[RussWestrem]

Here's another post I did earlier this year on the ROOter forum, which is now a bit difficult to find. I did this post in late January, and I've since moved from the Raspberry Pi 4B to a Qotom x86 (i3) industrial PC with 4x gigabit Ethernet ports, 2x USB3 and 2x USB2 ports. OpenMPTCProuter needs an Internet appliance with some horsepower, but it has proved amazing for true WAN bonding -- there's little else like it! With my low end VPS (1vCore, 1GB RAM, 25GB vSSD), I'm frequently topping out at about 235Mbps on the D/L. I'll bring a 2vCore VPS online sometime soon.

I'm happy to say after a number of failed attempts at getting WAN bonding working properly — I've finally got it functioning the way I want. Like many other projects, it's easy to get off track without a little guidance, and OpenMPTCProuter is a bit thin on tutorials. Here's how I did it, just in case there are others that have WAN bonding on their "to do" list.

OpenMPTCProuter can be found here:

https://www.openmptcprouter.com/

Download whatever version will work with that extra bit of kit you have laying around and get it up-and-running. It's OpenWRT based, so this should be familiar territory for everyone here. I used a Raspberry Pi 4 with 4GB of RAM:

https://www.openmptcprouter.com/download

ext4 images are preferred over squashfs, when both are offered.

Before you start configuring your OpenMPTCProuter device, you'll want to get your VPS (Virtual Private Server) running, as WAN bonding always depends on having a server on the other end with a very high speed Internet connection. You can bond up to 8 connections according to the developer, and those need to be reconstituted into a single connection (with a public IP!) on the server end.

The developer has written scripts for specific versions of Debian and Ubuntu, so be sure to use a supported Linux when you setup your VPS. I'm using a Vultr Cloud Computer VPS with 1GB of RAM, a 25GB SSD and a single CPU core. For Linux, I went with Debian 10 x64 for US$5/mo. For that half a sawbuck, you get your very own virtual server and public IP address:

https://www.vultr.com/

Once your server is running, you can SSH (PuTTY works great for Windows users) into it and execute the script as shown in the wiki:

https://github.com/Ysurac/openmptcprout ... te-the-VPS

Once the script is done running be sure to grab /root/openmptcprouter_config.txt as this contains keys and other useful information you'll need when setting up your WAN bonding router. WinSCP is perfect for connecting to your virtual server (before you reboot it!) to grab the file (use SCP protocol just like with ROOter). You can also keep a copy of the PuTTY output from running the script, as the information you need is at the end of the script execution too. Note that the script changes your SSH port to 65222.

Now that your VPS is online, it's time to get your router configured. OpenMPTCProuter uses 192.168.100.1 as an IP address. Set your default admin password just like in ROOter and then proceed to Network – Interfaces to setup WAN1 and WAN2. In my case, I already have both of my cellular WAN devices on a single Ethernet network, each using its own VLAN ID, as part of the "network of gateways" approach I've posted about previously.

https://github.com/Ysurac/openmptcprout ... er-install

I won't cover the specifics of that again here other than to say one of my WAN routers is on VLAN3 with an IP address of 192.168.3.1 and the other is on VLAN4 with an IP address of 192.168.4.1. So when setting up WAN1 and WAN2 in OpenMPTCProuter I used physical "custom interface(s)" of eth1.3 and eth1.4 with static addresses of 192.168.3.30 and 192.168.4.40 respectively. My LAN is on eth0.

https://github.com/Ysurac/openmptcprout ... figuration

After that navigate to System – OpenMPTCProuter – Settings Wizard and enter the IP address of your VPS server, along with "server key" (which auto-loads all the other keys you'll need). From there double-check that WAN1 and WAN2 have the appropriate IP addresses for your downstream ROOter routers. If these routers are on a common switch like mine you'll want to either use VLANs or OpenMPTCP router also supports something called "macvlan" which should do the same thing. If the various WAN routers that you'd like to bond are all physically separate (individual Ethernet or USB cables to your WAN bonding device, then no VLAN or macvlan is needed).

After that go to System – OpenMPTCProuter – Status, and you should see a webpage that looks something like this (drum roll please):

screenshot-192.168.100.1-2020.01.png

If not, the first thing to try is to reboot your WAN bonding router, as there are a number of moving parts here including a VPN.

In my case I'm bonding a Sprint connection using one of my bnhf outdoor directional ROOter setups (RBM33G routerboard, em7565 modem and a pair of directional weBoost wide band antennas cross-polarized) which has been typically giving me about 120Mbps down and 10Mbps up, and an AT&T outdoor omni-directional ROOter setup (RBM33G routerboard, em7565 modem and a pair of weBoost 4G-OTR omni-directional antennas) which has typically been giving me about 40Mbps down and about the same up. The results of the new, bonded, VPN protected connection, complete with a public IP (and port forwarding) are exactly what I had hoped for, with the bonded connection giving me Speedtest.net results of 165Mbps down and 45Mbps up:

screenshot-www.speedtest.net-2020.01.png

In the next week or so, I'll add another cellular provider (Verizon) and a 50Mbps+ hardwire connection I have available at the moment, to see how additional WAN bonding scales on the RPi4.

I'm curious why you use one modem per router on your gateway of modems. I just put all my modems on a single router (cba850) that lives outside and setup multiple gateways attached to wans on each of their own vlans and bring that into the house to my openmptcp router on a single cat 5e cable.

[tbenz]

This is fantastic! Thank you!

I got it going despite some head scratching moments-- I have a static IP on one WAN line (required setting OpenMPTCP to DHCP on that interface) and the physical ports on the Qotom device don't align with the logical ports in the software.

It works great now but before I integrate it into my network I thought I would ask those much smarter than myself:

Is it possible to leave my existing router in place and install the OpnMPTCP router in a (pseudo) bridge mode to the WAN of my Asus (with Merlin) router? Since I need to attach the Asus anyway for wifi/mesh, I would like to keep it handling all the NAT/routing functions and have the MPTCP router be the gateway and handle just the WAN bonding and delivery. Not sure if this is possible due to the "magic" that's happening. Pretty sure I could just disable DHCP on the MPTCP router LAN and assign the appropriate IP's to my existing router but then I suspect I would be having NAT issues. Could I set my existing router in a DMZ on the MPTCP router? Again - not sure how this would interact with the MPTCP magic. Is there a best way to do this or should I buck up and set up the MPTCP router as my main router and relegate my Asus as an access point


My primary objectives/uses:
-Ability to adjust ttl (currently done in my Asus router). I'm not sure of the effectiveness of ttl adjustments being that everything will now be run through the MPTCP VPN. Will this particular VPN affect ttl? Is there a way to be able to present the ttl that the provider is expecting when running this system?

-Ability to access my network externally through a static IP. This is currently done via my static IP on one of my WANs. I will now have another public IP I can work with via the VPS on Vultr.

-Streaming and general internet access. No gaming

Appreciate all the thoughts and advice!

[RussWestrem]

This is fantastic! Thank you!

I got it going despite some head scratching moments-- I have a static IP on one WAN line (required setting OpenMPTCP to DHCP on that interface) and the physical ports on the Qotom device don't align with the logical ports in the software.

It works great now but before I integrate it into my network I thought I would ask those much smarter than myself:

Is it possible to leave my existing router in place and install the OpnMPTCP router in a (pseudo) bridge mode to the WAN of my Asus (with Merlin) router? Since I need to attach the Asus anyway for wifi/mesh, I would like to keep it handling all the NAT/routing functions and have the MPTCP router be the gateway and handle just the WAN bonding and delivery. Not sure if this is possible due to the "magic" that's happening. Pretty sure I could just disable DHCP on the MPTCP router LAN and assign the appropriate IP's to my existing router but then I suspect I would be having NAT issues. Could I set my existing router in a DMZ on the MPTCP router? Again - not sure how this would interact with the MPTCP magic. Is there a best way to do this or should I buck up and set up the MPTCP router as my main router and relegate my Asus as an access point


My primary objectives/uses:
-Ability to adjust ttl (currently done in my Asus router). I'm not sure of the effectiveness of ttl adjustments being that everything will now be run through the MPTCP VPN. Will this particular VPN affect ttl? Is there a way to be able to present the ttl that the provider is expecting when running this system?

-Ability to access my network externally through a static IP. This is currently done via my static IP on one of my WANs. I will now have another public IP I can work with via the VPS on Vultr.

-Streaming and general internet access. No gaming

Appreciate all the thoughts and advice!

This might help you. This example is for pfsense but it should be the same.

https://github.com/Ysurac/openmptcprouter/wiki/pfSense

[tbenz]

Thank you.
I'll see what I can do with that

[tbenz]

I've integrated the OpenMPTCP router(OMR)device into my network and have an anomaly. I set up the OMR as a gateway to my existing router (Asus) as I want to keep the routing/NAT functions on my Asus - OMR LAN port to Asus WAN port

-I disabled DHCP on the OMR by selecting the "Ignore interface" checkbox in the LAN interface settings tab
-kept the default IP on the OMR - 192.168.100.1
-set a static IP in the WAN settings in my Asus: WAN IP 192.168.100.2 Gateway: 192.168.100.1

In the OMR status tab I get this:

OMR ovrvw.png

When I was setting this up, prior to attaching it to my Asus (I was dirctly connected to the OMR with my PC), and DHCP on the OMR LAN was enabled, I got the green checkmark and had internet access. When I disabled DHCP and set a static IP on my PC, I got the error message that's shown in the above pic - and no internet. But now, with the OMR connected to my Asus, with the same error message showing up, I have internet! (I'm connected to my Asus wifi with my PC). Is there something else that I need to do in addition to just disabling DHCP in the LAN interface to get the Asus IP leased by the OMR - and subsequently the green checkmark?
Thanks!

BTW- thanks for the link RussWestrem. Port forwarding is working great following those instructions.

[Yipzy]

Ignore the message. It's fine because you are manually setting the IP for your router instead of having OMR dishing out. You could also set TTL in the firewall's custom rules just like you would before in other LTE routers with OpenWRT. You just need to reference the correct interface for each rule.

[tbenz]

Thank you Yipzy!

[prideauxx]

Hello! I've successfully setup an OpenMPTCProuter on an Intel-i3 based SBC with 8 Gigabit NIC's (industrial PC/router enclosure) I imported from China (the usual suspects). Pretty much everything went without a hitch, save attempting to find a VPS provider that could potentially provide a residential IP and was not totally sketch. All of this done, I'm happy so far with the setup with very few remaining issues to resolve or complaints. It looks as if the streaming throttling from my two ISP MVNO providers has been bypassed, as fast.com provides about 50-60MB/s down now vs. 4-5MB/s down before. Latency can get fairly high (worst case in the 200ms range), but that is completely understandable given what I am doing, and I'm not a gamer anymore, so no big deal there either.

What I am running into (and what I LOVE about the OpenMPTCPRouter interface) is reported through the MPTCP router UI's 'vnStat Traffic Monitor' page. I'm using quite a bit of data, even at what should be low-usage times. Downloads appear to be in the 100MB/hour range throughout the night. Now this could be on my end regarding a device/entity on my network (hunting that down still), or it could be the MPTCP router's failover mechanisms being a little too-reliable (exchanging state between the VPS/sending pings too often?).

So here's my question regarding the MPTCP router side of things--it is mostly using all default values. It looks as if what I may wish to tweak regarding failover checks is under the UI's 'OMR-Tracker' page. Does anyone have experience with this who can provide some advice/counsel? Thank you in advance.

[Yipzy]

What VPS provider did you go with that its ips are not being blacklisted yet for streaming from Netflix? Could your excessive data be LAN instead of WAN data which doesn't matter? Mine uses very limited data on idle but I'm on the previous version and I haven't updated yet. Most of settings are at default as well.

[RussWestrem]

Hello! I've successfully setup an OpenMPTCProuter on an Intel-i3 based SBC with 8 Gigabit NIC's (industrial PC/router enclosure) I imported from China (the usual suspects). Pretty much everything went without a hitch, save attempting to find a VPS provider that could potentially provide a residential IP and was not totally sketch. All of this done, I'm happy so far with the setup with very few remaining issues to resolve or complaints. It looks as if the streaming throttling from my two ISP MVNO providers has been bypassed, as fast.com provides about 50-60MB/s down now vs. 4-5MB/s down before. Latency can get fairly high (worst case in the 200ms range), but that is completely understandable given what I am doing, and I'm not a gamer anymore, so no big deal there either.

What I am running into (and what I LOVE about the OpenMPTCPRouter interface) is reported through the MPTCP router UI's 'vnStat Traffic Monitor' page. I'm using quite a bit of data, even at what should be low-usage times. Downloads appear to be in the 100MB/hour range throughout the night. Now this could be on my end regarding a device/entity on my network (hunting that down still), or it could be the MPTCP router's failover mechanisms being a little too-reliable (exchanging state between the VPS/sending pings too often?).

So here's my question regarding the MPTCP router side of things--it is mostly using all default values. It looks as if what I may wish to tweak regarding failover checks is under the UI's 'OMR-Tracker' page. Does anyone have experience with this who can provide some advice/counsel? Thank you in advance.

I'd also like to know what VPS you are using that doesn't get blocked by the streaming services.

[Keymaster4225]

Ok, new to the board, and still learning a bit, but trying to jump into this. I have a nighthawk M1 with at&t ipad plan along with a tmobile phone that I want to usb tether that I want to bond with OMR. I also have a gl.inet ar750s router. Is there any way to use a VM through virtualbox to use as the OMR router? I know they have VM images but I can not for the life of me figure out how to get that setup with my two lte services.

If that's going to be too hard I want to get a raspberry pi to use. Seems like that would be easier. Just use usb to connect both the nighthawk and the phone, and run OMR on that. Am I on the right track here?

[topbilling]

Wow. This information is amazing! Thank you!

Question for the OP or anyone else with experience: using these tools, is it possible to track data usage on one WAN and completely failover to another once a cap is hit? I'd like to combine a wireless line of sight ISP which is capped daily with an LTE WAN. I'd like to set it so that all traffic falls back to the LTE once I've used X GB per day on the other connection.

[bnhf2]

Wow. This information is amazing! Thank you!

Question for the OP or anyone else with experience: using these tools, is it possible to track data usage on one WAN and completely failover to another once a cap is hit? I'd like to combine a wireless line of sight ISP which is capped daily with an LTE WAN. I'd like to set it so that all traffic falls back to the LTE once I've used X GB per day on the other connection.

Given what you're looking for, I'd recommend focusing on a failover/failback solution rather than going with WAN bonding. No reason to add a VPS into the equation if you're only looking to use one WAN at a time. ROOter would probably be your best choice (a fork of OpenWRT focused on cellular Internet).

I believe ROOter has what you need for managing data caps too. It can be done with vanilla OpenWRT as well, though ROOter is more of a ready to go GUI-based solution.

[topbilling]

Given what you're looking for, I'd recommend focusing on a failover/failback solution rather than going with WAN bonding. No reason to add a VPS into the equation if you're only looking to use one WAN at a time. ROOter would probably be your best choice (a fork of OpenWRT focused on cellular Internet).

I believe ROOter has what you need for managing data caps too. It can be done with vanilla OpenWRT as well, though ROOter is more of a ready to go GUI-based solution.

Thanks. I'm actually looking for both. Higher speeds through bonding and cap management.

[prideauxx]

Quick question--

Has anyone attempted to run a VPN or proxy at the VPS site to further obfuscate/secure connection and bypass blocking done for having a datacenter IP address?

Have some time off and may attempt this, but I'm honestly not sure if possible. Considering Pihole can be run from VPS site, this may be a promising alternative (if one is ok with the latency induced).

[tbenz]

Hello again.
Bonding has been working perfectly for a while now but I'm having issues getting OMR bypass working. I'm pretty sure I'm doing something wrong and not understanding routing functions thoroughly.
Here's a screenshot of my current status:

OMR cap.png

I'm trying to get my streaming device on my local network to bypass the bonding function and go direct out the master connection. The WAN side of my router receives it's IP (192.168.100.2) via DHCP from the OMR device. The LAN side of my router dishes out 192.168.0.xxx addresses, The IP of my streaming device is 192.168.0.30. I've tried putting this IP in 2 different locations in the OMR Bypass screen - in the "IPs and Networks" section and in the "Source LAN IP address or network" section. Neither are working for me.

I'm a little bit out of my realm and seem to be spinning my wheels. In my simplistic thinking it appears the OMR device is not recognizing the local IP of my streaming device and handling all traffic coming from my router the same way. Do I need to be doing something different with my local device IP to have the OMR device recognize it (static route, NAT changes, DHCP changes etc)? Thanks for your help!

[Dr-BroadBand]

You have T-mo, Starlink and ATT Cell??.....WoW

What are you using this setup for?

[tbenz]

Lol, yeah. Started with 1.5 DSL for 15 years. Then I got completely fed up and discovered LTE possibilities and ran with that for 8 months or so. Then, within a recent 2 week time period, I became eligible for T-Mo Home and Starlink Beta. When it rains, it pours I guess. I have speeds approaching 300mb down with this setup. So now I'm making up for years of missing bandwidth

I'm in an evaluation period at the moment. Monthly costs for this setup is $170 and not sustainable. The ATT $20 plan is somewhat weaker comparatively (20-30 down) and just along for the ride for now. That sim will be going back into my Ipad eventually. I will be settling on either T-Mo or Starlink once the coverage of Starlink sats is expanded in a month or so. For the time being Starlink is experiencing frequent beta hiccups with interruptions which will most likely get better once coverage is complete.

[bnhf2]

I'm trying to get my streaming device on my local network to bypass the bonding function and go direct out the master connection. The WAN side of my router receives it's IP (192.168.100.2) via DHCP from the OMR device. The LAN side of my router dishes out 192.168.0.xxx addresses, The IP of my streaming device is 192.168.0.30. I've tried putting this IP in 2 different locations in the OMR Bypass screen - in the "IPs and Networks" section and in the "Source LAN IP address or network" section. Neither are working for me.

You'll need to make some changes to the way you've set things up in order to have OMR Bypass work. With your current configuration your entire LAN is a single IP (192.168.100.2), as far as OMR is concerned, since you have another router (with its own DHCP server) in between OMR and your streaming device.

How about re-configuring your Asus router as an Access Point. That way it'll handle the WIFi for you, but OMR will become your primary router and DHCP server?

[tbenz]

I thought about that as sort of a last resort. As I have many port-forwards, other routing/firewall configurations plus an Asus mesh system that I would need to accommodate. I've resisted up to now as I'm familiar with the Asus (w/Asuswrt-Merlin OS) and have not been too befuddled by it's somewhat simplified interface. I'm new to the tremendously open platform of OpenWrt/GoldenOrb/OMR and have made a lot of ignorant mistakes in my explorations (lots of backup/restores). I know that once I dive in deeper I'll learn a lot more but it'll take a bit of time. Just being lazy I suppose. But I find myself with lots of time these days so I guess I need to jump in the deep end.
Thanks for your guidance (and forced education) with this whole process!

[bnhf2]

I thought about that as sort of a last resort. As I have many port-forwards, other routing/firewall configurations plus an Asus mesh system that I would need to accommodate. I've resisted up to now as I'm familiar with the Asus (w/Asuswrt-Merlin OS) and have not been too befuddled by it's somewhat simplified interface. I'm new to the tremendously open platform of OpenWrt/GoldenOrb/OMR and have made a lot of ignorant mistakes in my explorations (lots of backup/restores). I know that once I dive in deeper I'll learn a lot more but it'll take a bit of time. Just being lazy I suppose. But I find myself with lots of time these days so I guess I need to jump in the deep end.
Thanks for your guidance (and forced education) with this whole process!

As things stand now, unless you've also added those port forwards into OMR, and forwarded those ports to OMR from the VPS -- none of them would be functional. I.E., any ports you want forwarded from the Internet need to come from the VPS to OMR, and then from OMR to the ASUS, and then from the ASUS to the final destination on your LAN. Port forwarding can withstand your extra layer of NAT (if configured correctly), OMR Bypasss cannot.

I'm a big fan of AsusWRT-Merlin myself, so I understand your reluctance. The good news about OMR, as compared to vanilla OpenWRT, is that all of the packages you'll need to duplicate the Merlin experience are already installed -- and everything can be done through the GUI.

[tbenz]

Yes, I've forwarded the ports from VPS to OMR to Asus to device. They're all working at this point.
I'll probably jump into this tomorrow as I have an important Zoom call later and I don't need to mess up my internet right now.

I'll likely be back here with questions and problems begging for advice

[prideauxx]

I'm trying to get my streaming device on my local network to bypass the bonding function and go direct out the master connection. The WAN side of my router receives it's IP (192.168.100.2) via DHCP from the OMR device. The LAN side of my router dishes out 192.168.0.xxx addresses, The IP of my streaming device is 192.168.0.30. I've tried putting this IP in 2 different locations in the OMR Bypass screen - in the "IPs and Networks" section and in the "Source LAN IP address or network" section. Neither are working for me.

I'm a little bit out of my realm and seem to be spinning my wheels. In my simplistic thinking it appears the OMR device is not recognizing the local IP of my streaming device and handling all traffic coming from my router the same way. Do I need to be doing something different with my local device IP to have the OMR device recognize it (static route, NAT changes, DHCP changes etc)? Thanks for your help!

Hello--Hope this helps, for what it is worth. By the way, I have a similar setup, with a Samsung Smart TV. While I am certain there are likely alternate/better routes to go, I did try them and did not have much luck.

• On the OMR side of things, under Services->OMR-Bypass, I set 'Source lan IP address or network' to be my wireless access point. Note: Again, I tried to be more surgical than this, but this is how I got it to work. After setting the OMR bypass, I selected 'Save & Apply'. I then rebooted the router. I know, pretty extreme, but this was the only way I found that would work for the TV in question and Prime
• On the TV side of things, I hard rebooted the TV after the router reboot. By hard reboot the TV, I mean pulling the plug on it, or pressing power on the remote until the TV's OS reboots.
• After the TV rebooted, I no longer got the VPN nag from Prime
• After I was done watching TV, I reversed the process on the OMR side of things

[tbenz]

Thanks Prideauxx! Any and all suggestions are certainly welcome and appreciated.

I think I'm understanding what you're saying. The way I'm set up now, if I set bypass to my router IP, all traffic is bypassed since all traffic goes through my Asus router. That works but I'm trying to be as you say - more surgical - by avoiding the requirement to keep adjusting settings. I'd like to have just my streaming device bypassed to my fastest WAN and the remainder of my bandwidth hogs (security cams) and other devices connected to all the bonded WANs. I know I could run separate networks and incorporate in some VLAN strategies but again, I've tended towards finding the lazy route. This stuff takes a long time for me as I need to research and become educated every step of the way. It's been a long educational process up to now. It's a self-inflicted issue for sure.

I had a thought that maybe I could set up a static route(s) from my streaming device to the IP of my OMR device but I don't really know if it would work or is even possible (being the uneducated rookie that I am).

[mtl26637]

I had pretty much this exact issue and spent way to much time on trying to set it up through the double NAT. In the end I finally gave in and set my main netgear router as an access point and let the OMR device hand out all ip's via DHCP. Now that I have it switched over and working there is no way I'd go back to the double NAT setup. Things are so much simpler with the OMR device being the only boss.

On a side note, you could always keep your setup as is and just plug the streaming device in to the OMR. You wouldn't have access to the streaming device from the 'LAN' side but depending on what it is, I don't think you would need to in most cases.

[tbenz]

Thanks mtl26637!
I thought about going direct from the streaming device to OMR. Being that the OMR device has only 4 ports (1 LAN, 3 WAN). "bandwidth greed" set in and I didn't want to sacrifice a port and hoped for another simple solution.

I'll be doing what you did - switch routing over to OMR.

[mtl26637]

Thanks mtl26637!
I thought about going direct from the streaming device to OMR. Being that the OMR device has only 4 ports (1 LAN, 3 WAN). "bandwidth greed" set in and I didn't want to sacrifice a port and hoped for another simple solution.

Actually it may be as simple as you had hoped. You wont need to sacrifice a WAN port. Take a simple ethernet switch and plug into OMR LAN. The rest of the ports on the switch will now be 'LAN' ports. No configuration such as VLAN or anything needed. They are fairly cheap if you don't have one, you can get a 'dumb' switch for around ~20 bux or so. If you configure your current router as an 'access point' it will be serving the same purpose as a switch also.

[tbenz]

I decided to switch all routing functions over to my OMR device. So far, so good.
My T-Mo Home WAN is currently setup like this: Sierra Modem in USB sled to GL router to OMR Eth3. I'd like to eliminate the router middleman and run the USB sled directly into the OMR device (Qotom NUC) into one of it's USB3 ports. I don't have a clue how to set up the OMR device to accommodate this. Anyone have any guidance or somewhere to point me?
Thanks!

[topbilling]

This looks like a really fun project. Quick question before I invest the time... On my best WAN connection, I average around 50 ms latency to the closest VPS provider. I understand that this is my floor once I start bonding other connections. As far as overall bandwidth, will I still see an improvement after bonding even with such relatively high latency?

I saw another writeup that mentioned needing 20 ms max for an OpenMPTC instance to work well: https://milankragujevic.com/openmptcpro ... -for-cheap

Thanks!

[tbenz]

The latencies for the 3 WAN sources in my setup range from 35 to 60 ms individually (2 LTE and Starlink) Combined it is around 50 or so. Tremendous improvement in bandwidth. Pretty close to 100% efficient aggregation. Works surprisingly well.

[topbilling]

This is proving harder than I would have thought. I'm struggling to get the VPS setup. I lost hours trying to update Ubuntu (kept getting broken pipe message) and then gave up and switched VPS to Debian.

Now, I'm unable to get OpenMPTPC installed properly. I execute this command:

wget -O - https://www.openmptcprouter.com/server/ ... -x86_64.sh | sh

But then the config file with the server keys is missing. I'll take another look through the thread, but if anyone has ideas I'd be grateful.

[topbilling]

I figured it out. Have to use a KVM VPS instance. If the OP is still checking you may want to update the first post for the other noobs out there.

[topbilling]

I'm trying to bond two Verizon LTE connections, but the speed and latency are terrible. One of the Verizon connections (WAN1) passes the Multipath test. The other (WAN2), which is a Hotspot plan attached to a WG1608 running Rooter, times out. Do I need to add/enable TCP Multipath on the WAN2 Rooter or does the OpenMPTCP router handle all of that?

Image 12-22-20 at 10.37 AM.jpg

[crumps]

Found the domains for HBO Max by running a DNS log on my router.

conviva.com
hbo.com
hbomax.com



They showed up like this in the logs from a fresh start of the app on my TV:
1608692013,10.1.1.237,comet.api.hbo.com
1608692014,10.1.1.237,commerce.api.hbo.com
1608692015,10.1.1.237,telegraph.api.hbo.com
1608692021,10.1.1.237,artist.api.cdn.hbo.com
1608692022,10.1.1.237,markers.api.hbo.com
1608692027,10.1.1.237,b3bb7de118d3a2fed1dd1ee77ed5a2bac10fe77b.cws.conviva.com
1608692027,10.1.1.237,dash.pro42.lv3.cdn.hbomax.com


Also found these Network prefixes on bgp.he.net
157.166.126.0/24
206.208.177.0/24
206.208.182.0/24
206.208.178.0/24

All of the above can be used with OMR-Bypass

[mtl26637]

As far as overall bandwidth, will I still see an improvement after bonding even with such relatively high latency?
I saw another writeup that mentioned needing 20 ms max for an OpenMPTC instance to work well: https://milankragujevic.com/openmptcpro ... -for-cheap
Thanks!

I wouldn't worry much about latency, esp. on initial setup. I've played with OMR for awhile now and have never had issues due to latency on any of the providers. ATT is ~25ms and VZW ~75.

I'm trying to bond two Verizon LTE connections, but the speed and latency are terrible. One of the Verizon connections (WAN1) passes the Multipath test. The other (WAN2), which is a Hotspot plan attached to a WG1608 running Rooter, times out. Do I need to add/enable TCP Multipath on the WAN2 Rooter or does the OpenMPTCP router handle all of that?

I've never worked with trying to run my own 'server' so not sure if problems may be related there. At first I tried AWS and had troubles so switched to VULTR and never looked back at $5/mo. Since then I've seen AWS is now supported so rather than trying to run a server initially you might try a free AWS server or paid VULTR to get it up and going and then could switch to your own server to rule out that side of things. Also if hosting your own server it could be issues 'between' your location and the server location and the server location has to have a pretty fast connection to the internet or it will be the bottleneck and this will be all for nothing.

As far as speed/latency it looks like from your screenshot that the server isn't set up properly so your won't see correct/accurate speeds on the connection. TCP Multipath should be enabled on all of your WAN connections except for VPN as that is how it actually does the aggregation to/from the server. OMR can be shaky after changing settings and then back so best bet would be to start back from scratch with a clean install on both server and router if possible until you get things up and going. I've seen where clean installs clear up or fix problems for others many many times in the past.

[BillA]

I'm trying to bond two Verizon LTE connections, but the speed and latency are terrible. One of the Verizon connections (WAN1) passes the Multipath test. The other (WAN2), which is a Hotspot plan attached to a WG1608 running Rooter, times out. Do I need to add/enable TCP Multipath on the WAN2 Rooter or does the OpenMPTCP router handle all of that?

I would test each connection individually by running a speed test (https://myip.speedtestcustom.com/) without connecting it to OMR (hotspot > ethernet > PC / don't use WiFi, it's just another point of fault). Those cheesy Verizon hotspot boxes without an ethernet port can cause all kinds of issues, including packet filtering which cannot be disabled.

Not sure if this could help, give it a try (from FAQ):
Status say that Multipath is blocked on the connection, what can I do?
MPTCP is filtered somewhere on the network, so you can't use it. You can disable ShadowSocks and enable Glorytun UDP or MLVPN, they don't use MPTCP and can aggregate connection.

[topbilling]

Thank you for the suggestions. I'm having a really hard time figuring this out...

Following the above advice, I've isolated WAN2 which is a prepaid VZW sim with hotspot enabled. I started by testing with the phone (Samsung A51) tethered via WiFi to my PC. It seems like VZW is intentionally increasing the latency (500+ MS) on hotspot connections. Speed test on the phone itself shows regular <100 ms latency. I'm unable to tether phone to PC via USB because MAC does not support android tethering without a 3rd party driver and I can't figure out how to defeat Apple's security restrictions on 3rd party drivers.

I then tried connecting the phone via USB to WG1608 router running GO and changing TTL. This did not help because VZW somehow has locked down the device so you can't change TTL. The only way around this is to either root the device or use an app called EasyTether, but there does not seem to be a driver for the OEM modem that Samsung uses in the A51.

Then, I pulled out the SIM and put it into the WG1608 directly. Here's where things got weird. I am unable to permanently set the TTL values. I tried in both the WG1608 and the router running OpenMPTCP and it always reverts back to VZW's TTL setting which spikes the latency and makes the connection basically unusable. I changed the TTL setting in the WG1608, ran a ping test and watched as the TTL setting reverted back in the middle of the test.

Screen Shot 2020-12-27 at 9.58.46 AM.png

I've ordered test SIMs from PagePlus and Simple Mobile since VZW clearly does not want me doing what I'm trying to do. That said, is there anything I can do to make the this prepaid plan usable? Am I doing something wrong with the TTL? Do I need to be changing DNS servers? IPv6 disable? SQM?

[topbilling]

Finally got it working and stable. Disney plus, Prime and Netflix all working. Speeds are still pretty slow, but I'm in a highly congested area. Will try a speed test later tonight.

This is a game changer for anyone who relies on LTE for their primary connection. Will be making a donation once everything is setup and running smoothly. Thanks to all those who contributed and helped with troubleshooting.

[BillA]

Finally got it working and stable. Disney plus, Prime and Netflix all working. Speeds are still pretty slow, but I'm in a highly congested area. Will try a speed test later tonight.

This is a game changer for anyone who relies on LTE for their primary connection. Will be making a donation once everything is setup and running smoothly. Thanks to all those who contributed and helped with troubleshooting.

What exactly did you tweak to get it to work?
By the way, it's always a good idea to disable IPv6, or edit the custom firewall settings to also change the TTL for IPv6.

[xdavidx]

I'd highly recommend the RBM33G, as it has a speedy processor (for a router product), 2x mPCIe slots, a USB3 port, real gigabit Ethernet, an m.2 SSD slot, a wide range of power voltages including passive PoE, and a price point similar to the RPi. For me the 24V PoE was a huge plus, as I already use Ubiquiti for all of my WiFi.

Thanks @bnhf2 for putting this all together for others to follow, and for helping people along the way.

What do you see as the advantages of the RBM33G or RMB11G vs the various Raspberry Pis? And what do you see as the advantage vs the typical ZBT routers that many use? (I haven't compared the specs or prices yet)

Regarding the OpenMPTCProuter device, I believe you started with a Raspberry Pi and then moved away to a much bigger machine (Qotom). What was the reason for that? Was it the file system issues when power was lost, or something else? The reason I'm asking is because I was thinking of going with a Raspberry Pi, since it is inexpensive.

For anyone who has implemented this solution, I have the following questions:

1) Has anyone tested dropping one of the WAN connections when on a VOIP call? If so, does the call continue uninterrupted?

The added speed of having multiple LTE modems aggregated together is great, but my main reason for looking into this solution is to prevent dropped VOIP calls. A member of the family spends 8 hours a day on VOIP calls and when the modem switches towers or the tower disconnects the modem or other glitches cause a router reboot, that effectively hangs up on the customers, which isn't good.

2) Has anyone noticed any voice quality issues during a VOIP call? I found someone mentioning online that a shared vCPU on the VPS could possibly fall behind at times and garble the voice quality. A dedicated vCPU should fix this, but that seems to be a very expensive option for most VPS providers.

3) Has anyone tested using a VPN client on the VPS server in order to avoid streaming issues (Netflix blocks, etc.)? I know the router can be configured to bypass the connection to the VPS for devices on the LAN, but our family streams from many different devices, and I'd rather not have to configure it for each of them, Additionally, some streaming is done from phones and PCs, and I'd rather those benefit from the aggregated connection and high availability/stability of multiple WAN connections. Lastly, 4K streaming may sometimes not be fully realized if a single LTE connection slows down (switches to a tower with less bandwidth or is congested, or started withholding bands, thus reducing bandwidth). It would be rare for it to get lower than what is needed for 1080P, but even that could happen if multiple devices are streaming at the same time. Having two WAN connections aggregated for streaming helps this situation. Not as critical as a VOIP call dropping, but still a nice to have.

I'm aware this would add latency, but I'd plan on picking a VPN server to be in the same city as the VPS server to minimize this.

[xdavidx]

Nothing to do with the questions above, but for people reading who are interested in what people are using and how their network are set up, mine consists of the following currently:

- 11 down/0.8 up DSL (currently not being used at all and I'll probably drop it if I add another LTE connection)

- WE1326 router and EM7565 modem. The router is in an external enclosure purchased from The Wireless Haven. It is on a 10 foot pole on the roof, on an old DirecTV dish mount, along with two directional, panel antennas from The Wireless Haven (https://thewirelesshaven.com/shop/antennas/4g-lte ... l-antenna/). Ethernet cable runs from there down a couple levels to the basement. Power is by extension cord right now, but will be setting up something in the attic come spring, with a DC extender and running the power out of the attic to the router.

- An old Linksys E3000 is my primary router. This is running a Shibby MultiWAN build of Tomato. MultiWAN, because at one point in the past, I had 2 DSL connections that it was load balancing. It serves as my firewall and DHCP server. I planned on leaving this in place and and putting the OpenMPTCProuter between it and the LTE routers. However, if accessing the LTE routers' UIs from the LAN side is more difficult by routing with multiple hops, then the OpenMPTCrouter may take over these duties. I like the device list and realtime bandwidth graphs Tomato supplies. I'm not sure if I'll get those with OpenMPTCProuter.

- An Asus RT-AC5300, in Access Point mode, does just that. Powerful wifi device, reaching all areas of the house. Connected to the E3000 over a long ethernet connection, to a central location in the house, because the E3000 is in the basement and that area isn't wifi friendly.

Has a horrible issue with handling ARP between devices on the LAN after it has been running for a while. Devices on the LAN aren't able to discover and connect with each other often without jumping through a lot of hoops. I may have found the magic setting to reduce this issue just recently, but I need to do more testing with it. (Disable "Airtime Fairness" on each of the wifi radios, if anyone else has this issue)

- AT&T account. When I first temporarily set this up in 2019, I could fairly consistently pick the tower I wanted based on directional aim. That was in the summer, with a lot of foliage. Now that it is winter and/or due to tower changes, it likes to jump around between towers. This makes it much harder to maximize speeds, since it could jump to a tower with less bandwidth.

I can typically get between 10 MHz and 25-30 MHz of bandwidth in aggregate (up to 3 CA), depending on the tower and time of day. This gives me anywhere from under 10 Mbps down up to 70+ Mbps peak down. Typical is in the 20-30 Mbps range. I can get under 10 Mbps up to around 20 Mbps up. Typical is 15 Mbps or so.

If OpenMPTCProuter works well (I'll test it with my LTE connection and my old DSL connection), I'll get another AT&T account and ditch the DSL. I'll probably go with a directional parabolic grid antenna with a much narrower beam width. Either one of the cheaper ones paired with a mimo feed horn (cuts off at 1700 MHz, so no Band 12), or one of the more expensive ones (600 MHz to 6500 MHz) as primary and something more modest for the secondary antenna. That should reduce the modem jumping around between towers a bit, because it will hear the one it is pointing at much better than the others nearby.

Photos:

20201220_roof_mount_from_ridge.jpeg

20201220_roof_mount_steep.jpeg

[xdavidx]

Update on my implementation...

Received the following pieces in the mail Sunday afternoon:

- Raspberry Pi 4 B with 8 GB of RAM (in a starter kit that had case and other stuff) - https://www.amazon.com/CanaKit-Raspberr ... B08956GVXN

- 5 port managed switch - https://www.amazon.com/TP-LINK-TL-SG105 ... B00N0OHEMA

- Pack of 5 Cat6 ethernet cables (went with 14' length) - https://www.amazon.com/Cable-Matters-5- ... B00E5I7YYS

Saturday, I had already rented a VPS with vultr (smallest instance, located in Chicago) and installed the OpenMPTCProuter software there.

I should mention, they have a deal now, where if you put $10 in your account with them, they will give you a $100 credit. The catch is the $100 has to be used within 30 days. That does allow you the freedom to try different sized servers though.

Make sure if you upgrade the server to a bigger size, that you take a snapshot first. And you can't revert to a smaller size. You have to delete the VPS and create a new one to go back. Snapshots can only be used to create an instance that is the same or larger than when the snapshot was taken. So if you are trying multiple sizes, take a snapshot of each size, so you can always go back to that size.

The reason I chose Chicago is because when I hit the AT&T towers where I live in Minnesota, the public IP endpoint is in Chicago. So any systems that identify where you are coming from always thinks I'm in Chicago. I wanted a VPS also in Chicago to minimize latency when my packets pop out on the internet, until they get to the VPS.

whatsmyip_att_chicago.png

As you can see here, there is only one hop exposed between my home network and my public IP (an IP that can't be accessed from the internet, due to commercial grade NAT). I blocked out my IP for security reasons.

traceroute_att_ip.png

And here is the trace to the VPS in Chicago. IP blocked out again. And this IP can be used to connect back to my network from the internet. One of the added benefits of this solution.

traceroute_vultr_ip.png

Now, it is theoretically possible that having the VPS in some other city could result in fewer hopes and less latency than going from one location to another location in Chicago, but that would require a lot of trial and error to find that needle in the haystack.

Here it shows how long it takes to go from the VPS to my AT&T public internet. I was logged into the VPS and pinging the AT&T public address:

ping_att_ip_from_vultr_vps.png

Under 2 ms is pretty close.

(continued in next post)

[xdavidx]

(continued...)

When my order arrived Sunday, from Amazon, I did the following:

- Set up vlans on the switch. I'm doing a one armed router setup, where the OpenMPTCProuter will send and receive all data on a single ethernet cable to/from the switch. I've set up port 1 on the switch to have tagged packets for each of my vlans, going into the new router.

vlan 1 = a default vlan that I removed all ports from
vlan 2 = for my LAN (OpenMPTCrouter on port 1, and my LAN router on port 5 and an extra port (4) if I need to plug in another device sometime)
vlan 3 = my DSL WAN connection and OpenMPTCProuter (port 2)
vlan 4 = my AT&T LTE WAN connection and OpenMPTCProuter (port 3)

- Wrote the OpenMPTCProuter firmware to a micro SD card with Win32DiskImager (https://sourceforge.net/projects/win32diskimager/).

- Booted the Pi with the router firmware and connected it to a laptop to configure it.

- I had some trial and error here of connecting it to the switch and connecting a laptop to the switch to see if they could communicate properly. So I had to go back and forth between a direct connection to the router and laptop and then testing between them through the switch. Main problems were:

- A self-signed cert for the SSL for the router management browser interface throws a flag for chrome browser, and even after telling it to go ahead anyway, it wouldn't. Took a long time to figure out it was my anti-virus blocking me.

- Not having worked with vlans before, I may have had some configuration tweaks to make in the switch.

- Getting the vlan configuration right on the router took some trial and error. The Pi doesn't have an internal switch, so everything is virtual and not as visually accessible in the user interface. When you create a new network interface, you have to know to manually enter it as eth0.2, eth0.10, eth0.20, etc. Those 3 represent my vlans, with subnets of 192.168.2.x (for the router on the LAN side), 192.168.10.x (for the DSL router), and 192.168.20.x (for the LTE router).

- The last hurdle was a realization that no firewall being configured for network interfaces in the router means that no packets travel, not that all packets travel.

- After making sure the laptop and router could communicate through the switch, I plugged the DSL modem into the switch, after adjusting its subnet configuration to match what I wanted. I believe the router couldn't communicate right way, due to some settings on the interface or firewall. I adjusted those settings and it could then see the DSL connection and communicate with the VPS. This provided a single legged WAN connection to the VPS, and gave my laptop on the switch access to the internet through that tunnel. Very minor performance drain by going through the tunnel.

- I played with that a while and then plugged my LTE WAN connection into the switch (after configuring the LTE router for the right subnet). Success! 2 connections turned into 1. (screenshot from a day later)

openmptcprouter_success.jpg

- I then plugged my regular LAN router into the switch. It didn't work at first. I had to release and renew its DHCP address to get a new one from the OpenMPTCProuter. I could set this as a static value, but I need to be able to revert to only using the LTE router instead of the full blown tunnel approach (by swapping cables around). Since the LTE router and the OpenMPTCProuter are on different subnets, I need the old LAN router to dynamically get on one or the other by renewing its DHCP address.

- Most things worked well on my LAN. The exceptions were certain streaming providers, like Netflix and Hulu. I played around with trying to configure the router to have these bypass the tunnel, but that's not as easy as it should be. The router provides many ways to do it, but most don't seem to work.

(continued in the next post)

[xdavidx]

(continued...)

The only two ways I found to get video streaming (for Netflix, Hulu, etc.) to work when the multipath tunnel was in place were:

1) Specify the IP address of the machine that was accessing the streaming service (my laptop, for instance). The downside to doing this is that the any devices using this method would then lose the speed boost of the aggregated connection. I'm not sure if they would loose the high availability aspect as well, if the link they were on went down. It is supposed to pick the master link of the tunnel, but it wasn't doing that, so I had to tell it which link to use. I'm guessing if that one link goes down, then connectivity would be lost for any clients bypassing the tunnel through that link.

It also means having to configure this for each machine that wants to stream. I could put in the IP of the LAN router, but then everything on my network would bypass the tunnel.

A person could set up a second wireless access point and have all true streaming devices (TVs, Rokus, etc.) connect to that AP, and phones/computers connect when they want to stream, but otherwise connect to the main AP that uses the tunnel. That creates extra complexity for users and creates congestion on wifi.

2) Specify domain names for the streaming services. In order for this option to work, all devices must have the OpenMPTCProuter specified as the DNS server. I normally have all my devices set to learn the DNS over DHCP. That means they get the LAN router's IP address. And then in that router, I have it using google's DNS servers (8.8.8.8, 8.8.4.4). So all my devices on the LAN end up using google's DNS servers indirectly too.

That doesn't work for this use case. I had to manually set my laptop's DNS server to the LAN address of the OpenMPTCProuter. Even changing the DNS server that the regular LAN router uses doesn't fix the problem, because it caches results, so future attempts to stream won't tell the OpenMPTCProuter that a filtered domain name is attempting to be used, hence it won't bypass the tunnel for the streaming requests that follow. It seems like it should be able to cache the IP address for those domain names and still recognize that they need to be bypassed, but it wasn't doing that in my case.

The problem with the domain name approach is that streaming services use a variety of domain names. Finding them all and plugging them all into the system would be a bit of work. And new ones could be used by the providers at a later point in time. It would be nice if OpenMPTCProuter could read from an online database to make this all work. One of the other options is to filter on "service", where the values are things like "netflix". So maybe it is doing that very thing for those, but it doesn't work for me.

Although a browser connection to netflix can bypass the tunnel by plugging in netflix.com to the router configuration, a mobile app or app on a streaming device doesn't use netflix.com as the domain, so it won't work. Again, you have to know all possible domain names that might be used and enter all those to cover all your bases. And you have to do that for all streaming providers.

- Another thing to about bypassing the tunnel for streaming services is, if one of your links is below 15-25 Mbps, or dips below that, then 4K streaming will be affected. Some people may be using multiple WAN links to get enough bandwidth to stream higher quality content.

A possible solution for that is to combine the VPS with a VPN, where the VPN is smart and has ways to stay off the blocklists of the streaming providers.

- I signed up for ExpressVPN, picking Chicago again as my location. I installed the OpenVPN client on my laptop to test it. the newest OpenVPN client won't properly process the configuration file generated from ExpressVPN. I had to use an older version of the client (version 2.3.8) to get around this issue.

- I then installed the OpenVPN client on the VPS and started up the VPN tunnel. If you do this when you are ssh'd into the VPS over the OpenMPTCProuter connection, you'll lose your ssh connection due to how packets are routed. I found a few commandlines online to generate the right routing settings to avoid this. However, the OpenMPTCProuter server wasn't using the VPN path to get out to the internet. This requires some other routing changes in linux, and I didn't try to figure that out.

- Instead, I used the vpn client on my laptop to establish a VPN tunnel, using the multipath tunnel as the transport. This has the same logical effect that you are splitting your packets across multiple WAN connections, but the endpoint on the internet side is the VPN server, so you can access picky streaming services. Unlike the previous option, it does mean double encryption of the data across the link between the router on premise and the VPS server on the internet.

It also means the multipath tunnel may see the packets differently and change which method it uses to route them, or may not do multipath at all. This is what happened to me when I used the OpenVPN client for Windows. I was only going across one WAN link at a time, for a given video stream.

Now that I think about it, I'm not sure if it was going across one line to the VPS, or avoiding the tunnel completely. I was watching the bandwidth usage on my vlans in the OpenMPTCProuter UI, and could see it was only using one at a time. Same with doing a speed test when set up this way.

Rather than trying to muck with poorly documented OpenMPTCProuter features for adjusting how it does its magic, I installed the ExpressVPN windows client and that did allow it to work as desired. Both streaming and non-streaming data was split across the two WAN links, then on to the VPN server, and then out to the internet. Unplug a WAN link, and it just keeps on chugging along. I will say that speedtest.net would sometimes glitch when doing this unplug test.

I left the ExpressVPN client settings at the defaults. That means letting it determine the best protocol to use for the VPN connection. My guess is that some of those methods will allow the multipath router to use both paths and some of them will not (at least not without changing settings on the router).

There is a reasonably small, but definite hit in performance of the connection when doing this VPN over the top of the multipath tunnel. Whether this the doublepath encryption, or extra latency of the VPN server is hard to tell. Later on, I did some testing with just the slow DSL WAN active, and it was more apparent in speed tests that I was losing bandwidth. The multipath tunnel might knock it from 11 Mbps to 9+ Mbps. But adding the VPN on top of that knocked it down to 4+ Mbps an sometimes slower. VPN alone over that doesn't affect the performance that much, so it is the intermingling of the two tunnels with each other that is having an additive effect. VPN over the top of the tunnel with both DSL and LTE going didn't have the same percentage drop vs no VPN tunnel. Maybe there is something about the DSL connection it doesn't like.

I haven't tried playing with VPN client settings to see if I can improve the performance. If a person has 2 reasonably fast WAN connections, then it probably won't be a big deal. But definitely something to be aware of. My main concern is, at least in my current state, where one leg is a slower DSL, I could saturate that DSL connection and then some packets could be delayed across it, which could affect more interactive things, like VOIP. That partly depends on how well the multipath software handles that situation, when a link is at capacity.

The other approach to doing the above, and which doesn't require every device to run its own VPN client, is to run a VPN client on a router. This can be done on my old LAN router (which runs Tomato firmware), or can be done by installing the VPN provider's firmware on a fresh router that you put inline with your main router, or it can be done by configuring OpenMPTCProuter to launch a VPN connection over the top of its multipath tunnel. I haven't taken the time yet to try these other approaches to see how they compare, performance-wise.

No matter which way you slice it, when you get into a tunnel on top of an extra fancy tunnel, the variables start to add up.

(continued in the next post)

[xdavidx]

(continued...)

- The other thing I tested, and the main reason I'm doing all this, was voice calling over this multipath connection. I only had time to try a wifi call over the connection to a service where you can record your voice and play it back. Someone in the house is on the phone the entire workday, and anytime the LTE modem connection is broken, their calls drop. Not good. A wifi call isn't the same the type of voice calling they're doing over a softphone on a PC, but I tested it anyway. I do make wifi calls periodically (due to poor cell reception on my phone).

The quality wasn't the best, but I don't know how much of that was due to the service I was calling and how much was due to my phone and how much was due to the multipath tunnel. Again, there are settings that can be changed in the router that would affect various VOIP protocols differently, but I haven't tried changing them.

I was able to unplug one link and my call normally didn't drop. I did have a couple drops, so I'm not sure why that was. I have a feeling that's more a function of something on my phone. It seems to make better wifi calls if I have the cellular enabled, even if the cellular has no bars and isn't being used in the call.

I need to do more testing with video conferencing software between two computers, and I'll try to install a true softphone and test that as well.

I still have a lot of testing to do, but my end results are:

1) I may need to fiddle with settings to get VOIP calls to be clearer and to still have the multipath benefit of not dropping calls if one link goes down temporarily.

2) If different settings don't fix this, then it may also be related to my slow DSL. When I have two LTE connections going at the same time, maybe it will be better. However, my LTE sometimes drops down to single digit speeds, depending on which tower it connects to and the congestion on the tower. I can't have poor quality audio when that happens either. Usually the upload speeds are decent even if the download speeds decrease. With my DSL connection, I only have about 0.8 Mbps for the upload speed, so anything that reduces that speed can have a negative effect on call quality.

3) A different LTE antenna setup could provide for a stronger signal, for more consistency and faster connections. It could also reduce how many times the LTE connection is dropped. That doesn't seem to be due to super poor signal quality, however.

4) A different LTE modem and/or different router hardware for the LTE router could reduce these brief connection drops.

1 and 2 are about using the multipath nature of the tunnel to solve for the call dropping problem. 3 and 4 are about improving one of the connections so that it drops less frequently (and only using one of the connections for the VOIP calls). It may only be 1 or 2 times in a day that it drops, but that's still not good in this situation.

Regarding improved performance and high availability, the OpenMPTCProuter is a winner.

[BillA]

- The other thing I tested, and the main reason I'm doing all this, was voice calling over this multipath connection. I only had time to try a wifi call over the connection to a service where you can record your voice and play it back. Someone in the house is on the phone the entire workday, and anytime the LTE modem connection is broken, their calls drop. Not good. A wifi call isn't the same the type of voice calling they're doing over a softphone on a PC, but I tested it anyway. I do make wifi calls periodically (due to poor cell reception on my phone).

Regarding improved performance and high availability, the OpenMPTCProuter is a winner.

Wow, now that was one heck of a long and detailed test.

Over the years I have tested many VOIP providers, along with client devices such as soft-phones, Asterisk, and ATA adapters. I've noticed that a VOIP connection's reliability depends on several factors, which includes the VOIP provider's platform/proxy, the client device being used, and of course the internet connection. A call can drop at any one of those points of fault, and not necessarily due to the internet connection alone.

With a certain combination of VOIP providers and access devices, I can literally disconnect the internet source for a few minutes, and once I reconnected it, the call would just continue without dropping. For example using Vitelity.com as my VOIP provider with a Linksys/Sipura ATA adapter, calls will not drop if there's a glitch in the internet connection, but calls will drop using the same provider with a soft-phone. Also the Obihai OBI202 adapter with Google voice seems to be reliable even with a flaky internet connection (https://www.amazon.com/OBi202-2-Port-Ad ... B007D930YO), and as a bonus it's completely free from Google (well, besides the cost of the box;).

Seem like running a VPN on top of OpenMPTCProuter is a little over complicated, giving you a hit on both speed and latency. There has to be a better way to bypass the streaming provider's IP security without having to use a clusterfuq of VPN's on top of VPN's. lol

Also, upgrading to a 5G modem would not only improve your speed and throughput for your family, but also reliability. While 5G service may not be available in your area, rest assured it's coming soon to a tower near you (in that creepy movie announcer guy's voice... muaahh!), it's just a matter of time.

[docderwood]

Hello,

I've bene using openMPTCP for quite some time. The VPS is on google fiber at our primary house (it's robust) with
AES and the router runs at our lake place (also I5 with AES).

I've tried to figure this out, but haven't been successful: How do you run all the data through a VPN between the two? I've got two ATT and one Verizon plan.