WAN Bonding with ROOter and OpenMPTCProuter

[bnhf2]

Here's another post I did earlier this year on the ROOter forum, which is now a bit difficult to find. I did this post in late January, and I've since moved from the Raspberry Pi 4B to a Qotom x86 (i3) industrial PC with 4x gigabit Ethernet ports, 2x USB3 and 2x USB2 ports. OpenMPTCProuter needs an Internet appliance with some horsepower, but it has proved amazing for true WAN bonding -- there's little else like it! With my low end VPS (1vCore, 1GB RAM, 25GB vSSD), I'm frequently topping out at about 235Mbps on the D/L. I'll bring a 2vCore VPS online sometime soon.

I'm happy to say after a number of failed attempts at getting WAN bonding working properly — I've finally got it functioning the way I want. Like many other projects, it's easy to get off track without a little guidance, and OpenMPTCProuter is a bit thin on tutorials. Here's how I did it, just in case there are others that have WAN bonding on their "to do" list.

OpenMPTCProuter can be found here:

https://www.openmptcprouter.com/

Download whatever version will work with that extra bit of kit you have laying around and get it up-and-running. It's OpenWRT based, so this should be familiar territory for everyone here. I used a Raspberry Pi 4 with 4GB of RAM:

https://www.openmptcprouter.com/download

ext4 images are preferred over squashfs, when both are offered.

Before you start configuring your OpenMPTCProuter device, you'll want to get your VPS (Virtual Private Server) running, as WAN bonding always depends on having a server on the other end with a very high speed Internet connection. You can bond up to 8 connections according to the developer, and those need to be reconstituted into a single connection (with a public IP!) on the server end.

The developer has written scripts for specific versions of Debian and Ubuntu, so be sure to use a supported Linux when you setup your VPS. I'm using a Vultr Cloud Computer VPS with 1GB of RAM, a 25GB SSD and a single CPU core. For Linux, I went with Debian 10 x64 for US$5/mo. For that half a sawbuck, you get your very own virtual server and public IP address:

https://www.vultr.com/

Once your server is running, you can SSH (PuTTY works great for Windows users) into it and execute the script as shown in the wiki:

https://github.com/Ysurac/openmptcprout ... te-the-VPS

Once the script is done running be sure to grab /root/openmptcprouter_config.txt as this contains keys and other useful information you'll need when setting up your WAN bonding router. WinSCP is perfect for connecting to your virtual server (before you reboot it!) to grab the file (use SCP protocol just like with ROOter). You can also keep a copy of the PuTTY output from running the script, as the information you need is at the end of the script execution too. Note that the script changes your SSH port to 65222.

Now that your VPS is online, it's time to get your router configured. OpenMPTCProuter uses 192.168.100.1 as an IP address. Set your default admin password just like in ROOter and then proceed to Network – Interfaces to setup WAN1 and WAN2. In my case, I already have both of my cellular WAN devices on a single Ethernet network, each using its own VLAN ID, as part of the "network of gateways" approach I've posted about previously.

https://github.com/Ysurac/openmptcprout ... er-install

I won't cover the specifics of that again here other than to say one of my WAN routers is on VLAN3 with an IP address of 192.168.3.1 and the other is on VLAN4 with an IP address of 192.168.4.1. So when setting up WAN1 and WAN2 in OpenMPTCProuter I used physical "custom interface(s)" of eth1.3 and eth1.4 with static addresses of 192.168.3.30 and 192.168.4.40 respectively. My LAN is on eth0.

https://github.com/Ysurac/openmptcprout ... figuration

After that navigate to System – OpenMPTCProuter – Settings Wizard and enter the IP address of your VPS server, along with "server key" (which auto-loads all the other keys you'll need). From there double-check that WAN1 and WAN2 have the appropriate IP addresses for your downstream ROOter routers. If these routers are on a common switch like mine you'll want to either use VLANs or OpenMPTCP router also supports something called "macvlan" which should do the same thing. If the various WAN routers that you'd like to bond are all physically separate (individual Ethernet or USB cables to your WAN bonding device, then no VLAN or macvlan is needed).

After that go to System – OpenMPTCProuter – Status, and you should see a webpage that looks something like this (drum roll please):

screenshot-192.168.100.1-2020.01.png

If not, the first thing to try is to reboot your WAN bonding router, as there are a number of moving parts here including a VPN.

In my case I'm bonding a Sprint connection using one of my bnhf outdoor directional ROOter setups (RBM33G routerboard, em7565 modem and a pair of directional weBoost wide band antennas cross-polarized) which has been typically giving me about 120Mbps down and 10Mbps up, and an AT&T outdoor omni-directional ROOter setup (RBM33G routerboard, em7565 modem and a pair of weBoost 4G-OTR omni-directional antennas) which has typically been giving me about 40Mbps down and about the same up. The results of the new, bonded, VPN protected connection, complete with a public IP (and port forwarding) are exactly what I had hoped for, with the bonded connection giving me Speedtest.net results of 165Mbps down and 45Mbps up:

screenshot-www.speedtest.net-2020.01.png

In the next week or so, I'll add another cellular provider (Verizon) and a 50Mbps+ hardwire connection I have available at the moment, to see how additional WAN bonding scales on the RPi4.

[bnhf2]

And here's the follow-on post I did, about a month later, after moving to the x86 router box:

If you missed my original post on this amazing upgrade to your ROOter setup (assuming you have at least two decent sources of Internet available), you might want to have a look at this post first whrl.pl/RfZ7Tq

What's got me enthused today, is just how fluidly OpenMPTCProuter is working on an x86 platform, combined with a trio of ROOter routers running on outdoor RBM33G setups! One of the ROOter boxes is even linked using a gigabit wireless bridge, rather than Ethernet. All of the ROOter boxes are living on the WAN side of my primary router in what I've been calling a Network of Gateways.

Here's the 4-port "firewall appliance" I'm using from Qotom via Amazon:

https://www.amazon.com/gp/product/B074XNYHL4/

In addition to some nice hardware specs for a router, I'll also give these guys a big thumbs-up for having it set to boot off a USB drive out-of-the box. There's a 16GB SSD inside this unit, and maybe I'll "flash it" at some point, but it's really convenient to be able to write an image to a USB3 flash drive, and boot from that — perfect for those of us that like to be able to reconfigure our setups with a minimum of fuss.

This unit is working with software VLANs too, which is great given all the different subnets commingling on the WAN side of my primary router. By running VLANs, I'm also able to access each of my ROOter routers by IP, which I was not able to do using macvlan on my previous Raspberry Pi 4 OMR box. My interface setup looks like this:

screenshot-192.168.100.1-2020.02.21-14_35_23.png

On first boot of OMR, the unit was configured with 1 LAN and 3 WAN ports, each on its own physical ethernet adapter — these units don't have internal switches. The LAN port is eth0, so be sure to connect there to access LuCI. Since I have a VLAN interface setup and available for each of my ROOter boxes, I reconfigured the 3 WAN ports to use eth1.3, eth1.4 and eth1.6 which represent VLAN tags of 3, 4 and 6 respectively. This is done by entering a "custom" physical interface, as you're setting the appropriate static addresses for your ROOter routers.

The only other thing I needed to do was to put in my OMR server key and VPS IP address, as described in my previous post. The end result, after a few moments for everything to connect-up looks like this:

screenshot-192.168.100.1-2020.02.21-14_30_53.png

And of course, the first order of business on the new Multipath TCP setup is to run a Speedtest:

screenshot-www.speedtest.net-2020.02.21-14_34_08.png

A word of caution here, if your resulting speeds are anything like mine, you can use a LOT of data in a hurry running Speedtest.net on a setup like this. 3 tests and I had blown through a gigabyte of data! Also, some providers block multipath, the most notable one for me so far is Verizon. Not sure if they do it everywhere, but they are near me. AT&T and Sprint have been no problem, and they're the fastest anyway. Also, I get limited value out of my 3rd WAN as it's the same provider, on the same tower as WAN1. It's nice to spread my data usage across all of these SIMs I'm paying for anyway. Speeds are great, and it's a very responsive and robust connection.

[terryjett]

Man, that is cool stuff and a lot on information to digest. Thanks for sharing with us and I am intrigued... Just do not have the time now to try this, but one day soon, I have got to!

[mezmerizz0454]

i have read this 3 times i think it might sink in after 2 more times ,,and it is very interesting.
I'm pretty sure there are more than a few people here who carry the Old School Captain Crunch Whistle to make long distance phone calls keep the gold coming

[BillA]

Definitely an interesting project, however it may be a little too complex for the average user.
Having to run a virtual server in some remote data center incurs an extra monthly cost, and it's also an additional hop point for the data to take, including a potential security hole/snooping. The less hops the better, more reliable, lower latency, and more secure.

I have found a simple and inexpensive device for channel bonding using TP-Link's Safestream Multi WAN Router.
While it's sold as a 4 port load balancer, you can enable channel bonding by disabling "Application Optimized Routing" under Advanced > Load Balance > Configuration, to bond the bandwidth from multiple WAN inputs (up to 4 sources).

100Mbps/$60 - https://www.amazon.com/SafeStream-Broad ... B002T4D3L8
1,000Mbps/$160 - https://www.amazon.com/SafeStream-TL-ER ... B00A8NWU70

The basic connection setup looks like this:
Multiple Mobile/cable/DSL source's LAN port > into the WAN ports on the TP-Link Safestream then from LAN port > into the WAN port of a regular router/access point

There are a couple of ways to use the TP-Link box in order to bond multiple mobile connections, or cable/DSL in any combination. On all the internet source devices DHCP must be turned off since we don't want it to assign IP's.
1. The cheapest route (about $40 + the cost of them modem) is by connecting a mobile modem in an external USB enclosure to a GLI MT300N box to convert the modem's USB port into an ethernet port, connected to the TP-Link box.
2. For just a little more (about $50 + the cost of them modem), a more elegant all-in-one solution is by using an inexpensive WE826 router box with an internal modem connected to the TP-Link box. Course you'll need multiple WE826 boxes for multiple mobile modem connections, or in combination with a cable/DSL modem.

Technically any OpenWRT router could bind multiple internet sources as long as it has enough processing power, memory, and someone writes the necessary code for it. But for now the OpenMPTCProuter and the TP-Link box are the only viable solutions for true channel-bonding.

[bnhf2]

There are definitely plenty of good applications for the TP-Link multi-WAN, fail-over, load-balancing router. However, what that box does and what OpenMPTCProuter does are not the same thing. OMR is true link aggregation, i.e. if you have two Internet connections where each speed tests at 50 Mbps, the resulting combination of the two using OMR should be around 90Mbps (or better). Devices like the TP-Link box, can do a good job of providing additional bandwidth, but will never be any faster than the fastest attached Internet connection.

As it turns out there's lots of upside to having a full-time Virtual Private Server on the other end of an OMR setup. First, OMR uses a VPN between the two as part of its magic -- so users get the advantages of a full-time VPN, without the usual performance penalty. Second, and this is a big plus for many cellular Internet users -- you now have a real public IP, complete with port-forwarding! OMR also supports having a Pi-hole (ad blocking DNS server) running on the VPS, which eliminates the need for anything ad blocking related to run on your LAN.

I agree with you that OMR is not for the average user, though I will say the developer has made it relatively straightforward to get up-and-running. VPS VMs are amazingly cheap these days (I think I pay $5/mo for mine), and there's little setup required beyond running a script on the VPS. There's processing power required on the OMR side (more than the average router), but something as simple and cheap as a Raspberry Pi is supported.

[BillA]

There are definitely plenty of good applications for the TP-Link multi-WAN, fail-over, load-balancing router. However, what that box does and what OpenMPTCProuter does are not the same thing. OMR is true link aggregation, i.e. if you have two Internet connections where each speed tests at 50 Mbps, the resulting combination of the two using OMR should be around 90Mbps (or better). Devices like the TP-Link box, can do a good job of providing additional bandwidth, but will never be any faster than the fastest attached Internet connection.

As it turns out there's lots of upside to having a full-time Virtual Private Server on the other end of an OMR setup. First, OMR uses a VPN between the two as part of its magic -- so users get the advantages of a full-time VPN, without the usual performance penalty. Second, and this is a big plus for many cellular Internet users -- you now have a real public IP, complete with port-forwarding! OMR also supports having a Pi-hole (ad blocking DNS server) running on the VPS, which eliminates the need for anything ad blocking related to run on your LAN.

I agree with you that OMR is not for the average user, though I will say the developer has made it relatively straightforward to get up-and-running. VPS VMs are amazingly cheap these days (I think I pay $5/mo for mine), and there's little setup required beyond running a script on the VPS. There's processing power required on the OMR side (more than the average router), but something as simple and cheap as a Raspberry Pi is supported.

I love the concept, and I understand that it's more than just a load-balancing router which is mainly useful on multi-segment downloads. Having OMR baked into the standard GoldenOrb release, would make it that much more popular.
However I'm still not sold on the idea of having to run a remote VPN server to combine the packets. If packet reassembly could all be done locally in the mobile router like on hardware based load-balancing/aggregating routers, then it would be a clear winner. Unfortunately 99% of non-technical users would be lost on the server setup, let alone the client/server settings. You and I might think it's easy, but having dealt with non-technical users for decades, trust me it ain't, just being realistic. But it would make a great business setting it up for them.

[bnhf2]

Unfortunately 99% of non-technical users would be lost on the server setup, let alone the client/server settings

My apologies if I gave the impression that this was an "anyone can do it" sort of post. There's little documentation on OMR, and no step-by-step guides for deploying it, so for anyone who feels they would need something like that to make it happen -- probably best to look elsewhere for a solution.

Let me try to break down the skills that would be needed:

For setting up the VPS, anyone that's ever created a virtual machine using a tool like VirtualBox, VMware or Parallels will have no problem understanding the concepts. "Know to work" cloud platforms, and virtual machine specifications are listed on the OMR website.

Once the VM is setup, a tool like PuTTY is required to be able to login in and run the couple of commands required to set it up. The script the OMR developer has written to do all the real work is fantastic, and is the same script that one runs to update the VPS when needed.

There are only a few types of client router devices supported, but they include x86 boxes (you can use anything from an Intel NUC to an industrial firewall appliance) and the Raspberry Pi. For these devices, one needs to be able to flash an SD card or USB thumb drive using a tool like Win32DiskImager or Etcher. There are a couple of consumer router type devices supported too, which would need to flashed using the same procedures as OpenWRT, ROOter, DD-WRT, Gargoyle or any other open source router OS.

In between your OpenMPTCProuter and the VPS will be your Internet gateways, which can be pretty much anything. DSL, cable modems, cellular hotspots, phones, WiFi as WAN links, ROOter routers -- whatever. And here's where things get interesting, as you can combine up to 8 of them (I've had as many as 5), and get near linear speed aggregation. For the user with better-than-average skills, in Internet-challenged areas, this can mean putting together a bunch of modest connections (say 4 10Mbps links) and ending up with a single 35Mbps virtual link, that won't miss a beat if one gateway goes offline.

So you're absolutely right, OMR is not built for the masses. But for those comfortable setting up and using an open source router OS like ROOter -- or building a cellular router solution using individual components such as those available through The Wireless Haven, I don't think it would be much of a stretch. And for the power user, that wants a public IP to host a webserver, or make the content on their Plex Media Sever available to friends and family, or host an MQTT broker to unlock some Internet of Things magic -- there's almost nothing available like it.

[56387f]

So you're absolutely right, OMR is not built for the masses. But for those comfortable setting up and using an open source router OS like ROOter -- or building a cellular router solution using individual components such as those available through The Wireless Haven, I don't think it would be much of a stretch. And for the power user, that wants a public IP to host a webserver, or make the content on their Plex Media Sever available to friends and family, or host an MQTT broker to unlock some Internet of Things magic -- there's almost nothing available like it.

In your last post you summarized all things in a very nice way. Yes, I can see how having high bandwidth internet connection can be a certain advantage in case of hosting a web server because in this case "the more the better". I have dealt with load balancing/ failover configurations and a couple of times was thinking about multiple WAN aggregation scenarios but somehow every time in the end I simply could not justify the cost & complexity to achieve a decent result. Instead, every time I just switched to a faster/better service and eventually (170 Down/150 up FIOS) reached the level when I stopped caring about the speed - if it becomes 300/250 or 3000/2500 it will not change a thing for me.
So I wonder now how nowadays a power user who does not host a web server could potentially benefit for WAN bonding?

[bnhf2]

I have dealt with load balancing/ failover configurations and a couple of times was thinking about multiple WAN aggregation scenarios but somehow every time in the end I simply could not justify the cost & complexity to achieve a decent result. Instead, every time I just switched to a faster/better service and eventually (170 Down/150 up FIOS) reached the level when I stopped caring about the speed - if it becomes 300/250 or 3000/2500 it will not change a thing for me.

The equation changes of course for those with good residential Internet service available, as one just doesn't have the same issues and challenges as for those who depend on cellular, satellite or rural DSL Internet. I probably should have mentioned this in my original post, though it may have been implied by this being on the "Wireless Joint" forum and my use of ROOter (a cellular specific version of OpenWRT).

In my case, as a full-time RVer for the last decade, my Internet service is far from predictable, and changes every time we re-locate our motorhome. As such, OpenMPTCProuter has really been a dream-come-true, in terms of being able to better utilize the multiple cellular plans I pay for (Sprint, AT&T and Verizon ), of always having a public IP (which I would otherwise only get from Sprint) and a full-time VPN as a bonus. But it's the link speed aggregation that really makes it unique, and different from other ways of utilizing multiple gateways.

OMR is one of those projects that for those who understand what it does, have multiple Internet connections available, and are stuck behind carrier-grade NAT, can be game changing. More often than not though, it gets confused with basic load balancing and failover/failback solutions. Part of the reason for this I believe is that there are very few similar products. Peplink has their original SpeedFusion product (which is quite expensive, so not much used by individuals), there's Speedify (which has never shown me much speedwise), and not much else.

[mtl26637]

I just wanted to come back here and give a big thanks to bnhf2 for posting this information. I had no idea that openmptcprouter even existed and was always under the impression truly combining WAN connections wasn't possible. I do agree that this isn't for everyone and it would help to have a little knowledge of networking but I would think that would be a fair assumption if you read this forum. I already had a free online vps setup through amazon AWS for my reverse ssh tunnel so I was already familiar with the remote portion. Word of caution though, the only AWS instance I could get to boot was the t2.micro. Any upgrades to the instance rendered the instance unusable. I spent TOO much time with amazon before throwing in the towel and going with a vultr instance instead. That being said, even though I was extremely familiar with amazon's services, I'm glad I went with vultr in the long run. Ultimately I'm still not sure what the top limit is on network speeds with the free t2.micro instance, and I'm sure monthly bandwidth would have eventually come into play.

Now that its all said and done, I have 2 WAN connections. 1) ATT+WE826 and 2) ATT+WE1326. Those 2 are load balanced and ran to my main netgear home router to run the house. Off to the side and completely separate from that I have both of these WAN connections (direct to the lan ports of both routers and before load balancing) connected to a Raspberry Pi 4 using cheap gigabit 'usb to ethernet' adapters. The true gigabit ethernet port of the pi4 is connected to a separate gigabit switch where I connect my main workstations for the extra speeds offered by MPTCP. Of course the pi4 is running openmptcprouter and offers average download/upload speeds of around 110/65 Mbps. Not to mention having a true static public ip address now, I could ultimately ditch my reverse ssh tunnel setup, but I just can't do it knowing how much time and effort I've invested into that over the past year.

Once again, thank you bnhf2. If it wasn't for this post I would have never known of this. I'm extremely surprised that this isn't a MUCH more popular thread honestly. I think it might be mistaken or misunderstood what it truly is or something. I put in a lot of work probably but the payoff is worth it if you like tinkering with this stuff. And by the looks of it, MPTCP is still rather new technology and this will all be common and in place automatically in the next 5 years or so. Just to think, a little over a year ago I told my local dsl provider to piss off and got rid of my 5-10 down 0.5-1 up Mbps internet connection that cost more than both of the above combined. Still pisses me off thinking about it really, what a scam eh..

[docderwood]

Would be appreciative if you could recommend a VPS/hosting company?

[bnhf2]

Would be appreciative if you could recommend a VPS/hosting company?

Hard to argue with Vultr, if you're in the U.S -- my current OMR VPS is running me $5/mo. Use whichever of their datacenters that's closest to you.

Otherwise, have a look at the list of tested platforms in the OMR documentation:

https://github.com/Ysurac/openmptcprout ... te-the-VPS

Theoretically, any server that is running one of the supported operating systems can be used, including at a friend's or relative's house. Obviously, the upload and download speeds at such a location would need to be greater than whatever your aggregated WAN speeds could reach, or you'd be creating a bottleneck.

I'd recommend veering away from one of the tested commercial providers only if your networking skills are well above average.

[docderwood]

My skills definitely aren't above average!

Fortunate to have GoogleFiber (w/o static IP) at one place....but still think commercial VPS would be better. Thx!

[bnhf2]

If it wasn't for this post I would have never known of this. I'm extremely surprised that this isn't a MUCH more popular thread honestly

Glad you found the post useful! I'm amazed too at the relatively modest interest in what OpenMPTCProuter has to offer.

I'm currently in an area with very limited Internet options -- each local provider seems to be capped at around 4Mbps on the d/l and 1Mbps on the u/l. However, by combining a DSL connection with two cellular links, I'm able to get 12Mbps/3Mbps which is a nice bump over 4Mbps/1Mbps.

Whether I'm camped where my combined connections exceed 200Mbps, or in a spot like where I am now, OMR delivers.

[docderwood]

"Qotom x86 (i3) industrial PC with 4x gigabit Ethernet ports, 2x USB3 and 2x USB2 ports".....

Could you tell me which of these you have (ie how much RAM/SSD size etc)?

I've set up a Vultr VPS and installed Debian etc as above.

Please poke holes in what I'm hoping to do:

1) Currently have Verizon LTE unlimited plan (have had for years) pulls 15-30 (rarely 50) from dual Yagi's and Sierra Wireless EM7565

2) Use either a Tmobile or ATT plan in a MOFI SIM7. Leaning towards Tmobile since I have line of site 4-7 miles away. ATT not so much.

Use Qutom to bond the two before handing off to a router...have a Mikrotik or older Edgerouter.

Any advice? Anyone do this sort of work on the side to help a networking neophyte?

[mtl26637]

Unfortunately, (and thats a big unfortunately), Verizon doesn't support MPTCP as of yet. I can't remember exactly where I read this, but did confirm it today. Hopefully they will support this in the future though. Please someone tell me i'm totally mistaken?

[docderwood]

Unfortunately, (and thats a big unfortunately), Verizon doesn't support MPTCP as of yet. I can't remember exactly where I read this, but did confirm it today. Hopefully they will support this in the future though. Please someone tell me i'm totally mistaken?

Uh-oh....hope that's not the case.

I looked here to see if there were support issues but didn't find anything: https://github.com/Ysurac/openmptcprouter/issues/950

[docderwood]

bnhf2,

Just to confirm.....you aren't having any luck doing this with Verizon Wireless? Was hoping to bond VZ wireless with Tmobile.

If Verizon Wireless doesn't allow bonding, can I set a cap on VZ and have it kick over to Tmobile using this setup? thx!

[mtl26637]

Also, some providers block multipath, the most notable one for me so far is Verizon. Not sure if they do it everywhere, but they are near me. AT&T and Sprint have been no problem, and they're the fastest anyway. Also, I get limited value out of my 3rd WAN as it's the same provider, on the same tower as WAN1. It's nice to spread my data usage across all of these SIMs I'm paying for anyway. Speeds are great, and it's a very responsive and robust connection.

Looked back through to try to find where I read it. Here is where I seen it (look at first page of this thread) and can confirm it does not work for me either. Spent a couple hours yesterday trying to setup a third WAN connection. Got it all setup and connected but VZW would not allow any of the MPTCP connections .

[docderwood]

Looked back through to try to find where I read it. Here is where I seen it (look at first page of this thread) and can confirm it does not work for me either. Spent a couple hours yesterday trying to setup a third WAN connection. Got it all setup and connected but VZW would not allow any of the MPTCP connections .

Sadly I see it there. Major bummer.

Having said that....is there any way for me to configure things to flip over from Verizon to Tmobile or vice versa at a set bandwidth usage level a month.....or for me to divide usage between the two with this setup?

thx!

[docderwood]

FYI....I DM'd OpenMPTCRouter on Twitter.

Got this back:

Hi, tracebox can help to check where and if MPTCP is really blocked. As workaround, I added MPTCP over VPN in OpenMPTCProuter, not ideal but this work when MPTCP is blocked.

[mtl26637]

If looking to control bandwidth allocations and failovers etc., look into the mwan3 capabilities built into OpenWRT/GoldenOrb. You can setup fine grain control of % usage of each WAN interface. OpenMPTCProuter is more about combining the two WAN interfaces to maximize your instantaneous throughput. Example, if each of 2 WAN interfaces has an average of 50mbps download speed then with OpenMPTCProuter then end result is a download speed close to 100Mbps. Using built in mwan3 your max. stays 50, however, it is split percentage wise in what you set it up as. You can also set it up to use both, but that does get a bit tricky and one would need to further set up various rules in the mwan3 settings to pass certain traffic straight to OpenMPTCProuter. On my initial setup and before configuring additional bypass rules, I was pulling from same wan interface at the MPTCP router. IMO, i'd ditch VZW if set on using OpenMPTCProuter. Even if you went with 2 connections from same provider it shoudn't matter much. Actually where i'm at the ATT and T-mobile would pull from same tower anyway..

[mtl26637]

I forgot about trying MPTCP over the VPN connection. I may have to give that a shot since I have a work hotspot on VZW here wasting away.

[docderwood]

I'm clearly not understanding how this works.....

Looking at the diagram from: https://www.openmptcprouter.com

I assumed I would be running OpenMPTC Router at my place (Qutom I3 Internet router) which would connect to
1) Verizon Internet via one ethernet port and
2) T-Mobile thru the other ethernet port

and to the VPS hosted on Vultr (via a VPN)

The Qutom would direct traffic between the VPS hosted at Vultr (via a VPN connection) using either the VZ or Tmobile path.

In other words....a client on my network would request data from the internet....it would hit the router (Qutom) first.....then go to either the Verizon or Tmobile connection to the VPS (via a VPN hosted by vultr)....then to the internet....the return path would come back via the public facing IP of the VPS and back through via Verizon or TMobile.

It seemed like a VPN was baked into the package I installed onto the VPS with keys etc saved to a text file?


Sorry...I'm in way over my head.

[mtl26637]

for the most part that is correct, except on the return path rather than "Verizon or TMobile" the vps routes the return packets via multiple TCP connections over "Verizon AND TMobile". The built in mwan3 in OpenWRT/GoldenOrb (aka Load Balancing) is basically the return path being "Verizon OR TMobile". I'm not an expert on this stuff by any means, but pretty sure thats the basics.

But back to the VZW part, they don't seem to allow multiple tcp (MPTCP) so that is why this doesn't work with them.

[docderwood]

Hey, bhnf2!

Can you confirm you have had no luck with Verizon yet?

Unfortunately, it's my best provider so far.

thx!

[BillA]

Hey, bhnf2!

Can you confirm you have had no luck with Verizon yet?

Unfortunately, it's my best provider so far.

thx!

Verizon blocks multi-cast packets so it won't work unless you run it through a VPN tunnel, unfortunately it's going to slow it down significantly, and increase latency too. May you want to test both ATT and Tmobile service in and around your using a phone, and if you get at least 5Mbps speed, then chances are the router will do much better.

[bnhf2]

Hey, bhnf2!

Can you confirm you have had no luck with Verizon yet?

Unfortunately, it's my best provider so far.

thx!

I'm not entirely sure what to say about Verizon. I don't use my Verizon SIM that often, but I am using it at the moment. As I reported way-back-when, Verizon does appear to block MPTCP, and that can be confirmed by running the MPTCP Support Check in OMR.

On the other hand, I've been using Verizon, Viaero Wireless and a local phone company's DSL for the last month with reasonable results. When I run Speedtest.net, I can see traffic passing on all three of my WANs -- with similar numbers on the two cellular links. On top of that, I've used about the same amount of data on each of those two links.

OMR Status also indicates all is well (No message indicting that Verizon doesn't support MPTCP):

screenshot-192.168.100.1-2020.06.04-16_27_22.png

I'll probably ask the developer what he thinks about this when I get a chance. In the meantime, Verizon is clearly a working part of the mix for me with OMR. Perhaps things would be different if I was only using two WANs, rather than three. I guess my best advice at his point would be to try it, and see how it works for you in your location.

[bnhf2]

Verizon blocks multi-cast packets

You got me thinking with this post! A little off topic, but probably worth clarification in case somebody stumbles across this in the future:

Multicast IP and Multipath TCP are not the same thing. From Wikipedia:

Multicast

In computer networking, multicast is group communication where data transmission is addressed to a group of destination computers simultaneously. Multicast can be one-to-many or many-to-many distribution. Multicast should not be confused with physical layer point-to-multipoint communication.

Multipath TCP

Multipath TCP (MPTCP) is an ongoing effort of the Internet Engineering Task Force's (IETF) Multipath TCP working group, that aims at allowing a Transmission Control Protocol (TCP) connection to use multiple paths to maximize resource usage and increase redundancy.

I know Verizon themselves use multicast to broadcast live TV streams, not sure if they block individuals from using it.

[docderwood]

^^^^Thanks bnhf2!

I'm going to start playing with it. Wish me luck, getting in over my head. There may be some used gear for sale if I fail:)

[bnhf2]

I'm going to start playing with it. Wish me luck, getting in over my head. There may be some used gear for sale if I fail:)

The nice thing about an "industrial pc" (like the Qotom) configured like a router, is that it can be used for many purposes. Though I doubt you'll need to vector away from OMR, if you do -- OpenWRT, Gargoyle, DD-WRT, ROOter and pfSense are all options. And of course, it'll run Windows or one of the many, many flavors of Linux. It's a great box for those willing to try something new -- regardless of past experience. Best of luck, and don't hesitate to ask for help. Here is good, but the developer of OMR is also quite accessible. You can post questions/issues on the OMR Github, or chat with the dev on IRC -- keeping in mind he's in France timezone-wise.

[docderwood]

I had a Qotom ordered it and cancelled it yesterday since I wasn't sure Verizon would work. I'll pull the trigger on it or something similar now, I've been wanting to play with pfsense anyway if it doesn't work out (I have a unifi dream machine pro I regret buying, long story).

I sent the developer some money via Paypal, figured he/she deserved it. Good to know they are active on Github.

thx again.

[mtl26637]

On the other hand, I've been using Verizon, Viaero Wireless and a local phone company's DSL for the last month with reasonable results. When I run Speedtest.net, I can see traffic passing on all three of my WANs -- with similar numbers on the two cellular links. On top of that, I've used about the same amount of data on each of those two links.

OMR Status also indicates all is well (No message indicting that Verizon doesn't support MPTCP):

Well isn't that interesting. When I did have VZW hooked up the status page showed all green, however, when checking the MPTCP traffic (could have been normal Bandwidth traffic) it showed in the 'kb' ranges rather than the 'mb' ranges. I just assumed it was cause VZW was blocking something. If I could get a legit VZW connection up and runing that would be great! Maybe it was the way I had it connected as the OpenMPTCProuter wouldn't recognize my usb hotspot connection so I had it connected via the 'wwan' interface wirelessly.

Please let us know if you ever here any thoughts back on it. If I end up trying VZW again I will let you guys know the outcome.

[BillA]

Multipath TCP

Multipath TCP (MPTCP) is an ongoing effort of the Internet Engineering Task Force's (IETF) Multipath TCP working group, that aims at allowing a Transmission Control Protocol (TCP) connection to use multiple paths to maximize resource usage and increase redundancy.

I know Verizon themselves use multicast to broadcast live TV streams, not sure if they block individuals from using it.

I meant Multipath, brain slows down at 3am. he-he
It's possible that Verizon works with certain combinations, like a Verizon and non-Verizon connection, but not two Verizon ones.

[fcp818]

Well I gave this WAN bonding a shot and finally got it up and aggregating. Combined our unreliable 50meg cable service with our backup Sprint LTE. Getting in the 70s down with 15 to 20 up. Better consistency in speeds than before.

Before I take the next step and add ATT LTE to the mix I need to resolve an un anticipated issue. Our streaming services such as Netflix, Prime and HBO Max will not function (blocked by the shrive provider). I believe it's from the VPN function or the IP on the VPS side. I'm using Vultr state side in New Jersey. My ip is 45.77.xx.xx.

Anyone run it this and what route did you take to get past it?

Thanks.

[bnhf2]

Our streaming services such as Netflix, Prime and HBO Max will not function (blocked by the shrive provider). I believe it's from the VPN function or the IP on the VPS side. I'm using Vultr state side in New Jersey. My ip is 45.77.xx.xx.

Anyone run it this and what route did you take to get past it?

Thanks.

Streaming services blocking access from data centers is a bit of pain, but it's unfortunately pretty common. These services are trying to prevent people from geo-relocating, like when somebody from overseas wants to appear as though they're in the U.S. to access U.S. content and services. Legitimate customers also get caught in this "net".

OMR-Bypass is your friend here, and allows to specify devices or types of traffic you'd like to bypass the VPN. I don't do much streaming myself, but I used to use Netflix and Amazon Video specific bypasses under Services - OMR-Bypass - Protocols and Services. That stopped working for me at some point -- though I'd suggest you try those bypasses first. More recently, I've been using the IP address of my primary streaming device under Source lan IP or network, with the destination interface set to default. Streaming traffic from that IP is then routed to whatever interface is available.

How you set it up will depend on how you do your streaming, and which approach allows you to best utilize OMR for all your other traffic.

[docderwood]

Probably a dumb question:

I'm hoping to set this up in the next couple of weeks. I've got a low end Vultr VPS setup.

Regarding the VPN issue: Would Netflix Prime etc block you if you had a site to site VPN setup to your primary residence 60 miles away? I've got gigabit at our primary residence and have thought about running the VPS at the main house given the ample bandwidth (although I don't have a static IP there).

[bnhf2]

Would Netflix Prime etc block you if you had a site to site VPN setup to your primary residence 60 miles away? I've got gigabit at our primary residence and have thought about running the VPS at the main house given the ample bandwidth (although I don't have a static IP there).

I can't imagine you'd have an issue, as the streaming services block ranges of data center IPs to my knowledge, rather than actually being able to detect VPN usage on some part of your link. I'm pretty sure you would need a static IP though, as the VPS needs to be specified by IP, not FQDN. However, someone did request the use of FQDNs -- and IIRC the developer said it would be in a future release.

You could likely make it work for the time being with a dynamic IP, by setting up a Dynamic DNS service like DuckDNS.org to keep track of your Public IP (which probably only changes on a router reboot). Then if you're away, and the IP changes, you could login into DuckDNS to find out what the new address is and plug it in to OMR. Not perfect, but it should work.

[fcp818]

Streaming services blocking access from data centers is a bit of pain, but it's unfortunately pretty common. These services are trying to prevent people from geo-relocating, like when somebody from overseas wants to appear as though they're in the U.S. to access U.S. content and services. Legitimate customers also get caught in this "net".

OMR-Bypass is your friend here, and allows to specify devices or types of traffic you'd like to bypass the VPN. I don't do much streaming myself, but I used to use Netflix and Amazon Video specific bypasses under Services - OMR-Bypass - Protocols and Services. That stopped working for me at some point -- though I'd suggest you try those bypasses first. More recently, I've been using the IP address of my primary streaming device under Source lan IP or network, with the destination interface set to default. Streaming traffic from that IP is then routed to whatever interface is available.

How you set it up will depend on how you do your streaming, and which approach allows you to best utilize OMR for all your other traffic.

Thanks for the feedback, you got me in the right direction. I was using the domains area at first and that wasn't working.

The Source IP works for my smart tv for all streaming. I really was hoping the Protocols and Services would do the trick. It works for Netflix but Prime Video only works sometimes. Disney+ and HBOGO and HBOMAX are not listed and I'm not sure which other list would be similar.

[bnhf2]

I really was hoping the Protocols and Services would do the trick. It works for Netflix but Prime Video only works sometimes. Disney+ and HBOGO and HBOMAX are not listed and I'm not sure which other list would be similar.

If Protocols and Services would be better for you, I'd suggest reporting the issue on the OMR GitHub page. The developer is responsive, and I'm pretty confident he'll fix it. Since I don't do much streaming, I wasn't particularly motivated to report the issue, and provide the necessary logs and such to get to the bottom of it. The streaming services are something of a moving target as they continually expand, so OMR-Bypass needs to be updated to keep up.

[Yipzy]

Thanks for this solution, bnhf2! My dual wan setup wasn't cutting it for me. This is much better without paying for Speedify. I have a dilemma though like others, Netflix and other streaming services. The bypass would work with source IP but I need some sort of encrypted connection to bypass the video quality throttling from LTE. I don't see how to bypass that except to find a VPS provider that hasn't been banned by streaming service providers yet. Most popular data providers are already on their blacklist.

[mtl26637]

Using the OMR-bypass I've not ran into any problems with streaming via prime? Maybe check your TTL settings or log into your providers account and disable the video throttling? You may do a search on these topics as I know I've seen some info on various carriers and what is used to get around the video detections.

On a side note I thought i'd give an update on my setups with various carriers. I did finally get ATT / SimpleMobile / Verizon working with the MPTCP. I'm an idiot, I had by VPS firewall only letting packets pass with ATT IP's (166.0.0.0/8). I forgot all about doing this initially with the firewall. All providers that i've tried seem to work with no problems now.

One last thing, Does anyone see a reason that PI-4 running OpenMPTCProuter setup would be a limiting factor in top speeds if using VLANS and the single ethernet port only? I've yet to see anything better than ~150 down even using 4 separate LTE WAN providers. Not sure if the pi4 is the bottleneck or not. Using HTOP the pi4 never seems to go above 40% cpu usage so not sure what it is. Hate to drop $200 on something like a Qotom just yet even though i'd like to have one to play around with anyway.

[bnhf2]

I've yet to see anything better than ~150 down even using 4 separate LTE WAN providers. Not sure if the pi4 is the bottleneck or not.

I think you'll find that your VPS is the bottleneck. Try adding more RAM or cores, one or the other (or both), should get you a higher top end. It's fun to go fast -- isn't it?

[Yipzy]

Using the OMR-bypass I've not ran into any problems with streaming via prime? Maybe check your TTL settings or log into your providers account and disable the video throttling? You may do a search on these topics as I know I've seen some info on various carriers and what is used to get around the video detections.

TTL bypasses the hotspot detection. Encryption such as a VPN will bypass the video throttling. The plan's video settings cannot be turned off like AT&T's stream saver option that I know you are probably referring to that. It's a business Tmobile plan with 480p video streaming. I could bypass it with VPN for Youtube but not Netflix. The ip ranges from most of the data providers (VPN,VPS,etc) are blacklisted.

[mtl26637]

I think you'll find that your VPS is the bottleneck. Try adding more RAM or cores, one or the other (or both), should get you a higher top end. It's fun to go fast -- isn't it?

I did check with htop in the beginning and didn't see it being overloaded but that could have def. changed since I last checked. I will check that this evening for sure. I'm currently running the vultr $10/mo plan that features 1 core, 2GB ram, 2TB bandwidth? Do you think I need to bump it up and see what happens?

Ya after seeing speeds above 200 it makes me jealous, lol. I'm sure once I see the big numbers i'll move on and forget about it but can't until I see them, heh.

It's possible its my modems, I have EM7565 as my best, then an EP06-A, and an MR1100 running T-mobile and a sierra MC7700. So many variables makes it hard to track down the bottleneck. Could be distance from the different towers too. Once the store opens back up I'm gonna grab an EM20-G as i've wanted one anyway.

[docderwood]

I'm hoping to get rolling with this in the next two weeks. I'll have a MOFI SIM7 and two WG3526-P feeding into a Qotom running OpenMPTCP router Any advice on how to have these three feed into the Qotom? Maybe in bridge mode?

[bnhf2]

I'm currently running the vultr $10/mo plan that features 1 core, 2GB ram, 2TB bandwidth? Do you think I need to bump it up and see what happens?

I'm on the 1 core, 1GB RAM and 1TB of monthly usage plan, and I've been able to hit 230Mbps. However, that seems to be my top end -- so next time I have fast enough cellular links, I'll probably spin up a 2 core VPS just to see if I can break through that ceiling. My current plan is $5/month though, and it looks like the cheapest 2 core plan is $20/mo, so I doubt I'd keep that as my "daily driver".

Anyway, based on what you've got going now, bumping up to a 2 core plan would be an interesting experiment. Might want to be sure you can change it back though before you upgrade, as there's no reason to be paying for more VPS than you need if another core has no impact.

[bnhf2]

I'll have a MOFI SIM7 and two WG3526-P feeding into a Qotom running OpenMPTCP router Any advice on how to have these three feed into the Qotom? Maybe in bridge mode?

Two options seem best to me here:

Either, run each cellular router to its own Ethernet port on the Qotom. For example, your LAN interface could be eth0, and then use the remaining physical interfaces of eth1, eth2 and eth3 as WAN1, WAN2 and WAN3. You'd probably want to connect the LAN interface to one or more gigabit switches to accommodate whatever other Ethernet based devices you have on your LAN.

Alternatively, you could create a network of gateways on a single WAN interface using VLANs. This is the way I do it, as I only have one Ethernet cable that goes from where my OMR router is located to the area where I have my cellular routers. So in this case let's say that eth3 is where you want to connect all of your WAN devices. You could create VLANs on eth3.1, eth3.2 and eth3.3 -- then assign those to WAN1, WAN2 and WAN3. Each needs its own subnet, which could be 192.168.10.1, 192.168.20.1 and 192.168.30.1. Create VLANs on each of your 3 cellular routers using those subnets, and use VLAN names like VLAN10, VLAN20 and VLAN30. You'll want DHCP turned off on all LAN and VLAN interfaces. Back on the OMR router, use static addresses like 192.168.10.10, 192.168.20.20 and 192.168.30.30 on WAN1, WAN2 and WAN3 respectively.

The second approach is obviously more involved, but potentially reduces the number of long Ethernet runs. This can be especially important if your cellular routers are outdoors (which is often the best place for them), and PoE powered like all of mine.

[docderwood]

Curious about something as I slowly get around to set this up:

As I test, I have good speeds on Verizon (10-50); better on ATT (50-100); and abysmal on Tmobile (4-9). Can I bond 2 ATT connections with Verizon to increase total capacity (running all thru VPN point to point to main house 60 miles away)? Would probably have the 2 ATT pointed to different towers.

[docderwood]

Curious about something as I slowly get around to set this up:

As I test, I have good speeds on Verizon (10-50); better on ATT (50-100); and abysmal on Tmobile (4-9). Can I bond 2 ATT connections with Verizon to increase total capacity (running all thru VPN point to point to main house 60 miles away)? Would probably have the 2 ATT pointed to different towers.

[docderwood]

Success!

TMobile, ATT, Verizon.

VPS is @ primary residence with Google Fiber.

Screen Shot 2020-07-02 at 11.00.23 AM.png

Screen Shot 2020-07-02 at 11.01.41 AM.png

[bnhf2]

Success!

TMobile, ATT, Verizon.

VPS is @ primary residence with Google Fiber.

Sweet!

BTW, you should have your most reliable, fastest connection set as "Master" though -- which isn't T-Mobile, based on your previous posts.

[docderwood]

Sweet!

BTW, you should have your most reliable, fastest connection set as "Master" though -- which isn't T-Mobile, based on your previous posts.

Thanks for all your posts on this!

It's been interesting playing around with Master vs Enabled. I asked the developer....it seems that it's set up to go by latency...so when one connection is saturated it starts to utilize the other connections. Still playing with it, but when Tmobile is set as "master" it saturates it and then ATT/Verizon kicks in.

Pretty neat open source solution regardless.

[docderwood]

Hello again,

I'm trying to understand how the VPN works on OpenMPTCP. Then I login, there are 4 options: DSVPN; MLVPN, OpenVPN, GLorytun (Which I believe is a proxy server).

It seems like when I activate any of these VPN's...a message pops up "VPN connection is down" on the summary page.

What is the real advantage of the VPN between the router and VPS (which is on my home network at another location)?

Thanks!

[Ycarus]

Hi,

VPN are used for anything that is not TCP. For TCP shadowsocks is used, and shadowsocks is a socks5 proxy.

DSVPN and Glorytun TCP use MPTCP for aggregation and MLVPN and Glorytun UDP use own internal method for that.
By default Glorytun TCP is used.

[rifaterdemsahin]

https://github.com/Ysurac/openmptcprouter/issues/1094

I got stuck ion the virtualbox. If i can solve it i would test with 5 connections on 700MBits

[bnhf2]

Hello again,

I'm trying to understand how the VPN works on OpenMPTCP. Then I login, there are 4 options: DSVPN; MLVPN, OpenVPN, GLorytun (Which I believe is a proxy server).

It seems like when I activate any of these VPN's...a message pops up "VPN connection is down" on the summary page.

What is the real advantage of the VPN between the router and VPS (which is on my home network at another location)?

Thanks!

The VPN (typically Glorytun which is the default), is an important part of the OpenMPTCProuter aggregation magic. If you've disabled Glorytun for some reason, re-enable and wait a minute or so. The VPN should re-establish. If it doesn't, you still have some work to do on your configuration.

[docderwood]

Hi,

VPN are used for anything that is not TCP. For TCP shadowsocks is used, and shadowsocks is a socks5 proxy.

DSVPN and Glorytun TCP use MPTCP for aggregation and MLVPN and Glorytun UDP use own internal method for that.
By default Glorytun TCP is used.

Thank you.

I guess my confusion lies in what the tab labeled "VPN" does and when I should enable any of the VPN options (and which options I need to enable) or are they already activated as needed? For what it is worth, my VPS is at our primary residence and not in a datacenter.

Thanks for your work on this/joining the forum.

OpenMPTCProuter - Status - LuCI.pdf

Settings.pdf

[docderwood]

The VPN (typically Glorytun which is the default), is an important part of the OpenMPTCProuter aggregation magic. If you've disabled Glorytun for some reason, re-enable and wait a minute or so. The VPN should re-establish. If it doesn't, you still have some work to do on your configuration.

Gotcha....I see it is enabled.

So, Verizon fails the Mulipath TCP test. When I look at the realtime bandwidth usage, it is definitely being used for data...with slightly less throughput then the two ATT accounts.

Is it worth it to turn on the VPN option for the Verizon connection? If so.....what does it use for the VPN and do I need to enable anything on the "VPN" tab?

Thanks!

Screen Shot 2020-07-19 at 9.50.23 AM.png

[bnhf2]

Gotcha....I see it is enabled.

So, Verizon fails the Mulipath TCP test. When I look at the realtime bandwidth usage, it is definitely being used for data...with slightly less throughput then the two ATT accounts.

Is it worth it to turn on the VPN option for the Verizon connection? If so.....what does it use for the VPN and do I need to enable anything on the "VPN" tab?

I see the same thing when I use Verizon, in that it fails the Multipath test -- yet is still passing traffic and clearly contributing to the overall performance. I've tried the VPN option you referenced with no improvement. Verizon remains an enigma, but my link has been better with it than without it, so it's continued to be part of my aggregated WAN.

[Yipzy]

I see the same thing when I use Verizon, in that it fails the Multipath test -- yet is still passing traffic and clearly contributing to the overall performance. I've tried the VPN option you referenced with no improvement. Verizon remains an enigma, but my link has been better with it than without it, so it's continued to be part of my aggregated WAN.

I concur with bnhf2's assessment as well. My Verizon link does the same thing and contributes to overall performance.

I did check with htop in the beginning and didn't see it being overloaded but that could have def. changed since I last checked. I will check that this evening for sure. I'm currently running the vultr $10/mo plan that features 1 core, 2GB ram, 2TB bandwidth? Do you think I need to bump it up and see what happens?

I don't think you need a better performance VPS. I'm using the $5 one just fine. It's limited by my Verizon tower which only does 65DL/45UL with 2CA (66+13 No 4x4 MiMo) but my AT&T does 4CA 150DL/20-30UL (30+2+12+66 w/ 4x4 MiMo). Tested on 5Ghz WiFi channel on my phone too.

[Ycarus]

Gotcha....I see it is enabled.

So, Verizon fails the Mulipath TCP test. When I look at the realtime bandwidth usage, it is definitely being used for data...with slightly less throughput then the two ATT accounts.

Is it worth it to turn on the VPN option for the Verizon connection? If so.....what does it use for the VPN and do I need to enable anything on the "VPN" tab?

Thanks!

Screen Shot 2020-07-19 at 9.50.23 AM.png

VPN between router and server is enabled by default, nothing to change in default settings if you don't have problems.

The MPTCP over VPN is not related, it make a VPN using OpenVPN in UDP mode between the router and the server. This is only when the ISP really block Multipath TCP.
For some unknown reason (for now), tracebox report Multipath disabled for some ISP that are in fact correctly aggregated. When multipath is really blocked, the connection is not used at all and you have only bandwidth used by ICMP ping test visible in bandwidth graph.

[bnhf2]

For some unknown reason (for now), tracebox report Multipath disabled for some ISP that are in fact correctly aggregated. When multipath is really blocked, the connection is not used at all and you have only bandwidth used by ICMP ping test visible in bandwidth graph.

That explains the mystery with Verizon. Thanks @Ycarus for joining this thread, and for your excellent work on a fantastic project!

[docderwood]

Kind of feeling like the village idiot, can't figure this out for the life of me.

I have 3 WG3526. They are all connected to an intel box with 6 ports running OpenMPTC router. I am using POE...so all are connected via the WAN port to power them.

Everything is working great....the one thing I wold like to do is access each router from my local network. For the life of me, I can't figure it out.....tried giving one of them a static address of 192.168.100.2...no go. I can access them by unplugging the them from the OPenMPTCP router and plugging into a laptop and typing in their IP's.

Any help appreciated.

IMG_9757.jpg

[bnhf2]

Kind of feeling like the village idiot, can't figure this out for the life of me.

I have 3 WG3526. They are all connected to an intel box with 6 ports running OpenMPTC router. I am using POE...so all are connected via the WAN port to power them.

Everything is working great....the one thing I wold like to do is access each router from my local network. For the life of me, I can't figure it out.....tried giving one of them a static address of 192.168.100.2...no go. I can access them by unplugging the them from the OPenMPTCP router and plugging into a laptop and typing in their IP's.

A few questions about your config first:

Am I correct that you've reconfigured the PoE port on each WG3526-P, which is labelled "WAN", so that it functions as a LAN port? This would be done by modifying the Network - Switch settings in ROOter.

I see from your attachment that each WG3526-P is on its own subnet, but it looks like you're using DHCP for the OMR router's IP. So the question then, is each router connected with its own Ethernet cable to an individual port on the OMR router that's been assigned to WAN duty (no switches on the WAN side)? Presumably so, or things wouldn't be working.

If the above are true, then it should also be true that you're not using VLAN IDs or MacVLAN.

If you've done all of the above, each of your WAN routers should be accessible by its IP from your LAN as 192.168.5.1, 192.168.1.1 and 192.168.3.1 respectively.

[docderwood]

Yes, I have configured each of the WAN ports to function as a LAN and each ethernet cable runs from the WAN ports to physical ports on the MPTCP routerc(Eth1/2/3....Eth0 is connected to the LAN with all the devices on our network).

On each of the 3 WG3526's I have the IP set (in the WIFIX touter software) as
1) 192.168.5.1
2) 192.168.1.1
3) 192.168.3.1

Not using VLAN's or MAClans.

If I unplug any of the 3 and plug into a laptop, I can then reach the routers individually with the above IP addresses.

I've been screwing around with it for a couple of days, all in all OpenMPTCP has been working great. It wold be nice to be able to access each router for rebooting/troubleshooting....my reptilian brain just can't seem to make it happen. Is this a subnetting issue?

Anyway...anyone know of some good online networking courses? I need to understand this better.

A few questions about your config first:

Am I correct that you've reconfigured the PoE port on each WG3526-P, which is labelled "WAN", so that it functions as a LAN port? This would be done by modifying the Network - Switch settings in ROOter.

I see from your attachment that each WG3526-P is on its own subnet, but it looks like you're using DHCP for the OMR router's IP. So the question then, is each router connected with its own Ethernet cable to an individual port on the OMR router that's been assigned to WAN duty (no switches on the WAN side)? Presumably so, or things wouldn't be working.

If the above are true, then it should also be true that you're not using VLAN IDs or MacVLAN.

If you've done all of the above, each of your WAN routers should be accessible by its IP from your LAN as 192.168.5.1, 192.168.1.1 and 192.168.3.1 respectively.

[ciarlill]

In your initial post you said you have posted about the "network of gateways" approach elsewhere and so did not go into depth on it. I checked out your post history, but was unable to find it. Can you provide a link?

My main concern is I am wondering if there is anything I need to do in R00ter to "bridge" to the OpenMPTCP Router. This has become more of an issue than I anticipated in my initial explorations and setup (even without introducing OpenMPTCProuter) so I want to make sure this will work before proceeding. My original plan before I knew bonding was possible was to have 2 raspberry-pi based R00ter devices setup in a dual-wan configuration to my Ubiquiti edge router x, and then from there provide 2 VLANs for dedicated services (work and streaming+gaming). Even before I got to the dual-wan stage I ran into issues in that bridging just a single R00ter device to the ER-X was not as straightforward as I thought it would be, and the "best" solution is to is to use DMZ instead of a true bridge mode, but that causes double NAT issues which is problematic for gaming.

So now I find this post, and think maybe I can avoid all of that hassle? I can ditch the ER-X as my main router appliance and instead rely on OpenMPTCProuter running on a beefier x86 setup and just bond the connections together as I expand and add them. But will I run into the same issues when it comes to bridging to the OpenMPTCP device as I would with the ER-X? I am still confused/unsure on this point.

I am not really worried about multiple cable runs vs. not, as long as it just works. I want my "home" to have the public IP of my VPS and be able to do port-forwarding and management just like I had a regular static IP. That's my primary goal (along with better bandwidth) to avoid issues with NAT when it comes to gaming.

I am a software engineer and occasionally wear a sysadmin hat, so setting up the VPS side of things looks like the simplest part of all this. I am just pretty shaky on the network stuff once it goes beyond the most basic NAT and subnetting. Any direction you could give is greatly appreciated.

Thanks

[bnhf2]

In your initial post you said you have posted about the "network of gateways" approach elsewhere and so did not go into depth on it. I checked out your post history, but was unable to find it. Can you provide a link?

My main concern is I am wondering if there is anything I need to do in R00ter to "bridge" to the OpenMPTCP Router. This has become more of an issue than I anticipated in my initial explorations and setup (even without introducing OpenMPTCProuter) so I want to make sure this will work before proceeding. My original plan before I knew bonding was possible was to have 2 raspberry-pi based R00ter devices setup in a dual-wan configuration to my Ubiquiti edge router x, and then from there provide 2 VLANs for dedicated services (work and streaming+gaming). Even before I got to the dual-wan stage I ran into issues in that bridging just a single R00ter device to the ER-X was not as straightforward as I thought it would be, and the "best" solution is to is to use DMZ instead of a true bridge mode, but that causes double NAT issues which is problematic for gaming.

So now I find this post, and think maybe I can avoid all of that hassle? I can ditch the ER-X as my main router appliance and instead rely on OpenMPTCProuter running on a beefier x86 setup and just bond the connections together as I expand and add them. But will I run into the same issues when it comes to bridging to the OpenMPTCP device as I would with the ER-X? I am still confused/unsure on this point.

I am not really worried about multiple cable runs vs. not, as long as it just works. I want my "home" to have the public IP of my VPS and be able to do port-forwarding and management just like I had a regular static IP. That's my primary goal (along with better bandwidth) to avoid issues with NAT when it comes to gaming.

I am a software engineer and occasionally wear a sysadmin hat, so setting up the VPS side of things looks like the simplest part of all this. I am just pretty shaky on the network stuff once it goes beyond the most basic NAT and subnetting. Any direction you could give is greatly appreciated.

Thanks

Here are the links to the Network of Gateways posts I did on the ROOter thread:

https://whrl.pl/Rfcpb6
https://whrl.pl/RfcriK
https://whrl.pl/RfcCEn

The main advantage to this approach is to be able to have all of your WAN devices on a separate, common network, which minimizes cabling, and maximizes flexibility.

I'll re-read those posts to see if I'm doing anything different now, but the essence of the configuration is to setup VLAN IDs on all of the WAN gateways, except one, which can be left "standard" (no VLAN ID and DHCP on). All other gateways will have DHCP turned off.

Edit: I did those posts back when I was using a Cradlepoint router, with failover/failback, load balancing and WAN affinity (seems so "last decade" now!) -- but this same setup served me well to spin-up my first OMR based router configuration.

[ciarlill]

Thanks so much! I have read and re-read both this post and the one on whirpool a few times now and think I am very close to having something working. I have my (single) RPi ROOter device communicating with the OpenMPTCP device over the vlan and I can access both the ROOter device (192.168.3.1) and the OpenMPTCP device (192.168.100.1) when my laptop is plugged into a common switch. I have my VPS setup and am just waiting on another SIM card before proceeding with the WAN bonding side of things. Just trying to get as much configured and learn as much as possible before fully committing to building out a secondary ROOter device.

In the meantime I noticed something strange... I wanted to disable WiFi on the ROOter device to cut down on extraneous networks floating around. Although I can access the configuration page at 192.168.3.1 I cannot actually save any configurations, it just times out and appears to be redirecting me back to 192.168.1.1. It works fine if I plug directly in and access on 192.168.1.1. Wondering if you have run into this and have any ideas on anything I might have missed in setup. It seems like one of the additional benefits of this approach is actually being able reach all the devices on their own VLANs for additional configuration as necessary. Please let me know if you have any ideas. Thanks

[bnhf2]

In the meantime I noticed something strange... I wanted to disable WiFi on the ROOter device to cut down on extraneous networks floating around. Although I can access the configuration page at 192.168.3.1 I cannot actually save any configurations, it just times out and appears to be redirecting me back to 192.168.1.1. It works fine if I plug directly in and access on 192.168.1.1. Wondering if you have run into this and have any ideas on anything I might have missed in setup. It seems like one of the additional benefits of this approach is actually being able reach all the devices on their own VLANs for additional configuration as necessary.

Problems with not being able to save changes is usually related to a corrupt SD Card filesystem (which can happen easily on an RPi). Do you see "Unsaved Changes" in the upper right corner? If so, more recent RPi versions of ROOter include a filesystem check and repair menu item. Run that and reboot.

Whether that turns out to be the issue or not, be sure to always do a proper shutdown from the ROOter menu system. Pulling the power cord on an RPi actively running any version of OpenWRT, very often results in the need to do a filesystem repair.

If that's not the culprit, I'd suggest confirming your new VLAN is part of the LAN firewall group. This can be easily verified by checking the header colors under Network - Interfaces. Those that are part of the same firewall group will have the same color header. Mine is green for the LAN group and pink for WAN.

[ciarlill]

Problems with not being able to save changes is usually related to a corrupt SD Card filesystem (which can happen easily on an RPi). Do you see "Unsaved Changes" in the upper right corner? If so, more recent RPi versions of ROOter include a filesystem check and repair menu item. Run that and reboot.

Whether that turns out to be the issue or not, be sure to always do a proper shutdown from the ROOter menu system. Pulling the power cord on an RPi actively running any version of OpenWRT, very often results in the need to do a filesystem repair.

If that's not the culprit, I'd suggest confirming your new VLAN is part of the LAN firewall group. This can be easily verified by checking the header colors under Network - Interfaces. Those that are part of the same firewall group will have the same color header. Mine is green for the LAN group and pink for WAN.

Thanks. That was not the issue (although I did run into this issue on my initial ROOter setup, so good thought!). I'm not quite sure what resolved this, but after hooking up a working modem my changes started saving. I don't think that was the cause, I think it is more likely I had something cached in my browser which was wanting to target or reload from the non-VLAN IP address (192.168.1.1 vs 192.168.3.1).

Anyway, onto my next hurdle! The run I want to do is from house -> detached garage -> barn (where antennas and modems will be). I want to put a switch in the garage to add another wireless AP for more range. So I got the following 2 devices: TRENDnet TPE-TG50g (5port gigabit PoE+ switch) and TRENDnet TPE-P521ES (5 port PoE+ powered w/ passthrough, EdgeSmart). So the 5 port PoE+ switch would be put in the garage and powered from there, and then I could do a single run to the barn, plug into the pass through switch to power it, and 2 modems - sounds great. But as soon as I introduced the PoE passthrough "smart" switch I was unable to access the device on VLAN3. Somehow the managed device, even totally unconfigured (so it should be acting just like an unmanaged device?), is screwing up the VLAN traffic. I verified this same exact setup works with 2 unmanaged switches, I can pass VLAN traffic across both of them no problem. Same exact cofiguration, just replacing the smart switch with a tp-link dumb switch I had laying around. I know this is getting pretty far off topic from the original post, was just wondering if maybe you had some ideas I could try.

Thanks again for your help!

EDIT: Think I got it working. I made the mistake of only adding the port the rpi was connected to to VLAN3. I had to add both that port, and the port connected to the unmanaged switch so that VLAN3 traffic could exit the switch.

[ciarlill]

The quest continues...

Wondering if you ever ran into any latency issues with the VLAN on a ROOter device? For some running in this configuration is giving me latency spikes. Even just pinging from OpenMPTCP router directly to the ROOter device. More info here if you are curious: https://github.com/Ysurac/openmptcprouter/issues/1123

For a test I just disabled the VLAN on ROOTer, and plugged the modem directly into eth1 on the OpenMPTCP router device and configured a static interface on 192.168.1.1 as WAN1. This works and I have internet, and the ping issue is gone. But the worrying part is I cannot seem to actually modify the default LAN interface address in ROOTer, it just doesn't apply, so I'm not sure this will be a valid solution.

[bnhf2]

Wondering if you ever ran into any latency issues with the VLAN on a ROOter device? For some running in this configuration is giving me latency spikes. Even just pinging from OpenMPTCP router directly to the ROOter device. More info here if you are curious: https://github.com/Ysurac/openmptcprouter/issues/1123

For a test I just disabled the VLAN on ROOTer, and plugged the modem directly into eth1 on the OpenMPTCP router device and configured a static interface on 192.168.1.1 as WAN1. This works and I have internet, and the ping issue is gone. But the worrying part is I cannot seem to actually modify the default LAN interface address in ROOTer, it just doesn't apply, so I'm not sure this will be a valid solution.

I have had problems with VLANs on the Raspberry Pi 3B+ -- so I'd be sure your issue doesn't lie there. Check the log for errors related to the Ethernet port. I'd also suggest verifying your Ethernet run to the garage and barn aren't part of the problem. Cable testers are cheap these days, so if you don't have one -- get one. I typically verify my cables using a tester, and also by using a pair of unmanged switches that have different colored LEDs based on the speed of the link.

[ciarlill]

I have had problems with VLANs on the Raspberry Pi 3B+ -- so I'd be sure your issue doesn't lie there. Check the log for errors related to the Ethernet port. I'd also suggest verifying your Ethernet run to the garage and barn aren't part of the problem. Cable testers are cheap these days, so if you don't have one -- get one. I typically verify my cables using a tester, and also by using a pair of unmanged switches that have different colored LEDs based on the speed of the link.

Okay good to know. I did some googling but didn't come across anything about pi 3b+ vlan issues but that is the device in question. I'll take a peek in the logs after I reconfigure it again.

No cables have been run yet, everything is still on my workbench until I'm 100% sure it is and will work how I want, and I have swapped out all cables involved to eliminate that possibility as well.

At this point I'm also considering trying to run the modems directly of the USB ports from the qotom appliance and just sticking the whole kit out in the barn (there is power out there to some lights, would just need to wire in an outlet) and reversing the whole run back to the house lol. Just a little worried about the qotom dealing with 90+ degree days out there.

I am having doubts about the ability to power the pi+modem off PoE, or at least with the switch I got, so I might just need to bite the bullet and do some wiring regardless.