My Ultimate LTE router using a Raspberry pi 4B

[poodad]

I set out on an adventure to dump my 3Mb/768k DSL in favor of LTE data. Little did I know the depth and breadth of the rabbit hole I was about to plunge into.

My first big break was finding @swwifty's thread in this forum on building his router. Inspired by his RPi 3 build, I decided to replicate his setup, but use the newer Raspberry Pi 4B and a cat12 Sierra Wireless LTE modem in a USB enclosure. I intended on using GoldenOrb for the software (GoldenOrb only supports up to RPi 3, but the RPi 4 specs say "fully backwards software compatible", so it should work, right?).

I want to document my build story here in case anyone decides to do something similar.

First the TLDR version:
A RPi 4B with a PoE hat can just barely ensure enough power for itself and an LTE modem in a USB enclosure.
Said USB enclosure requires TWO USB connections to ensure enough power.
As of 08/06/2020 there is no GoldenOrb version for the RPi 4.
There is a community build of OpenWRT for the RPI 4.
Find some way of zeroing out your SDCARD before or when writing RPi images!!!

Ok, the details:

My first concern was how to adequately power everything. Since this rig is going to in an enclosure on the antenna mast, I wanted to use PoE to power it. After seeing the large current draw of Sierra Wireless cat12 modems, I immediately became concerned. The RPi PoE "hat" (that's what PRi daughter cards are called) can supply 5V at 2.5A. That's 12.5W.

Long story short, that's enough to power the RPi at max load, and supply the almost 5W max the LTE card can demand with a couple hundred milliamps to spare. I figure I'll have some additional power headroom by the fact that the RPi won't be doing much work as a firewall - just NAT and routing - no traffic inspection.

The big "a ha" here was that some of the Sierra Wireless modems can draw so much max current that a single USB 3 connection can't supply enough juice (USB3 is 900mA max). This is going to require a USB 3 enclosure with 2 USB connections - one for power and data, the other for additional power. Fortunately, the The Wireless Haven USB enclosures have exactly this capability.

First problem: The Wireless Haven and their associated facebook store are closed, so I can't get the modem and USB enclosure. But I order the the RPi and PoE hat from Amazon.

Once I get the RPi, I discover that GoldenOrb will not run on a RPi 4. It turns out that GoldenOrb is a specific release of the ROOTer project, which is itself a "community release" of OpenWRT. A bit of googling shows that while there is not a stable release of OpenWRT for the RPi4, there is a daily snapshot.

Further research turns up my next huge lucky break: Someone who has major OpenWRT skills is doing his own community build of OpenWRT for the RPi4. He's done a huge amount of work into building a Cadillac package. You can find it in this thread: https://forum.openwrt.org/t/rpi4-community-build/69998

I hit another snag. As I was experimenting with OpenWRt installs, it would start doing crazy things, including just not booting. I discover that the windows tools I'm using (Rufus and Win32DiskImager) to write the images to the sdcard apparently skip over "unused" areas. This leaves data hanging around on the sdcard that interferes with the system you just installed.

My solution is to use Windows Disk Manager to:
1. Create a partition in the tail end unused area of the sdcard.
2. Delete the first two partitions on the disk.
3. Create a partition in the empty space just created.
4. Format it as fat32 with the "quick format" option turned off. This causes every block to be written to.
5. Write the image to the sdcard.

NOTE: If you are using dd under linux to write your images, it does not skip areas and you don't have to worry about this.

Next problem: The fan on the PoE hat doesn't spin up and the RPi gets a bit warm. This is a real concern considering this is going to be in an outside enclosure in southern Indiana. Turns out, this is easy to fix. In the OpenWRT web interface (called luci, BTW) there's a place to install software packages. Just install kmod-hwmon-rpi-poe-fan.

As of now, I'm stuck until The Wireless Haven or their associated facebook shop opens back up and I can the modem and enclosure. Whenever I get these in, I'll continute the thread.

BTW, there also interesting developments on this OpenWRT thread https://forum.openwrt.org/t/raspberry-p ... /39444/258

[poodad]

Thanks! I'll check it out!

[poodad]

Some things to report:

Wifi is broken in the RPi4 builders of OpenWRT. There's some patches that fix this issue in the mainstream linux kernel needed to fix this, but they haven't made it over into the OpenWRT version yet. I'm not using wifi, so no worries for me, but I thought I'd pass it along.

[poodad]

Ok, time to update the build thread. I received my Sierra Wireless MC7565. I've been waiting for The Wireless Haven to open to buy their M2. USB3 enclosure, but I went ahead and bought a cheapo M.2 adapter off Amazon figuring I could at least get the modem software updated, which did indeed work from my Windows laptop. The firmware update utility did crash at the end of the process, but the modem reports the expected version.

Next step was to connect it to the RPi4. Success again. The build of OpenWRT I'm using had modemmanager installed, and modemmanager recognized the modem.

I wanted to poke around with AT commands, but there was no /dev/ttyUSBx device files. After a bit of digging and experimentation, I found that I needed to install kmod-usb-serial-qualcomm. Installed that package as well as minicom for a tty based terminal emulator and I'm now able to get information on the modem.

I connected up a single WIlson yagi and pointed it out the window in the direction of the tower. I got an IP address, but not a good enough signal for reliable connectivity.

Next step - get this stuff up on the roof and get both antennas connected.

I've been on a real Grafana binge lately. If you are interested in stats and dashboards, you should definitely check Grafana out (as well as InfluxDB and Telegraf), I've had some success in configuring OpenWRT's collectd to forward stats to Telegraf (which has a collectd listener plugin). Telegraf then writes the stats to InfluxDB where Grafana cann display them in a dashboard.

Next step is to write a small perl program to query the modem for connection stats and write them to InfluxDB.

[Didneywhorl]

Great work, That's super cool.

[R1250GSA]

P.D.,
How has your project been going?
It has been about a month since your last update.
I have an RPi4B that wants to go in my attic...

[poodad]

That's weird - I updated the thread a while back, but the update is gone.

The project has felt like one step forward and two steps back.

Anyway, after the initial indoor testing, I tried taking the rig up on the roof but a slight mishap ripped both antenna wires out of the connectors. I decided no more risking my $200 modem until I received the real USB3 M.2 enclosure that I ordered from China (while The Wireless Haven was closed).

I eventually got the enclosure, but had a nasty bout of gout which prevented me from getting up on the roof.

After the gout subsided, I took the temp antenna rig with the bare RPi and USB modem enclosure up on the roof. Using a temporary SIM, I was getting mixed results. Anywhere from almost no bandwidth up to 10Mb down and 6Mb up.

Running minicom against /dev/ttyUSB2, I could do AT!GSTATUS? and see that occasionally the modem would get carrier aggregation, but mostly was sticking with one band.

About that time, my largish enclosure from Amazon (again, The Wireless Haven was closed) arrived that is going to be the mast mounted home for the RPi and the USB enclosure. I had assumed that the machine screws and standoff hardware the RPi uses was the standard M3 standoffs that PC motherboards use which I have a bunch of. I was wrong - RPi uses M2.5 which is fairly rare and not stocked anywhere locally. So, back to Amazon for M2.5 hardware.

I'm also now trying to find a provider. I had assumed this would be the easy part. Turns out, it appears to me that most, if not all of the rural/RV LTE plan providers are, um, a bit less than fully legit.

I will have more updates soon.

[poodad]

It's alive! Frankenrouter (as I've come to call it) is put together and installed.

IMG_5604.JPG

Inside is the Raspberry Pi 4B, the USB3 M.2 enclosure with a Sierra Wireless 7565. Also notice the heat management system. I drilled some tiny holes in the bottom of the case (you can see the inlet holes on the left), I took a 1 1/4" sink drain tailpiece, cut it to length, hot glued a small fan on top, then hot glued that over the holes on the bottom right. The fan pulls hot air from the top of the case and pushes down the tube and out the bottom. Not optimal, but it appears to work since the RPI4 temps are the same with the case open and closed. The fan is powered from the 5V GPIO pin on the RPI.

IMG_5605.JPG

Here's a shot of my Grafana dashboard. I've got perl scripts that queries the Raspberry Pi 4 temperature and fan speed and the modem stats and writes them to an InfluxDB database. You can see that the modem starts heating up pretty dramatically when it is under load.

grafana1.jpg

Next move is to remove the top of the USB enclosure case for improved airflow. I may add a heatsink to the modem.

BTW, if anyone wants the perl scripts, let me know.

[R1250GSA]

OUTSTANDING!
Well played Dr. Frankenstein.

What are the speeds now that the smoke has settled?
Which carrier?
Any lessons learned from your modem choice, the Sierra CAT-12?

Yes, I would like to see your Perl scripts.

[SliverGT]

Nice work, do you have a better picture of how your mast is attached to the roof?

[poodad]

Perl scripts can be found here: https://1drv.ms/u/s!AjjTUdb1APffjjDKI5W ... Q?e=TO8gKe

[poodad]

Nice work, do you have a better picture of how your mast is attached to the roof?

I made the telescoping post from two pieces of metal tubing from Home Depot that fit one inside the other almost perfectly. One was a section of electrical conduit and the other a chain link fence top rail section.

I drilled a 1 1/8" hole in the top of the ridge vent which was exactly the same diameter of the outside pole.

The bottom of the pole rests in a "cup" made from a small bit of 2X4 with a 1 1/8" hole drilled 90% through. This is screwed to the attic floor. I made a locking collar that attaches to this block of wood and when tightened, prevents the pole from rotating.

[poodad]

OUTSTANDING!
Well played Dr. Frankenstein.

What are the speeds now that the smoke has settled?
Which carrier?
Any lessons learned from your modem choice, the Sierra CAT-12?

Yes, I would like to see your Perl scripts.

Right now, I'm testing with a business class AT&T SIM using the i2gold APN. Speeds are good (12Mb/3Mb), but latency is terrible at around 1s. I guess I'm hitting AT&T routers on the moon or something.

Learnings:

1. The Raspberry Pi 4 is not ready for prime time as a router. If it weren't for for some guru named Wulfy23 who is building a top-notch community build of OpenWRT for the RPi4, I'd have been stuck. The stock daily snapshots for the RPi4 suck.

2. The EM7565 seems like a top notch LTE modem. I'm seeing a bit of weirdness with AT commands, but everything seems to be working. I'll know more when I start trying band locking.

3. I need to play around with QMI vs MBIM. I've heard rumors the QMI works better for Qualcom based devices.

4. The EM7565 gets HOT when under load. My modem temp shot up to 65 degrees C while testing. It may get a heatsink

5. Grafana is awesome stuff!

[R1250GSA]

I do not see any grounding of your roof setup.
It looks like a very juicy lightning rod.

[poodad]

I do not see any grounding of your roof setup.
It looks like a very juicy lightning rod.

I have a couple of VERY tall trees close to the house and the antenna should fall within the cone of protection of those trees. However, I am going to add a ground just to be safe, just haven't done it yet.

[poodad]

Update: I've been testing some more. When left to defaults, the modem will do carrier aggregation with band 66 and some other band which escapes me at the moment. I do remember that the secondary band was one of the 700-800Mhz bands. This resulted in download speeds of around 16Mb, but upload speeds of only 1Mb. The modem temp went to 65 degrees. Testing using RDP into a terminal server resulted in a not great experience.

I'm about 5 miles from the tower (I think - the cell ID I'm seeing doesn't seem to exist on any of the locator web sites), and I have heavy amounts of trees and a hill obscuring the path. When using CA with B66 as the primary, my RSSI was around -90dB which I'd expect given that B66 is 1.7Ghz.

I tried band locking to the couple of low frequency bands this tower supports, and B5 seemed best. RSSI in the low to mid -70dB range. Downloads dropped to 10Mb, but uploads jumped to 6Mb. Overall experience was better. RDP was markedly better.

Now, I need to find a provider to replace the testing SIM I'm using. Nomad used to be a good choice, but they've recently gotten a terrible reputation of disrupted service and continuing to charge credit cards after canceling. They are expensive now to boot.

Visible seems to be a popular choice, as does the AT&T post paid tablet plan. Both of these require some extra hoops to jump through, but I guess I'll do whatever it takes.

[lordrylen]

Update: I've been testing some more. When left to defaults, the modem will do carrier aggregation with band 66 and some other band which escapes me at the moment. I do remember that the secondary band was one of the 700-800Mhz bands. This resulted in download speeds of around 16Mb, but upload speeds of only 1Mb. The modem temp went to 65 degrees. Testing using RDP into a terminal server resulted in a not great experience.

I'm about 5 miles from the tower (I think - the cell ID I'm seeing doesn't seem to exist on any of the locator web sites), and I have heavy amounts of trees and a hill obscuring the path. When using CA with B66 as the primary, my RSSI was around -90dB which I'd expect given that B66 is 1.7Ghz.

I tried band locking to the couple of low frequency bands this tower supports, and B5 seemed best. RSSI in the low to mid -70dB range. Downloads dropped to 10Mb, but uploads jumped to 6Mb. Overall experience was better. RDP was markedly better.

Now, I need to find a provider to replace the testing SIM I'm using. Nomad used to be a good choice, but they've recently gotten a terrible reputation of disrupted service and continuing to charge credit cards after canceling. They are expensive now to boot.

Visible seems to be a popular choice, as does the AT&T post paid tablet plan. Both of these require some extra hoops to jump through, but I guess I'll do whatever it takes.

How do you send AT commands to your modem? I cannot get socat to work properly.

[Didneywhorl]

How do you send AT commands to your modem? I cannot get socat to work properly.

try adding picocom, simplest for me. That or minicom

[lenn0x]

Great Project!
I'm looking to do something similar with an EM160R-GL, The Wireless Haven 4x4 MIMO antenna, and now a raspberry pi4. I tried to use a B1300 but kept running into power delivery issues. Dairyman recently released a Goldenorb Build for the Pi4.

http://www.aturnofthenut.com/upload/RaspberryPi4-GO2021-03-23.zip

[techdeisel]

It's alive! Frankenrouter (as I've come to call it) is put together and installed.

IMG_5604.JPG

Inside is the Raspberry Pi 4B, the USB3 M.2 enclosure with a Sierra Wireless 7565. Also notice the heat management system. I drilled some tiny holes in the bottom of the case (you can see the inlet holes on the left), I took a 1 1/4" sink drain tailpiece, cut it to length, hot glued a small fan on top, then hot glued that over the holes on the bottom right. The fan pulls hot air from the top of the case and pushes down the tube and out the bottom. Not optimal, but it appears to work since the RPI4 temps are the same with the case open and closed. The fan is powered from the 5V GPIO pin on the RPI.

IMG_5605.JPG

Here's a shot of my Grafana dashboard. I've got perl scripts that queries the Raspberry Pi 4 temperature and fan speed and the modem stats and writes them to an InfluxDB database. You can see that the modem starts heating up pretty dramatically when it is under load.

grafana1.jpg

Next move is to remove the top of the USB enclosure case for improved airflow. I may add a heatsink to the modem.

BTW, if anyone wants the perl scripts, let me know.

Hello thanks for all your test and documentation, wondering if there's a specific guide, you followed maybe you could point me that direction or a little help setting it up ? The granfa and stat portion, I have everything else licked

[albert90]

Whew, just came across this thread, seems like it was a really cool project. I'd love to hear how it's doing now, all this time later.

[Didneywhorl]

@poodad

How is it coming along? Still on it?

[albert90]

Hm, OP hasn't been on this thread for all of 2021. Wonder if he's still around actually.

[Didneywhorl]

@poodad was pretty active for a bit there.

[poodad]

Came back to find information on how I made this work. I should have posted a followup.

Basically, after I got this all set up, I realized that there's no legit way to get an unlimited data plan on AT&T or Verizon. Sprint/T-Mobile has no tower anywhere close to me. Starlink put a nail in this project's coffin by allowing me to pre-register for their service which would be available in my area "real soon now".

A year and a half later of Starlink continually putting me off, I signed up for their service, but with mobility mode enable ($135/month)

Anyway, I've abandoned the Frankenrouter project and I'm going to sell it off (hopefully all of it at once, but I may have to part it out).

@lordrylen to get AT commends to the modem, I installed kmod-usb-serial-qualcomm which makes the modem show up as a tty port. To talk to it interactively, I installed minicom.