Our equipment - update #4 for 2018

Looks like it has been around 18 months since we last updated our equipment information and there have been a couple significant changes.

We changed our upstream ISP about 12 months ago. We used to buy fiber and Internet access from Comcast. While this worked fine, we started looking for more of a strategic partner for our bandwidth. We found another WISP in Colorado that was expanding their business connection footprint into Golden and after some discussions, we realized it made sense to buy bandwidth from them. They took over our Comcast contract to the Mountaineering Center downtown and then they added a 10G fiber link to our tower on Lookout mountain. This gave us access to significantly more bandwidth at cheaper pricing than Comcast. It also gave us redundancy between the two locations.

With this change, we had to move away from the Peplink core router. It was time to finish learning the Mikrotik and switch back to them. We started off with a CCR1009 at both the Mountaineering Center and the Lookout Mountain tower. These were connected via 1G fiber to our ISPs routers in each location (also Mikrotiks).

IMG_20171104_091020.jpg

We also removed our AirFiber 24 link from the Mountaineering Center to Lookout and replaced it with a Siklu 1 Gbps link operating in the 70 GHz band. This Siklu link has two Ethernet ports on it. We use one to route our management VLAN between locations. Our ISP uses the other to build a redundant link for both of our networks incase of a fiber failure at either location. Our ISP took care of BGP routing of our public IP space - one less thing for us to worry about and it makes sense since they own the public range we use.

We recently upgraded both of our edge routers to Mikrotik CCR1016-12S-1S+ routers. We did this to remove a single point of failure at each location - the edge switch - and move all of our connections to fiber. Now, each CCR1016-12S-1S+ is connected to 2 or 3 Netonix switches that in turn hit all our radios. These Netonix all are home run wired back to the CCR1016-12S-1S+ via redundant fiber (using RSTP). If we loose a switch now, we only loose the radios attached to it. In the past, we had one main switch that fed other switches via copper. If we lost our main switch, we lost everything. As we keep adding radios and backhaul links, I needed to increase throughput to each switch, remove single points of failure, and get away from RF interference the longer Ethernet runs were experiencing on our towers. Plus, I like being electrically separated from switch to router on our towers by using glass.

Now that we have Mikrotik routers in both edge locations, we have our ISP routing around fiber failures via BGP and we are routing around failures of their hardware using simple internal route rules in the Mikrotiks.  In other words, if there is a fiber cut out in the wild, our ISP will self heal to the other fiber line via their link on the Siklu.  If our ISP has their own router failure or the connection between us and them fails at either location, we will self heal over our side of the Siklu link. The two fiber paths (Mountaineering Center and the Lookout tower) each take separate physical paths back to the main connection in an Internet hotel in downtown Denver.

So, while we are not 100% immune from problems, we are well protected from a large percentage of potential issues.

IMG_20180415_133906.jpg

We were able to re-use the AirFiber 24 radios to become a point to point to a large condo building in Golden. From this building, we placed an IgniteNet 10G Omni (60 GHz radio) to supply 2 Gbps connections to a number of other condo buildings in the area - allowing us to grow our "on network" building footprint and offer speeds well in excess of 100 Mbps in each of these buildings. Many can now support eventual gigabit speeds with no additional hardware upgrades. We just need to buy more bandwidth...

In each of our "on network" buildings, we place another Mikrotik router (various models) to provide both private and public IP routing to the customers in each building. This allows us to bandwidth shape at each building as well.

Let's talk about network management now... That was also a big change in 2017. The network finally outgrew QuickBooks and manual management. Both from a simple invoicing and money collection standpoint to an automated work flow point. We contracted with Sonar and now use them for 100% of our customer management, billing, traffic shaping, sales, installation, etc. A perspective member signs up on our website via a form that links to TowerCoverage.com  That form pre-qualifies the person and then dumps all their information into Sonar for us. We then schedule roof visits and installs via Sonar and kick the work orders out to our installers automatically. Once installed, we activate the customer in Sonar, it assigns them an IP address and then programs the correct router and places them into the correct speed package and builds the interface queue. It bills them and allows them to pay online. If a customer is delinquent, it will automatically slow their connection to 1 Mbps and allow access only to our billing portal. The moment they pay, it automatically releases that restriction and they are back online. Everything is automated! Took a bunch of "busy work" from the admin.

IMG_20180215_115614.jpg

We have added a couple more licensed point to point links as well as a handful of unlicensed 60 GHz links capable of 2 Gbps of throughput. We are removing as much 5 GHz point to point as possible.

2018 has brought a few new buildings on-line as well as expansion into two additional neighborhoods we did not service before.

No Point to Multi-point equipment changes are on the horizon. We continue to be very happy with the Cambium ePMP gear and look forward to the ePMP 3000 line coming out this year.

My speeds seem slow... What is going on?

When members contact us regarding slow or intermittent speeds inside their house, 99% of the time this is related to the wireless connection inside the house - from your router to your computer. There are a number of causes of this. The first thing we will ask is for you to plug your computer directly into the router and run a speed test. This will rule out the Internet connection almost all the time.

Read More

AUWireless Equipment - part 3

It's been a little while since we last updated our equipment. I know a number of smaller WISPs out there that have followed this. One of our goals as a co-op has been to provide as much information as possible to help others interested in this model deploy gear successfully.  And, one of my motivations behind this is I like to test new gear. We have a very good working relationship with many of our vendors and as a result, get some test gear from time to time. Some of it works out great, some of it not so much, but we do deploy it all in a production environment and put it through the paces.

Read More

Easley Rd site is up!

Member antenna looking towards the tower on the house in the distance

Member antenna looking towards the tower on the house in the distance

After weeks of preparation, four volunteers helping with the install of the repeater site and a week of testing the site with an existing member, we finally added our first "new" member to the Easley Rd antenna site. We are officially calling this antenna "Easley-East" since there are already plans in the works for Easley-West. However, at this point, Easley-East should have no problems servicing the residents that have asked for service from us. Five additional new members are scheduled for installation prior to the July 4 holiday.

This is AU Wireless' second mini-pop (or neighborhood tower). The first was added in Harmony Village. We have plans in the works to add a third mini-pop in the Mountain Ridge neighborhood later this summer. Stay tuned for details on that.

Testing, testing and more testing

When we started this project in August, 2015, I wanted to make sure we were going to use the fastest wireless system we could find for the money.  That last bit is key - for the money. We did not want radios on members homes to go much past $100. For the access points on the towers, we wanted to stay under $500.  So, that narrowed the field to a handful of vendors.  I had years of experience with Ubiquiti hardware. I have deployed systems of a dozen radios up to 100+ radios  using their hardware. In my mind, it was going to be the direction we were going to go.

We started by deploying a couple Rocket 5AC PtMP radios with UBNT 60 degree AC sector antennas. We bought some Nanobeam 5AC radios (NBE-5AC-19) as well as a couple PowerbeamAC radios (PBE-5AC-300-ISO) and began testing signals from various points all over our intended coverage area.

Let me back up slightly and describe the setup...  We are a town in a valley. Our main antenna site is about 1700' above the town and coverage starts 1 mile out from the tower and goes about 5 miles out from the tower. For those that understand most sector antennas, you see an immediate issue.

2015-09-24 13.37.31.jpg

However, we had other problems. First was firmware issues. Ubiquiti has been plagued with firmware issues. They develop hardware and software very quickly. Sometimes too quickly I think. Next came plain old performance issues. I was expecting to get 100+ Mbps radio to radio throughput with this set up. We were not seeing that in most locations we tested. We were getting 50 - 80 Mbps download and a little worse upload. The primary testing area was between 1.5 and 3 miles from the antenna. We made sure the narrow vertical beam of the sector was hitting where it needed to be. RF was clean on our 20 Mhz test channel.

This made me begin to doubt my immediate feeling to go with Ubiquiti and I started reading and reading. We tried LigoWave and a couple smaller vendors. Nothing we were happy with. I kept hearing Cambium get mentioned over and over again. After a long conversation with them, we decided to deploy some ePMP test gear. Through a local vendor who also had some expert advice for us, we started testing an ePMP 1000 GPS access point into a Cambium 120 degree sector. We started with the ePMP 1000 integrated CPE and then upgraded to add some new Force 180 radios along with the Force 110 into the mix. This was a very close match to the same CPEs offered by Ubiquiti.

Not to get into the weeds with details, I was amazed by how rock solid the ePMP gear is. We were getting 100 Mbps down (30 - 60 Mbps up) from nearly all of our test sites and there were no firmware issues. Also, signal to noise with the Cambium sector was about 10 to 15 dB better than what we were seeing with Ubiquiti.  Keep in mind, we did test using the same frequencies with antennas pointed in the same direction with CPEs in the same location.  It was a well orchestrated apples to apples test. And extremely time consuming. These tests ran nearly non-stop from late August into early December.

2015-12-05 15.39.20.jpg

Then it came time for some antenna tests. The narrow vertical beam width of most traditional sectors was causing an issue due to our high antenna.  I contacted RF Elements and ended up on the beta test for the new TwistPort antennas. I'll let you do the reading on them but they (on paper) solved exactly what we needed to solve.  So, I deployed two of these, one on an ePMP and one on a Rocket 5AC.

The TwistPort performed significantly better than the UBNT AC sector. No lobes, great SNR and good vertical performance.  CPEs that would not lock up on the AC sector worked with the TwistPort (which was also a 60 degree version). With the ePMP gear, it also performed very well but it drops off around 5.4 Ghz. The 5.7 and 5.8 is very crowded in our area so we need to be in 5.1 and 5.2.  The TwistPort adapter is not yet tuned to go that low. This was a major bummer because the size, price and performance are there for this antenna - just not yet at the low bands.

But, we decided to buy a couple RF Elements carrier class sectors that have fantastic 5.1 performance. Putting these on the ePMP radios (we had decided to move past Ubiquiti by now) brought our signals from around -65 to -55 and added 10 dB to the SNR. Speeds increased around 25 Mbps. This is just by putting these sectors on the AP.  I am very pleased with them and would recommend anyone needing a sector antenna, look at RF Elements and if you are 5.7 / 5.8 Ghz, try a TwistPort. You won't be disappointed.

We created a detailed spreadsheet (thanks to the RF Elements beta program) were we tracked upload and download signal strength, SNR, radio to radio speed tests and speed tests to the Internet over a 100Mbps connection. We ran 2 full sets of tests with the Rocket 5AC PtMP radio - one into the UBNT AC sector and one into the RF Elements TwistPort. We had about 12 test sites in the coverage area and tested both the NBE and PBE radios at each location.

Then, we repeated this with the Cambium ePMP radio. Two full suits of tests with the Cambium 120 sector and then the RF Elements TwistPort.  CPEs were Force 180 and Force 110 radios at each test site. These are almost exact matches to the NBE and PBE products from Ubiquiti - except Cambium does not have an AC product.

After all this, two things were very clear:

1) My wife was getting very tired of the UPS guy stopping by every day with boxes.

2) Cambium ePMP products out performed Ubiquiti AC equipment in our test environment 95% of the time (there was one radio location where UBNT gear was slightly better in speeds).  

If you are interested in seeing the full spreadsheet, contact us and I will be happy to send it along.  It has been shared with RF Elements (as part of the beta program) and Cambium.

While none of this was written as a paid endorsement, I am very pleased to be working with Cambium for our co-op WISP and thank RF Elements for a well performing antenna line.

December update and a new partnership in Golden

Finally, news to report and our December update!


In July, we started the process of looking for a building in downtown Golden with fiber optics in it for our backbone connection to the Internet. I can count on one hand the number of buildings that have fiber. Three of them are banks and are not interested in talking to co-op Internet providers. One is managed by a national leasing company that does not return five phone calls (looking your direction Source Gas building). The fifth is a building many of you know and have been in numerous times.  The American Mountaineering Center on 10th St has fiber they never hooked up but it is still in perfect condition.

I started the conversation with the building management back in August. They are a building full of like minded co-op and non-profit organizations. It was a perfect fit for us.  After a number of meetings and email conversations, yesterday we signed a deal with them for access to the fiber lines as well as roof rights to place our antenna to get the Internet up Lookout Mountain.  To say the least, we are very excited to be working with them. Next time you stop by the center, let them know you appreciate them working with us!

So, where are we with timelines? The bad news is my wife has to endure even more UPS packages on the front door as additional equipment arrives.  The fiber has been ordered as have the routers and radios to get it up Lookout Mountain. We just received 2 more (and final) test radios that would go on members homes. We are committed to finding the fastest radio equipment available today. We are down to two vendors and hope to narrow it down to one in the next 7 to 10 days. You may see me around town with a tripod full of antennas and a laptop...

I expect to transition to our high-speed line from the Mountaineering Center in mid to late December. We are adding our last test customers this week and next with our current temporary line.

If you live in the city of Golden (north of the ~ golf course), I think we can start connecting new members right after the holidays. Just in time to hook up any new electronic toys.

For those in the Easley Rd area, we are working on a relay site on the hill over there. In addition, some new radio equipment is shipping in early January we want to deploy in your neighborhood.  Our goal is to get your service up and running in late January.

We will have some more updates and information on how the co-op will work in the next week or two once we know the radio vendor and the pricing of the equipment.