slazer2au

cross-posted from: https://lemmy.ca/post/32248097

These stories are originally posted over the past decade on Reddits TalesfromTechSupport so I am copying over to Lemmy to help bring some life into this /c/

---

Some of you know I work for an ISP in a land down under. This incident took place a few ~~months~~ years ago when Apple ios 7.1 came out

---

Just got back from lunch one day and one of our layer 2 wholesalers call up to log a "fault"

Me: G'day slazer speaking
ResellerIT: Hi mate, I am wanting to log a speed fault with one of our private schools.
Me: no worries mate. What school?
ResellerIT: RegionalPrivateSchool. Your favourite one, they are only getting really high latency and between 5 to 10mb/s

damn it not those guys again

Back story. When this school went live their hardware firewall had a bug where after x amount of data was pushed, it could only do about 20mb/s in either direction.

Me: Considering previous problems with that school have they rebooted their firewall?
ResellerIT: Yes, odd thing happened though, when the firewall came up it ran at the 100mb/s for about 10 - 15 min before dropping back again.
Me: Odd, let me check it out.

I log onto the radio and see the school usage is bouncing between 80 to 100mb/s.

Me: Mate, have you looked at their current usage?
ResellerIT: No, why would I?
Me: Just look. You will work it out.
ResellerIT: Bugger me, that's quite a but of usage. I'll take it from here, sorry to call you mate.

/call

I kept the radio screen open in the background in case he called back and went back to my "active internet monitoring" AKA Reddit while listening to LRRLive on Twitch.

A few hours later I get an email from my boss asking what is happening at RegionalPrivateSchool, he got a call from the account manager. The only time the account manager gets involved is when he isn't getting in info out of his IT team (ResellerIT).

I flicked him an email back recapping my chat with ResellerIT and look at the radio it is still flatlining 80mb/s both ways.

I decided to take a look as to why a school with no students in it is still using 80% of their bandwidth in both directions. So I run the SuperSecretSexySpecial command on the radio that shows the top 20 source and destination IPs along with packets per second in real time.

When looking at the SuperSecretSexySpecial output I do some reverse look ups on the addresses. The school seemed to be pulling an arse tone of traffic from the local Akamai cache and pushing just as much up to addresses that map back to dsl services.

I start thinking, why is the school doing so much data? First thought, second Wednesday of the month Windows updates. But then I thought surely a school should run WSUS in case a bad patch comes out. As for the upload maybe some of the staff have discovered torrents aren't blocked on the firewall and let them run overnight.

I shoot my findings though to my boss, the account manager and ResellerIT. I include in the email that this is all speculation as well as some pointers for fixing it they can pass onto the schools IT guys. I get an email back from the account manager with some comments from the schools IT people saying they don't run windows, it is an Apple school and they are already running the apple version of WSUS. They also boasted that their school was one of the ipad trial schools. 1,300 students all with ipads, my second worse nightmare.

Then I remembered what my work iphone did this morning and an article I was reading at lunch, ios 7.1 for iphone, ipad and ipod came out a few days ago and we all know what happens next. The flood of app updates.

I decided to call the school and talk with their IT guys about running some tests for me. First step was to remove the apple update server network cable. When he did, the traffic dropped back from 80mb/s both ways to about 15mb/s. I asked them to plug the server back in and surely when it came back online the usage started again.

At that point I speculated that the student devices are calling back to the school to get the ios7.1 update and any apps that also require updates.

---

The following Friday I get an email from the account manager, thanking me for helping with the issue at the school. It turns out I was spot on with the student devices calling back to the school for app updates. After the schools IT guy reconfigured the apple server their speed tests were back up to 100mb/s both ways and sub 15ms response times.

The boss was so happy with my work he let me off early on Friday with a bottle of something special.

Microwaves tend to come in 2 types, ones with a rotating plate and ones without. Assuming everything else is equal about a microwave does rotating the food assist with the reheating?

Talking about food and why he should get more.

Another tale from the the land downunder. This time for all you RF geeks. I apologise in advance if I use dB, dBm, and dBi incorrectly, I tend to use them interchangeably at work.

---

One of those random things I have to do is support wireless gear that our ISP sells on the side to system integrators for point to point wireless between buildings.
It is fairly easy work, we over engineer the links to perform better than the system integrators expect. This is a story about how the original engineer over engineered the link too much.
The link was installed about 6 years ago and from what I understand hasn't performed as expected.

---

In the office, at my desk working on how one of our transit providers fudged up their route map and was advertising our address space back to us, a story for another time maybe.

phone rings

Me: G'day slazer speaking.
Customer: Hi, its Customer from [redacted], we bought a wireless link from your firm few years ago and it has been working mostly well till last week when it fell over and we haven't been able to get it back.
Me: Ooooookk, let me grab your details and I will give it a crack.
Customer: The box in the rack says Redline AN50E and the link light is off.
Me: all right, do you still have management access to the radio?
Customer: I do on this side, not the remote site obviously.
Me: Makes sense. on the status page what are the RSSI and SNR values
Customer: RSSI says -86 dbm for all 3 values and SNR is 0 dBm
Me: Is the other end powered on?
Customer: Yes, the guys in the other office can login to the management as well.
Me: That's good, can they tell you the values on that side too?

hold music starts

Customer: They are seeing the same values.

damn

Me: Do you mind if we come down and have a look?
Customer: No worries mate, just ask for me at reception.

I make a list of kit we will need for the job and "delegate" it to my minion to load into the van and we head out.

---

We get to site and Customer shows us around the master end of the link. I spot the first of many problems. The ethernet is running in half duplex mode (may account for their poor performance.) and the radio is running at 20dB transmit power.

I turn to Customer.

Me: have you played with any of these settings?
Customer: When it was originally installed the tech said if we have any problems with the link we should turn the transmit power up to 20.

I stare blankly at him for a few seconds before double checking I'm not going insane. I make note of the usual misconfiguration suspects, frequency, channel size, encryption enabled, correct encryption key and drop the transmit power down to 1 dB. We head over to the slave end.
Most of the settings are correct, with the exception of transmit power, again it is running at 20dB. I drop it back to 1dB and see the SNR come up above zero for a few seconds before disappearing.

We do a test on the indoor coax cable going to the roof and see no RF coming back down the cable. Damn a faulty outdoor unit. So we head up to the roof and see what we can do about the outdoor unit.

I let my minion and Customer go up the ladder first and as I pop my head out of the roof access hole I see a disaster.

The original tech installed a 60cm panel for a rf link which is no more than 50M. Rf geeks will know why this is a disaster. 20dB of transmit power along with a 28dBi antenna, no way that is legal in Australia.

We swap out the outdoor unit on the slave site, because we were on that side, and as soon as we plugged in the new outdoor unit it started chirping away with its alignment buzzer saying it has the maximum modulation.

Me: That's not good.
Minion: What do you mean? The link is working with this new outdoor unit, so we found the faulty part.
Me: Yea, but where is the antenna connected at the moment?
Minion: In the faulty unit.
Me: Yes, so with 1 dB transmit power on both end and only one 30cm panel on the master side we are forming a link.
Minion: So?
Me: What do you think will happen when we attach the 60 cm panel and put the transmit power back to 20dB?
Minion: It will get saturated and the link will fail.
Me: Yes, so all the drop outs they are talking about is because the link was overengineered too much.

We reattached the panel and looked at the management RSSI -36dB, SNR 30dB.

Me: That has sorted it.

phone rings, Customer comes up on caller ID

Me: Hi mate, we got it back up, how is it looking?
Customer: The link light is on, but I cant ping across the link.

damn it Rf is up and talking but no traffic is passing, the encryption key must be wrong. I get him to correct the encryption key and his traffic starts flowing again.

I confirm the modulation and transmit power are ok and head back over to the master end to talk with Customer.

---

Me: The outdoor unit is most likely to have burnt out because the RF levels were too strong.
Customer: I notice now when I put the transmit power to 20dB the link goes offline.
Me: Never change that value to above 1 ever
Customer: Ok then. The speeds are better, before it was running between 6 and 12 Mb/s now it is saying 54Mb/s
Me: Yes, because of RF magic we turned the signal power down to get a better signal.
Customer: I'll accept that.

And with that done, Minion and I went back to the office.

---

Context for those who aren't in the RF world. Imagine having a conversation with someone across an alleyway with one person shouting at the top of their lungs and the other using a megaphone. At some point hearing damage kicks in.

When using new wireless kit, never assume the vendor knows what they are doing, most of the time they do not know what the local laws regarding wireless equipment even are. We have some vendors ignore standards while others follow the standard so closely the kit becomes unusable.

---

We installed a new 900Mhz radio to a customer who was in a particular bad spot. All seem well, the customer was getting the speed over the wireless and the latency was rather good.

A few weeks after install I get a call from the customer.

ring ring

Me: G'day slazer speaking.
Cus: Hi, this is [manager] calling from [customer] we have a guy here saying the radio on our roof is interfering with [national mobile carrier] in the area.
Me: Ooook, that doesn't sound good. Can I talk with him?
Cus: Sure. I'll shoot the call down to reception where he is.

call transfer

Me: G'day this is Slazer, we run the kit on the roof, what is the issue?
CarrierTech: This is CarrierTech from [contracting firm] we have been sent out by [national carrier] to find out why their customers are experiencing call problems in this area.
Me: I see, is [Cus] still hanging around?
CarrierTech: Yes,
Me: Sweet, I need to have a quick word with him and we can sort this out.

Phone passed back to Cus

Me: Hi mate, Thanks for calling us. We will handle everything from here and you wont have to do anything.
Cus: Ok, sounds good, I will pass you back to CarrierTech

Phone ping pong finishes.

Me: Right mate, lets get this sorted. What are you seeing and how can we resolve it.
CarrierTech: I noticed the radio on this roof and our kit is saying it is running in the 900Mhz band. What brand and model is the radio?
Me: It is a Ubiquiti Nanobridge M900.
CarrierTech: Is the firmware up to date and you are running in the Australian country code?
Me: Yes.
CarrierTech: Ok, so it looks like it currently isn't complying with Aussie rules because it is sitting in the middle of the 900Mhz band assigned to [national carrier].
Me: Not good, What is there band?
CarrierTech: [freq band]
Me: Yea, we are sitting in the middle of that, luckily this is a backup link so I can mess with it during business hours. Let me lock out those frequencies and reboot the unit.

few min later

Me: Ok, I have gone as far away as I can from their band, how is it looking?
CarrierTech: I will have to check from outside. Can I have a number I can call you back on?
Me: sure, [insert company number]
CarrierTech: OK, I will call back a little later.

---

About 20 min later he calls back.

CarrierTech: It looks like that has cleared up the problem. Where does this link go back to?
Me: [insert address from city 10Km away]
CarrierTech: sigh I spent the entire day there yesterday chasing down the same problem and narrowed it down to that street. I should of started at this end.
Me: Well, my apologies mate, I will have to get in touch with the vendor and get this fixed for the next firmware release.
CarrierTech: Yes. I am sure [National Carrier] will also push them and the ACMA about it.
Me: On that note. I assume because the problem is fixed we won't be getting a call from them?
CarrierTech: No, if they complained to the ACMA it would be 6 months before they could do anything about it.
Me: Sounds about right for a government department, just out of curiously how many sites were affected by this?
CarrierTech: About 20 to 30 sites.
Me: wow, now I am really glad you called us first.

insert ending formalities

/End call

---

I let the boss know what happened and he was glad how it worked out.

Last time we had a run in with the ACMA it ended badly for them, but that is another tale for another time.

cross-posted from: https://lemmy.world/post/19180701

My incident over ~~2~~ 9 years ago involves the federal regulator making impossible claims.

---

Working in the wonderful world of Wireless Internet Service Provider (WISPs), you get those calls once in a blue moon that makes you question everything.

phone rings

Me: G'day, this is slazer.
Caller: Hi, this is Fred calling from the ACMA (the Aussie version of the FCC). Can I talk to your senior radio engineer please.
Me: We don't have one, but I am the senior network engineer. I will do what I can do help.
Fred: Ok, I am at [site] and we are detecting some interference on the local council 80Mhz band and we believe your equipment is responsible.
Me: I am sorry, run that by me again.
Fred: We believe the equipment operated by your company on [site] is interfering with the local councils 80Mhz emergency push to talk system.
Me: Ooook. That sounds impossible our equipment is running at 5Ghz. How did you get to that conclusion?
Fred: Well, we have shut down all the other wireless operators on the tower but the interference is still there. In your cabinet there is what looks like an amp which takes up about the bottom 6RU. Would you be able to turn that off?
Me: We don't have an amp in our cabinet. That is our UPS in case there is a power outage.
Fred: A UPS? That explains why your equipment didn't go down when we turned off your breaker.
Me: It also kept beeping at you till you turned the power back on didn't it?
Fred: Yes. So is there a way we can turn your kit off so we can finish our tests?
Me: Not at this time of the day. We have clients actively using the service.
Fred: Ok, I will run some more tests and get back to you.

/call

I take down his number in case he calls back and let my minions know that if he calls put him directly though to me. I call our vendor rep, just to make sure I am correct.

Vendor: Hello this is (dude) from (vendor)
Me: G'day , it is slazer from (WISP). Do you have some time to chat, I just got off the phone with the ACMA.
Vendor: Oh boy, whats up?
Me: Well one of the ACMA "engineers" have said the kit we have installed is interfering with an 80Mhz push to talk system.
Vendor: That doesn't sound possible. If it were possible, we would have people all over the world complaining.
me: I know, just doing a sanity check. I will let you know if it turns out to be your stuff, which I doubt.
Vendor: No worries mate, thanks.

/call

I also call the boss and let him know what is going on. He has the same mind set as the vendor, impossible for us to interfere with an 80Mhz system.

A couple hours pass and he calls back.

Me: g'day mate, how did you go?
Fred: You have a radio pointed between 50 and 60 degrees off the tower, I think that is responsible for the problem.

I look up the radio in question and it is a 5.4Ghz radio.

Me: That can't be. It is a 5Ghz radio.
Fred: can you turn it off so see if the interference goes away?
Me: Like I said before I can't turn off any of our radios unexpectedly during the day, that particular radio goes to the school in [suburb].
Fred: Hmm, when can we turn it off to test?
Me: provided the school is OK with the outage, 2 weeks from now at 3AM.
Fred: Your shitting me?
Me: No, part of the contact we have with the school says we have to give 2 weeks notice for any planed maintenance that could impact their service.
Fred: But why 3AM?
Me: Because that is the time when it will disrupt the schools service the least.
Fred: There has to be a better time then 3AM.
Me: Not really, the schools nightly backup goes from 8PM till 2AM.
Fred: Seriously?
Me: Yes. I will call the school now and organise the outage. I will give you a call back when I have confirmed everything.

/call

---

I organised the outage with the customer and kept everyone in the loop.

---

Outage window came along and I got a call from Fred.

Fred: How far off are you?
Me: I am ready to go.
Fred: Eh? Aren't you meeting us here?
Me: No, why spend 2 hours travelling up there at night when I can do it from the comfort of my home?
Fred: OK, well lets get started.
I turn off all the radios except the the one I am using to log into the site via.
Me: They are all off except one, how is it looking?
Fred: Still seeing the interference. When you say they are off, I am still seeing the same amount of lights on your gear in the hut.
Me: I have turned off the radio unit on the outdoor unit. So at the moment all our radios bar one are not transmitting.
Fred: Which one is on?
Me: Our backhaul, if I turn it off I wont be able to turn it back on remotely. What I can do is bounce it. Have are you looking at your kit?
Fred: Yes.

I reboot the final backhaul radio.

Me: OK, you have about 2 min before it comes back online. How is it looking?
Fred: No different...... What in the world is causing this interference.
Me: No clue mate, we operate in the 5Ghz band. Seeing as you haven't found anything I am going to turn our kit back on now.
Fred: but we haven't finished testing yet.
Me: Yes we have, all our kit was off and you said there was no difference in the interference.
Fred: It must be your kit. It is the only unlicensed kit in the area. Everyone else is using licensed spectrum.
Me: ............. I would ask how you came to the conclusion of they don't use licensed spectrum so they must be the problem, but it is 3AM and I would like to go back to bed.
Fred: But we aren't done yet.
Me: Yes, we are. Good night.

/call

I turn on our equipment again and write up a report for the boss, then return to bed.

---

A couple days later, we received a warning notice from the ACMA about the events that transpired. Sadly, this is where my part in the story ends and the boss picks it up.

After several back and forth between the boss, our lawyers, and the ACMA rep. The warning is withdrawn and the 80Mhz kit gets moved to another tower a couple hundred meters down the road only to run into the same interference problem.

I don't know if they ever fixed the problem, it has been a few years and it doesn't bother me.

These stories are originally posted over the past decade on Reddits TalesfromTechSupport so I am copying over to Lemmy to help bring some life into this /c/

---

Sigh, I had one of those Mondays. As per the rules all names are replaced to protect the identity of the stupid and ill informed.

Some auzzie slang/humour may come off as offensive, I apologise, its just how we roll in the land down under

Back story, I work for a fixed wireless ISP. I deal anywhere between integration firms and the onsite IT bloke. This particular incident took place at the HQ of a multi site medical center group

---

Get a call at 6:30AM

Me: G'day slazer speaking.
Customer IT guy (Lets call him Steve): Hi mate, its Steve from Medical Group our head office is offline at the moment. We had a really bad storm go though last night, it may just be power but can you guys be on stand by just in case?
Me: nyaaa, all right. I'll do my usual morning stuff and get into the office asap. Can you check out HQ and let me know?
Steve: no worries mate.

2 min later

imessage from the boss: slazer, the HQ of Medical Group is down. whats going on.
imessage to the boss: Just got off the phone with their IT bloke and he is going in to checking power. I'll get to the office early and prep our spare radios.

no reply from the boss.
[insert usual morning stuff of shower, shave, and shi....]
While driving to the office I get another call from Steve

Me: G'day Steve, how is it looking on your end?
Steve: Well, we have lost a UPS and a switch to last nights storm, you may of lost your radio though, there is no up-link light on your Cisco NTU.
Me: bugger, I guess you have tried power cycling it?
Steve: Yea, the light is on the power injector but no light on the NTU. Our sparkie (aussie slang for an electrician) is coming in to check everything else is OK, I'll get him to check your cable too.
Me: Cheers mate, I'll get a spare radio configured and head straight up to you.

3 accidents on the motorway D: a normal 45 min trip takes 2 hours but I get there eventually.

CIT: You took your time mate.
Me: Traffics a female dog.
CIT: Fair call, the sparkie had a look at the run from the server room to the radio on the roof, he said everything is fine. where do you want to start?
Me: Well lets make sure the POE injector is OK first.

We head to the server room and I notice there is no light on the POE injector. I do the usual troubleshooting and the light on the POE will only stay on while the cable to the radio is not plugged in. I check the injector by plugging in the replacement radio, lights stays on and the radio turns on and starts squawking while it searches for a base station to connect to. The port on the NTU also comes on ruling out the POE and NTU as the cause of the fault.

Me: Well the problem is not down here. Lets go for a sticky beak on the roof.
just as i finish saying the sentence, the sparkie appears out of nowhere.
Sparkie: Everything is fine on the roof, I have checked the cable and the radio is powered up
Me: ......... its not that I don't believe you, its just that..... no bugger it, I don't believe you.
Sparkie: hmmf

the sparkie walks off.

Steve: Little rude there mate? Me: Only because he lied.

Stevelooks confused

Me: By how the light was behaving on the injector, there is no way everything is fine.
Steve: Fair enough mate, let me know what you find.

He goes back to checking the servers and I head up to the roof alone. Once I get onto the roof I notice there is no light on the bottom of the radio...

I remove the waterproof bung and saw the rj45 head had been...... I don't have a word that will get passed the profanity filter for how the head looked.

Now, I have seen RJ45 heads shorted before from either over voltage (doing 54v to a 24v device) or water getting into the bung but nothing this bad.

It takes me a moment to collect myself and I begin repairing the cable. YAY for service loops!!! I install the replacement radio and get off the roof to make sure the customer is back online.

---

Warning: PUT YOUR DRINKS DOWN BEFORE OPENING THE PICTURE

---

I find the Steve in his "office" (read cubby hole)

Steve: back online are we? Good, What was the problem.
Me: May wana get the sparkie in for this.

Stevelooks confused, but pages him to his "office".

Sparkie: Whats up?
Me: When you said you checked the cable, what did you do?
Sparkie: I put a RJ45 tester on both ends and it tested OK.
Me: Again, I do not believe you. Tell me, how did you "test" this?

I gave both Steve and the sparkie a moment to collect their jaws from the table.

Steve: You can go slazer, thanks for getting the connection working. May I keep that head?
Me: Sure mate, I have a pic, that is all we require.

---

I am not sure if I will find out what happens with that sparkie, but I doubt I want to. On the bright side, because I had to travel before 7AM the company paid for my breakfast :D

To those of you who saw the pic before my warning of putting your drinks down, I am sorry. For those of you who blandly ignored it.... well, I am still sorry, but you were warned.

---

Update Time

So it turns out the sparkie vocabulary is smaller than both myself and Steve thought most sparkies have. When he was told to check the cable going to the radio on the roof he thought they were talking about the Wifi Access Point on the 3th floor.

His reasoning: Because ground, 1st and 2nd have floors above them they have ceilings. 3rd floor is the top floor so it is not a ceiling, it is a roof...... I'll let that logic sink in for the rest of you too.

This is a more recent story while working for an MSP in Europe compared to my time working for an ISP in Australia

the cast:
Me: Slazer
OT: Other Tech

---

I get a message on slack

OT: Hey, I am seeing something weird in the French office for customer, can you help me look into it?
Me: Sure

Queue the Teams call.

OT: So all the Access Points in that office are reported as offline in cloud.vendor.com portal but the customer is not reporting an issue.
Me: Ok, that is odd. What is the monitoring system saying?
OT: Monitoring says everything is OK, I can ping them and do SNMP calls to all the AP, they are just reporting as offline in the portal.
OT: The other thing is the firewall says the AP are trying to access cloud.vendor.com but the local in policy is denying the traffic.
Me: That is rather strange.

I log into the firewall and check the logs and see the APs are in fact trying to access cloud.vendor.com but the destination is 255.255.255.255. Not the expected IP from the vendors documentation.

Me: Well I want to say it's a DNS issue what happens when you reboot the AP?
OT: Rebooting from the portal doesn't work but I rebooted on from the switchport and the same thing happens.
Me: Is the on prem DNS server working?
OT: Yea, the domain controller is the DHCP/DNS server and it has no issue with access, the customer hasn't reported connection issues. It looks to be just the APs.
Me: Ok then, are they being allocated the right DNS servers?

OT logs into the domain controller and everything is looking good.

Me: dafuq?..... Wait, do these even use the DNS server from DHCP or do we set one via the device template?
OT: Not sure, never had this happen before. When we provision these they are plug and play.

I log into the vendor portal and start poking around and notice all the APs have the same DNS server of 208.67.222.222 (OpenDNS)

Me: Ok, well the AP aren't using the local DNS server they are using openDNS. Lets start a packet capture to see what is going on.

I setup a packet capture on the firewall and limit it to the IP of the AP we are looking at and let it run for a bit and crack open the capture in Wireshark.
I just start laughing at the error

OT: I know that laugh, what did you find?
Me: what do you make of this error?

Every single DNS query had this as the response.

The OpenDNS service is currently unavailable in France and some French territories due to a court order under Article L.333-10 of the French Sport code. See https://support.opendns.com/hc/en-us/

OT: Wha????
Me: Yea.... Now for the hard part.
OT: Hard part?
Me: How do we fix this? There is no ssh logins to the AP, we can't push config because the devices are offline according to the portal, and there is no way we are getting console to each of those units.
OT: I see.

Then the dumb idea occurred to me.

Me: I have a dumb idea. We DNAT any traffic destined for OpenDNS to Googles DNS so we can reconfigure the units to use the local DNS servers.
OT: Would that work?
Me: It should.... I hope.

We then setup DNAT for the AP specifically to rewrite the DNS request destined for OpenDNS and forward it to Googles DNS.
After activating the config we start seeing the devices come online in the portal as if nothing happened to them.

OT: Hey, it worked.
Me: omg, it actually worked...

I am somewhat sill shocked it worked.

At some point I will get some time to clean up that DNAT and finish reconfiguring the APs.