Operations bought an EMS/IoT system without involving IT — now they just need “access to the router

[u/carpe_diem2022]

Hi there :) ,

Need some advice from people who have dealt with similar situations.

Our Operations Dept decided to install IoT system/digital energy meters across a fairly large factory site.

They found the vendor, agreed on the solution, signed the contract and started the project.

IT was not involved at all.

Apparently nobody discussed things like:

• How these devices spread across a large factory are actually going to communicate
• Network infrastructure, switches, fiber/cabling, VLANs, etc.
• Network/security segmentation
• Server/VM requirements
• Database requirements
• Backup and monitoring
• Internet connectivity
• Vendor remote access
• Firewall rules
• Cybersecurity

Now that the project is already moving forward, IT gets an email saying they need “access to the router” so they can put the system online.

That's it. Access to the router. :)

And somehow, when IT starts asking questions about architecture, security, server requirements, ports, protocols and who is responsible for what, the perception becomes that IT is delaying the project.

I don't want to become the guy who simply says “No”, but I'm also not going to give a third-party vendor access to our corporate router/firewall and connect a bunch of unknown industrial/IoT devices to the production network just because someone already signed a contract , the IT network must be always separate from OT network.

How do you handle situations like this?

Interested in both the technical approach and the organizational/process side of this.

[u/dvr75]

This is a pure management issue , failing to communicate and coordinate project.
To be realistic about the situation probably no one will cancel it.
So you need to be SMART and not JUST. you are now the "guy" who Throws a wrench in the works.
My advice for you is try to cooperate with this project but bring in your terms like security. Ask for layout , plans , protocols etc. and try be helpful (i know it hurts but eventually you got to play it smart and resposible).

[u/project2501a → u/dvr75]

OP this is the smart way.

and if they say "you are delaying things", the answer is "we are trying to implement the appropriate level of security for these devices , given the advance time given"

help them with a IoT vlan, make sure it works and then file a memo with your boss:

this project did not have IT involved and since it was dropped from heaven, all the other things you wanted are on hold for X months.

ps: i worked my butt off, give me

[u/ApplicationHour → u/project2501a]

I work for a contractor that sells cloud based access control, surveillance and intrusion systems. Also platform based video conferencing systems. I have on file in nice, organized documents explanatory "one-pagers" that enumerate every protocol, port and IP destination used by everything we do including exactly what it does.. I cannot tell you the number of times the non-technical people just try to slide it through as if these things are going to light right up in a secure corporate network environment.

Small and mid-sized regular old business outfits? Yeah. Sure. They usually let out whatever traffic that originates inside the firewall. But the bigger shops? Law firms? Financial? International? manufacturing? No. Not so much. I needed to talk to those guys about 3 minutes after the sales order got turned in. There is paperwork to fill out before we even start opening boxes.

[u/roronoa_zoro0101 → u/project2501a]

yeahh

[u/Reverent → u/dvr75]

Yeah, time to go full Wally reflector.

[u/Nesman64 → u/Reverent]

For those unfamiliar: https://swizec.com/blog/the-wally-reflector/

(I'm linking to this random blog that embeds the comic because the dilbert guy has refactored his website and broken the old links. Apparently it's a subscription model.)

[u/Unusual-Obligation97 → u/Nesman64]

He has also completely lost his marbles.

[u/twforeman → u/Unusual-Obligation97]

He's also dead.

[u/DrGirlfriend → u/Unusual-Obligation97]

He's also dead

[u/Unusual-Obligation97 → u/DrGirlfriend]

Well, TIL. Thanks.

[u/noobtastic31373 → u/Reverent]

Honestly, it's best way to not be seen in a negative light. "No, because of x,y, and z." Vs. "Sure thing, i just need x,y, and z taken care of. Oh those are too expensive? Sorry, my hands are tied." Now it's the situation, not you.

[u/slashinhobo1 → u/dvr75]

These people dont care about security. You have to hit them where it hurts which is their pockets. You tell them to implement this we will ha e to make a few purchases. Have new runs ran for all these items, purchase a dedicated switch. Make sure you get all licenses and 5 year warranty. If you require vms are get additional need storage. Do everything new.

When they get the bill their management will remember to brong IT to the meeting.

[u/Sinsilenc → u/slashinhobo1]

It might be multiple switch's since its across the floor depending on run length. Cable costs are stupid high right now as well.

[u/chron67 → u/slashinhobo1]

These people dont care about security. You have to hit them where it hurts which is their pockets. You tell them to implement this we will ha e to make a few purchases. Have new runs ran for all these items, purchase a dedicated switch. Make sure you get all licenses and 5 year warranty. If you require vms are get additional need storage. Do everything new.

When they get the bill their management will remember to brong IT to the meeting.

You really have to handle all of that carefully and it really depends a lot on the personalities of the people involved. Some places react well to this approach. Some will laugh at you and just say "make it work" and you have to deal with it.

[u/Papfox → u/chron67]

Don't forget having all the cabling and wiring closet work done by a professional structured cabling contractor. You don't have time to do all this work and it's important so I'm sure they want it done by a professionals for quality. If it's a factory, is it an electrically noisy environment? If so does it need screened cables or fibre optics for interference immunity? This could get very expensive...

[u/smokinbbq → u/dvr75]

As a Project Manager for a 3rd party software system. I HATE when someone on the business side decides to buy our product, and it's not until installation time that we start to find out all of the things that are going to be done.

IT obviously has their back up about this project that gets thrown in their lap. Business side is pissed that "I can't just make it work", and why are there are these delays.

[u/RevLoveJoy → u/smokinbbq]

I did a tiny bit of sales engineering about a million years ago and very quickly learned to just keep asking of our customer (potential or otherwise), "Where is your IT contact?" "Shouldn't IT be in this meeting?" "Well, we really need input from IT on these and other matters." -- some version of that remark ALL the time.

I guess my take away was nearly all businesses see their IT departments as blockers, not enablers.

[u/ddasilva08 → u/RevLoveJoy]

I had a client that really enjoyed not being the IT team in until after they had already signed the contract for a project, and then asked us to make it work. We did make it work, and after several of these instances, I pulled reports of how much more they spent on our services to get the equipment set up on the network in a manner that matched their required security compliance. (Between the delays and the sudden ask for an on-site AD controller that was never scoped for their office, it was a fair number of billed hours) They eventually learned the hard way that it was better to involve us from the jump; it just took showing their finance folks how much they were wasting due to lack of communication.

[u/pinkycatcher → u/RevLoveJoy]

I guess my take away was nearly all businesses see their IT departments as blockers, not enablers.

Stick around here long enough and you'll realize many people in IT are blockers.

"Oh they wanted to do something insecure, fuck em, we won't allow it"

"Oh they only gave us two weeks notice, we work on our time schedule not theirs"

"Oh they haven't figured out every single little process they need, ignore them until they figure their shit out"

[u/RevLoveJoy → u/pinkycatcher]

I've been doing this going on 35 years. I'm painfully aware. :D

[u/billndotnet → u/RevLoveJoy]

I had a CISO once who worked from the adage: "My job is to help all of you go as fast as possible while keeping us off CNN."

[u/chron67 → u/pinkycatcher]

Stick around here long enough and you'll realize many people in IT are blockers.

"Oh they wanted to do something insecure, fuck em, we won't allow it"

"Oh they only gave us two weeks notice, we work on our time schedule not theirs"

"Oh they haven't figured out every single little process they need, ignore them until they figure their shit out"

This is all too common. One guy on my team has this attitude and I suspect it is only a matter of time till someone higher up the food chain forces his removal. He doesn't report to me so I can only offer friendly advice but I know his manager has gotten complaints. I am honestly surprised he hasn't been removed already.

[u/pinkycatcher → u/chron67]

One guy on my team has this attitude and I suspect it is only a matter of time till someone higher up the food chain forces his removal

As a director anyone who's a flat no is getting worked around or getting removed. I come with business problems, if you have no solutions you're not helping the team or the company.

Even my cybersec team comes with an assumption of "we're doing this, here's the best way to make it safe and secure."

The moment IT becomes the wall that says no to everyone is the moment you just create shadow IT.

I try to be easy to work with and help people solve their problems, I want them to come through me, if they can come get problems solved and good answers then they'll continue to work through IT.

One problem I also see is many IT people think it's their responsibility to protect the company. It's not, it's whatever manager's responsibility. The responsibility for failures falls on me, not my team, if we have a major cybersec breach the failure is mine, and I'm the one with my neck on the line if we're non-compliant when we say we are. Unless of course someone on my team just straight up lies. My cybersec was anxious and stressed all the time when I arrived, now he's relatively chill, because we generally try to do the right thing and we all know sometimes there needs to be risk involved, but we're accepting of it rather than ignoring it.

[u/Mostlyamoron → u/pinkycatcher]

I want my team to be responsible for protecting the company. I just make sure I am accountable for it. I want them to have a sense of ownership while knowing that, as you said, failures are on me as their leader.

[u/chron67 → u/pinkycatcher]

I try to tell anyone working with me (or for me) that we should almost never say no to the business. The goal is to almost always say "yes, and" or "yes, but" or "what if we tried this instead" or similar. If the business is happy and making money then life is good. If the business isn't happy then IT can hit the chopping block surprisingly fast. We are all replaceable whether we want to think so or not.

[u/Papfox → u/RevLoveJoy]

This is the intersection of advertising and reality. The vendor ad is always doing to paint the system as being good and omit or gloss over the hard bits because they want the new customer to think it will be easy and buy it. When IT insist on obeying company policies and asking hard questions, the ad meets reality

[u/tdhuck → u/dvr75]

I've been in this scenario so many times, management never learns, unfortunately. I just CC my boss on any requests that are out of scope, a security issue, etc. and I do what they end up deciding if my boss also agrees. CYA.

[u/ThatOneWIGuy → u/dvr75]

Also, in emails. So when you raise a security issue and they force you to ignore it, you have protection when it causes issues.

[u/Spagman_Aus → u/dvr75]

It's also now - as IT are coming in at the last minute - IT's responsibility to make the other stakeholders aware of the risks created by not including IT earlier. But, that's on the IT manager, not any Sys Admins.

At the end of the day, this will go ahead. IT can get in the way, or at least get the risks of this implementation included on the Org risk register with a high residual score.

[u/chron67 → u/dvr75]

This is GREAT advice. I know a lot of folks in IT that get this attitude that everything has to go through IT first and that their time is more valuable than anyone else. Outside of very niche scenarios, the business doesn't generate profit through the IT department. Rather, IT makes the rest of the business work better. IT protects the business, speeds up the business, etc. But the business often sees other segments as making the money and IT as spending the money.

Basically everyone in IT is in customer service whether they realize it or not. Their customer just also happens to be their employer a lot of the time.

[u/PkRavix → u/chron67]

That's why it's better to work in industries with compliance requirements. You just say no and sick compliance on them.

[u/Archer007 → u/PkRavix]

*sic

[u/SamOakTree → u/dvr75]

I promise you no one's going to give them that information. Some manager is just going to bully them into doing that for them.