r/sysadmin • u/carpe_diem2022 • 1d ago
Question Operations bought an EMS/IoT system without involving IT — now they just need “access to the router
Hi there :) ,
Need some advice from people who have dealt with similar situations.
Our Operations Dept decided to install IoT system/digital energy meters across a fairly large factory site.
They found the vendor, agreed on the solution, signed the contract and started the project.
IT was not involved at all.
Apparently nobody discussed things like:
- How these devices spread across a large factory are actually going to communicate
- Network infrastructure, switches, fiber/cabling, VLANs, etc.
- Network/security segmentation
- Server/VM requirements
- Database requirements
- Backup and monitoring
- Internet connectivity
- Vendor remote access
- Firewall rules
- Cybersecurity
Now that the project is already moving forward, IT gets an email saying they need “access to the router” so they can put the system online.
That's it. Access to the router. :)
And somehow, when IT starts asking questions about architecture, security, server requirements, ports, protocols and who is responsible for what, the perception becomes that IT is delaying the project.
I don't want to become the guy who simply says “No”, but I'm also not going to give a third-party vendor access to our corporate router/firewall and connect a bunch of unknown industrial/IoT devices to the production network just because someone already signed a contract , the IT network must be always separate from OT network.
How do you handle situations like this?
Interested in both the technical approach and the organizational/process side of this.
530
u/-Enders 1d ago
Connect them to an IoT VLAN and be done with it.
Unless they are asking for admin access to the router, that’s a hard no.
245
u/vmeldrew2001 1d ago
I'd also throttle a limit on bandwidth too. IoT should be minimal, but jic.
199
u/Longjumping-Twist615 1d ago
Agree that's how you get bitcoin mining refrigerators
64
u/penisandorvagina 1d ago
Top 5 fridge models to mine eth on?
16
9
u/Wynter_born 1d ago
"Honey, why does the fridge have scorch marks around the screen?"
→ More replies (1)→ More replies (1)13
u/CanWeTalkEth 1d ago
Should really be any model since they moved to proof of stake. Ethereum is a green blockchain now.
→ More replies (2)→ More replies (1)16
27
u/KetteringChrismon-55 1d ago
Egress control kinda fits her too. If devices only need a few cloud endpoints, allow those and log the rest so anything unexpected is easier to spot later
14
u/FelisCantabrigiensis Master of Several Trades 1d ago
I would not manage such a connection without a say in what is deployed behind it. As soon as I actively manage it, I'm responsible for the failures behind it.
"You want your own door to the building? Sure, I'll get one installed with a standard lock and you get the keys. Sign here to say you will control all use of the door yourself."
17
u/purplemonkeymad 1d ago
They just also need to open a port to every device on that iot network as well. No, they don't have a static ip for their office or infrastructure why would i ask?
→ More replies (1)81
u/BananaSacks 1d ago
"And be done with it" -- yeah, not even close.
Change mgmt, security review, architecture review, DPIA if under GDPR, the list goes on.
90
u/Top-Perspective-4069 IT Manager 1d ago
The fact that their Operations group was able to do this doesn't read like they have change management at all.
9
u/BananaSacks 1d ago
Agreed, but ya never know. And regardless, compliance can quickly become liability and consequences.
→ More replies (1)29
u/Top-Perspective-4069 IT Manager 1d ago
That's why this whole thing should be a management issue and not a sysadmin issue. Sysadmin provides pertinent information, management decides what to do.
8
3
u/Nuxi0477 1d ago
"We already paid, give them what they need." :(
5
u/Top-Perspective-4069 IT Manager 1d ago
And that's a decision. But just doing a thing without raising the flag first is dumb.
27
u/awful_at_internet Helpdesk Manager 1d ago
IT wasnt involved. That sounds like a whole lot of someone else's circus, someone else's clowns. Golly gee shucks, Operations. I wish we had the time to do all of that for you, but no one told us we'd need to plan for additional staffing..
→ More replies (1)8
3
u/ka-splam 1d ago
DPIA if under GDPR
It's an energy meter in a factory, why would it be processing personal data?
5
u/BananaSacks 1d ago
It processes data. End of. From an audit, legal, compliance, DP perspective - you still need to go through the motions.
It very may will not. In that case, the DPIA paperwork will be short, simple, and unimportant. It still needs to be addressed!
•
u/ka-splam 19h ago
It processes data. End of.
The GDPR only covers personal data which relates to humans and can identify them; from the Information Commissioners Office (ICO) in the UK website section: What is personal information?:
Personal information, also known as personal data, is any information that:
- relates to you; and
- you’re identifiable from, either on its own or when linked to other information.
It’s important to know if something is personal information as data protection rules only apply if it is
My bold, not theirs. Smart energy meters do not process personally identifiable information about people, they process electricity and gas use by machines. If they somehow are under the GDPR then the website has a page: When do we need to do a DPIA?
Article 35(1) says that you must do a DPIA where a type of processing is likely to result in a high risk to the rights and freedoms of individuals:
Their bold, not mine. Their examples of high risk are "systematic and extensive profiling", "processing on large scale of special categories of data" [racial, ethnic origin, political opinion, genetic data, etc. etc.], "systematic monitoring of a publicly accessible area on a large scale".
34
u/Rainmaker526 1d ago
This. Give them a couple of access ports to a new VLAN and inform management. You don't want to start receiving tickets for those device when the vendor doesn't get its telemetry.
Let management delegate maintenance somewhere that isn't your desk.
24
12
u/Careful_Dimension233 1d ago
Second this. If only meters and measuring with CTs. Give them an isolated VLAN on the understanding that it cannot communicate with anything else in your network.
8
u/floswamp 1d ago
This is the answer, unless you own the company and you are IT as well you just work there and do as you are asked. Have them sign a paper that their needs were met and go on with your life. Sysadmin jobs don’t pay enough to worry about every single system. Not to say that you should open the network to anyone but all you do is give them just exactly what they need and nothing more.
→ More replies (1)4
u/GingerPale2022 1d ago
This just kicks the can down the road and implies that anything in the future is fair game. Reality and experience tells me that this poor soul will be told to “just do it”, but stinks need to be raised, at the very least, to them give pause for something like this in the future. At the bare minimum, it shows someone’s paying attention to the stewardship of the company’s infrastructure security.
Like I said, it probably won’t amount to squat and one does need to pick their battles, but rolling over and just tossing them in a VLAN and being done with it sets a precedent that will cause SO MUCH heartburn in the future.
→ More replies (5)
46
u/j0mbie Sysadmin & Network Engineer 1d ago
"Giving you admin access to the router voids our cybersecurity insurance. Here's a (throttled) wired VLAN with standard (guest network) internet controls. My laptop browses the internet fine on it. If the equipment doesn't work through that, it's a vendor issue."
128
u/Celebrir Wannabe Sysadmin 1d ago
When they say "IT is delaying the project" just say "Operations did not involve IT in a timely manner and forgot to factor in the time needed for IT to assess security considerations"
34
u/NeinJuanJuan 1d ago
"IT is delaying the project"
"Please refer to my previous email for a timeline documenting the durations, information, and resources required by IT to complete the requested works. Work is progressing acording to the agreed schedule.
"There is no such email."
"Exactly."
•
u/surveysaysno Sr. Sysadmin 20h ago
Maybe more like "At kick off we should have done up requirements and a timeline, when was the kick off meeting so I can find it in my notes.
I can't find that kick off meeting, was IT not included? We're going to have to do the whole assessment process. If timing is critical on a project consider looping in IT earlier."
And make that point EVERY time someone complains about waiting.
24
u/ncc74656m IT SysAdManager Technician 1d ago
Yeah but for that to work well you need to have it all in writing. Project management and change control must be documented and publicized to the point where people have no excuse for not knowing.
13
u/sadmep 1d ago
Assuming there is someone who gives a fuck higher up. Not always a good bet.
16
u/Celebrir Wannabe Sysadmin 1d ago
I have absolutely no problem with throwing people under the bus which wouldn't hesitate with throwing me under the bus. They decided the rules so I'm just playing along.
6
u/sadmep 1d ago
I'm sure you have no problem with it, all I'm saying is that for it to be effective you need someone higher up that actually agrees with you. Otherwise, you just become the odd one out and get cut out even further.
→ More replies (1)3
u/mike9874 Sr. Sysadmin 1d ago
No, just escalate it to management and they can argue out the process and decide if Ops needs to wait and follow process or if IT needs to rush it in and bypass process
62
u/Vermino 1d ago
Declare red lines.
Explain why they are red lines.
Say what you can do instead.
Explain how similar issues in the future can be avoided.
I'm sorry, we can't give 3rd party access to critical network infrastructure like our routers. IT is responsible for safeguarding the company against cyberthreats, and routers are important entry points to our company's infrastructure.
We'd be willing to meet with the 3rd party to see what their needs are, and see which we can implement for them.
In future, having these meetings sooner, rather than later, helps us find mismatches between your IT needs and our IT policies faster. We're always open to listen to your needs, and aid with our expertise.
Technical solution : VLAN, open only ports that are requered between the various VLAN's & internet
33
u/Pork_Bastard 1d ago
Yep even better if you have a written policy stating such that is filed with your cyberinsurance carrier. That helps us a ton. This is against our insurance carrier policy, sorry matt. Always protect against matt.
→ More replies (2)
102
u/Le_Vagabond Senior Mine Canari 1d ago
get an LTE modem / router, give them that, wash your hands of everything. anything less is going to look like you're the problem, which they already think since they bypassed you entirely. obviously get that signed off on by your hierarchy first if you have one.
57
u/Speeddymon Sr. DevSecOps Engineer 1d ago
This honestly is the right solution. Keep it completely off your network. You have no idea if it's going to end up being C2 infrastructure that opens a channel back to China, Russia or NK.
33
u/screampuff Enterprise Architect 1d ago
It is. We have IoT devices and they need to meet insurance requirements. The ones we have don't. So we bought separate fibre lines for 20+ locations
And this is still cheaper than getting devices from a vendor that meet insurance requirements and putting them in a VLAN on our corp network.
4
u/picflute Azure Architect 1d ago
No one in their right mind would ever propose giving someone an unmanaged LTE Model + Router that has an enterprise to manage. Operations is trying to solve a problem and IT not being consulted is a management issue. Advocating for something as risky as a LTE model with no oversight or management is only adding gas to the fire.
A separate VLAN with no outbound connectivity is perfectly fine to support operations as they are tied to a business revenue. So long as you can perform oversight of the work and inform legal + security (risk department) of telemetry coming from said devices then policies can be drafted and implemented after.
→ More replies (3)7
u/Tfire327 Jack of All Trades 1d ago
Give them full control of that device too and make it entirely their problem. Eventually you'll have to do something with it but it will shut them up now and you can prove your point later with better documentation.
4
→ More replies (2)2
30
u/gadhalund 1d ago
IOT vlan, limited to like 9600b/s so if they do exfiltrate files, itll take years, but still enough for iot packets to work fine
→ More replies (2)3
u/itskdog Jack of All Trades 1d ago
Why 9.6kbps rather than a round 10kbps?
•
u/BatemansChainsaw 22h ago
But 9.6kbps *is* the round number as the progression of dialup speeds as, ime, the third iteration.
- 2400 bps, Early modems circa mid-1980s
- 4800 bps
- 9600 bps (9.6k), Late 1980s
- 14,400 bps (14.4k), Early 1990
- 28,800 bps (28.8k), Mid-1990s
- 33,600 bps (33.6k), Late 1990s
- 56,000 bps (56k), The practical maximum late 1990s
•
10
u/TaterSupreme Sysadmin 1d ago
And somehow, when IT starts asking questions about architecture, security, server requirements, ports, protocols and who is responsible for what, the perception becomes that IT is delaying the project.
You're damn right IT is delaying the project. That's the expected and desirable outcome when you have somebody that needs IT resources for their project, and they don't bring IT in during the project planning phase.
61
u/_stinkys 1d ago
An implemented change management process would likely have stopped this from occurring. At this point you should document your case to executive management and let them make the call. I would never give them access to the network admin - Give them their own vlan with internet only access and inform ops that you are not responsible for supporting the product.
21
u/FlibblesHexEyes 1d ago
I wish that were the case. Changes come from management and when challenged we’re told to just do it. And yet that same management holds us all to the change management process.
→ More replies (1)14
u/hipshaps123 1d ago
OT VLAN’s with internet access?
Oh no.
2
u/rosseloh wish I was *only* a netadmin 1d ago
It seems that basically every single piece of OT I am involved with, "needs" it.
→ More replies (1)6
u/naosuke 1d ago
OT VLANS don't get internet access. If there is a service that needs internet access, it talks to a proxy that lives in a DMZ, but opening up the entire VLAN to the internet violates all sorts of industry standards and best practices.
4
u/rosseloh wish I was *only* a netadmin 1d ago edited 1d ago
We're about 15 years behind "best practices". It's moving, but baby steps.
We're also small-scale. Or at least the vendors we buy OT equipment from seem to behave that way.
Anyway if you've got a "here's how to implement all of this on a shoestring with negative-one people able to dedicate time to it" thing to read, I'm all ears my friend.
Edit: I also might be misunderstanding something here. What does a proxy in a DMZ do that a firewall doesn't in this specific scenario? Are you assuming I mean my OT VLAN is wide open to WAN traffic? Because no, that is not the case, nothing gets through from the outside...but things inside it can dial out just fine, which is what they "need" (phoning home to the manufacturer, etc).
→ More replies (1)
22
u/FelisCantabrigiensis Master of Several Trades 1d ago
Option 1: Pass a question to your legal / compliance / (cyber) risk insurance manager, asking if this has been correctly evaluated for risks and is approved by them before you connect it. This should, in any functional organisation, cause a bureaucratic shitstorm that will impede the fools but not be blamed on you. If your organisation is not functional, then proceed to option 2 below.
Option 2: Reply stating you will give them a network port with direct access to the Internet with standard outbound NAT and no other configuration, that they will be free to manage all other security and connectivity, and ask them to acknowledge that this meets their needs. When they say yes, do that and step back to let them do it all themselves. Ensure that their technology remains completely isolated from the rest of your network. If someone says "hey, let the office net access this thing on the EMS/IoT network", then make a trombone connection on your firewall so they are treated as untrusted external traffic.
Or, if your upper management actually backs you:
Option 3: Go to your upper management and get them to address the issue with the operations department that the IT department's involvement with networking and security is required.
→ More replies (1)8
u/TallGuyTheFirst 1d ago
+1 for option 1.
It also means that you don't become the blocker of the thing, you are just waiting on insurance to continue and I mean that's out of your hands at that point.
5
u/FelisCantabrigiensis Master of Several Trades 1d ago
If you have a good relationship with your risk, legal, etc, people (and I very much recommend doing so) then they'll get back to you with "hey, can we discuss the technical side of this" and you then advise them on the technical risks, mitigations, etc, and suggest how this could be managed to meet their risk and compliance objectives.
I've shot down more than one mad idea that way, and the risk people think I'm helpful and protecting the company.
2
u/TallGuyTheFirst 1d ago
Oh absolutely, the only reason I was able to stop a few very bad ideas (some in progress at the time) at my last employer was because we had a compliance contractor who was on the same page as I was. Crazy idea comes up, I go yeah I can see how that would work, let's see what we'd need to do to get it compliant and insured.
8
u/DehydratedButTired 1d ago
What project are you blocking?
How can you block a project you were not added to?
There is no scope or funding for you to operate, therefore you can’t operate.
Pushback hard and keep flipping it back on then.
If you give them an inch, it’s the same as blessing the project and they will keep asking for your input. Except they never asked for your input, paid for your time or submitted to your operating rules. This is a shadow IT operation and should be treated the same as someone trying to host their own WiFi or server on your network without your permission .
6
14
u/ThrobbingMeatGristle 1d ago edited 1d ago
I don't want to become the guy who simply says “No”, but I'm also not going to give a third-party vendor access to our corporate router/firewall and connect a bunch of unknown industrial/IoT devices to the production network just because someone already signed a contract , the IT network must be always separate from OT network.
If this really was your call, then you wouldn't be posting this thread.
This comes down to risk and who will accept it.
In any case my first step would be to ask for the approved change request.
If your company has a risk officer, they need to be filled in on the fact that there is a risk coming that will need to be added to the risk register.
That risk will need to be accepted by the executive officer who has that authority.
4
u/Perfect-Escape-3904 1d ago
But you know they don’t have one. Why is your first instinct to poke the wound and not to work together within your own business?
5
u/ThrobbingMeatGristle 1d ago
That is a matter of perspective.
Cyber security has evolved in most companies such that this is no longer an issue that falls on IT shoulders.
IT Policies dictate what can and cannot be done and they are de-facto internal law because those policies are owned and signed off on by the relevant C suite officers.
Change control officers implement those policies, and everything grinds on like a well oiled machine (/s).
If OP works for a company lacking that maturity, then things like this will often catalyze change for the better.
Most of the suggestions on this thread on various easily implemented networking compromises just kick the can down the road and worry more about politics than safety. OP understands the risks, he instinctively hits the nail on the head, but he thinks its his problem. It is not, and he has already got some political blowback for it.
Risk appetite is a budget item at the top level and only the top level can accept the ownership of new risks - even if that risk is a documented exception to policy - which is essentially what this situation will likely become for a while.
7
u/bstevens615 1d ago
I’d be the bad guy. I’d say NO. Too many security risks until you get a better understanding of what is needed. Someone has to keep the company safe. IT should be happy to assist but not at the expense of security.
26
u/Wolfram_And_Hart 1d ago
Literally none of that is your problem.
Isolate a port on the router and run a line to it from their rig. It’s not that hard and is done all the time. Hopefully it’s all internet based.
The moment their network needs to touch yours however then start complaining.
7
u/Sasataf12 1d ago
The moment their network needs to touch yours however then start complaining.
Even if they ask for network access, just go through the proper onboarding steps - assess requirements, risks, etc and go from there.
5
u/Wolfram_And_Hart 1d ago
Doesn’t sound like they have any of that. So isolate the line and move on. They were not important enough to include it’s not important enough to worry about.
→ More replies (2)4
u/cloudAhead 1d ago
All well and good until they ask for 'just one rule - any/any/any'.
Fast forward and you're filing the 8K about your 'material cybersecurity incident'...
→ More replies (3)11
u/dustojnikhummer 1d ago
Or just tell them to pound sand and bring their own cellular. If you are going to bypass me like that I'm not letting you onto my network. Yes, if I'm legally responsible for it then it's my network.
9
u/FatBook-Air 1d ago
Unfortunately, at most places, even if you bypass it with cellular, it will become "your network." Now you have two networks. Isolating may still be better from a security point of view, of course, but I don't think it's going to make it any less your network.
→ More replies (2)2
u/Wolfram_And_Hart 1d ago
That’s fair. But running a line outside of your firewall to the providers router is just as good.
2
u/fahque 1d ago
It's not that easy. OP said they want to install these meters across the site, meaning there are many. There would be many runs, just like after a taco bell dinner.
→ More replies (1)4
u/Wolfram_And_Hart 1d ago
They said they have a whole IoT network that they already installed. I’m assume they are wireless sensors like most of them reporting to special WAPs
2
u/dustojnikhummer 1d ago
We can talk about me giving you a VLAN that will end at your provider, maybe, but OP wanted to avoid the (quote) "the perception becomes that IT is delaying the project."
4
u/Wolfram_And_Hart 1d ago
Yep and that’s why you run the cable. Hand it to them and be done. “I don’t know what the delay is they have a dedicated like to the router and the ISPs phone number.
And if any of it messed with my crap I would light them up.
4
u/dustojnikhummer 1d ago
Ah, you meant run another physical line, sure, that can be worked with.
"Bring your own router, internet connection from an ISP and we will run the cabling for you"
3
7
u/Llew19 Used to do TV now I have 65 Mazaks ¯\_(ツ)_/¯ 1d ago
In the short term, defo make a BMS vlan with DHCP enabled on it, make sure their devices all live on it, and I guess you could hope for a list of ports to open for them but you might just have to sniff them yourself - potentially after allowing it to connect if time is so precious.
Not sure what it's called in the States, but where I work the cause of this would be a lack of 'business partnering,' basically IT being left out of company plans. IT management failing to engage with this is pretty poor tbh, and it's one of the things I actually really enjoy about my current job in a factory that's growing reasonably quickly.
7
u/KB4MTO 1d ago
When it comes to non IT asking for access to configure the router/firewall, be that guy. I would refuse access entirely to someone outside of IT. And if I am forced to give access, which I was once, I made the asking party to sign a release of responsibility for anything that happens because of them.
6
u/thatpaulbloke Cloud Engineer UK 1d ago
Imagine how the Finance department would react if another department announced that they "just need access to the bank accounts". Assuming that you have a process to go through to get access like this then make them go through it and anyone who asks why IT is "delaying the project" gets advised that the processes are being followed and the project could have remained on schedule if IT had been involved at the correct stage.
I've accommodated and gone out of my way to get things like this working and what you are teaching people is that doing things properly isn't necessary and they can just abuse you, so the sooner you put your foot down the better. If someone higher up wants to force the issue then they take responsibility in writing, at which point they usually find that they don't care as much as all that after all.
4
u/justwant_tobepretty Sr. Sysadmin 1d ago
Vlan the IOT system and put the vlan in a DNS blackhole. If data isn't reaching the vendor they can request (in writing) access.
Worth mentioning that IT needs to be consulted for any IT infrastructure amendments so you can protect the organisation from unregulated access.
•
u/ComeSwirlWithMe 23h ago
Dear Operations,
I understand your department wants unfettered access to the router so you can bring IOT devices online. Considering the IT department had no involvement or discourse with your department or the vendor, Id advise against this due to security implications, installation variables, etc. However, I will contact (BOSS) and request how to proceed.
Thanks,
IT Department.
Dear Boss..
Basically operations dept did all this dumb it stuff without speaking to us. Has all these implications. IT department will do as you request, but do not want liability if you decide to proceed.
Thanks,
IT Dept.
→ More replies (1)
•
u/Grrl_geek Netadmin 23h ago
Do you have a cyber team? Have them do their thing and find out how insecure the IoT is! 😁🤣 I am one evil *itch... hahaha
→ More replies (1)
3
u/Majik_Sheff Hat Model 1d ago
They didn't involve IT because the salesman knew those questions would come up and that interferes with the flow of country club steaks and cocaine.
3
u/jordantwalker Sysadmin 1d ago
Sounds about onbrand for medical IT. Some doctor just assumes that he's super smart and can have somebody else set everything up for him.
5
u/lethallunatic 1d ago
Classic thing.
I would say: lack of planning on your part does not constitute an emergency on my part.
Make it very hard. I mean iot devices on a network? How many devices? Is the infrastructure even ready for it? Who will manage security? Network congestion, throughput etc. It's silly to make it someone else's problem. I would push back really hard.
→ More replies (1)
5
u/K3rat 1d ago
Cc my boss and the bosses boss. I usually point out the fact that someone outside of IT really screwed the pooch when it comes to project definition and key personnel and tasks. Then I give them the proper workflow for requesting a new implementation and a pretty diagram to boot. Then I link to the form where they can make the request and submit the proper information.
5
u/Papfox 1d ago edited 1d ago
Honestly, the correct answer should be, "No. Send this project back to the drawing board so we can plan this properly with IT involved."
This does sound like a situation where a different approach may be a good idea as the people who ordered it are clearly demonstrating they're going to paint you as the bad guy if you don't roll over and give them what they want. I would respond with a qualified "yes" whilst making yourself sound helpful.
"For security reasons, we don't allow outside companies to configure our routers. Please send us details of the VLAN(s), ports and routings required and we will get it done for you." If they won't play ball, ask them to get the vendor to confirm they will accept full responsibility in writing for any commercial losses that result from any security breaches their system or their configuration of the router causes. They should immediately refuse, which makes them the ones saying "No."
If they still push back, ask your C-Suite to sign off on the risk this project might enable a cyberattack or data protection violation and that they are happy for the vendor to have unsupervised admin access to your routers and firewalls.
•
u/MelonOfFury I’m not trained in managing psychosis 22h ago
This went all the way through contracts, GC, and procurement along with an earmark from financial to approve the budget for it. If IT is not aware, leadership needs to be getting a step added to the approval process.
•
u/jimethn 18h ago
That's heinous that they didn't involve you, but I think your job at this point is to make it work. You need to get involved in the implementation and not just be a gatekeeper for access. Figure out how to make it infrastructure as code, document the heck out of everything, and basically come in as a leader instead of a complainer. It doesn't matter if they're incompetent or not, you need to be the competent one and you need to be INVOLVED. This is your network, and the project isn't going away.
As far as how you're perceived, I think this needs to be face-to-face/zoom meetings. Emails about a bunch of technical details are going to alienate leadership, even if they're 100% grounded in the facts on the ground. You need to figure out the key technical points of contact and get on their calendar and come at them with a collaborative attitude. It doesn't matter if they're within your company or without, this is a big technical project and you will get access if you reach out in a friendly way. Do your research about the tech to prep for the meeting, and ask them for any documentation you should read.
6
u/Justan0therthrow4way 1d ago
I would ignore the business owner and deal with whoever is the product owner/manager and ask for a meeting with the vendor. You need to know exactly what the system needs access to.
I’ve been in this exact situation and I’ve said “no we can’t do that, and listed the exact reasons including security issues why. I was very clear about not agreeing to cloud or database infrastructure without oh idk the fucking DBA being engaged.
Luckily for me an executive of the area doing the project (despite being an absolute bellend) realised my point of view and gave “IT” time to figure out our role since we had to support the damn thing.
The person who reported to this exec was furious lol
It’s fine to say no or push back. If you’re more junior get your manager’s support. Don’t say “what do I do”, email or direct message them saying you want to push back for
A)
B)
C)
Reasons. Unless they’re desperate to please upper management they’ll hopefully agree with you.
Then, next time in a job interview you are asked about difficult people/stakeholders/a situation that was tricky, you have a good example.
4
4
u/ccsrpsw Area IT Mgr Bod 1d ago
We are a mature organization on that - and have backing from the CEO, Board and others on projects like this:
Certain projects, not going through the correct approval process - send it back to the vendor. Period. This includes, right now, AI projects using Hardware onsite, and large scale SaaS projects too! Its amazing to see when it happens. Our CEO reams out the various managers.
Other projects, where its okay, and not a risk:
a. Even if its running late, if it needs IT changes - Submit a ticket
b. IT Security Reviews and approves or caveats as needed
c. Network team builds out the requirements (and Firewall Rules) - usually an IOT type network
d. Relevant team does the remainder.
e. Handed back to the business
We always, unless there are big revenue implications, work on a "your lack of planning is not our emergency" rule. Period. And if you dont like that - you are free to ring up the CIO, CEO or President. That never goes well (in 10+ years, I think I've seen.... 3 people do it... and never successfully). I've only seen it expedited when the number has at least 6 zeros (maybe the odd 5 zero one for important customers). But never without some level of "feedback" to managers from up high.
4
u/varmintp 1d ago
I would have the IT Manager call a meeting with everyone involved from Operation's side of procuring this and everyone from IT's side that would need to be a part of implementing this out. Then it would be a project planning meeting, so make it a good 4 hour meeting, to come up with how to roll this out and when each piece would be implemented, and who is responsible for what after rollout. Essentially the meetings that should have happen before the product was purchased and done over a bunch of smaller meetings. Don't just start implementing, call the meeting and setup the timeline that should have happen.
Then for implementation, they get their own vLAN that doesn't touch anything else.
•
u/BasicallyFake 23h ago
IT is delaying the project because the project did not follow what I hope are established processes and procedures.
Own the delay, be transparent on the why and how to make sure it doesnt happen again.
•
•
u/Brad_from_Wisconsin 21h ago
What does "access to the router" mean"
It could mean that they need to have the address of the router so that they can configure the devices that they will be installing.
Access could be granted the ability to have their equipment pass traffic to and from the router.
It could mean that they need to have a configuration made on the router to create a "guest" network for their devices to live on.
You need more information.
If they are asking for access to configure the live router, you need to step in and tell them that for security purposes, the configuration needs to be reviewed prior to it is put in and it needs to be set up by a network admin from IT.
You may gain a lot of credibility by stepping up to make sure that this implementation works. Just demand documentation and sign off prior to implementing anything.
→ More replies (1)
•
u/Intrepid-Staff-4532 21h ago
Ask them if the remember the 2013 hack of Target during the holiday season that infected all of their POS terminals and resulted in the hackers stealing 40 million credit cards and debit cards. And then ask them if they know how the hackers got into their system, and send this to everyone: https://www.tunedsecurity.com/the-2013-target-data-breach-an-analysis-of-one-of-the-largest-retail-cyberattacks-in-history/
•
u/chompy_jr 19h ago
You put EVERYTHING in writing. Scope, project expectations. Get all of your questions asked. Can’t promise they will be answered. Make your concerns known. The response isn’t important.
If they press and you haven’t been given ownership, you write up the risk and give them the keys.
If you aren’t empowered with ownership of your network then all you can do is advise and roll with it.
•
u/nyckidryan 17h ago
Give them a T-Mobile 5G connection that's all theirs. Don't mention anything about CGNAT. "You needed a connection, so I gave you a dedicated one. 🤷♂️"
Tmo has a 30 day return guarantee btw...
→ More replies (1)
10
1d ago edited 1d ago
[deleted]
2
u/Perfect-Escape-3904 1d ago
It’s an AI post, but thanks for still not reading it and giving the usual “no” answer. It helps the LLM learn more about why IT is the department most often left out of things.
3
u/slugshead Head of IT 1d ago
Energy meters are usually part of a BMS. They'll hardwire the meters back to a headend (or a few) and that's what you put on your network.
IOT or dedicated VLAN and perhaps an ACL to allow http/https to be accessible.
BMS doesn't usually need internet access, unless you expand it further. e.g. Siemens Desigo > Building X. Then it's all cloud.
3
u/hipshaps123 1d ago
If you follow Purdue (and anyone with large factories should), internet is only available in the enterprise layers, and no OT layer should have any sort of internet access.
Most small customers do - what others suggest here - is simply avoid the OT layers and setup an entirely ancillary network with a 4G router or whatever, and let them play around. This of course includes their "servers" and what not. Weakness is lax oversight. If the devices only read data and have no further NW connectivity, it might be palpable for it sec.
The right way of doing it, is of course to use local non-routed OT vlan's for the equipment, connected to a application tier, which is routable (but not internet exposed) and let this tier have servers running on structured VM environments, with various types of segmentation in play. The system is of course then connected to from corporate network via a rdp/citrix platform in a intra-connected dmz like zone - and whatever admins from the vendor side is first connected to corporate using regular consultant access.
This way you'll end up with a sandwich from top to bottom like this:
- Corp external user, connect via vpn or citrix/rds
- Secondary login / jump to interconnect/dmz zone towards factory IT
- connect to management ui
- devices connect to management sw
- everything microsegmented
→ More replies (3)
3
u/harbinger-nz 1d ago
Had this happen once but as a fully fledged replacement voip system for the enterprise, imagine my surprise when some guy turns up to deploy half a dozen VM machines into a local government organization. Problem was there was no accountability or financial constraint, being in the energy sector. An absolute clusterfuck.
3
u/AdamoMeFecit 1d ago
This scenario repeats itself infinitely within weakly run organizations. The only answer for it is stubbornly to say NO and thereby inflict organizational pain.
Ideally, leadership eventually gets the message and starts demanding IT involvement up front. In my experience, that never actually happens.
Responsibility for building out the project cleanly always devolves to the lowest and most at-risk tier on the organization chart.
3
u/i_am_voldemort 1d ago
I'll take the downvotes but have you considered:
Why didn't they come to IT for this obsentsibly IT requirement in the first place?
2
u/Bright_Arm8782 Cloud Engineer 1d ago
Because we like things done properly, to standards and in a structured fashion with all compliance and change management thought about, all of the necessary people informed and all the work planned out.
People don't want that, they just want to wang some shit together and call it done.
→ More replies (3)
3
u/stuartsmiles01 1d ago edited 1d ago
Ask them for their business case / reason so it can get approved to be allowed to be on the network, then forward to it, business and security for sign off by each team as part of approved change control with appropriate risk remediation applied, and stop ability to buy anything not within an approved supplier list.
https://x.com/starsandstripes/status/2094213583781699864
Internet connected Freezers offline...
https://x.com/AndrewCurran_/status/2094204559614947836
control systems advice
3
u/pdp10 Daemons worry when the wizard is near. 1d ago
You have several, inter-related problems here. One of your problems is that the harder you make it for them to do the right thing with this implementation going forward, the more-incentivized you'd make them to try to route around you somehow.
When it comes to connectivity, a typical threat to route around is to use mobile WWAN services instead. Or, to just avoid the Layer-1 implications and costs, to use some WiFi instead of fiber, which could potentially be disastrous in a factory.
How do you handle situations like this?
This situation is a cliche -- extremely common.
The reason why people intentionally choose to not coordinate -- to "ask forgiveness, instead of permission" -- is because of the idea that it will get them what they want, faster. They can get away with it when their peers and bosses don't rebuke them for failure to coordinate.
3
3
3
u/1z1z2x2x3c3c4v4v 1d ago
Just create a new VLAN that only has a path to the router. I would not do anything extra, like opening ports or modifying any firewall rules...
P.S. Just wait until they ask for remote access from the Internet too...
3
u/realityhurtme 1d ago
I would be asking for a properly architected internal design setting out the requirements and security controls so it can be evaluated.
2
u/realityhurtme 1d ago
And until thats been provided and signed off Im going to be knocking back any change requests that would allow that system to go live
3
3
u/RickkeeC Sr. Sysadmin 1d ago
Cite a few articles about IOT control devices are at the top of the "what to exploit first list" need to make sure no CVE exists. What is their firmware update schedule. If they bought "close out" model, it's probably not secure, using http access.
•
u/Danowolf 22h ago
And of course you have it governance documentation on your side.
Your Position : Sounds like an excellent way to save money! Your happy to assist.
Facilities owns the equipment and vendor relationship. IT owns network approval, security requirements, identity/access, and connection. Eureka everyone is happy.
•
u/sleepmaster91 15h ago
One of our customers is running a manufacturing Factory and what we've done for this is set up a "internet only" vlan with no access to the corporate network whatsoever. Then they can do whatever they want on that vlan
•
u/Turbojelly 13h ago
VLAN now!!
Best way to keep them seperate. Make sure you save at least triple IP compared to current numbers, you know how shit likes to grows.
Don't let that shit touch your beautiful network.
4
u/Fred_Stone6 1d ago
It will be fine they only want to use 192.168.1 network not like you are using it. /s
4
3
u/wb6vpm 1d ago
Remember, “no” is a complete sentence. If they don’t understand that, then escalate to “fuck no”…
→ More replies (2)
5
u/LooseEthernet 1d ago
it is wild how "access to the router" is the universal corporate phrase for "we have no idea how networking works but we want it to just work" lol. i had a similar thing happen where a vendor wanted a static ip and a dmz for a "simple" sensor array that ended up being a massive security hole. the only way to survive this without becoming the office villain is to give them a completely isolated physical port or a dedicated cheap gateway that has zero paths into your actual production environment. if they complain about the lack of connectivity to your internal servers, you just tell them that the vendor documentation didnt specify those requirements during the procurement phase. usually that shifts the blame back to the people who signed the contract without calling you first. it turns the conversation from "it is delaying the project" to "the vendor didnt provide the specs"... which is a much safer place to be lol
6
u/awful_at_internet Helpdesk Manager 1d ago
Higher Ed gets this a lot. It's simple: IT isn't responsible for the bill, IT isnt staffed to support it. You grant access in whatever way is appropriate, but you do not commit departmental resources to solve some other department's problem. Especially if there isnt even a ticket.
Give it a few years. Incompetent system administration and the consistent "we cant touch that. the vendor won't talk to us because it was purchased outside IT. We aren't trained on it. We aren't staffed to support it." Messaging will eventually drive home that yes, actually, you do need to talk to IT.
5
u/anomalous_cowherd Pragmatic Sysadmin 1d ago
Get them to put a completely new and separate ISP connection (and router) with a solid signed off agreement that IT have zero responsibility for any of it, and that it cannot be connected directly to the corporate network.
It's a terrible idea with lots of room to be subverted in future but just pushing for it might make a point.
4
u/dustojnikhummer 1d ago
Make them bring their own cellular, the only thing it will share with your company's building is a power outlet.
2
u/MasterIntegrator 1d ago
You got some shitty leadership to be blamed for slowing things down by asking questions. Those are the right questions.
2
u/StrategicBlenderBall 1d ago
Do you have a dedicated cybersecurity team, or does that just fall on IT? Either way, take a look at NIST SP 800-213, it’s for federal systems but includes a lot of useful information that you can use to protect your corporate and production networks.
2
u/mmckibben 1d ago
This is where forms are nice. Make one for them to fill out. I personally start these conversations off with something like in Happy to help. In order to maintain uptime and a secure network please fill out this form so that I can insure my work is being done correctly and meets the proper specifications for the introduction of the new solution. It puts alot of the burden back on them to give you what you originally needed and creates your paper trail without you looking like your the roadblock.
2
2
u/Unable-Entrance3110 1d ago
I think that it's pretty common for vendors to just want to do it since they do these kinds of things all the time and they are heading off unknown problems down the road.
That said, I always push back and say, "give me your configuration documentation and I will do it"
It's almost never a problem.
2
u/evantom34 Sysadmin 1d ago
State your case on the risk, push back where you can, and ask the Ops director if he assumed the risk if compromised. Don’t be passive aggressive, but cover your ass.
2
u/distrbthpce 1d ago
Please do the needful….
Seriously though, shove it on its own vlan, svi on the firewall on its own guest zone and don’t allow it access to anything.
If you want more you need to go through processes that you ignored by not involving the network team. I’m not afraid to upset anyone not in my direct reporting line. Not sure of your size business but I would say that this is all that can be done without infosec approval and make them deal with it. It’s the only thing I can do to make infosec inconvenienced after how much they inconvenience me with pointless drivel.
2
u/stiffgerman JOAT & Train Horn Installer 1d ago
The BOFH would say, "sure thing, boss!" and provide them an unconfigured port on the router. No IP config, nada. Just an ethernet port.
Being literal has its advantages. You weren't given any specs so how can they expect you to configure stuff?
2
u/Mental_Beginning_698 1d ago
I feel like this is a good instance where you separate IoT onto some consumer grade internet like spectrum. Refuse to punch holes through the firewall until they can answer some questions. Inform management its like sleeping with your windows open. In Chiraq.
2
u/CaptainZhon Sr. Sysadmin 1d ago
of course they really need access to the firewall so the device can call home and they will expose the network to the internet in the process.
2
u/phobug SRE 1d ago
Dude, just give them netgear router (not access, the literal router, get one from bestbuy with your own money and hand it over) and tell them good luck.
•
u/mic2machine 15h ago
Give 'em an old WRT54 from goodwill... I still have a couple gathering dust somewhere.
→ More replies (1)
2
u/doglar_666 1d ago
I would not connect the IoT kit to my network. My team had a similar issue. We had the luxury of being able to say "No. Purchase your own kit." - Obviously, this isn't always politically possible. If you are in a bind, simply quote for your best guess at what they need, plus a timeline for delivery. Make it clear there's an financial and time based cost when bypassing IT. If you had been properly consulted, there wouldn't be a delay.
2
u/Diligent_Buster 1d ago
Your boss needs to have your back as some people are already telling you. Remind everyone that anything that touches the firewall or goes through the firewall needs a security assessment and you are not trying to impede the project. Instead you are working to make sure it is implemented securely and correctly so that it works and performs as expected. Make sure your boss understands and has your back.
Tell them they do not get access to the firewall but if they give you their system requirements you will take care of it.
For me every vendor gets their own VLAN or in some cases their own firewall (if they seem problematic). At a minimum partition them off into their own VLAN. Possibly put them on their own switches that they pay for and you control. The switches are needed to "keep things secure and performs as expected" and have their budget pay for it as part of this project. I would think twice before handing this off to them because they didn't get approval. You don't need a bunch of adjacent networks all over the place that you don't control with their own wifi/ap's and god know what other shit they connect. I've seen all sorts of crazy crap because of this.
Be "nice" but insistent that this needs to be done right and you are working on it but you NEED them to supply requirements and you need time to implement correctly. Remind them that in the future if they bring it by you first you can be prepared next time instead of scrambling to get it done.
And document everything. Back everything up in emails to protect yourself. And insist that you need ADMIN access to this equipment. If it has wifi radios you may need to change channels, settings etc so it coexists with your other equipment. Also you need to MONITOR it, audit it, check logs, etc. Hold your ground. Be nice but insistent. If you can pull off being a dick that is fine, but your boss has to have your back or you have to be willing to accept the consequences. I was usually a dick about this stuff but I knew my bosses knew I could leave at any moment and my entire staff would follow me, etc.
2
u/mouarflenoob 1d ago
Management issue. Explain the situation in an email and ask someone above you to engage their responsibility. You're not the guy who says no anymore, you're the guy who is trying to help and waiting for the go ahead.
•
u/Own_Error_007 23h ago
Keep it segmented and submit that it would be "more efficient" if it had its own router.
•
u/mercurygreen 17h ago
WHY would they need access to the... Oooooh.... they want to punch a hole through it on 8080 so their devices can talk directly out.
Get a crappy T1 and a home router and let them go to town. When they have it "functional," take control and figure out if it can be made safe.
Name and shame - what products are these?
•
u/Dsnake1 16h ago
What's your policy say?
Mine says this would have to go through review, and compliance would have my ass. So honestly, I'd probably loop in dept heads (compliance, security, ops obviously, possibly facilities depending on your procedures) and ask for assistance expediting the review. It takes work for this to not come across as passive aggressive, but it's probably the best way forward.
In addition, I'd probably inquire from ops about budget flexibility to potentially involve an MSP. Ops would absolutely have to make the choice between cost and speed, since I don't have enough staff to make the project happen overnight, and again, per policy, I can't just hand the vendor access to the firewall.
It's also probably worth asking about scope, too, if you haven't already. Maybe it's as simple as adding some access points on a vlan.
•
•
u/Alarming-Injury7611 11h ago
I'd look to get a couple of meetings with the vendors project team. Unless they are cowboys, they will have the answers your to your questions. From my experience the "access to the router" request will be a from the non-technical ops manger on your side. Asking for a meeting and working out a solution will also signal to management the value of involving IT early in projects. That move you from the "guy that say no" to the guy that provides a solution to a business problem.
→ More replies (1)
6
u/redthrull 1d ago
How do you handle situations like this?
become the guy who simply says “No”
I don't know why you don't want to. You're the IT expert, not them. Operations stalled? The company losing money? Good. Let them. They have to learn.
Tell them it's not that you don't want to, but you have to understand first what they're actually trying to do and wanting to accomplish. Otherwise, you'll be opening yourself to a whole other can of worms when something goes wrong and now "IT just has to fix it ASAP!"
2
u/BananaSacks 1d ago
This really should be a Senior Mgmt fight. IT should be involved in EVERY IT related project, along with security, data protection, and sometime legal.
IT isn't the blocker in this case, the Operations Exec/Sr. team is purely responsible here for their actions.
This should be baked into the PMO life cycle before any project gets off the ground. This should NEVER have made it anywhere without change management either.
In your case, I would have a conversation with your senior leadership alongside the same fore the security department (if you have one). See if you can get on board to start those conversations while you and your peers begin the planning and legwork that you need to do regardless.
5
u/ISeeDeadPackets Ineffective CIO 1d ago
This isn't an IT problem it's a leadership problem. If you're in charge of IT, then it's your problem and you need to learn how to communicate with the other business units. IT's job is to enable the business's objectives and a huge part of that is building a healthy relationship and not throwing some hissy fit because you weren't read in at the appropriate time.
This is a great opportunity to build a foundation for being more proactively included in other projects, or just being the obstacle they currently see you as. Write up a brief message in terms that aren't overly technical explaining what you need to get this going.
That doesn't mean "Hey bob, we need the vendor to provide ports and blah blah blah" it should look more like "Bob, I'm glad your team has found a solution that works for you, IT will do it's best to get this going. Before we can do that though, we need some information from the vendor to make sure the implementation will be successful and as non-disruptive as possible. Can you introduce me to your representative with the company so we can get a call with their technical team setup?"
Once you have that going you can build out a bit of a project timeline and you'll probably have some backup from the vendor reinforcing it. If you've got competing work then you can lay that out to: "Bob, I've been able to work with their technical team and I've made up a project plan. This constitutes about "x" hours of work based on what the vendor has told us. Right now we're also tasked with "x" along with normal daily support/maintenance, can we get together and figure out which items to prioritize?"
Or you can just toss up an IOT vlan and tell them to have fun while doing nothing to prove you can be a useful asset to the company. One of these approaches is a lot more likely to get you read in sooner than the other for future projects.
3
u/_haha_oh_wow_ ...but it was DNS the WHOLE TIME! 1d ago
1) Warn in writing that not consulting IT on an IT purchase is not recommended and caution against untested system that IT had no say in. Note that this system may not be supportable.
2) Give them "access to the router" and buy some popcorn. When problems occur, refer them to the warning from step 1.
4
u/collinsl02 Linux Admin 1d ago
And by access make it "that port there" which will give bare internet access. Nothing else, no inter-site services, no switching etc.
2
u/RunningAtTheMouth 1d ago
Oof. At least it's not a printer.
But seriously, the job is to support the company. Keep that in mind and move forward as best you can.
In my org, there are only 3 entities that may access our firewall - Me, my systems counterpart, and our MSP, because we know all the little things that make it work. You REALLY don't want someone who doesn't know your systems to mess with that. I'd lead with that.
Then plan on how to make it work. Off the top of my head, a VLAN and a network just for this IoT setup with whatever access they need. Don't let it touch your production network - you don't need that kind of thing.
Finally, talk to your manager about the planning of such things. A heads-up ahead of time allows you time to plan and handle the prerequisites necessary. You're not holding it up. You're working on the infrastructure that is absolutely necessary to make the system both reliable and secure.
2
u/kadusus 1d ago
Access to the router is a hard no. As a compromise, setup a dedicated vlan to talk to the devices and meet minimum specs. Since IT was not involved, that VLAN is walled off from the rest of the network, with no access network wise to any devices that are currently talking to production. This way you are not "delaying" the project while continuing to keep your security posture. If they want more access, then they are going to answer your questions so you can ensure you can support the design and still keep everything secure. Management should then hopefully be having the conversations to retrospect the fact IT was not involved and how to avoid issues going forward. This should spark a conversation within leadership on how to include IT in the future. Until then, least privileged access with a crap ton of guard rails.
2
2
u/bv915 1d ago
Unpopular opinion: It doesn't matter that you weren't involved from the jump. You're now there. Do whatever is needed to implement this system as if you were involved from day 1.
Any perceived resistance from IT will be reported to top leadership as, "We're trying to work with IT to get this implemented but they're <insert something negative here>." You'll be labeled in some offensive manner and it will only compound your "they don't come to me with things," problem even more.
•
u/dartheagleeye Jack of All Trades 23h ago
The more I hear tales like this the more I am considering a career change
3
u/feel-the-avocado 1d ago
"The router doesnt have a local management interface, it downloads its config from our cloud servers. What change do you need me to make to that server configuration?"
1
u/Snoo-95788 1d ago
Had something similar happen at a place i worked at.
Sensor lights/cameras only problem was the rf signal from sensor to light and camera cut the signal from the APs. Devices would drop randomly. Found out it was the shiny new toy someone decided was a great cost cutting venture.
Separate vlan was suggested and it is the best/easiest way.
1
u/ranjop 1d ago
Few thoughts:
- make Operations to be responsible for the maintenance and cybersecurity of the IoT devices
- strict NO to connect any OT system
- separate VLAN, no routing between any other internal network
- strict FW rules inwards and outwards
A bigger issue is why Operations never contacted IT in the first place. Some relationship building needed, maybe?
1
u/I-Love-IT-MSP 1d ago
They want to open ports I bet. Yeah tell them to give you the information and you'll make the changes. If it's an open port to a public static it's fine but open to any is a death sentence. This is why I hate low voltage idiots. They have zero clue about anything yet have the strongest opinions.
1
1
u/The_Wkwied 1d ago
If they can't specify what 'access to router' they need, they need too much access.
If you want to play, give them an IP on a vlan all by itself with access to nothing. Then when they say it doesn't work, well, what IPs does it need to talk to? What ports does it need open? If they can't tell you that, they don't know. They just want everything. That's not happening.
399
u/dvr75 Sysadmin 1d ago
This is a pure management issue , failing to communicate and coordinate project.
To be realistic about the situation probably no one will cancel it.
So you need to be SMART and not JUST. you are now the "guy" who Throws a wrench in the works.
My advice for you is try to cooperate with this project but bring in your terms like security. Ask for layout , plans , protocols etc. and try be helpful (i know it hurts but eventually you got to play it smart and resposible).