r/sysadmin • u/TruePace3 • 4h ago
Work Environment UPDATE: Hospitality Guy in IT
previous post (got removed by mods, but its the same post)
So basically, i joined today and after the onboarding, i met with the current IT guy (who is on his notice period)
The situation is precarious to say the least.
IT budget is severely limited, a bunch of systems are on Active Directory (controlled by an older IBM Intel Xeon machine running Windows Server 2008) , a bunch of systems are not on Active directory
There are 3 headless Windows Machines around the offices acting as fileservers, disk management is messy all around, the entire network is flat with no segmentation or separation of any kind, no NVRs, just 2 DVRs
All Windows installs are not genuine/cracked versions (not by massgrave but the sketchy iso you get from shady websites)
The primary database of the Dealership lives on a 1TB SATA HDD on a headless windows PC , which holds data of a tally server , file server and an apache based website that is used for storing purchase information
This disk has NO BACKUPS OR REDUNDANCY! and this disk is accessed constantly everyday for 9hrs
There is a FortiGate 50G Firewall standing between this network and the wide open web
After work hours, they shut down all systems including the servers.
Now, im not an expert, but this felt like it was one disk failure away from complete catastrophe.
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
Now, the total number of clients in the network is about 60 systems, running anywhere between Windows 8.1 to Windows 10 and about 5 printers
Now, a lot of the data was stored on premise, however in 2018, the OEM mandated a lot of the data stored on cloud via their proprietary website, due to which they retired a server, which is sitting in the closet collecting dust.
Now, kindly tell me if what im thinking is stupid, but
I was thinking to recommission it, setup Proxmox to fire up a Windows Server VM to handle AD and migrate the Win Server 2008 to something newer, and a Debian based VM to unify all these scattered fileservers (and hopefully setup something like snapraid+mergerfs so that disk failures=me getting fired)
•
u/VisaExpired 4h ago
I stopped reading after the first few lines, bro run for your life, like how you have toxic girlfriends this is a toxic workplace.
•
u/Responsible-Alps-500 4h ago
I stopped reading after “they turn the servers off every night”…. WTF
•
u/ScriptThat 4h ago
If you know just enough to be dangerous, this sounds like a great idea.
•
u/Flatline1775 3h ago
When I first started at my current job we had a site that went down every night. Turns out they were unplugging all the switches every night to save power. My initial response was pretty much the Frye squinting gif. They don't do that now.
→ More replies (1)•
•
u/_Lucille_ 3h ago
"they air gap the servers every night"
•
u/TheRealLazloFalconi 2h ago
Can't get hacked if your servers are off!
•
•
•
•
u/Significant_Sky1471 4h ago
And I was like: what the hell I’ve just read
It’s totally not okay, all their infrastructure looks like a total mess
→ More replies (1)•
u/w0lrah 3h ago
You know what, I would never suggest that someone do this themselves, but it does have some advantages if you truly don't need those machines after hours.
It obviously means no one can remotely tamper with them after hours, you'll never have a system that's been broken to the point it can't reboot for long enough that all the backups are just as bad. If something breaks boot, you know immediately. You'll never have uptime-related bugs either. If a service memory leaks, you might easily have enough to never notice.
Again I'm not saying I'd ever encourage shutting servers down at night, but I'm not inherently against that part either. The disadvantages are obvious but if you don't care about the disadvantages there are some real advantages to be had.
Now, the rest of this environment does sound absolutely batshit, and I wouldn't want to be responsible for anything there until there were at least backups of all critical systems and a signed contract to replace all computers (including servers) with EoL operating systems.
•
u/confirmedshill123 2h ago
I think it's more your just rolling the dice every morning that those drives spin up again.
•
u/dah_pook 1h ago
This was my thought too, that HDD is gonna quit and probably soon. I'm gonna guess they didn't splurge on a high quality drive.
→ More replies (1)•
•
u/1z1z2x2x3c3c4v4v 2h ago
Turning them off at night is recommended to prevent the hackers from slipping in at night when the dealership is closed... /s
•
•
u/gigaspaz 2h ago
I agree with this reply. I understand getting a new job is tough, but you are going to hate yourself for staying OP.
→ More replies (6)•
•
u/Stonewalled9999 4h ago
hah I knew it was a car dealer when you starting writing about the crap network and old ass stuff they had.
•
u/OhioIT 3h ago
Yes! Why is it always car dealerships that have the oldest and cheapest duct-taped equipment available
•
u/f0gax Jack of All Trades 2h ago
Car dealers, doctor offices, and small/medium law firms. Always running on the ragged edge of IT.
→ More replies (2)•
u/Capable_Event720 3h ago
I've worked in automotive, and...it...was...interesting. Mostly on the manufacturing side.
And I'll never trust a Full Self Driving vehicle.
An automatic scan finds a vulnerability, automatically opens a Jira ticket, I get assigned to fix it...and I get fired because I used the "vulnerability word"?
•
•
u/texas_County850 4h ago
I have seen a lot of crazy IT guys but at least 80% of the time they tried and begged to unfuck it and management is actually the problem.
•
u/two4six0won 2h ago
If all of their windows instances are cracked, yeah, I'm betting it's tight-fisted management.
•
u/rm-rf_regret 4h ago
You need to put into pretty pictures with colors and numbers what it will cost them if they do not listen to you.
Email it so you have it documented. You are sitting on a bomb.
Priority 00000 is get some type of backup solution in place. Anything.
•
u/statikuz start wandows ngrmadly 4h ago
Priority 00000 is get some type of backup solution in place. Anything.
Plot twist: the extra disk activity from the backups running is the straw the breaks the cameldisk's back.
•
•
u/rm-rf_regret 4h ago
He just over there humming along on hopes and dreams. The only thing we gonna be able to help with its thoughts and prayers lol.
•
u/statikuz start wandows ngrmadly 4h ago
What about good vibes? Although in this case you probably want 0 vibes.
•
•
u/GremlinNZ 2h ago
Had that once. Doing full disk imaging on industrial machines during a scheduled maintenance window (installing more machines).
Oh, can't image because it's too corrupted. Uh-oh. Put it back in machine... Won't boot now.
Thankfully the installer (flown into the country for this work) has suitable imgs and between their knowledge of software and mine installing the OS, we get it functional.
If that happened when they weren't around... Doesn't bear thinking about.
•
•
u/joeshmo101 1h ago
I mean honestly I wouldn't be too surprised if their 1TB storage is just a cheapo AliExpress hard drive formatted to read 1TB but truly only 32GB and running any sort of backup that tries to access old data totally collapses the house of cards.
•
u/SirLoremIpsum 2h ago
Email it so you have it documented.
I think you're beyond Cover Ya Ass...
Any business this shit to have this bad an environment is not going to have the big boss go "Oh yeah, that's right IT dude, you did war us. Guess that's on me!"
•
u/soundtom "that looks right… that looks right… oh for fucks sake!" 2h ago
I think at this point, it's just to be able to hand it to a lawyer when the managers decide to sue OP for "destroying their business" after shit hits the fan.
•
u/LankToThePast 2h ago
With people like this, I don't even think documenting it would matter. OP is going to get blamed for anything that goes wrong anyway. I wouldn't even bother doing up a professional presentation, I would just tell them what is failing and what you need to go buy. (They will not spend anything, I know)
•
u/redd1t_is_de4d 4h ago
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
How? Management doesn't seem to care. They won't care about anything you say. Just run, this is no place to work at.
•
u/MetalSufficient9522 4h ago
If there was no IT budget, you should have never taken the job. This is one of the first questions.
You are now the fireman.
•
u/Stonewalled9999 4h ago
I turned down a job once where they were willing to pay the "IT Erector" 180K a year, me $75K a year but refused 3 grand for a firewall instead of a Linksys WRT54G.
•
u/thrownawaymane 35m ago
IT Erector?
"I architect the best systems but for no more than 4 hours at a time"
•
u/MetalSufficient9522 4h ago
People sometimes see value in different things. They may believe (due to something they were told) that firewalls "are a scam".
I mean, honestly- some of them are.
•
u/Middle-Water-7879 4h ago
- Report them for the stolen/cracked software.
start looking for a new job while youre waiting for the investigation to happen about the stolen software.
Collect the 10% bounty for the stolen software report.
•
u/LostRams 4h ago
They actually have a bounty system?
•
u/Middle-Water-7879 3h ago
Yup. Look up Business Software Alliance or Software & Information Industry Association. They both pay out bounties if they collect from the business who is doing the piracy.
•
u/Nu-Hir 3h ago
I reported a company 3 years ago for using 3rd party software in Windows 10 to force it to allow multiple RDP sessions so they wouldn't have to actually buy a Server OS and set up a Remote Desktop Server. Still haven't heard back.
→ More replies (2)•
u/Middle-Water-7879 3h ago
It also helps if you provide evidence of piracy instead of just an accusation
•
u/Drew707 Data | Systems | Processes 3h ago
Yes. The guy that reported my old company told me he was paid out, but it wasn't as much as he thought it was going to be.
→ More replies (1)→ More replies (1)•
u/PotentTurnip Jack of All Trades 4h ago
Microsoft has long stopped caring about pirated Windows.
•
•
u/Drew707 Data | Systems | Processes 3h ago
Eh, I went through an audit right before COVID and we got dinged to the tune of $100k for overextending Office licenses by ~10.
•
u/First-Structure-2407 1h ago
I walked into a company got boned for about £70k for grey copies of NT4 and Office2003
•
u/TacodWheel 4h ago
Worked in a car dealership. Most scummy people I've ever encountered. Good luck with that.
•
u/Significant-Belt8516 4h ago
They need to be concerned with PCI compliance. One breach or suggestion of a breach will cause a world of hurt to come on them. You can't fix companies and this one is working on a broken philosophy. Good luck.
•
u/nico282 3h ago
Lol talking about PCI compliance to a company running cracked Windows 8 ...
•
u/SirLoremIpsum 2h ago
Lol talking about PCI compliance to a company running cracked Windows 8 ...
I know right!
The house is on fire, but if we see electrical wiring out of compliance we'll rain hell!!!
•
u/fizzlefist .docx files in attack position! 1h ago
Feels like you touch one thing the wrong way and the entire system will cascade. GTFO
•
u/Anonymo123 4h ago
IT is seen as an expense until things stop working and people stop being productive, then its an emergency and why this and why that. Design a replacement environment with pricing and the what and whys especially the disaster they will be in once (not when) it goes down. Explain each of the points of failure and give them a price. If your feeling nice see about some used gear, price out opex vs capex if they want to go that route...
If they say no, keep applying duct tape and bailing wire.. start looking, you will be the fall guy.
Been there, done that... left so i wasn't holding the bag when it blew up, and it did a few months later exactly how i said it would.
•
•
u/Plastic_Willow734 Jr. Sysadmin 4h ago
Fuuuuuck no. Unless you're making a very handsome salary I'd start looking at jobs.
•
u/Sportsfun4all 4h ago
In Mom and pop Car dealerships the management is the problem. They see it as a cost center instead of a force multiplier until the network goes down and they can’t make any transactions and then blame it for losing 10x their sales when down but when up they only have 1x of sales. I’d seriously start looking for another job if possible. Really no room for growth
•
u/Thunderb1rd02 4h ago
No budget = no success.
This is an environment that will only change when catastrophe happens. The question is do you want to be there for it?
•
u/dalgeek 4h ago
The primary database of the Dealership lives on a 1TB SATA HDD on a headless windows PC , which holds data of a tally server , file server and an apache based website that is used for storing purchase information
This reminds me of a customer who ran out of space on their Exchange server, so they plugged in an external Buffalo drive then extended a dynamic disk over the external drive. It was basically poor man's RAID0 and they could have lost their entire Exchange database if someone unplugged that drive.
→ More replies (1)
•
u/silentwhim 4h ago
I suspect that the only thing you will get from this is the blame when things understandably break.
•
u/K0T3T5U 3h ago
You need to learn how to talk to executives who run a business based on numbers. Right now they are capturing revenue with what they have, any IT disturbances have likely been minor thus far, so theres zero upside to upgrading things as that cuts into profits.
Instead of saying 'we need to upgrade this server because its out of support and it'll cost XYZ to do so" you need to ask "how long can you run the business if the computers stop functioning?" Most will say they can't suffer any downtime as that will lead to lost revenue. Now you can have a conversation. You get to find a middle ground between costs and doing things correctly, making a plan on what needs immediate fixing vs what can wait. You can also suggest a shutdown test, or unplug the internet or something and see how it goes, and then show the results, and then plan upgrades because youve demonstrated the value of the IT system and keeping it running is paramount to the business.
Its all about framing the conversation into things they will understand, IE operations and business impacts. Ive talked with many CEOs who dont see a lack of IT spend as a problem until its put into a frame of reference they understand. They dont understand tech, so dont talk tech. Im positive some funds can be redirected if the right discussions are had.
•
u/nico282 2h ago
Bold of you that a car dealership owner will "redirect funds to IT" when in 2026 they are running cracked windows 8.
You are 100% right if this was any other sensible business, but I see 0 chance of anything good happening in this specific shitshow.
→ More replies (2)→ More replies (4)•
u/LankToThePast 2h ago
I've had and seen these conversations with people like this. OP: "how long can you run the business if the computers stop functioning?" Car salesman: "That's what we pay for you to prevent, if you can't do that, I'll find someone else who can"
→ More replies (1)
•
u/SeraphicalChaos 3h ago
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
There's the answer to your request. I'm kindly telling you your thinking is stupid. You'll never fix this without buy-in from upper management. If they refuse consider the impact this mountain of liability will have on their business, then they don't deserve a proper IT staff and they absolutely deserve what's coming their way.
I'd personally start looking for a new job because I wouldn't want to be around when the inevitable finally happens...
•
u/wiisucks_91 4h ago
We talking used dealership?
I've seen plenty of dealerships that have a regular enterprise network.
•
•
u/-Satsujinn- 3h ago
I'll echo everyone else. Run for the hills.
This isn't going to get better until it completely fails and when it does, it will be your fault.
This isn't just bad. This isn't even horrendous, toxic, stupendously idiotic, or any other adjective you can throw at it. This is so bad that I honestly question whether or not it's real.
Literally, if I were to write a troll post about the worst possible environment you could walk into, it still wouldn't be this bad.
You aren't going to be the hero that turns this around and brings them into the 21st century. You will only be the scapegoat.
•
u/SirLoremIpsum 4h ago
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
I would blame them a little bit.
The big question - how do you think you can succeed in modernising this when they failed?
Were they bad at their job or was it an impossible task?
This is all salvageable if someone lets you spend money... If they don't then you can watch how this guy hands off to you and you can hand off to the next guy in 6/12/18 months
•
•
•
u/evilkasper IT Manager 4h ago
Come up with a list of issues, solutions, potential liabilities and cost. Explain, factually what happens when specific systems fail. Ask them what they would like to prioritize. This is ultimately their ...network... Horrible as it is.
Start looking for a different job, while you're there. I very much doubt they will allow funds to fix this and trying to without a budget as a one man team will be a bad experience.
•
u/CryptographerLow7987 4h ago
I honestly would not mind taking on such a challenge to bring this network up to enterprise level as long as management is on board with a proper budget. If they just wnat you to maintain this cluster fuck, walk away.
•
•
u/UMustBeNooHere 4h ago
Get the fuck outta there. Now. No budget and how things are run? Nope. That’s a nightmare.
•
u/OverthinkingAnything 4h ago
Wow. That's the worst things of all the various environments I've ever seen all rolled into one....10 years ago.
The technical debt here is astronomical. You have about six weeks as the new guy to have any chance at doing this right. Don't lead with the fallback plan - lead with the right plan. These guys won't upgrade the proxmox server when its time or appreciate the risk involved in any of it.
Hate to say it but this is the kind of client that should just be on O365/SaaS; yes subscriptions suck but it takes the ball out of their court to make shit decisions.
•
u/Balthxzar 4h ago
Hey OP, don't take on any responsibility for any of this. If it goes wrong, and it will, it'll be on your head.
Write up a list of recommendations to management, including estimates of potential losses and downtime if something goes wrong (perhaps with drive failure statistics) and hand it to management.
They won't read it, or make any changes, but when something goes wrong, you can forward them the original email and say "this is in line with the expected issues I brought to your attention" and leave.
•
u/BadSausageFactory beyond help desk 3h ago
once you get it unfucked they're going to get mad at you for trying to make it better because now they can't just turn it off at night so nobody can break in and steal the server
•
•
u/Every-Ad-5267 3h ago
You are getting on a ship with gigantic holes in it.
Don't be surprised when it sinks with you in it.
•
u/CharcoalGreyWolf Sr. Network Engineer 3h ago
If you’re a glutton for punishment and decide to stay, your priorities are wrong.
Backup is your first priority. There are no other priorities until you have known good backups.
•
u/theITguy 2h ago
Getting your first IT job can feel impossible and the market is shit right now, so.......
Do what you can with what you have to get some experience. Also, immediately start looking for your next job. This job will be great for story telling in future interviews, as long as the stress doesn't kill you first.
I've lead dumpster fire IT before and it's painful, but you do learn quickly. Assuming you're in an "At Will" state, you can walk away from this job anytime you want. So, take some comfort in that, earn a few paychecks and for shit's sake, look for another job.
As far as the infrastructure, I would start with backups. JUST BACK IT UP HOWEVER YOU CAN. From there, start fixing and building.
→ More replies (1)•
•
u/Beautiful_Ad_4813 eh, I just love what I do. 4h ago
I have so many questions.
The big one is “what the fuck”
•
•
u/Jeff-J777 4h ago
I would run. Don't be the fall guy, start looking for a new job. I had a job with a movie theater company and their whole network was in disarray. The had 7 locations across mutiple states. Some were on one DC with one domain. Some were on another DC with another domain. Some had no DC. Some has VPNs between the sites some did not.
It was crazy. A lot of their network gear was Ubiquiti. I don't have an issue with Ubiquiti but you can't feed one theater with just one AP. Some network gear was consumer grade crap. No backups for any of their servers. Then come to find out a few months before I started they got hit with Ransomware and lost everything.
I stayed there for 2 months then I booked it.
RUN RUN RUN RUN!!!!!!!!!!!!!!!!!!!!!!!!!!
•
u/deefunkt01 4h ago
Yeah, get out as fast as you can. This is a nightmare scenario and they will *never* spend more to fix it, only yell at/blame you when it inevitably fails.
•
u/GradeAccomplished322 3h ago edited 3h ago
I read "dealership" already got me nervous, those places only come in amazing and dreadful places to work, there is no middle ground
The dealer brands suuuuck to work for if you aren't raking in the money, they have no time or patience for small timers
Your idea isnt bad but that many cracked windows systems who knows what devils are in the woodwork
I would be getting everything onto modern operating systems and as much of it off local assets as possible before something happens
Windows 2008r2 and Windows 8.1 dont even support some modern protocols, just using them for anything is gonna suck
•
u/jason_abacabb 3h ago
Get a quote to do data recovery so they know what it will cost when the hardware fails.
•
u/defaults-suck 3h ago
These folks don't need AD, Servers or even Windows. They need Chromebooks with Google Workspace and Wireless Printers.
•
u/_nethack 2h ago
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
This is the part where you run
•
•
u/TheLionYeti 4h ago
Tell your bosses exactly how much its going to cost when this stuff breaks. Exactly how much blame the hell out of the old guy and get the budget to fix it.
•
•
u/Disorderly_Chaos Jack of All Trades 4h ago
Should we be worried about whatever this company does? Are they protecting us from disaster or something?
•
•
•
u/defiantleek 4h ago
You need to get out of this job as soon as possible it is going to destroy your life.
•
•
u/The82Ghost DevOps 3h ago
Run! Just run! This WILL blow up in your face! Don't waste any time trying to fix this mess, just RUN!
•
u/ChampOfTheUniverse 2h ago
Does "management" (outside of IT) know about all of this? Like I'd start documenting the FUHUCK outta all of this and send this up the chain as soon as you can just to CYA. For all you know, the departing admin could be good at lying to stakeholders and you do not want to be the scapegoat when things fail because you are definitely set up to do so.
•
u/kristoferen 2h ago
If they had a budget this would be a fun project to fix. Without a budget, you can't un-clusterfuck this mess... run.
•
•
u/LowIndividual6625 2h ago
I sort of love walking into environments like this and have a relevant story….
In 2018 I took a solo system admin job for a midsize business, and I had three days with my predecessor before she left, she had only been there for six weeks and was under skilled so those three days didn’t help much.
At the end of my first Friday they gave me my building key and alarm code….. about 10pm that night the Active directory went down and there were no remote access tools at that point so I drove into the office.
I discovered the AD migration they said was complete was never actually finished and the sole copy of AD was running on a beige-box gateway server (moo) running windows server 2003 that had crashed.
I discovered the Carbonite (lol) backups were not working and it was a miracle that the server rebooted cleanly and came back online.
The first thing I did was an image backup of the entire server. The second thing I did was compose an email to my boss (who was C level) explaining how close the company was to a catastrophe and that we needed to meet first thing on Monday to talk about this.
Fortunately, they took this seriously all C level management, including the owners were present in the meeting.
I was very honest and direct because with less than a week on the job, I had nothing to lose.
This could’ve gone one of two ways - but I’m fortunate enough to say that they took this seriously and essentially wrote me a blank check for the equipment and services I needed to get things stable and secure.
TL;DR - have a frank and honest conversation with your supervisors about how serious this neglect is and the potential loss of business of what comes down to multiple points of failure. If they listen to you, then it will be a great opportunity for you, but if they are cheap or unwilling to address the issues, you should leave right away..
•
u/johnno88888 2h ago
Jeez it sounds like where I used to work 16-20 years ago.
No budget, everything had to be a bodge. They bought a Nissan dealership from a well known company who was selling up to focus on water cress! They needed to export all their customer data from one DMS server that was hosted in their head office to a new shiny dms hosted in house. The cost of the server, software, support AND export and import of the database was £31k. Considering pinnacle dms was £100000 a year for the rest of the sites.
I got a telling off from the CEO and MD that it was far too expensive. Considering they’ve just spent £00000s on the new dealership.
We had a dodgy copy of mdaemon. Only got support for it when I “accidentally” uninstalled it.
In the other head office (we had two for two different franchises), they had a windows server 2000 sat there doing just dhcp. Asked if I could buy a new server as the dreytek vigor 2600+ was getting ddos’d with the amount of dns requests every lunch time. “No”
And don’t get me started on the Linux file server on a HP ML120. That was cutting edge tech. Two disks out of unmonitored r5 failed. As you could imagine… lucky I had a rsync job set up for offsite backup.
After 3.5 years of fighting for more, gave up, moved to London for big hedge funds IT, got shouted at, drunk beer with the people that shouted on Fridays. and now working in proper IT elsewhere. Happy days
•
u/radicldreamer Sr. Sysadmin 2h ago
Ive worked this exact setup.
You will be expected to fix this, you will not be given the budget to do so.
Once you have a good evaluation In place, get ballpark quotes to remediate and make sure it's in writing. That way WHEN, not IF it fails you can fall back on the fact that you did your due diligence and laid out the risks and what needed to happen and were denied.
Hospitality puts on a good facade, but it's mostly put together with duct tape and bailing twine. It can be a fun job, but they will throw you under the bus when their lack of investment in IT fails them. Also remind them their server OS is almost old enough to drink beer, assuming you are American of course :)
•
u/namocaw 4h ago
Step 1: quickly document everyrhing as you found it.
Step 2: get a backup in place asap.
Step 3: patch firmware on firewall
Do these now. Do not ask permission, ask for forgiveness later.
Step 4: plan for improvement immediately.
If your plan falls upon deaf ears or budget is declined or timeline is pushed out and not begun immediately. Then you quit. Simple as that. This environment is a s*** show. That is ready to collapse.
•
u/Stosstrupphase 4h ago
Noting the bosses of all the issues in writing. The Proxmox plan is kinda decent, but you’ll probably need to throw out most of the hardware, unless you turn it into thin clients.
•
u/BalderVerdandi 4h ago
On average, the cost of a cyber security incident was between 4.9 million USD (global average) to as high as 10.4 million USD (US based average).
Ask them if they have $5 million sitting around to fix the incident, and then how much it would cost to get legal and proper equipment since you're operating on licenses that aren't yours.
Proxmox to create a new AD environment on a Windows VM using the Win2008 server.
Import anything needed from the 2008 box running AD.
Decommision the 2008 box.
Use one of the Proxmox Windows VM's for a local fileshare, DHCP, etc., then push everything else that needs to be backed up into the cloud.
•
u/Happy_Kale888 Sysadmin 4h ago
so you knew all this before you signed on? at least the bar is low...
•
•
u/Repulsive_Initial308 4h ago
Make sure that Fortigate has got a local-in policy before you do anything else :D
•
u/BloodFeastMan 4h ago
If management doesn't want to spend money, there's still ways you can prevent disaster using a few old desktop computers, copies of Linux, and rsync.
•
u/Affectionate-Cat-975 3h ago
Get their rejections in writing and explain that when these risks come calling, you are NOT accountable for the failure and inability to recover the systems. That will either put their necks on the cutting block or scare them to supporting you. Either way, bide your time and then run
•
•
•
u/cyr0nk0r 3h ago
just get a private cloud. then tell the business they never have to worry about server hardware ever again.
If you find the right provider, it will be much less expensive than trying to convince them to upgrade anything. Just shift it into an OPEX cost and you maintain control of the day to day mangement.
disclosure: I own a private cloud business that helps companies like that every day do exactly what you're trying to do. Upgrade to modern platforms without the fight of spending tons of money.
•
u/awetsasquatch Cyber Investigations 3h ago
I'd go back to hospitality until you end up in a better situation. If this is your first IT job, I say this with love, you're in WAY over your head. That's a catastrophe waiting to happen and it'll fall on your head.
•
•
u/firestorm201 3h ago
Outline the costs they’ll pay if they continue to neglect the IT environment. For the data side, get a quote for data recovery. Share what the expected rate of success for said recovery is. Give them a timeline of how long that’ll take. For the licensing side, explain the risks of running shady versions of software (ie backdoors, risks to their customers, potential lawsuits, not to mention a dreaded MS audit where they bend you over both directions.)
And if that still makes no difference on their budgetary stance, you should run for higher ground.
•
•
u/Sweaty_Weight_2486 3h ago
still running windows server 2008 and windows 8.1, fukkin run, they arent going to invest in your schemes to modernize otherwise your predecessor would have done so.
•
u/_Lucille_ 3h ago
When it comes to not having a budget, I would go and explain how the lack of backups and redundancy is a business risk. What will the company do if the system fails at any point? They will have no business and take a terrible reputation loss.
A hotel may skimp on cleaning by offering it every other day, but you still need working vacuum cleaners and detergents to keep the place clean.
•
u/Capable_Event720 3h ago
Proxmox and RAIDZ3 (ZFS, 8 disks, 3 can fail without data loss) is my poison.
The old trash (two former "servers") are backup nodes, synced every hour for the effing mission critical Windows Terminal Server and every minute for the mail server (yep, Debian, as well as the other LXCs).
Main server is...pretty old, dual Xeon server class hardware. Eighth Enterprise Data Whorehouse Edition SSDs, which made up more than 75% of the system's price (the main box, without drives, was just a bit above 1000€).
The "old trash" unit with a consumer spinning disk outperforms the "old trash" unit with a consumer SSD. Don't use consumer SSDs for ZFS in systems with lots of writes!
Firewall is an HA dual pfSense system. Primary connection is fiber, backup is LTE (and cheap Netgear switches as WAN switches). Yep, if the primary connection fails, the whole datacenter runs on Aldi Talk LTE; cheap at 15€/month for 50GB, and if it runs out of traffic, I can book additional traffic via my phone. One additional GB is free. Repeatedly! Basically unlimited traffic, but if a failover occurs, I prefer to book additional traffic in advance. Last week was Gamescon in Cologne, so they handled a lot of live streams.
Cheap HP 1820 LAN switches. How many VLANs do you want?
Yes.
😂
→ More replies (1)
•
u/bschmidt25 IT Manager 3h ago
Woo boy... I worked as a SysAdmin in hospitality in the mid-2000s so I'm familiar with the mindset, although my company was much larger than this. They don't really want to spend much money on the back of the house, only stuff the guests see. But we always put the ball in management's court on decisions. It only took a few extended outages of their property management systems (a simple flat file system back then) to crack open the wallets and modernize everything. First thing I would concentrate on is backups, although I'd probably stay away from Linux. It's already going to be hard enough to find people willing to walk into this should you leave or if they want to add another guy, and paying extra for Linux expertise isn't something they're going to want to do. Not a good time to be buying hardwar, but they're going to need to cough up some dough on that too.
•
u/uuuuuh 3h ago
I thought that said “hospital” at first and I was getting ready to report it.
Getting out of there is a good idea but if you want to give it a shot, make a report for them comparing the cost of a proper IT setup with the cost of rebuilding everything they will lose when that one single hard drive fails.
If you really want to put some oomph behind it, talk to people and get a good idea exactly how bad it would be when that drive fails. Have solid numbers to compare (100 man hours to rebuild vs 2k for a NAS, etc). If they can’t get their shit together when it’s put that plainly, run.
•
u/dreniarb 3h ago
unplug that server from the network at the switch - see how they feel about costs then.
though probably once you "save the day" they'll go back to not thinking it's worth the cost again.
•
u/Academic-Proof3700 3h ago edited 3h ago
- Don't touch old server, if it shits itself, you are the one to be on a pyre
- move them to some cheapass cloud with dedicated server, beef it up, move data on separate raid drives.
- Motivate it that it will be cheaper in the long run cause (pick one): some regulations will require you to upgrade, lack of support or such. That ws2008 is a walking security gaping hole, because no amount of fortinets (that are recently getting bad press iykyk) will protect you from some dude opening malware mail.
Its one of the examples, where - if not for the lost data - it would be cheaper to just "commit arson" and watch it all burn.
OH! And for any denial, have it on paper (not email, if you are hosting in in there :D), cause when it ultimately goes tits up, you want to have it all nicely printed.
•
•
u/RadiantWhole2119 3h ago
If leadership has no intention or desire to improve anything they’ll see any changes made as a problem. Even the slight disruption will bring backlash to you and I wouldn’t be surprised if you get let go because of it. You’re not fighting tight budgets. You’re fighting people who do not value technology or understand the importance of simple things like redundancy.
•
u/LankToThePast 3h ago
This looks like an awful place to start working if you’d like to work in the IT field. Someone without any experience in IT shouldn’t be deciding what technology to roll out. You need to get something on paper that says you’ve studied IT in some capacity.
A simple analogy is, I’ve driven a car for 20 years, but the person who just finished mechanic school is more qualified on paper to fix a car. You need the education paperwork for someone to hire you because they need to have shown due diligence in picking a candidate.
On paper, you are not qualified to take on this responsibility, and anyone hiring you to do this level of task is setting you up for failure. I was offered a similar position shortly out of school, and I had a year experience under my belt. I luckily took something else and only realized years later how unprepared I would’ve been for that role.
Years in the field would help you look at the bigger picture, how the fuck does an environment end up like this? Who set up this insanity? It comes from being told “no money”, “figure it out”, and “what are we paying you for?”. That’s why the previous person left, and why if someone showed me that environment during an interview, I’d say “thank you for your time”. Then I’d get in my car and never consider working there.
Please only stay there long enough to get your education under you, then find a better place to work.
•
u/EffectiveEquivalent 3h ago
Shit this is one of the worst situations I’ve seen on here.
No budget? I got nothing.
•
u/Nova_Nightmare Jack of All Trades 3h ago
If you do run from this place, you might make a bit of cash by reporting the pirated software. If you actually tried to stick it out and fix the mess, you will likely be resisted at every point and blamed for anything that doesn't work, especially if they get caught later on.
•
u/Away_Chair1588 3h ago
You're holding a bomb with very little fuse left.
You're never going to be able to implement any upgrades. You will be perpetually putting out fires until you quit.
•
u/bseaman77750 3h ago
Never work at a site with unlicensed windows machines - biggest red flag of all.
•
u/lontrachen 3h ago
Reading “no backups or redundancy” after you wrote that the windows machines have cracked versions of the OS does not even sound real. Who do you work for? Your uncle? 😂😭
•
u/d00ber Sr Systems Engineer 2h ago
What's your budget? I worked in a similar US healthcare situation in my younger years and budget was always a nightmare. You're going to need to look at ebay for servers. You'll never be able to buy new. You just need many redundant servers. Nothing is ever going to have warranties, so spend capital when you can on replacement parts/servers/spares.
•
•
u/Mental_Beginning_698 2h ago
Wow. Any SBS servers? I bet they are with that level of cheapness and likely unwilling to upgrade out of their "package deal"
•
u/Electronic-Aide5833 2h ago
Se não fosse pelo Windows pirata eu tentaria uma conversa a alta gestão para trazer conformidade. Mas todo mundo sabe que pirataria é errado e mesmo assim continuam quer dizer mau-caratismo está instaurado e não vale a pena perder tempo com mau caráter.
•
•
u/SeaworthinessHead613 2h ago
It sounds a nightmare, but schedule each bit, concentrate on the important bits, ie backups first server, firewalls, av scans then ws builds, & then the rest.
•
u/ThePathOfKami 2h ago
wouldnt everything be easier if you fully convert to cloud in this situation ?
Like from every standpoint, security, control ,etc etc.
Apart from the licences all the problems would be solved xD
WTH env is this even
•
u/undefined261 2h ago
Top initial priority is documentation of everything, and CYA. Write down on all the risks of failures and exploitation. After that, make sure to have known good backups so the business doesn't go down the tube.
Purchase information? Are they accepting payment card data as a form of a payment? The PCI DSS standard is there for a reason, if they're collecting and storing payment card info on prem they have to put safeguards in place, or the business could have their ability to use it as a payment method revoked. Not to mention heavy legal liability. Look up the PCI DSS as the bare minimum for standards and controls. I have no idea, but I assume a dealership/franchisee situation likely has company policies that should be compliant with as well.
Look up the read/writes on the SATA data for that hard drive using a utility and look up what the manufacturer says are it's max lifespan to get it in their heads. Look up what happens to a company who doesn't implement PCI DSS standards, and throw in a little FTC Act 15 lawsuits for those businesses who are negligent with data. There's real risk there they can be liable if a breach happens.
•
u/kaiser_detroit 2h ago
Having worked in an automotive marketing environment for 10 years and having interactions at the dealer level (and even at the OEM level) it's amazing any of those companies function. Having to maintain Internet Explorer compatibility on websites into 2020 because they were still running Win XP or ::shudder:: earlier in their dealerships was just absolutely mind boggling. Combine that with the shit fest of data feeds they'd send us and complete lack of understanding on how DNS works.... I'd never work in that space. The blood pressure spike alone makes it a hard no. And if you're trying to hack together a solution with old parts, you're just going to make it worse in the long-term. You're quite literally building this: https://xkcd.com/2347/
tldr; run, do not walk, to the nearest exit.
•
u/Redacted_Reason 2h ago
I would only take that job if it was to do a fresh start, complete overhaul. Oh you want me to maintain? Yeah, nope.
•
u/Britzer 2h ago
You can do a lot of stuff with second hand hardware and free software. The 60 clients need to be replaced. Why not go to a large refurbisher and ask if they have refurbished notebooks that have licenses for Windows 11 Pro in the UEFI? Those might run only 400 bucks a piece. User will love you for getting new machines.
Get used servers and do everything open source? Proxmox is the right idea. You can also use Univention UCS based on Debian to replace Windows DCs.
https://en.wikipedia.org/wiki/Univention_Corporate_Server
Or go full RHEL (Rocky Linux 10) for DCs and file servers. With AI support, you can a lot more these days.
It is important to get management backing. Explain to them that they could/should spend a lot of money. But if you save them a lot of money for being smart they need to compensate you and trust you. If you spend a lot of money you are usually worth more and earn more. Open Source tools allow us IT folks to save a lot of money, but that doesn't help us, because when you don't spend the big bucks, you aren't worth much. But with tools like Proxmox, Rocky Linux, ZFS, ... you can create systems that would cost millions of dollars if you bought them.
•
u/1z1z2x2x3c3c4v4v 2h ago
Since you are new, I will give you some sage advice...
You only work to get skills. Once you get enough new and in demand skills, you move up or out. And you keep doing this for as long as you can, until life or family obligations prevent you from changing companies.
That said, you are in a real shit show. The fact you are using illegal copies of Windows, old copies, pretty much assures that some of those machines are compromised.
So learn as much as you can, as fast as you can, and continually be looking for bigger and better companies to jump to.
This will be your "baptism by fire". It will be the best of times and the worst of times. Don't take any of this shit show personally. You didn't create it; you were only hired to support it and try to keep it running.
Good luck. Have fun. Learn as much as you can. And thank them for the opportunity when you leave.
•
u/Key_Zombie7967 2h ago
Prepare for a resume generating event in the near future. Get out while you can still walk. These fuckers are going to get what they asked for...
•
u/Candid_Ad5642 2h ago
If management have understood the danger, and are willing to spend the required money to fix this, yeah, it sounds like it could be a really cool case to fix
But it sounds like management do not understand the issues, so I expect the budget will be shoestrings, bailing wire and duck tape, and that when this eventually fails, you'll get blamed
In other words, just run
•
u/Bratwurst1981 1h ago
They spent their whole budget on the firewall…. A horrible time to be refreshing hardware too. First, review and find out what isn’t being backed up. Fix that.
•
u/ITMORON IT Manager 1h ago
Been there done that, have the mental scars from lost sleep and stress. I would suggest telling them to get the budget or get fuct. Its fun fixing things, properly. Ive done my share of cloud lifts, from file servers to entire enterprises and migrations from coppere to VoIP. When you get a budget it is amazing, when you are sheostringed and treated like a ncessity, theres no point. They will nto change, not even when shit breaks, theyll just want another patch, shitty hacked license, stolen gear whatever. Te unlicensed Windows products are highly suspect.
•
u/GoofyGoffer 1h ago
Hospitality is the worst for this type of thing. I used to work for a bunch of hotels and fought for every line item. Then they would ask if I could do it for cheaper. They see IT as a necessary evil, not something to invest in. If you stick with it try to pitch it in a way that will improve the business.
Lol misread the title as the job type, same applies though.
•
u/WideAreaNetworker 1h ago
Normally I would say document every recommended upgrade/change and get the rejection signed. But, if anything goes sideways they’re just going to get you to fix it and likely fire you for breaking it in the first place. Run!
•
•
u/Wooden_Alarm9529 Sysadmin 1h ago
Run. You are being set up to fail. If owners and managers don't want to fund the IT, they never will.
•
u/Randalldeflagg 1h ago
I worked for a MSP and one of the clients was a car dealership. sketchy as hell hardware all around. Office 97 that everyone shared the same product key. everything was in a pst with the option to leave in the cloud. no idea what they did once MS started requiring tighter security around accessing emails. Everyony ran Windows 7. The hand scanners for the vehicles ran CE and were unstable and flaky. refused to invest in new ones. Or even the latest vehicle software from the manufactures.
No idea how these places stay working at all.
•
•
•
u/Lordcorvin1 1h ago
Hospitality? Ask about what Insurance they have.
That shit will cause their insurer to withdraw. Make a probable Ransomware higher cost to the company then just upgrading.
They only understand numbers.
Look into PCI-DSS, if it applies to them.
•
u/enjaydee 1h ago
I was in a similar situation when I was getting started in IT. Smaller set up than what's described here, but pretty much the same attitude. They were one crash away from losing all their finance and client records.
I tried to get funding to set up proper backups and redundancy. They said no, so I left.
I get OP is young and trying to get their foot in the door, but surely there's better options to get experience than this clusterfuck.
•
u/Broke_Dick_Honda 1h ago
Leave and report them to microsoft auditing anonymous give it a few months. That's they only way these people will get it that you need to do things right. If there business shuts down because of it. O well they are exposing people information to cracked software and prolly 100 different back actors.
•
u/_stinkys 1h ago
Licensing is going to be your most difficult issue. You need to become compliant but the business will take time to adjust to the cost. Make sure you don’t leave yourself open on that one.
•
u/Joy2b 1h ago
Start with demanding the very reasonable funds to buy a couple of backup hard drives for essential files. If they can’t afford that, they can’t afford you.
Next, start building relationships and collecting complaints. What you are listening for is “I lost a sale or two because my computer froze”. P
Then start building a bracket of what you want to upgrade over the next 18 months, like server/workstation/server/workstation, and start getting VIPs to do nudging for better spots on the bracket and faster progression.
Talk to finance about depreciation.
To sell network upgrades, bundle with wireless.
Always have a clearly marked section on your projects for nice to have, so if the boss wants to haggle, they’re taking off extended warranties, or an extra wireless access point for the shop.
•
u/_Do_The_Needful_ 4h ago
Man, seeing the environment, knowing the current guy is leaving (for a good reason), and that they don't want to spend any money on IT.... things will not get better. Ask him if his new employer is hiring.
I have been there before. Shoestring budget. IT is just a cost center that doesn't help the business mentality. Consumer-grade hardware. Everything in disarray. You might spend months un-fucking this environment and doing them a huge favor, but considering they have no budget for equipment, I can't see it buying you a raise of any kind.