Inspired by the very similar thread about school incidents.
My company called all lab staff “pandemic heroes” for coming in every day during the pandemic and taking on extra work to compensate for management and office staff who stayed home for years.
Then shortly after return to office, they closed the lab and laid off all lab staff.
Sounds like your company took the Veterans Affairs approach to “hero response”.
Worst part is that they did it mostly to boost the IPI right before we went public by driving down operating costs.
We weren’t even able to buy in u til 6 months after going public and the price leveled off at 6 months
An IT contractor at my government job was one of the people that tried to kidnap Michigan governor Gretchen Whitmer.
INC-224, never forget.
I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.
15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.
15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.
I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.
We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.
We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D
I love it.
All the other comments had guns sex and drugs.
Your story had mySQL.
Mad lad.
I understood almost none of that.
Database (thing that holds and retrieves bunch of data) broke when it reached a size of 16 Terabytes because the underlying filesystem (Thing that lets you store data on a physical disk like a hard drive or SSD) has a maximum possible size of 16 Terabytes by default (ext4)
16 TiB is roughly 16,000 Gigabytes which is roughly 16,000,000 Megabytes
Ty. I understood the tb but I didn’t know what a lot of the other abbreviations meant.
Working at McDonald’s at the time. The HR manager went on bereavement leave and a replacement was brought in. The day the HR manager came back she was told she was demoted and was put as the DriveThru order taker for a couple months before finally being fired and given severance.
A month or 2 later the old restaurant manager who was now the “Systems Manager” and in charge of all the admin tasks stopped doing unpaid overtime, so all of his duties were taken away and he was put as DriveThru order taker.
For 3 months he came in for exactly 8 hours every day, only did order taking in DT, and left. He was still being paid his restaurant manager’s salary during this time, the new restaurant manager was in over his head and would not ask the old restaurant manager for help. Eventually the old RM left to work for a competitor working with the old HR manager.
Apparently the owner called the competitor to scream at them for stealing his staffI hurt my back working today. That was an incident.
The overnight IT guy was caught watching porn while working (this was over a decade ago, he was in the office every night and not a remote worker). How was he caught? He was saving the pornographic photos on a shared network drive…
When confronted, he didn’t try to deny anything, his explanation was simply, “That’s just my thing.”
Boss: “Were you looking at porn in the office?”
IT guy:
One of my very first tech jobs, there was a guy who watched porn in his office, as far as I could tell just continuously. I saw the DNS logs so I was aware. I didn’t care as long as it wasn’t interfering with my ability to get things done.
One day I had to go in his office and talk to him about some of his code that was breaking, and he went OH and tried to hide all the blatant porn on his screen. Like dude I know. I don’t care. I am here to talk with you about your shitty code not your personal failings and issues; those are purely your own problem IMO.
The ceo of a client viewed porn pretty much all day every day during the pan when everyone was wfh. Our screen sharing tool showed a stamp-sized preview of the client device before connecting. One of the interns said after over a week of trying to engage him with the chat feature and closing the connection due to porn before sending a chat, he gave up and accepted that there was no time when porn wasn’t on-screen.
I saw this so often when I was client facing. CEOs, doctors, and sales people were the biggest offenders.
We had a gyno who had a huge pile of porn on his file server. It was all from the waist up. Seriously, he had half a terabyte of titty pics.
Separately, there was a sales guy who was juggling like 5 women (poorly) at any given time. He was fucking gross and would try to show them off to anyone who came to work on his continuous computer problems that were all caused by him.
Separately from that, we had a “troubled boys ranch” as one of our clients. One of the C Suite was caught with porn and we had to go over it with a fine toothed comb to make sure none of it was of any of the kids. There wasn’t (thankfully) but there was a whole lawsuit about it and he was charged with showing it to some of the kids.
A gyno with just tit pics either feels like he went 50/50 in career path and made a bad choice, or has become so desensitised (or put off) by his work that he just cant bear to look below the belt
When your hobby becomes your job!
We should be able to look at a liiittle porn at work.
You’re a rock star!
As a treat.
Aside from the obvious don’t watch prom at work, I don’t see how anyone does it with company property. Doesn’t pretty much everyone have a phone or computer these days? Shouldn’t it be common sense to use your phone rather than the work computers need by your employer?
I would say no. But there is a part of me that sees he’s an overnight IT worker, and then I’m like “what the hell else would he be doing?”
WoW
EvE Online until he perfecta his spreadsheets
As a former overnight IT worker, I always just assumed my browsing history would be reviewed by other IT people. Maybe they wondered why I spent so much time on horror game forums, maybe they already knew I was a disturbed mind.
That time the food cart pod exploded:
https://www.oregonlive.com/portland/2017/10/food_cart_pod_on_fire_in_downt.html
Them including that pic of Audrey Donovan is hilarious
Depends. Had a client pull a knife on me once, and another dragged me around the facility for an hour while he tried to break down a door to “kill” another client because he had stolen the change from a $5 Taco Bell gift card.
The other incident being was a coworker harboring one of the fugitive kids at her house with her like…6 children while her husband was away in Nebraska for work. Randomly saw her in family court a year later while I was working another job, hopefully while her husband fights her for custody of the kids…
Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new “current” version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.
More impressive than the fact that you saved a repo once is that the same repo still exists today with the complete git history. At the rate companies abandon products for new ones, old repos are rare.
Our repo is old as time. Carried through from SourceSafe to TFS to Git
I used to work at an accounting/consulting firm who were dead set on writing business applications in VBA within Excel. The code was embedded in the notebook, and to distribute the software was sending the latest version of the Excel file. This made version control virtually impossible, and we would instead combine our work manually.
I cannot recommend having tech-illiterate people lead software projects.
The amount of times I hear people telling me that “I should just do it in Excel”. Excel. Is not. A database.
Excel is a single-assignment dynamically-typed functional programming language with a really obtuse editor.
Stop… Stop… I’m already dead
Excel is a whole OS unto itself. Like Emacs except you can get out of it.
Close enough when your actual database system is written in fucking COBOL.
Subversion has existed probably for longer than your company, the fucking managers couldn’t be arsed to read a damn book?
They were using SourceSafe back then. But any source control that isnt decentralised has the same problem. If the central server gets deleted so does all history
I had a worse experience. My first internship was doing web development in ColdFusion. Why that language? Because when the company was first starting, none of the funders wanted to learn Linux/Apache administration and CF ran on Windows.
Also, the front end development team did not have version control but shared code via a file server.
Gotta respect that save. Reminds me of the Toy Story 2 assets being lost from a server failure and they were saved by one employee having a copy on their personal computer at home.
It wasn’t a server failure. Someone rm -rf on the root of the server. The server did what it was told.
That’s complicated to answer in my case, as nobody gets along (I’m one of the few people with a relatively stable work relation), so there’s an incident everyday, though there are also occasional ones that stand out a lot. I for some reason have a lot of bad rep without any actual cause for it and remember people storming into our operations more than once and demanding I be exiled from the place. There are two types of people in this situation whenever it has happened: those who are almost about to oblige and fulfill their wish, and me who calls authorities and ends up dealing with the situation before they can do so before everyone just forgets all that happened.
I bet you have done good stories about that
It depends on what you mean by good, as they aren’t things I like to think happen to me often, though one consistent theme reveals itself out of it, even when merely talking about it.
I was supervising filling in a pit we had dug on the edge of a forest. We had dump trucks coming in dumping gravel. One particular driver wasn’t great at his job and there had been issues with him in the past.
That driver came in and dumped his gravel, but then he drove off with his bed still raised and almost immediately smashed into electric lines that ran off into the forest. One telephone pole even snapped at the base and fell over.
Within 30 seconds multiple cops came speeding onto the job site. It turns out those electric lines ran to a radio tower in the woods that ran the police radio. The idiot in the dump truck had taken out the police comms for the whole town.
Note: if you’re planning a crime in that town, you only have to cut one wire to disable all police communication.
That’s some lacking infrastructure
You’d be surprised, how fragile critical infrastructure often is. There was an incident in Europe a few years ago, where a single miscalculation in a planned power line shutdown almost caused the entire European grid to split.
It slowed down a bit, and then we quickly learned that maintaining the perfect 50hz wasn’t actually necessary anymore. Few people still have clocks that depend on it
Clocks, true.
Computer systems in general, however, will start acting very squirrelly outside of an approved MHz range. Wall warts and power supplies can handle only so much deviation from the norm. It’s why high-end UPS systems do power conditioning to provide a pure sine wave.
No worries we’re talking like 49.9995 hz
I’m not talking about the incident in Romania, but in Germany.
A shipyard needed some wires over a river deactivated and that caused an overload cascade, because the river was the border between two providers who had different assumptions about the capacity of the power lines connecting them.
oh damn, ain’t something. I will be looking into that, thank you!
That’s some lacking infrastructure
They probably had plenty of infrastructure for normal operations.
What they were lacking was a BCDR plan.
…which includes having backup lines or a more robust installation. Police officers aren’t engineers or system administrators for public infrastructure.
You’re right tho, a backup alone would not be sufficient
This is why we shld bury our lines, much more effort to dig down six feet than get a ladder and snip
Buried lines of all kinds are frequently severed by excavators because their position isn’t properly or fully documented.
The best set up I ever saw was a sewer tunnel, almost 12 feet tall, that handled all the services. From sewage to water to electricity to data; it held everything and was trivial to maintain and run new lines in.
line sounds like a really interesting idea, although I feel like documenting where you put things should be a basic task. Probably why it’s not done properly
And this is how a micro quake severed our T1 line from LA to Phoenix and shut the network down in our office for a week.
Honestly never thought of that, sounds like there would need to be some sort of protective channeling, with space to allow some shifting
A wild backhoe appears!
It used Dig! It’s Super Effective!
What does a network engineer bring on a hiking trip in the woods? Water, snacks, extra sunscreen, a first aid kit, bug repellent, bear spray … and a folding shovel and a piece of fiber-optic cable.
(What’s the fiber for?)
Well, if you get lost in the woods or need to be rescued, you take the shovel, dig a trench, put the fiber in it, bury it … and within an hour, someone with a backhoe will show up to tear it up. Then you can just follow the backhoe tracks back to civilization.
Couldn’t they sell a few of their spare MRAPs to buy a backup generator and a redundant microwave link? Sheesh.
One day a coworker of mine was walking into our huge office building and thought he saw a mitten on the ground of the lobby. When he picked it up it was actually a pair of lacy women’s underwear. Ostensibly it fell out of someone’s gym bag or got caught in their pant leg in the laundry and dislodged there. He drops it immediately and comes into the office. He doesn’t mention this to anyone.
Two hours later the main receptionist comes in with the underwear in front of our whole group and says she saw him drop these this morning and she wants to return them. He’s denying the whole thing and at this point none of us have the previous context and all locked in to the conversation and silent laughing. She says, “We just want to give these back in case they have sentimental value!” and the the whole group is dying laughing now. He eventually convinces her he isn’t interested in a stranger’s underwear (which she bare handing) to which she says she’ll keep them in case he changes his mind (???).
It’s been 5 years and it gets brought up nearly daily
Does he still work there and does he laugh with you? Otherwise this sounds like bullying
Sounds like the receptionist did this on purpose lol
Funny if they were her panties all along. Turned the embarrassment from “Guess who dropped her panties in the lobby” to " Guess who was playing with panties in the lobby."
This adds a whole new dimension to the lore.
Someone must’ve summoned Shenron the day before he found them and got underwhelmed by the wish fulfillment.
Oolong back on his shit
Used to work at a local resort. One time, Kid Rock rented out the whole top floor of the hotel, and requested no staff go up there during his stay. Of course, it’s a hotel full of minimum wage teenagers, so intra-staff communication is abysmal. A maid ended up running into a naked Kid Rock holding a bag of cocaine lol
Jason spilled a barrel a barrel of ink and fell in it. 🤣
Guy found a gun in the customer’s stuff
Guy starting waving it around and playing with it, pulled the fuckin trigger, almost shot one of his coworkers
Cops came, guy said he was moving a cabinet and it went off which obviously no one believed, somehow he wasn’t arrested, idk
Guy was fired over the phone before he left the customer’s house
Was this one on the news? This is very very familiar
Another:
Big awful dude starts working, among other issues he was SUPER upset that the girls at the gym are allowed to have their own separate area to work out where he can’t ogle them, he felt this was grossly unfair and was angry about it
So anyway my boss goes back to the truck to get something, at like 9 in the morning on the job site, opens up the back, the ENTIRE truck is filled with weed smoke which billows out because big awful dude is in there getting high. Boss is upset, obviously, but big awful dude is just laughing
I think they had to finish out the day with him but the boss was definitely irritated about it
Oh shit! I forgot one from another job.
One of the busboys walked into the office, found no people and a satchel with about $30,000 in cash, picked it up and walked out, clocked out like normal, went home.
Guy SHOWED UP TO WORK THE NEXT DAY. Just assuming I guess, they won’t have cameras or anything, if I just don’t say anything there’s no way they can know who it was and they’ll probably just move on if I play it cool.
I guess the management was pretty aware of his level of planning skills because they had cops waiting at the restaurant at the time of his scheduled starting time and he was taken away in cuffs, presumably not to return for quite a long time.
I guess in his defense, he knew damn well if he stopped coming to work the day after $30k went missing, they’d know it was him.
I mean obviously the smart thing to do is not to fucking touch the money, but I’ll give the guy showing up to work the next day. It’s not like $30k is flee-to-Argentina-and-start-a-new-life money.
No, the smart thing to do is to not leave 30k unattended around people who aren’t paid well.
