CrowdStrike issue causes mass global tech disruptions

[Eric66]

Curious how you did this? Did you go hands on & delete the offending .sys file manually?

I have hundreds of remote systems that are doing one of the following:
-Stuck at a BSOD
-BSOD but building a crash log file, crashing again before it gets to 100%, rebooting and building a log file again
-Systems displaying a bitlocker pw required page, enter the pw & it crash reboots back to the bitlocker screen

I've had to give out the bitlocker pw & a local admin username & pw then talk the users through getting into SAFE mode. Getting into SAFE mode has been a real PITA!

Honestly, I'm not sure what the server side of the recovery looked like. I'm in the group responsible for direct and indirect control of the production lines, vehicle tracking and data collection, once all of the VMs were back up it became our responsibility to get them into a production ready state.

 

[indyblue]

They actually use a backdoor to push their updates and there is no way to stop it short of unplugging from the internet.

Why can’t it simply be firewalled off at the corporate level? Have one server in a DMZ that gets the updates and then internal staff can copy those and start the process internally so it can either be automated or manually distributed.

 

[BigRed]

 

[ancjr]

I was in the ER when they announced that they couldn't handle any more patients because of this mess. Drove 30 miles to an ER that still was able to operate pen and paper and was better off all the way around.

 

[Hop]

Well Corporate pushed out an emergency group policy change that will allow us Field Service IT administrators to create an unencrypted Windows PE boot stick on any still functional remote PC (we could have manually done this on a remote PC). We have a tool on that stick to pull the Bitlocker pw and also pull the local admin login pw.

This is SUPER dangerous imo.

Our users, right now, don't know this USB policy has been pushed out. This puts our pi at risk & is a really bad idea imo.

 

[firecadet613]

I'm amazed so many fellow remote folks on my team (and other teams) are still affected with laptops down, while I wasn't impacted.

Do that many folks leave their work laptops on 24/7? I shut mine down when I'm done for the day, so I'm assuming I never received the faulty update?

 

[jkaetz]

Well Corporate pushed out an emergency group policy change that will allow us Field Service IT administrators to create an unencrypted Windows PE boot stick on any still functional remote PC (we could have manually done this on a remote PC). We have a tool on that stick to pull the Bitlocker pw and also pull the local admin login pw.

This is SUPER dangerous imo.

Our users, right now, don't know this USB policy has been pushed out. This puts our pi at risk & is a really bad idea imo.

Is it possible to free up some disk space on the systems and plant the recovery PE image on the disk itself rather than on USB? Of course you still have an admin PW and bitlocker key in plain text but at least not on a USB drive. Hopefully all your systems have different local admin PWs. Could also mass change the local admin passwords after recovery.

I'm amazed so many fellow remote folks on my team (and other teams) are still affected with laptops down, while I wasn't impacted.

Do that many folks leave their work laptops on 24/7? I shut mine down when I'm done for the day, so I'm assuming I never received the faulty update?

My computers never get turned off. Fortunately we're not using CrowdStrike though even if we were I would have had the knowledge and ability to recover myself. I recognize that most don't. I don't need to save the tiny bit of electricity that they use while idle and makes it easy to pickup the work wherever it left off. It also facilitates the fluidity of work happening around life. Again, other situations may be different.

 

[WebSnyper]

My computers never get turned off. Fortunately we're not using CrowdStrike though even if we were I would have had the knowledge and ability to recover myself. I recognize that most don't. I don't need to save the tiny bit of electricity that they use while idle and makes it easy to pickup the work wherever it left off. It also facilitates the fluidity of work happening around life. Again, other situations may be different.

Same here, I leave it run as there are times I need to jump over to a PC from phone in the evening to handle something work related. Even though my machine could boot fast enough, it just makes it easier to leave it on as I do with my home machines, etc.

 

[Hop]

Is it possible to free up some disk space on the systems and plant the recovery PE image on the disk itself rather than on USB? Of course you still have an admin PW and bitlocker key in plain text but at least not on a USB drive. Hopefully all your systems have different local admin PWs. Could also mass change the local admin passwords after recovery.

My computers never get turned off. Fortunately we're not using CrowdStrike though even if we were I would have had the knowledge and ability to recover myself. I recognize that most don't. I don't need to save the tiny bit of electricity that they use while idle and makes it easy to pickup the work wherever it left off. It also facilitates the fluidity of work happening around life. Again, other situations may be different.

I got some more info about this Corporate Win PE drive today. It has a time bomb built in so it won't boot PE after a predetermined amount of time.

We also use a LAPS tool that rotates the local admin password. I can expire that password after fixing a remote machine.

This has been a Monday from Hell. I have people, even in this day, year 2024, that cannot tell the PC from a monitor from a docking station. People are CONSTANTLY calling the PC "box" a modem. I have people calling in new PC setup tickets for a laptop docking station thinking it's their new PC. I have people that "turn off & on" their computer only to see the same frozen screen because they only power cycled the monitor & not the PC.

They live amongst us.

 

[DoggyDaddy]

I got some more info about this Corporate Win PE drive today. It has a time bomb built in so it won't boot PE after a predetermined amount of time.

We also use a LAPS tool that rotates the local admin password. I can expire that password after fixing a remote machine.

This has been a Monday from Hell. I have people, even in this day, year 2024, that cannot tell the PC from a monitor from a docking station. People are CONSTANTLY calling the PC "box" a modem. I have people calling in new PC setup tickets for a laptop docking station thinking it's their new PC. I have people that "turn off & on" their computer only to see the same frozen screen because they only power cycled the monitor & not the PC.

They live amongst us.

Can you help me with my VCR? The clock keeps flashing 12:00. I've turned it off and back on with no luck!

 

[BigBoxaJunk]

Can you help me with my VCR? The clock keeps flashing 12:00. I've turned it off and back on with no luck!

The best permanent fix for that is a piece of black electrical tape over the LED.

 

[WebSnyper]

The best permanent fix for that is a piece of black electrical tape over the LED.

Actually the answer for that on a vcr is just to unplug it... And leave it unplugged.

 

[DoggyDaddy]

Actually the answer for that on a vcr is just to unplug it... And leave it unplugged.

But I need it! I only have a limited Beta Max collection!

 

[foszoe]

But I need it! I only have a limited Beta Max collection!

VCR is a different format. That was JVC.

Beta was Sony.

 

[DoggyDaddy]

VCR is a different format. That was JVC.

Beta was Sony.

I know...

 

[TheGrumpyGuy]

I know...

View attachment 367941

I don't think it even left skid marks

 

[BiscuitsandGravy]

I got some more info about this Corporate Win PE drive today. It has a time bomb built in so it won't boot PE after a predetermined amount of time.

We also use a LAPS tool that rotates the local admin password. I can expire that password after fixing a remote machine.

This has been a Monday from Hell. I have people, even in this day, year 2024, that cannot tell the PC from a monitor from a docking station. People are CONSTANTLY calling the PC "box" a modem. I have people calling in new PC setup tickets for a laptop docking station thinking it's their new PC. I have people that "turn off & on" their computer only to see the same frozen screen because they only power cycled the monitor & not the PC.

They live amongst us.

Add in a cloud based PAM solution and it gets even more spicy.

 

[foszoe]

Add in a cloud based PAM solution and it gets even more spicy.

PAM. That is so old fashioned. Just use a modern non stick pan.

 

[DoggyDaddy]

PAM. That is so old fashioned. Just use a modern non stick pan.

Pam... When I was 4 or 5, these people from the hills of Tennessee moved in next door. They had a daughter named Pam. I had a crush on her even back then, and she was probably 6 or 7 years older than me. She babysat me a few times. She had incredible yayas. Probably not the same PAM.

 

[foszoe]

Pam... When I was 4 or 5, these people from the hills of Tennessee moved in next door. They had a daughter named Pam. I had a crush on her even back then, and she was probably 6 or 7 years older than me. She babysat me a few times. She had incredible yayas. Probably not the same PAM.

It's okay to have a crush on your babysitter, but to be infatuated with her grandmas is a little over the top.