if anyone is caught up with this bug here's a possible solution:  

 

https://www.news.com.au/technology/online/everywhere-i-look-blue-screen-of-death-how-to-fix-crowdstrike-bsod-windows-crash/news-story/615cfe835a86b4d35c92312cc41c95b3

 

Boot in Safe Mode

2. Navigate to Windows\System32\drivers\CrowdStrike

3. Find and delete “C-00000291*.sys”

 

Or according to Microsoft; restart your computer at least 15 times

 

 

When to a Shell station yesterday desperate to fill up only to find that none of the pumps were working.     So I had to go to another 5km down the road, not that I was aware of any outages.   When i got  home this was all over the news,  it looks like its effected air travel hard; especially JetStar.    Then I realized that at my last call the Surface Pro said that the organisation has implemented a number of updates and I need to reboot!   Well immediately started the tablet but I'm still going.   thank god!    Even though not being effected just as well that all the products are Apple at home!  Goes without saying is that its a blessing to have a number of operating systems working in case something like this occurs.   As they said not many organisations operate with Apple.   

 

I was affected in a different way. On a family trip to Bali, we left Melbourne at 8 am for Sydney to catch a connecting flight. We were at Sydney airport all day until 3 pm when our airline, Jetstar, was hit by the disruption. There were no updates until 9 pm when they finally grounded the flight—this should have happened much earlier. The situation escalated, and the Australian Federal Police were called in to manage the crowds. We had to pass through immigration and collect our luggage. There was no communication regarding accommodation or alternative flights. The announcement was simply: plane grounded, flight cancelled, rebook on Jetstar.com.au. There was no consideration for travelers with families needing emergency accommodation.

 

My partner and I managed to secure two rooms at Kingsgrove Pub for the night and settled the kids. Attempts to contact Jetstar were futile, so we reached out to Qantas directly and had them rebook our flight to Bali. However, all flights were cancelled on Saturday, and we couldn't get on any, so we were rescheduled for Sunday. Again, we received no assistance for accommodation in Sydney; we were just told to file a customer complaint later and keep all receipts for reimbursement.

 

We contacted our travel insurance company and were informed they didn't have a formal charge code for the global CrowdStrike outage, so we kept all tax invoices and were advised to keep expenses reasonable for later submission. We found a reasonably priced room near the airport for Saturday, and now I'm just taking a moment to relax and share this experience.

 

@MrBurns84

 

sorry that you are caught up in this.  Hope it gets better for you. 

3 hours ago, MrBurns84 said:

I was affected in a different way. On a family trip to Bali, we left Melbourne at 8 am for Sydney to catch a connecting flight. We were at Sydney airport all day until 3 pm when our airline, Jetstar, was hit by the disruption. There were no updates until 9 pm when they finally grounded the flight—this should have happened much earlier. The situation escalated, and the Australian Federal Police were called in to manage the crowds. We had to pass through immigration and collect our luggage. There was no communication regarding accommodation or alternative flights. The announcement was simply: plane grounded, flight cancelled, rebook on Jetstar.com.au. There was no consideration for travelers with families needing emergency accommodation.

 

My partner and I managed to secure two rooms at Kingsgrove Pub for the night and settled the kids. Attempts to contact Jetstar were futile, so we reached out to Qantas directly and had them rebook our flight to Bali. However, all flights were cancelled on Saturday, and we couldn't get on any, so we were rescheduled for Sunday. Again, we received no assistance for accommodation in Sydney; we were just told to file a customer complaint later and keep all receipts for reimbursement.

 

We contacted our travel insurance company and were informed they didn't have a formal charge code for the global CrowdStrike outage, so we kept all tax invoices and were advised to keep expenses reasonable for later submission. We found a reasonably priced room near the airport for Saturday, and now I'm just taking a moment to relax and share this experience.

 

 

Holy crackers, that sucks big time.  Hopefully you can get to your destination, kick back and eventually have a laugh about it.  Very disappointing to hear about Jetstar, I would have thought that considering you were at the airport during transit they'd automatically put you up and cover transit and at least breakfast.

 

I got stuck in Melbourne years ago waiting to jet to Sydney, then off to Chile.  As is so often the case, Sydney airport was fogged in so I got stuck in Melbourne for 3 hours.  When we took off, so did my connecting flight from Sydney to Chile.  I had to stay in Sydney for 3 nights because flights to Chile aren't run every day.  The airline arranged transit both ways, hotel accomm and all meals without even asking, because I was in transit.  That was a One World affiliated airline, I forget which one.  Fingers crossed you'll get your costs reimbursed.

5 hours ago, Addicted to music said:

here's a possible solution:  

 

https://www.linuxmint.com

 

🙂 

It’s a proper shitstorm of an outage but the amount of uninformed commentary about it online this past day has been epic.
 

Add to that the ambulance chasers and general shitbirds trying to make money from others misfortune and there has been a really distasteful display of human nature.

 

while a relatively unknown brand to many, crowdstrike are a $100b market cap security vendor. Their technology is widely used, not only in enterprise customers, but also heavily in our Australian government’s systems. 

Their technology has saved countless businesses from serious cyber impacts. this is indisputable. I’ve been there responding to major breaches with their tool set and forensics tools.

 

The fact that this outage has impacted so many major organisations shows you how important and trusted they are globally.
 

I’ve been at the coal face the past 24 hours with many impacted customers and while they are upset, they also know that this tool is hands down the best tool for the job we task it with.

 

i’m sure crowdstrike will come out with a robust explanation (they have already started this) and remediation actions.

It’s unfortunate that it has impacted so many, but the inconvenience of the lay person on the street really pales in comparison to the tireless work many in our industry have been doing since 3pm yesterday to recover systems and endpoints all over the globe.

 

maybe take some time to think how they have spent their days, likely without overtime or extra pay, sorting this mess out for everyone.

12 hours ago, BugPowderDust said:

It’s a proper shitstorm of an outage but the amount of uninformed commentary about it online this past day has been epic.
 

Add to that the ambulance chasers and general shitbirds trying to make money from others misfortune and there has been a really distasteful display of human nature.

 

while a relatively unknown brand to many, crowdstrike are a $100b market cap security vendor. Their technology is widely used, not only in enterprise customers, but also heavily in our Australian government’s systems. 

Their technology has saved countless businesses from serious cyber impacts. this is indisputable. I’ve been there responding to major breaches with their tool set and forensics tools.

 

The fact that this outage has impacted so many major organisations shows you how important and trusted they are globally.
 

I’ve been at the coal face the past 24 hours with many impacted customers and while they are upset, they also know that this tool is hands down the best tool for the job we task it with.

 

i’m sure crowdstrike will come out with a robust explanation (they have already started this) and remediation actions.

It’s unfortunate that it has impacted so many, but the inconvenience of the lay person on the street really pales in comparison to the tireless work many in our industry have been doing since 3pm yesterday to recover systems and endpoints all over the globe.

 

maybe take some time to think how they have spent their days, likely without overtime or extra pay, sorting this mess out for everyone.

no doubt the QA teams will be scrutinized at a PIR..lol

 

As usual, most commentary is about the detail of the incident and how it happened, but the real interest should be in the wider architecture and how our commercial and civil systems are allowed to become vulnerable to this extent with seemingly little in place in terms of mitigation strategy, and how we reportedly lacked a speedy means of recovery.  

On 20/7/2024 at 4:23 PM, Kaynin said:

 

Holy crackers, that sucks big time.  Hopefully you can get to your destination, kick back and eventually have a laugh about it.  Very disappointing to hear about Jetstar, I would have thought that considering you were at the airport during transit they'd automatically put you up and cover transit and at least breakfast.

 

sadly kaynin, times have possibly changed ... i saw the same with Qantas a year or so ago.. same thing.. walked into adelaide airport at friday lunchtime to catch a flight home and as i am seeing a text message from them saying flight cancelled see customer service.. so walked up to them instead of going through security to catch flight..

 

they said yep cancelled and they have nothing else, will have to wait till tomorrow... I asked them if Qantas was going to book my next flight and what about accomodation and they said ..sorry sir we have no one to arrange all that will have to do yourself  - that was it .. go fend for yourself  

 

rang our work travel agent they got me on a flight the next day and organised accomodation at the airport. bummer i had handed my hire car in as would have just headed for barossa valley or something for the night  . the flight next day got moved too..eventually left Saturday afternoon to get home saturday evening.. most of weekend wiped.. wasnt in good books when finally returned home hehe

 

this wasnt even in a crowd strike event or anything just usual to expect now i think... 

On 20/7/2024 at 1:12 PM, MrBurns84 said:

We contacted our travel insurance company and were informed they didn't have a formal charge code for the global CrowdStrike outage, so we kept all tax invoices and were advised to keep expenses reasonable for later submission. We found a reasonably priced room near the airport for Saturday, and now I'm just taking a moment to relax and share this experience.

 

hope all gets sorted MB, can put a real mess into things for trips with flight delays with the domino effect on all else planned. hopefully can get back on it 

On 20/7/2024 at 1:12 PM, MrBurns84 said:

we were just told to file a customer complaint later and keep all receipts for reimbursement.

 

yep qantas same.. just gave me their customer complaint card and suggest i contact them  

i am not sure with jetstar, but with qantas they will only reimburse for certain things and not others..things like my car park hire with my car stuck there an extra night at X amount wasnt covered for instance.. if i had accomodation booked that was non refundable that was my own problem... 

On 20/07/2024 at 5:08 PM, aussievintage said:

 

https://www.linuxmint.com

 

🙂 

 

I have Crowdstrike running on Linux and MS-windows servers (all the MS ones died on Friday and I unjammed them).  It has worked extremely well on both platforms for my limited "newb" experience up until 4PM last friday afternoon.

 

Crowdstrike have pushed out buggy versions of their Linux software as well in the past, though it didn't hit as badly as this incident. 

https://www.neowin.net/news/crowdstrike-broke-debian-and-rocky-linux-months-ago-but-no-one-noticed/

 

That said recent updates of how Crowdstrike works on Linux should make this harder to happen:

 

https://nondeterministic.computer/@mjg59/112816011370924959

 

 

 

 

 

 

On 20/07/2024 at 7:59 PM, BugPowderDust said:

It’s a proper shitstorm of an outage but the amount of uninformed commentary about it online this past day has been epic.
 

Add to that the ambulance chasers and general shitbirds trying to make money from others misfortune and there has been a really distasteful display of human nature.

 

while a relatively unknown brand to many, crowdstrike are a $100b market cap security vendor. Their technology is widely used, not only in enterprise customers, but also heavily in our Australian government’s systems. 

Their technology has saved countless businesses from serious cyber impacts. this is indisputable. I’ve been there responding to major breaches with their tool set and forensics tools.

 

The fact that this outage has impacted so many major organisations shows you how important and trusted they are globally.
 

I’ve been at the coal face the past 24 hours with many impacted customers and while they are upset, they also know that this tool is hands down the best tool for the job we task it with.

 

i’m sure crowdstrike will come out with a robust explanation (they have already started this) and remediation actions.

It’s unfortunate that it has impacted so many, but the inconvenience of the lay person on the street really pales in comparison to the tireless work many in our industry have been doing since 3pm yesterday to recover systems and endpoints all over the globe.

 

maybe take some time to think how they have spent their days, likely without overtime or extra pay, sorting this mess out for everyone.

Thanks for all the hard work!

 

I've been so glad that I moved from production support to supporting testing/development environments (about 20 years ago). Only one of our clients was affected (and by that I mean I couldn't get past the MFA) and so I finished up at 5pm on Friday and slept like a baby on the weekend.

 

Still the same shitfight that was there before 2:36pm on Friday, but I've learned to expect that from this one...

23 hours ago, betty boop said:

 

sadly kaynin, times have possibly changed ... i saw the same with Qantas a year or so ago.. same thing.. walked into adelaide airport at friday lunchtime to catch a flight home and as i am seeing a text message from them saying flight cancelled see customer service.. so walked up to them instead of going through security to catch flight..

 

they said yep cancelled and they have nothing else, will have to wait till tomorrow... I asked them if Qantas was going to book my next flight and what about accomodation and they said ..sorry sir we have no one to arrange all that will have to do yourself  - that was it .. go fend for yourself  

 

rang our work travel agent they got me on a flight the next day and organised accomodation at the airport. bummer i had handed my hire car in as would have just headed for barossa valley or something for the night  . the flight next day got moved too..eventually left Saturday afternoon to get home saturday evening.. most of weekend wiped.. wasnt in good books when finally returned home hehe

 

this wasnt even in a crowd strike event or anything just usual to expect now i think... 

 

This is the reason I no longer travel for work.......😃

 

Even as recent they asked me to do a training course in Sydney......stuff that!    Unlike other workplaces,  we don't get compensated for after hours travel, meal allowance only limited $50 for dinner!   Unlike when a previous CEO was running it....  And if i fell into the situation like above,  i'd be stuffed!

BTW,

 

I was at this school out in the west,  after i left on Friday,  they were hit hard!  Apparently all there endpoint: teachers, kids and admin were all effected.  There IT came in on Saturday to reset a number of steps to get them going, by doing this they had reduced there workload for this morning.

56 minutes ago, Addicted to music said:

This is the reason I no longer travel for work.......😃

 

Aaah, mate - you need to move onto the next stage of life ... where you no longer work!  

 

28 minutes ago, andyr said:

 

Aaah, mate - you need to move onto the next stage of life ... where you no longer work!  

 

I would  not know what to do with myself,  I also jog in the morning b4 going to work everyday….keeps me alive 😂

 

Edited July 22, 20241 yr by Addicted to music

3 minutes ago, Addicted to music said:

I would not know what to do with myself,  I also jog in the morning b4 going to work everyday….keeps me alive 😂

 

 

Excellent Peter!  👍  6 minutes a km is laudable.

 

I used to go running - from when I was about 30 ... to when I was about 60.  5kms around Albert Park Lake.  But then I gave it away - as a result (at 75) I have no knee/hip problems.  

 

Now I go to the gym - but, yes, my aerobic fitness is pretty ratsh!t.  

 

11 hours ago, andyr said:

 

Excellent Peter!  👍  6 minutes a km is laudable.

 

I used to go running - from when I was about 30 ... to when I was about 60.  5kms around Albert Park Lake.  But then I gave it away - as a result (at 75) I have no knee/hip problems.  

 

Now I go to the gym - but, yes, my aerobic fitness is pretty ratsh!t.  

 

Ah, my knees were already problematic by 18, so walking is as far as I'll go - but the gym is a necessity.

@andyr

 

Awesome to hear that you're active

Preliminary PIR has just been published: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/

 

There's a few nasty bits in there. 

Quote

"Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production."

"We did something similar before, the CI/CD content validator passed it, no need for testing, let's push this to production"

5 hours ago, BugPowderDust said:

Preliminary PI

"We did something similar before, the CI/CD content validator passed it, no need for testing, let's push this to production"
 

 

Famous last words......

 

Get use to it as  more products and updates of products get use to this practice..  Boeing is a great example of what can go wrong will!    Rumour has it that the person who ran Macfee did similar interruptions and is the same personwho approved this update to get pushed into field.    

On 24/7/2024 at 9:30 PM, Addicted to music said:

Rumour has it that the person who ran Macfee did similar interruptions and is the same personwho approved this update to get pushed into field.   

This sentence reminds me of the Simpsons episode where Ralph Wiggum says "Mrs. Krabappel and Principal Skinner were in the closet making babies and I saw one of the babies and the baby looked at me"

Crowdstrike's CEO, George Kurtz worked at McAfee. He joined McAfee in 2004 when the company acquired Foundstone, a cybersecurity firm he founded. At McAfee, Kurtz held several senior positions, including Chief Technology Officer (CTO) and Executive Vice President (EVP) of Enterprise, before leaving the company in 2011.

In April 2010, McAfee released a faulty update that mistakenly identified a critical Windows system file as malware, leading to widespread crashes and system reboots, particularly affecting Windows XP machines. 

 

It is distinctly unlikely that the CEO was the one who approved the update to be pushed to the field and caused last week's outages.

What happened to all the things we got taught in Uni/Tafe with software, I remember the SDLC book which I still have...

 system development life cycle,  SDLC phases: problem analysis and planning, designing, development, testing, and deployment.

 

Unfortunately most of those practices have gone out the window in some business's due to costs involved in todays IT world, which implement software into production systems that can sometimes causes issues without enough testing in the development system, before deployment.

 

 

14 minutes ago, Bass13 said:

What happened to all the things we got taught in Uni/Tafe with software, I remember the SDLC book which I still have...

 system development life cycle,  SDLC phases: problem analysis and planning, designing, development, testing, and deployment.

 

Unfortunately most of those practices have gone out the window in some business's due to costs involved in todays IT world, which implement software into production systems that can sometimes causes issues without enough testing in the development system, before deployment.

 

 

 

The excuse being used is a claim for the need of 'agile' software development and deployment.  The fear of being caught by a day-zero threat has enabled IT people to convince management to take risks.

a decent read 

https://www.wsj.com/tech/crowdstrikes-ceo-george-kurtz-failure-global-tech-outage-microsoft-windows-07d27a4a?st=iqw4o58dffcu04s&reflink=desktopwebshare_permalink