And it looks like OpenAI’s models tried hacking other places too.
With any luck, this incident will actually trigger some behavioral or regulatory changes that makes security paramount.
TMI was an interesting case. One of the things that always stuck with me was the truly horrendous HSI they had in the control room (e.g., “red” meant “cold” and “blue” meant “hot”; minor alerts would trigger dozens of annunciators, training operators to ignore most of them…). Really, high risk systems should always have a review by human factors specialists.
1 Like
Lots of human creations have global impacts that are catastrophic, and most of those aren’t technological. Consider the causes of the world wars.
1 Like
Aerospace and military agencies have developed a Project Management technique called “Lessons Learned” designed to analyze why things go wrong and what can be learned to prevent the problem from happening again. From what I’ve seen it can be a useful exercise.
It may go by a range of names, but just about any organization with reasonably mature technology practices will have a “Lessons Learned” component to its incident response and threat management policies and procedures, including the roles or names of individuals who are responsible for executing it.
Further, a high-quality organization will run “table top exercises” periodically to verify its capabilities. Certainly, it is expected for any company that is publicly traded or expects to be publicly traded. It is a huge red flag if an organization cannot identify specific policies and procedures for running “Lessons Learned” analyses.
1 Like
And Anthropic did it too. Sigh…
Then there is “Murphy Law” which is actually based on a failure analysed by engineer Edward Murphy Jr where the electrical connection for a rocket sled test was reversed, meaning no data was collected. (Dr John Stapp subjected himself to very high decelerations for medical research)
Murphy’s observation was:
“If there are two or more ways to do something and one of those results in a catastrophe, then someone will do it that way.”
Replace “someone” with “AI”!
1 Like
Lessons learned from Tailscale’s perspective: