This is Fine! A podcast about resilience engineering and software

If you want to come to the event to chat with Layer Aleph folks about Crisis Engineering you can sign up here: https://resilienceinsoftware.org/events/272451

A small snippet of our communication with Riverside (we’re pretty sure it’s AI, folks!)

—---------------------
Hi Colette,

Thank you for pushing back on our last reply, and I owe you a correction.

You were right about pressing record

Our logs confirm you did start a recording. The countdown you saw was real, and telling you the record button wasn't clicked was wrong. I'm sorry for that.

What the recording data shows

Here's the difficult part. That first recording started and then stopped 13 seconds later. Everything after that point wasn't being captured, even though your studio session stayed open and you carried on recording your episode.

You then started a second recording about eight minutes afterwards, which is the 8-minute 48-second take with Clint.

Those two are the only recordings that exist for that day. There's nothing else stored and nothing recoverable from the period in between, because no content was being written during it.
—----------------------

You can find Layer Aleph here: https://layeraleph.com/

And a good roundup of their recommended reading that was referenced for Crisis Engineering is here: https://crisisengineering.layeraleph.com/reading-list/

Some writing on Mikey’s Pomona College course is here: https://www.pomona.edu/news/2024/02/12-cover-time-back-campus-mikey-dickerson-01-pilots-return-pomona

Weaver referenced Karl Weick early on as an influence. Some of Karl Weick on sensemaking can be found here: https://pubsonline.informs.org/doi/abs/10.1287/orsc.1050.0133
And he also is part of the HRO (high reliability organization) group, some of that writing is here: https://journals.sagepub.com/doi/abs/10.2307/41165243

Obviously if we talk about Richard Cook we start with https://how.complexsystems.fail/

Downs’ Inside Bureaucracy seems to have limited printing options: https://a.co/d/08dulLA1

A small plug of Pandoc if you’re looking to get markdown into a .doc format: https://pandoc.org/

It has been 0 episodes since we have mentioned the Dependency comic from xkcd: https://www.explainxkcd.com/wiki/index.php/2347:_Dependency

If you want to come to the event to chat with Layer Aleph folks about Crisis Engineering you can sign up here: https://resilienceinsoftware.org/events/272451

Chad’s thesis from Lund is here: https://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=9076274&fileOId=9076276

The great Barry Turner book Man-Made Disasters from the 1970s has recently seen 2nd edition released, it used to be a very rare and difficult book to find: https://www.amazon.com/dp/0750620870

Some Patterson and/or Woods on Handover Communications in medical settings: https://www.jstor.org/stable/45126978 and https://journals.sagepub.com/doi/abs/10.1177/2327857919081001 and https://qualitysafety.bmj.com/content/21/Suppl_1/i19.short and re: NASA - https://journals.sagepub.com/doi/abs/10.1177/107118139704100155

Communication coordination during handovers for railway dispatch: https://rosap.ntl.bts.gov/view/dot/9020

Chad’s talk on Handover Communications at SRECon can be found here: https://www.usenix.org/conference/srecon23americas/presentation/todd

The Law of Fluency comes from Woods’ work in Cognitive Systems Engineering, here’s a paper that mentions it: https://www.researchgate.net/profile/David-Woods-19/publication/224753269_The_Messy_Details_Insights_From_the_Study_of_Technical_Work_in_Healthcare/links/0deec51bb2fc4d4e3d000000/The-Messy-Details-Insights-From-the-Study-of-Technical-Work-in-Healthcare.pdf but there are many other books and papers that do as well.

Common Ground in Joint Activity is another theme that comes up in the Cognitive Systems Engineering world. Gary Klein has written about it lots of places, perhaps most notably here: https://ieeexplore.ieee.org/abstract/document/1363742 and here: https://books.google.com/books?hl=en&lr=&id=371wV4dI7ckC&oi=fnd&pg=PA139&dq=common+ground+joint+activity&ots=1RKpCK7p7g&sig=XkyZygexYgU36UC-QtIzM3S67Cg#v=onepage&q=common%20ground%20joint%20activity&f=false

A Tale of Two Safeties is not by Allspaw/Cook, it’s by Erik Hollnagel: https://hf-sts.com/wp-content/uploads/2018/07/3689b-a_tale_of_two_safetieshollnagel.pdf

Aerobarrier is super cool if you want to check out how it works: https://glbipro.com/aerobarrier-2/

Colette isn’t drinking alcohol, it’s a delicious herbal mocktail by Casamara Club: https://www.casamaraclub.com/products/como
(Maybe we can get them to sponsor us someday)

On the passive house that survived the Palisades fire: https://www.reddit.com/r/Damnthatsinteresting/comments/1hy22ui/house_designed_on_passive_house_principles/

Layer Aleph https://layeraleph.com/ folks have written Crisis Engineering: https://bookshop.org/p/books/crisis-engineering-time-tested-tools-for-turning-chaos-into-clarity-marina-nitze/44736d1287a7da6e and there will be an RISF event with them in September https://resilienceinsoftware.org/events/272451 , which means we’re hoping to get them on the pod in August, FYI!

More about the Lund program here: https://www.humanfactors.lth.se/

Dr. Woods’ OSU program is here: https://ise.osu.edu/human-systems-integration

Our semi-academic, semi-practical space is the Resilience in Software Foundation slack - lots of folks from all ends of the spectrum talk there about RE concepts and practical applications for how to solve things (and pet pictures). You can get access to it by joining the Resilience in Software Foundation: resilienceinsoftware.org

The program in Australia is NOT in Melbourne, it’s in Brisbane at Griffith University, you can see Sidney Dekker’s profile there: https://experts.griffith.edu.au/19027-sidney-dekker as part of the Humanities and Social Sciences school. Drew Rae is also there, and his podcast is here: https://safetyofwork.com/

https://www.adaptivecapacitylabs.com/ and https://www.uptimelabs.io/ are very practical approaches to a lot of these things. Blackrock 3 https://www.blackrock3.com/ does training for incident command.

The new website is live! thisisfinepod.com

You can find John Allspaw at Adaptive Capacity Labs: https://www.adaptivecapacitylabs.com

Mike McGill, the skateboarder: https://en.wikipedia.org/wiki/Mike_McGill

Annie Duke’s Thinking in Bets, referenced by our question-asker is a great one: https://bookshop.org/p/books/thinking-in-bets-making-smarter-decisions-when-you-don-t-have-all-the-facts-annie-duke/31466984521c3d8a?ean=9780735216372&next=t

Naturalistic Decision Making has its own association, which has a ton of resources (and a conference!) - https://naturalisticdecisionmaking.org/
They also have a podcast! https://naturalisticdecisionmaking.org/new-podcast/

Gary Klein is the NDM guy - https://bookshop.org/p/books/seeing-what-others-don-t-the-remarkable-ways-we-gain-insights-chief-scientist-gary-klein/c4ae5e017fe005ff?ean=9781610393829&next=t

We contrast him and his style of approaching cognition and decision making with Kahneman and Tversky.
Kahneman and Tversky wrote a lot, but Judgement Under Uncertainty is probably the most famous? https://www.science.org/doi/abs/10.1126/science.185.4157.1124

And Kahneman wrote Thinking Fast and Slow: https://bookshop.org/p/books/thinking-fast-and-slow-daniel-kahneman-phd/83a544fe6f98df87?ean=9780606275644&next=t

It has been zero episodes since we’ve mentioned Lisanne Bainbridge’s Ironies of Automation: https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf

But also she has Verbal Reports as evidence of the process operator’s knowledge: https://www.sciencedirect.com/science/article/abs/pii/S1071581979603075?via%3Dihub

And the Etsy Debriefing Guide is super great: https://extfiles.etsy.com/DebriefingFacilitationGuide.pdf

Sidney Dekker and The Field Guide are foundational: https://bookshop.org/p/books/the-field-guide-to-understanding-human-error-sidney-dekker/3a4209dfc8b3a721?ean=9781472439055&next=t

From Dekker’s field guide (pg 47) there is a list referencing Gary Klein’s questions for an incident investigation:

Cues:
What were you seeing?
What were you focusing on?
What were you expecting to happen?

Interpretation:
If you had to describe the situation to your colleague at that point, what would you have told?

Errors:
What mistakes (for example in interpretation) were likely at this point?

Previous experience/knowledge:
Were you reminded of any previous experience?
Did this situation fit a standard scenario?
Were you trained to deal with this situation?
Were there any rules that applied clearly here?
Did any other sources of knowledge suggest what to do?

Goals:
What were you trying to achieve?
Were there multiple goals at the same time?
Was there time pressure or other limitations on what you could do?

Taking action:
How did you judge you could influence the course of events?
Did you discuss or mentally imagine a number of options or did you know straight away what to do?

Outcome:
Did the outcome fit your expectation?
Did you have to update your assessment of the situation?

John mentioned Uptime Labs, who do staged worlds for software incidents: https://uptimelabs.io/

Facets of Complexity in Situated Work is here: https://www.researchgate.net/publication/345523195_Facets_of_Complexity_in_Situated_Work

On the Jamie Zawinski quote: https://regex.info/blog/2006-09-15/247

If you don’t know the parable of the blind men and the elephant: https://en.wikipedia.org/wiki/Blind_men_and_an_elephant

Mitchell Hashimoto’s post on leaving Github: https://mitchellh.com/writing/ghostty-leaving-github

The Reddit post on Github’s availability historically (that we find questionable): https://www.reddit.com/r/github/comments/1rnvhs9/githubs_historic_downtime_scraped_and_plotted/

A reminder, the Messy 9 are: congestion, cascade, conflict, lag, saturation, friction, tempo, surprise, tangles

We have sometimes loved his stuff, but Gergely is annoying us with these posts: https://newsletter.pragmaticengineer.com/p/the-pulse-is-github-still-best-for?r=78c7k&utm_medium=email

https://x.com/GergelyOrosz/status/2048017382036082706

You can find the RISF store with Hindsight Bias merch here: https://www.bonfire.com/store/risf/

You can find a copy of Richard Cook’s Two Years Before the Mast at Lorin’s Blog: https://surfingcomplexity.blog/wp-content/uploads/2026/03/twoyearsbeforethemast.pdf

A reminder, Richard Cook’s How Complex Systems Fail can be found at http://how.complexsystems.fail

Some writing on the 1996 Annenberg conference: https://www.researchgate.net/publication/351953417_Coming_Together_The

Folk models paper (not by Woods, by Dekker and Hollnagel), which is specifically targeting Situational Awareness as being a folk model: https://link.springer.com/article/10.1007/s10111-003-0136-9

Some stuff about SNAFU Catchers: https://www.snafucatchers.com/
And https://snafucatchers.github.io/

Eric referenced our conversation with Beth Long about Building and Revising Adaptive Capacity, which she co-wrote with Richard Cook about New Relic’s real-life example of resilience engineering: https://youtu.be/A_rU4-M61Hk and https://www.sciencedirect.com/science/article/abs/pii/S0003687020301903?via%3Dihub for the paper

Erik Hollnagel’s RAG get’s referenced: https://erikhollnagel.com/onewebmedia/RAG%20Outline%20V2.pdf

Once again, we link you to Lorin’s Law: https://surfingcomplexity.blog/2017/06/24/a-conjecture-on-why-reliable-systems-fail/

Eric is referencing Lund, that is their Human Factors and Systems Safety program: https://www.humanfactors.lth.se/

Check out Crisis Engineering! https://crisisengineering.layeraleph.com/crisis-engineering-the-book/

The upcoming RISF event on Practice of Practice Gamelan: https://resilienceinsoftware.org/events/245030