Estimated reading time: 10 minutes
Her family rushed her to the Emergency Room twice within a couple months, each time resulting in a multi-night stay. Doctors diagnosed her with a heart arrhythmia each time but never located the root cause. They posed the standard list of questions, which included asking what medications she regularly took. But nothing ever stood out. After a couple-night stay her symptoms would subside, after which she’d be released. Now, a few weeks later, she’s back at the hospital with a new case of heart arrhythmia. But this time, a young intern accompanies the doctor. The doctor instructs the young novice to go through his list of questions with her while he observes:
INTERN: What medications do you regularly take, ma’am?
PATIENT: The same thing I told y’all the other 2 times I was admitted – just Vicks VapoRub before bed.
INTERN: How do you take it?
PATIENT: Like this. . . (she opens her mouth, takes her open palm, and demonstrates wiping the ointment on her tongue)
Just like that, this intern, naturally driven by a beginner’s mindset, uncovered the source of this patient’s recurring ailment through careful questioning! It turned out that her illness would always resolve itself during her overnight stays simply because she would be without her regular snack of Vicks VapoRub during those nights.
This intern, unburdened by the doctors’ many years of experience, wasn’t inclined to take shortcuts in the process. He didn’t assume she understood the proper administration of this common OTC medication. But the doctors themselves, having seen so many patients who’d regularly used meds like Vicks VapoRub, never imagined one would apply it so incorrectly. Yet, that failure of imagination would not have been a factor had they remained inquisitive while sticking to the standard list of questions.
And therein lies the pitfall in any area of mastery. Years spent gaining experience and honing craft can often, and counterintuitively, limit the scope of the practitioner’s imagination, creativity, and focus. Even more counterintuitive is the fact that cultivating a “Beginner’s Mindset” is often the fix.
But first, why does this phenomenon exist? How might it manifest in the lives of Performance Engineers? And what can we do to combat it in our quest for guru status?
Table of contents
The Trap of Experience
Our practices leave grooves over time. And eventually, those grooves become walls that box us in.”
— Chuck Rhoades in Showtime’s BILLIONS
Along the journey for greatness in any endeavor, our “beginner’s mindset” motivates us to study all the available literature, seek out mentors, drill the exercises, and accrue valuable experience from applying our newfound knowledge under real-world conditions. Over years of deliberate practice and execution, we gradually notice recurring patterns which our minds unconsciously form into shortcuts, rules-of-thumb, and best practices. In time, these mental “cheat codes” hew ever-deepening grooves into what was once a wide open pasture, walling us into a fixed mindset which stifles creativity and problem-solving ability.
Psychologists refer to this phenomenon as Einstellung Effect, a cognitive bias which describes the negative effect of prior experience when solving new problems. It creates a disposition to solve a given problem in a specific way despite the presence of more optimal alternatives. Psychologist Abraham Luchins’ “Water Jar Experiment” demonstrated it back in 1942. From the aforementioned hyperlinked article:
“Given 3 water jars of sizes A) 21 units, B) 127 units, and C) 3 units, measure out exactly 100 units of water.” Participants correctly figured that they’d fill B, then pour out enough to fill A, and then pour out enough to fill C twice, or B – (A + 2C).
After a subsequent series of similar tests, he then asked them to measure out exactly 18 units given 3 water jars of sizes A) 15 units, B) 39 units, and C) 3 units. 81% of participants replied with the same method used in the first test, B – (A + 2C), despite the fact that just adding jar A and jar C into jar B (i.e., A + C) would be much simpler. Their prior experience closed off their imaginations to the better solution.
Yet, of the participants who were never given the initial series of tests, 100% chose the simpler A + C solution. Their minds had not been mechanized by the prior experience of the first series of tests. Luchins proved that what you already know can hurt you.
Frustratingly, the further along we progress from our Beginner Stages, the more prone we become to this cognitive bias.
Personal Anecdote: The Lab Experiment
As Performance Engineers, we’re also prone to this tendency as our expertise grows. Well, not you, of course – you’re as perfect as your mom says you are! But it’s happened to me as recently as a year ago. Let’s gather ’round in a sharing circle as I give my testimony. My name is Mark E. Dawson, Jr., and I’m an Einstellung Effect survivor. This is my story.
Lab Setup
My R&D Lab replicates the basic building blocks of our production environment. For the systems under test (SUT), the lab comprises multiple CPU-type machines: some Xeon Gold-based systems, and some overclocked i9-based systems. Any change made to the software/hardware stack is benchmarked across all systems. Nanosecond HW timestamping at the software, NIC, and network levels, coupled with TICK stack monitoring and app logging instrument the entire lab. If I need to pinpoint where something went wrong, I have ample data to aid in diagnosis.
The Benchmark
One day after running a set of benchmarks, I noticed that the MAX against an i9-overclock system was uncharacteristically higher than that of the Xeon-based system by tens of microseconds. Yet, the rest of each system’s latency distribution appeared as I would expect relative to one another. So, I checked the timeseries-based latency scatter plot for the overclocked system. BOOM! All the outliers occurred within the first minute of the benchmark!
“Ah, I’ve seen this before <past experience made me short-circuit my process of viewing all logs and system telemetry>! This overclocked i9 is a retired prod machine that’s long in the tooth. It’s probably losing its ability to maintain frequency in the core and/or uncore area. So, when a new load is initially applied, it freaks out a bit before settling back down. Let me double-check that with an experiment.”
So I re-ran the benchmark, but this time I had “perf stat” running on the i9 SUT before and during its runtime:
mdawson@ocsys ~ $ sudo perf stat -e core_power.throttle -I 10000
# time counts unit events
10.000137852 151,361 core_power.throttle
20.000247285 170,164 core_power.throttle
30.000346693 196,967 core_power.throttle
40.000447125 267,580 core_power.throttle
50.000544173 139,964 core_power.throttle
60.000642237 366,877 core_power.throttle
70.000749019 25,325,734 core_power.throttle
80.000835257 44,627,468 core_power.throttle
90.000950532 43,687,140 core_power.throttle
100.001058887 43,527,594 core_power.throttle
110.001150044 43,629,874 core_power.throttle
120.001255150 44,091,517 core_power.throttle
130.001363649 44,730,550 core_power.throttle
140.001467374 6,275,039 core_power.throttle
150.001563901 201,453 core_power.throttle
160.001663608 137,391 core_power.throttle
170.001762296 168,459 core_power.throttle“BOOM! Those spikes at the beginning correspond *exactly* with the multiple orders of magnitude increase in throttling on the overclocked i9 <I’ve given Tech Talks about how correlation doesn’t indicate causation – but it didn’t matter to me now for some reason>! I’m gonna ask the SRE Team for a replacement server.”
Why would I ignore a wealth of telemetry data and short-circuit my usual process all because of a hunch? Due to my extensive experience where I’d observed things like this in the past, I didn’t stop to consider that there might be a common root cause for both the latency spikes *and* the core throttling!
Accidental Discovery
And what did I find HOURS LATER while looking at my TICK graphs for a reason completely separate from this benchmark issue? A short burst of Socket Memory Controller (MC) Throughput usage (from Telegraf’s intel_rdt plugin) of 3GB/s, which then dropped to ~100MB/s. Typically, MC Throughput for this benchmark never peaks above 150MB/s or so. That led me down the path of checking the app logs from the timestamp of the spikes, where I discovered a network connectivity issue in the i9 SUT setup which caused a heavy Huge Page copy-on-write storm that eventually resolved itself. . . 1 minute later. Yes, the same 1 minute during which both the latency spikes and the CPU throttling occurred! Fixing the benchmark network setup for the i9 system resolved the issue.
Lesson Learned
Just like those doctors described in the intro, I could have avoided my misdiagnosis had I just stuck to my established process. I instrumented my entire lab from the outset so I don’t even have to think about it. After every benchmark, I have all the error logs, network timestamps, and systems telemetry I need. But my prior experience made me skip steps, make assumptions, and ignore my hard earned lessons about correlation and causation.
You know who wouldn’t have skipped steps? Who wouldn’t have made assumptions? A novice Mark E. Dawson, Jr. An open, curious, more inquisitive version of myself. And therein lies the answer to combating this pitfall of experience – cultivating a Beginner’s Mindset.
Cultivating A Beginner’s Mindset
“If your mind is empty. . . it is open to everything. In the beginner’s mind there are many possibilities, but in the expert’s mind there are few.”
— Shunryu Suzuki
It’s a tough ask for someone who has spent years gaining valuable expertise and hard-earned experience to suddenly approach problems from the perspective of a novice. Luckily, we don’t have to struggle with this alone. A Google search for “beginner’s mindset” demonstrates a widespread understanding of its benefits, with prescriptive articles from areas as varied as Self-improvement to Business Management.
However disparate the domains from which the admonition stems, large agreement exists among them regarding steps to achieving that goal. We’ll first outline common advice between these writings before delving into more concrete tips applicable in Performance Engineering.
Achieving Beginner’s Mindset: General Tips
Among some of the better articles written on the subject1How to Cultivate Beginner’s Mind for a Fresh Perspective – https://mindfulambition.net/beginners-mind/27 Tips on How to Have a Beginner’s Mindset – https://www.vunela.com/7-tips-on-how-to-have-a-beginners-mindset/3A Beginner’s Mindset: Leading organizations in new directions – https://www2.deloitte.com/us/en/insights/topics/leadership/beginners-mindset-decision-making-for-leadership.html, the following list represents recurring concepts among all the tips given:
- Ask questions: What if our assumptions are wrong despite the best evidence at hand? What if solutions drawn from our past are no longer relevant? An excellent example of someone questioning prevailing wisdom/assumptions by running his own experiments is Travis Downs’ Deep Dive into Intel AVX/AVX-512-related Frequency Scaling.
- Go slowly: We tend to operate on autopilot in areas where we have the most knowledge and experience. This can take us out of the optimal discovery process and cause us to skip steps.
- Consider answers as a grey area instead of black and white: Dogmatic absolutism is the antithesis of open-minded curiosity. The most often uttered technical response from exemplars of the “beginner’s mindset” is “It depends.”
- Avoid pre-judgement: Can you really know how something will happen? Or are you too focused on how you believe things “should” work?
- Detach from “the expert” ego: Attachment to the “expert” identity traps us into offering answers before crafting questions.
Following these general tips can help any of us, no matter what our domain of specialty happens to be. But I publish this blog specifically for Performance Engineers. So, let’s talk about some actionable tips more pertinent to our everyday work lives.
Achieving Beginner’s Mindset: Performance Engineers
At the start of our careers, we may have learned our trade under the tutelage of a grizzled old mentor. Or by being thrown into the fire at one of those unscrupulous IT Contracting Sweat Shops. Or from spurts of self-directed learning regularly interrupted by tough work schedules. Maybe it was some combination of these and other means. And, in time, you may have ignored subtle gaps in your foundational understanding while, simultaneously, growing more rigid in your views. You may have been the recipient of Urban Myths handed down from ’80s era bearded UNIX gurus which you never thought to question. Oh, here’s something that’s much more likely: rapid advancement in technology obsoleted your old rules of thumb and mental models.
Cultivating a “beginner’s mindset” with the following tips can, therefore, do wonders for you as you grow in experience and expertise in Performance Engineering:
- Work with an intern: Few students learn the more specialized areas of IT at school. So, when they work with you, these novices tend to ask the kind of questions that may challenge your concept of areas you work with everyday. Can you explain the little details this inquisitive student has asked you about? Did the answer spawn more questions for both you *and* the intern? Congratulations, you’ve both entered the “beginner’s mindset”!
- Present on a Foundational Topic: This tip follows from the previous one. When crafting a presentation, you methodically build upon a series of ideas until you logically reach the main topic. This process forces you to consider the mind of the beginners in your audience. Also, during that process, you may encounter difficulty in explaining some prerequisite topic from your outline. It’s then that you remember that old saying, “If you can’t explain it simply then you don’t know it well enough.” I’ve spoken to many presenters who’ve stated that they learned as much delivering a Tech Talk as many of their listeners.
- View articles and videos targeted for beginners: I won’t lie to you and say that this is 100%. But I *will* tell you that there have been times I decided to watch an intro webinar about tools I’ve used all my career (e.g., Intel VTune, Linux perf, etc.) where I picked up something I never knew was available. Or times I’ve read a whitepaper explaining the inner workings of some area of the OS that I thought I already knew inside and out. Consider that the next time you skip past CppCon’s “Back To Basics” Track.
- Participate in Denis Bahkvalov’s Performance Contests: These contests are geared toward software developers who want to learn about coding with Mechanical Sympathy. “But I understand all those concepts already. Why would *I* wanna enter?” Umm. . . we’ve been chatting about this for 10 minutes already – do I really need to answer that for you now???
SIDENOTE: I met Denis from participating in a few of those Performance Tuning Contests. A year later we were collaborating on a book about performance analysis and tuning.4Paid affiliate link
Think Like A Beginner
It’s paradoxical advice, isn’t it? Thinking like a beginner to achieve mastery? But this “beginner’s mindset” would’ve saved that patient from much anguish and wasted time had her doctors realized it. It would’ve saved me a few hours and emails had I understood it. And my sincere hope is that it will remove the barriers from your imagination that your immense experience has walled you into over the years, so that the quality of your work brims with problem-solving prowess and engineering creativity.
- 1How to Cultivate Beginner’s Mind for a Fresh Perspective – https://mindfulambition.net/beginners-mind/
- 27 Tips on How to Have a Beginner’s Mindset – https://www.vunela.com/7-tips-on-how-to-have-a-beginners-mindset/
- 3A Beginner’s Mindset: Leading organizations in new directions – https://www2.deloitte.com/us/en/insights/topics/leadership/beginners-mindset-decision-making-for-leadership.html
- 4Paid affiliate link
