Are You Shifting Left Enough?

Estimated reading time: 5 minutes

I looked Death in the face years back

I held tears back

I gathered myself and stared back

I’m from where you don’t crack, the weak don’t live

You gotta bounce back, homie, the streets don’t give

— Jay-Z (Hip-Hop Billionaire – debut album released at age 26)

“I was 17 and already I’d become a joke to some people. Everyone recognized me so I couldn’t just get a job at Blockbuster. It was a very difficult time.”

— Corey Feldman (embattled former child star – famous since age 3)

History is replete with cautionary tales of kids who were born with such undeniable star quality that Show Business couldn’t wait to get its hands on them, hastily shoving those callow talents into the spotlight only to chew them up and spit them out once time had proven they were too immature and inexperienced to withstand its glare. Bucking that trend, Jay-Z had nearly lived an entire lifetime by the time he acquired his first taste of stardom, not releasing his first album (“Reasonable Doubt”) until the age of 26 – an old man in Hip-Hop Years. He’d been tested all his life, navigating the perils of growing up in the streets of ‘80s Brooklyn to beat the odds and reach adulthood virtually unscathed yet battle-hardened to anything the music biz or super-stardom could throw at him. He’s now a mogul and half of a “Billionaire Net Worth” Power Couple.

Likewise, GitHub and your own internally managed Source Code Control System (SCCM) are full of promising software borne of elbow grease and ingenuity. You get excited after the success of a Proof-of-Concept (PoC). Your confidence grows with every PASS result from a Unit, Regression, or Functional Test. But thrusting that application into the Production spotlight w/o vetting its ability to withstand a Production workload is a recipe for disaster that risks an infrastructure full of “strung out, has-been” software. That’s where Performance Testing comes into play – Performance Testing puts an application through its paces and assesses its readiness for the Production “streets [that] don’t give”, where “weak [software] don’t live.”

Last Stop – CI/CD Pipeline?

Up until this point, industry focus has been honed primarily on Shift-Left processes, whereby non-functional testing is integrated earlier in the Software Development Life Cycle (SDLC) by means of automated Performance Unit Tests and Regression Load Testing. This is in contrast to the role Performance Testing historically has played as the final checklist item before Production deployment. But is even THAT too late in the game? Are we shifting left enough? Could a seasoned Performance Professional make an even bigger impact at the Architectural and Software Design Phases?

Performance Engineers earn battle scars over decades mining log files, analyzing workload characteristics, and designing relevant load test scripts. Modeling application performance and correlating reams of data. Alerting software development teams to root cause issues and advising on remediation. They’ve seen all sorts design patterns across dozens of client engagements and have become familiar with their performance implications.

Eventually, probable bottleneck sources becomes predictable just by reviewing the software architecture at the outset of a project. In fact, a background in software development is not even necessary for this level of insight. All that is required is the one skill essential for any Performance Engineer worth his salt – pattern recognition. Therefore, Performance Professionals must shift their efforts even further left into Architecture/Design phases of IT projects.

Parallels in IT Security

Organizations first stumbled upon this realization in the area of IT Security. Years ago, companies regarded Security as some add-on feature:

“But wait – there’s more! If you’re one of the next 5 callers, we’ll throw in the Security attachment for free! Operators are standing by. . .”

Soon it became clear that plugging security holes as an after-thought to backlash from publicized breaches would no longer fly. Security Architects, then, became fixtures in project Design Phases. Likewise, perennial Black Friday/Cyber Monday headlines detailing embarrassing outages at online retailers continue to underscore the importance of performance consideration earlier in the Design phase, as well (check out the PerfBytes News of the Damned Podcast for entertaining commentary and review of lessons learned from such incidents). How much public embarrassment and refactoring effort could be saved if a Performance Professional is included in Architecture/Design phases?

Performance Design Antipatterns

Software Developers have long studied Design Patterns (popularized by the Gang-Of-Four authors of the book Design Patterns: Elements of Reusable Object-Oriented Software) and Anti-patterns. Between 2000 – 2003, Connie U. Smith and Lloyd G. Williams augmented the existing research in technical papers cataloguing a list of Patterns and Anti-patterns in Software Performance Design, each of which describes common performance anti-patterns such as The God Class, Tower of Babel, Falling Dominoes, Sisyphus Database Retrieval, Circuitous Treasure Hunt, etc., while providing solution design patterns for each. A library of these Software Performance Patterns and Anti-patterns would prove invaluable to a Design Team.

Shift Further Left

Everyone realizes that leaving performance considerations to one End-To-End Load Testing Phase before Production deployment won’t cut it anymore. It’s time to Shift-Left. Just make sure you’re Shifting Left enough.

CAMPAIGN 2020: Less Corey Feldman – More Jigga Man.

[1] Software Performance AntiPatterns – https://www.perfeng.com/papers/antipat.pdf

[2] New Software Performance AntiPatterns: More Ways to Shoot Yourself in the Foot – https://www.spe-ed.com/papers/moreanti.pdf

[3] More New Software Performance AntiPatterns: Even More Ways to Shoot Yourself in the Foot – https://spe-ed.com/classic-site/papers/evenmore.pdf