DevOps World Chicago 2023: Empowering Enterprise Engineers

I have a renewed sense of hope for the condition of the enterprise software developer.

For anyone familiar with me or my career, this is quite a statement. For the rest of you, to whom I’m some internet rando, I won’t bore you with more details about me than are absolutely necessary to understand my take. Suffice it to say that in 2017, I wrote a book whose central thesis was that software developers should initiate an exodus from the enterprise.

Fast forward 6 years to now, when I just spent the day as “press” at DevOps World, listening to a series of talks and interacting with participants and sponsors. Based on what I’ve seen here, my outlook for the future of enterprise developers is far less bleak than it was back then.

I’ll dive into the specifics of why, but the overarching theme is that it seems organizational change in the enterprise has evolved from a long history of something being done to software developers into something being done for them.

At this conference, I heard and saw a lot of innovation aimed at earning developer buy-in, relieving their toil, and sending them back to their various enterprises with pragmatic, actionable ways to improve their situations.

Where It Started: The Endless Agile Transformation

One more biographical detail, and then I’m done. I promise.

In the mid-aughts, starting about 10 years ago and running up until I wrote my book, I earned my living as a management consultant in software, almost exclusively in the enterprise. I specialized in static code analysis for strategic decisions, but often found myself in enterprises, helping them answer the basic question, “why is our agile transformation not working?”

Back then, DevOps was scarcely on the radar for enterprises. Instead, everyone was doing a so-called “agile transformation,” which generally meant having a different, Scrum-flavored series of meetings and changing very little else. These firms were usually in year 6 of a 2 year transformation, and were still working on the “define what agile means” OKR. The Bad Old Days of Shifting Burdens to the Left

If that sounds cynical, fair enough. But ask anyone in those orgs at the time, and they’ll almost certainly laugh ruefully and call it accurate.

One of the day’s speakers, Thomas Haver, referred to these sorts of institutions as “technology museums,” and I can’t think of a better turn of phrase. When you’re gluing Perl scripts to mainframes as part of some internal line of business initiative, you’re not using the latest and greatest, and you’re probably not knocking out that agile transformation any time soon.

But I think the central problem that engineering groups in these organizations encountered back then was neither the comically old technology nor the glacial pace of organizational change, per se. Rather, developers found themselves buried under the crushing weight of endless todo lists, as their employers heaped an ever-growing mountain of burdens on them.

Organizational Change Done To Developers

I can remember some interesting characters that I met during my consulting travels:

The “schema specialist,” who did nothing but review developer-proposed changes to any database, anywhere, and give them notes on what to do instead.

The “environment administrator,” who did nothing but review (and usually reject) requests to move JARs from one non-prod environment to another, and make sure the proper digital paperwork had been submitted.

The “integration architect,” whose contributions I never managed to figure out, but could reject seemingly anything that created an interface between two dev teams.

The “demo guy,” who engineers would have to work with to create power points for their sprint demos (instead of the customary “working software”).

I could go on, but I’m trying to make a point, not supply fodder for xkcd. And my point is that everything these organizations did heaped work on the engineers, including the creation of roles that seem like they should theoretically specialize work away from those same engineers.

To earn a living as an enterprise developer in the age of the agile transformation was to be an endless collector of bureaucratic toil. Write your software, submit compliant schema requests, fill out the JAR movement form in triplicate, please the integration architect, and make sure the power point guy knows that you’ve been delivering enough story points.

By the end of my consulting years, I hit peak cynicism. I thought developers should leave and not tolerate this situation, and I thought enterprises should give up on creating software and delegate that activity to vendors and future acquisition targets.

I’m pleased to see that things look different than when I left consulting 6 years ago.

Moving to Organizational Change For Developers

DevOps World had a common, refreshing theme. Tracy Bannon captured it well during a discussion panel: “The idea of shift left is not ‘dump it on the development team’.”

In opening remarks for the event, Cloudbees CEO Anuj Kapur pointed out that a poll of CloudBees customers found that developers only spend 30% of their time, well, developing. The rest is lost in some flavor of dark work or another.

Everyone at the event seemed to agree that this is a wholly unacceptable state of affairs, and that the path forward is one that involves a rethinking of what is asked of developers and what is done for them.

Removal of Toil

The first main event theme that I observed centered around eliminating toil. Rather than schema and integration specialists forcing checklists on the developers, conference participants explored the use of tooling and organizational tactics to eliminate as much toil as possible from the world of enterprise engineers.

In the conference keynote, “Go Big, Say Yes,” CloudBees announced product changes with a direct impact. They made CloudBeesCI highly available and highly scalable, meaning that organizations have immediate relief from problems such as bottlenecked jobs, needless time waiting for workspace downloads, and infrastructure-related build failures. All of this rolls up to a very simple value proposition: developers using their time instead of killing it, waiting for infrastructure.

When the engineers present saw the new Pipeline Explorer feature demonstrated, they burst into spontaneous applause, even though there was no break in the presentation. And the reason is obvious: they could see an end to loading some gigantic text file into an IDE so that they could search awkwardly for the reason a build bombed out. Pipeline Explorer lets them get there immediately, without the pain.

This idea wasn’t limited, either, to product announcements or even the talks. I watched the folks at Gradle, one of the sponsors, demonstrate a way to identify and troubleshoot flaky unit tests and the ability to use machine learning to prioritize executed tests based on changes to the codebase.

And Redgate did a demo of an offering that allowed source controlling database schema changes and keeping them consistent, in sync, and drift-free across environments. And all of this without a “schema specialist” in sight to scold them – just engineers safely changing the schema as needed.

Developer Buy-In

The engineer enablement theme wasn’t limited to tooling, either. One of the main takeaways that a recovering management consultant like myself couldn’t help but notice was the theme of getting buy-in to broader goals from the folks tasked with delivering them.

For instance, Katie Norton, a Senior Research Analyst at IDC, explained the concept of a software bill of materials (SBOM) and the broader concept of a software supply chain. Rather than a checklist of context-free “best practices,” this talk, aimed at an engineering audience, used practical analogies and diagrams to illustrate the challenges around and the gravity of managing the risk presented by using open source components.

This seems a lot better to me than how this risk was typically managed 10 years ago in enterprises: “don’t use open source.”

There was a talk that explained the idea of attestation and why it matters, encouraging teams to move compliance from a rote series of tasks to an objective, auditable and, most importantly, automated set of data. The mission was to avoid what presenter John Willis referred to as “governance theater,” wherein engineers would rename incidents to “cases” because of the reduced scrutiny that invited, or simply upload the same screenshot of a test run for 2 years to demonstrate that they had achieved code coverage.

Instead of tsk-tsking, this deceit was recounted with understanding. Of course teams did this when they were trying to ship features on time – it was the only way to get their work done. The developers here don’t need to change; the organization does.

Practical, Actionable Advice

Perhaps the most powerful motif of the event was that it distilled advanced practices and industry insights into easily actionable takeaways for participants.

For instance, speaker Julia Furst talked about the impact of generative AI on DevOps practices and suggested a series of tools that attendees could go and investigate. Ali Ravji and Mihir Vora from Capital One distilled hard-won experience building scalable CI/CD pipelines into concrete suggestions: focus on reusability, parallelization, and failure planning.

And, how’s this for practical and actionable? “Contribute to open source and adopt a plugin.”

Mark Waite, Senior Engineering Manager at CloudBees gave a really cool talk about how contributing to open source goes well beyond putting collateral good into the world and is actually good business. He relayed his experience building a CI solution for his XP team back in 2003, before Jenkins existed, and how that required a dedicated team member to maintain.

A much better solution, had it existed, would have been to contribute to Jenkins and use it. So he encouraged attendees to go contribute to open source to improve it, fix defects, and patch vulnerabilities that can help their organizations upstream, before things become expensive.

The Joy of Being Less Cynical

While I certainly enjoy a good, cathartic rant from time to time, it’s not exactly fun being cynical. Usually, it’s a sign that you need to take a vacation or start a different job.

So it was nice to circle back to the world of enterprise software development, after years away from it, and find it far more promising than I remember. It was nice to see an enterprise less focused on crushing engineers under the weight of endless compliance tasks and more focused on helping them ease that burden.

If scheme specialists and build administrators must exist, at least they can do so in supporting roles and with powerful tools, instead of the promise of toil for their colleagues. DevOps World was a fun event, sure, but it was also a philosophical palette cleanser for me.

The next stop on the tour was in Silicon Valley October 18th and 19th, check out what happened there!