DevOps Days Detroit Day 2: Handling Disasters and Rollouts
By Laura Cowan
Laura K. Cowan is a tech editor and journalist whose work has focused on promoting sustainability initiatives for automotive, green tech, and conscious living media outlets.
George Miranda talks about handling crisis from a devops perspective.
It's day 2 of DevOps Days Detroit at CCS overlooking the downtown skyline of Detroit from New Center, and once again the theme is trending toward a common challenge for devops teams: handling disasters and rollouts. Devops folks are no strangers to stress, and the speaker lineup this morning covered a number of recognizable topics to people on teams who have been faced with an onslaught of challenges.
George Miranda, Community Advocate at PagerDuty and author of the O'Reilly book The Service Mesh: Resilient Service-to-Service Communication for Cloud Native Applications, kicked things off with a talk on "The Perfect Storm - How We Talk About Disasters." "It's been mind-blowing to see other parts of our organization peeking in on parts of engineering," Miranda said of PR folks who wanted to learn about how devops handles crisis incidents and how blameless post mortems have helped PagerDuty teams learn from incidents after the fact.
"As a result, I've learned a lot about the PR process as well," he said of another department that has best practices for handling crisis. Miranda emphasized that building bridges between silos helps build empathy for other areas of deep expertise in an organization, which allows for future collaboration and learning.
The downtown Detroit skyline east of CCS at New Center on a Thursday morning in Detroit. Manholes steaming already as fall sets in.
DevOps Crisis Response Resources
Miranda listed a number of resources PagerDuty has developed for devops folks looking to learn more about how the company has developed response policies for post mortems and more. We offer them here so you can dig in:
business-response.pagerduty.com for Business Incident Response
pagerduty.com/ops-guides Ops Guides for Diving Deep
response.pagerduty.com Before, During & After Incidents
postmortems.pagerduty.com Post Mortem Process
Complexity & Emergent Behaviors
Seattle-based Ramin Keene spoke on day 2 of DevOps Days Detroit on "Production as an Experiment Lab."
"All software comes down to complexity and change," he said. "Change is this: I'm about to break this." The audience chuckled nervously.
"Complexity is not something to be avoided," Ramin continued, "but rather it is the side effect of business success."
Ramin Keene speaks about complexity and change and how they are emergent characteristics of business growth as well as how to handle them from a devops perspective.
DevOps Recruiting Trends & Expansions
We spoke with a number of sponsors and exhibitors at DevOps Days who were there either to recruit DevOps talent or move into a new market in Detroit, which is now being widely recognized as a hot emerging market for tech. Instead of thinking of it as cutting edge knowledge, many team leaders present acknowledged it as common knowledge these days within their space and expressed some frustration or amusement over the fact that Detroit and Ann Arbor in particular have been so slow to gain visibility as emerging tech hubs for software and devops talent in particular.
We asked several of these sponsors about their reasons for expanding into the Detroit market. Answers varied from general expansion across the United States and the world to needing embedded talent in Detroit because of connections with tech-related industries in the city. An attendee we spoke with was there from a major automotive OEM learning the ropes on devops to help a team handle processing massive amounts of data.
Sponsors present included Detroit unicorn StockX, whose recruiting team told us about how the company has expanded in the last year, and how their verification process works for assuring that the luxury goods and sneakers sold on the StockX marketplace are verified quality. "Sellers agree to sell a product on StockX for a particular price," recruiting team member Jeff told us, "and then they ship sneakers to a verification center, where employees check everything from the authenticity and scent of a high-end sneaker or other luxury item to the original packaging before shipping to buyers." We have recently started hearing other local companies refer to being the "StockX of Y Market" so were very interested to hear about the company's continued establishment in this space as Detroit's hottest startup.
Speaker Quintessence Anx stands in front of the throng getting lunch as she sits with the AppDynamics people, a Detroit embedded company purchased last year by Cisco. "We kept our culture," AppDynamics sales rep Ghirag Desai told us of the successful acquisition.
Other sponsors at DevOps Days were more in the open source software space, and told us they were in Detroit because their companies were either expanding, looking for more talent, or integrating Detroit offices into national teams.
"I was there during the Duo exit," Lightstep employee Jay Wallace now of Austin told us of his history working with Detroit and Ann Arbor tech companies. "Why are we here now? Think about what Detroit is doing. I saw people from Duo sprinkled into really good companies around Ann Arbor like Censys. There's a lot of influx of tech here. The market's really hot. When I developed the strategy to go to market, Detroit was a top 4 market to get into only behind Austin, Dallas and other major metro areas. We're paying attention to companies here that aren't tech focused but are heavily into devops focus, like Little Caesars."
DevOps organizer and elusive Cronicle co-founder Archie Cowan MCs ceremonies at DevOps Days Detroit 2019.
Open Spaces at DevOps Days
One of the favorite features of attendees at DevOps Days Detroit is the Open Spaces section of the afternoon, where the audience pitches ideas for devops-related subjects they would like to discuss in breakout groups.
DevOps Days Detroit organizer Rex Roof told us how it works.
"There's no set agenda," he said. "We poll the audience for topics, then split them into groups to go into separate rooms."
This is one of the most popular features of the smallish DevOps Days conferences because it splits a small conference into 7 tracks for the afternoon but is still small enough for everyone to be involved and connect if they want to. It resemble some larger tech/admin-oriented and devops conferences out in Silicon Valley before they grew into much larger expo events. There is still a very real possibility of connecting with anyone you want at DevOps Days Detroit, particularly in these Open Spaces groups where people dive deep into technical issues on their mind.
A bust of Harley Earl of General Motors fame flanks the room backstage at DevOps Days Detroit, which was hosted at the College for Creative Studies on the 11th floor of the Taubman Center. Photos of classic Cadillacs line the walls.
Rollouts & Handling Data
In the afternoon, Dan Garfield, full-stack engineer, Google Developer Expert, and member of the Forbes Technology Council, started off the afternoon with a speech on "Visualizing Canary Rollouts with Istio and Helm".
Finally, Technical Director at Endgame Inc. Steve Ross gave a talk about about accidental DataOps: "How we matured our organization with data, and how you can too!"
It's been a whirlwind couple of days, and remember you can check out the full set of talks on Youtube once they are processed by DevOps for rollout, though video folks tell us that won't be likely for another 2-3 weeks. Check back to see when the Detroit 2019 video goes live here.
We will be following up with sponsors and speakers alike to bring you more behind the scenes perspectives from devops folks around the Great Lakes region and from Philadelphia to Seattle and Silicon Valley as we have more time to talk after the event. Stay tuned for interviews with the speakers on their work in devops, community trends, emerging markets, and more.
Thanks, DevOps Days and Detroit, for a great week.