Agile DevOps
My Team journey.
The Beginning
In October 2019, after about a year and half into my Product Manager adventure in Deloitte Global Technology, my Boss - Carolyne (London, Canada) and her Boss Mike (Glen Mills, US) decided to schedule a call with me (Potterspury, Northampton Shire, UK) at 6:00 CET. I was eager and exited to hear about my mid-pandemic salary review, but to my surprise I was asked, if I am ready for a challenge...
Leadership discussions acknowledged the complexity and priority Deloitte Technology Infrastructure development program, recognizing that it required a dedicated focus. To address this and ensure adequate support for Deloitte's infrastructure teams, the decision was made to create a new DevOps Team.
I was entrusted with the responsibility of organizing and spearheading this initiative, leveraging my expertise as a Product Manager and Agile Product Owner. My role encompassed not only building the new team but also overseeing its integration into the existing framework. I ensured that the team embraced Agile principles, adhered to Scrum methodology, and followed the DevOps cycle to facilitate seamless product delivery.
Customers and Service Scope
Names:
Various departments and teams in Deloitte Technology; various teams in Deloitte member firms world wide.
Among others:
Firewall teams world wide
Architecture teams world wide
Cloud provisioning and support teams (Azure, AWS)
Identity and Access Management teams world wide
integration with 17 member firms - Splunk
Integration with 6 global monitoring systems - 22 member firms
Asking for:
new API integrations with ServiceNow
DevOps tasks for 200+ of workflows (Web Applications, Websites with Forms, automation workflows)
Interesting fact:
Within Deloitte Global, ServiceNow offered users approximately 600 workflows. This placed a significant responsibility on the DTI ServiceNow Team, who managed over 30% of the total workflows.
Notably, just 200 of these supported workflows handled a staggering 30,000 requests annually, representing roughly 90% of all requests processed through ServiceNow globally.
SaaS DevOps Team
Name:
Deloitte Technology and Infrastructure - ServiceNow (DTI-ServiceNow)
Roles and Locations:
4 Developers (3 India, 1 UK), 1 Quality Analyst (India), 1 Business Analyst/Product Owner (India), 1 Product Manager (UK)
Reporting to:
1 Director (USA), 2 People Managers in India (legal requirements - non delivery)
Interesting fact:
The United Kingdom is 5 hours ahead of Glen Mills, Pennsylvania - which is 9.5 hours behind New Delhi, India...
Initial Challenges and Issues
'What is a point of joining this call, if your Team will not be able to deliver it anyway...?' - very frustrated Customer
'I feel trapped, if I will quit as a junior after 2 years here- it means I've waisted this time... but staying with no help to progress - has no sense as well...' - desperate Team Member
Backlog
Inherited projects and tasks:
Firewall Configuration Change Request - consolidating firewall change management workflow globally; High Priority, Complexity and Urgency
Common SSDLC workflow - introduce consolidated way in architecting applications world wide. High Priority and Complexity
Integration with 17 instances of 3rd party software e.g. Splunk - to consolidate incident management world wide. Ongoing and resource consuming.
Approx. 90 Agile Points worth of smaller requests - some 1 year old...
Customers
Customer experience:
Extremally dissatisfied due to conflicting priorities.
Distrust and low expectations prevailed as customers were unable to rely on agreed delivery dates, among other factors.
Poor understanding of ServiceNow capabilities.
Low level of Agile/SCRUM adoption and understanding.
Poor or non-existent communication channels and rituals.
Very low quality of service ownership mapping.
The Crew
Impact on the Team:
Chaotic changes of priorities, schedules and deliverables caused low Team morale.
Development and delivery outside Agile/SCRUM processes - 'who shouts the most, gets it'
New Team overwhelmed with high and complex workload.
Mediocre Agile/SCRUM adoption - no agile points, limited UAT, among other factors.
High resource rotation on a very competitive employment market (India).
Skill gaps not addressed and unclear career advancement capabilities.
Solutions and Changes
'Thank you for the session today, it was very informative. I will make sure that the rest of my Team will attend the second one.' - happy Manager
'It is good that we can focus on development now. We have learned new things and it is fun to use it in practice.' - happy Developer
Agile DevOps
Split projects into delivery phases and manageable chunks.
Use Agile/SAFE to optimize design, signoff, sprint planning and development cycle.
Use data analysis to initiate service review - from 200 workflows, identified 40 responsible for 90% of user engagement (impact).
Follow up with extensive ownership review and stakeholders communication refresh.
Created Customer friendly dashboards to present backlog, workload and channel an open and honest communication.
Introduced automated testing for selected application.
Transparency and Reliability
Established clear and transparent ways to communicate progress and delays.
Introduced familiarization workshops about ServiceNow capabilities, Agile/SAFE/SCRUM, dashboards, UAT requirements etc.
Utilized retrospection activities to focus on CSI and to reduce impactful priority service failures.
Always keep promises, if not possible, then never 'leave Customer in the dark'.
Continuously engage service owners and decision makers. We are all in it together.
The Happy and Efficient Crew
Combined Agile, SCURM and DevOps cycle and stick to it. Agile points rules!
Shielded developers from direct access by Customers. Escalate to PM/BA immediately.
Introduced 3 golden rules:
Mistakes are good as long as we learn from them.
Do not keep quiet, if you in trouble or delayed - we are all here for each other.
Whatever impacts the agreed dates and deliverables - Customer must know (PM/BA)
Initiated skill-share and training programs to fill the gaps.
Invested in career development support and helped to identify clear progression paths for Team members.
Outcomes and Lessons Learned
Why this Team can deliver in 2-3 weeks, and on a specific date? Other Teams keep telling us that they have 90 days delivery target. - curious Director
Thank you for creating this opportunity and helping me to believe I can do it! - QA after She was promoted and become an excellent BA
Science and technology
Using known methodologies along with innovative technologies helps to limit the cost/time and to optimize product delivery.
Thanks to automation, data analysis and visualization we can focus on design and development.
Steady delivery of 40-50 Agile points per month.
Yearly reduction of development defects by 70% and UAT defects by 90%.
Trust is everything
Focus on communication and education helped Customers to understand design, development, testing and deployment procedures and limitations. It helped to optimize each step, reduce frustration and build trust.
Having stakeholders prepared and more knowledgeable after each co-delivery makes a huge positive impact on product quality.
Yearly reduction of production high impact incidents by 50%.
We win as a Team!
Prioritization is a key to manage rapid and successful development.
Understanding own mistakes is a key to continues improvement.
Best way to measure of success is to see the growth of People around you.
1 QA becomes BA and promoted
1 Apprentice becomes QA (permanent)
1 Senior Developer promoted to Manager