Delegates at the recent DevOps Summit from V3's sister site Computing were treated to a story about developers at one organisation being given cricket bats with which to threaten testers, so that their code would get into production faster.
But more of that later. Before he got to that anecdote, Andrew Fullen, solution director at Sogeti UK explained that when trying to implement a DevOps culture within an organisation, it's important to test at every point in the lifecycle.
He described the philosophy as "contiunuous testing".
"Testing fits in every point in the lifecycle," Fullen began. "Every time you make a decision you need to prove it's working."
That testing doesn't need to be onerous, Fullen added, but he explained that it does need to be used, validated, recorded and auditable, and that organisations need some form of automated testing at every point.
"It needs to be part of the fabric of your entire structure. Start adding tools piecemeal, and you'll get into trouble," he said.
He then described his vision of what an organisation with a strong record of continuous testing would look like.
"Every time you make a decision, before you put any code together, you capture the idea and the requirements, and there should be no differentiation between functional and non-functional testing. If something's secure, that's an important functional requirement. Is it learnable, useable, can people with poor eyesight use it? These are all important functional requirements," said Fullen.
But the problem, he continued, is time. Fullen explained that in the past, organisations used to run projects over longer periods than today, with even two to three year projects reasonably common.
"Now you have one or two week sprints, and in the future this idea of continuous delivery will kick off more, or you could even take the Amazon model of releasing every 13 seconds. Not all of those changes are good though. Anyone using AWS recently saw that they can have issues too, despite all their automation and checks."
Fullen then gave an example of a firm he worked with in the past, where the senior stakeholders decided to automate everything.
"Requirments gathering, approval, coding, security checks, repositories, backups, APIs, everything was automated," he said. "You name it, if there was a tool they bought it, if there wasn't they wrote it. It was brilliant, everything was automated, and it all ran on this lovely rig which was a phenomenol sight with these flashing and blinking lights."
But whilst many aspects of this system were impressive, there were problems too, as he described.
"Then our old friend time came back to haunt us. The weekly build started on a Friday night, and finished at 8am on a Monday morning. Then the testers would come in and spend two weeks working out what actually happened over the weekend. There were hundreds of thousands of log files, all in different formats, created by different tools, and with no centralised reporting because that was pretty much the only thing which hadn't been automated. And some of the most critical things were left until last in the automation chain."
He explained that developers were measuered on how quickly they made changes and got them out into the live environment. To this end, according to Fullen, management went so far as to buy cricket bats for those teams so they could threaten, one hopes jokingly, the testers, and get them to rush their work so as not to delay the code.
"The developers and testers were mostly married to one another too, so home time wasn't a good situation either!" he joked.
If anything went wrong with a release, it would take two weeks to find the log file which would enable them to find and fix the problem, and with malfunctioning code in the live environment, obviously further releases would be delayed.
"We decided to take time out," said Fullen. "We did a two day off-site meeting to work out where we'd gone wrong and what we could do to sort it out. We spent all day breaking it all down into simple problem statements, sticking them all on a wall and working out what was important and when it needed to run."
[Turn to next page]
Instapaper to 'go dark' in Europe until it can work out GDPR compliance
James Robbins of ArrowXL says that AI is no longer 'tomorrow's technology'
Staff told to beware of "unusual sounds" after an employee reported mystery symptoms
Sophisticated malware comprises code previously used to attack Ukraine