Recently I have been thinking and writing about corporate governance failings at the Post Office during the two decades of the Post Office Horizon scandal. Having worked in software development, testing and IT audit I have experience that is relevant to several aspects of the scandal. I have a further slice of experience I have not yet commented on publicly. That is largely because I should not talk about experiences with clients when I worked for IBM. However, I have decided to break that rule, and I feel justified for two reasons. Firstly, I think it offers a useful insight into failings at the Post Office and Fujitsu. Secondly, my clients all set, and met, a far higher standard than we have seen in the long-running Horizon scandal. Nothing I write will embarrass them or IBM, quite the opposite.
I keep going back to the management letter, [PDF, opens in new tab] issued by Ernst & Young (E&Y), the Post Office’s external auditors, following the 2011 audit. The letter was commented on in the Horizon Issues court case, Bates v Post Office Ltd (No 6: Horizon Issues), [PDF, opens in new tab].
To normal people this 43 page letter is incomprehensible and boring. It lists a series of major and minor problems with Fujitsu’s management of the IT service it provided to the Post Office. Only people who have worked in this field will feel comfortable interpreting the letter and its significance.
The letter draws attention to problems that E&Y came across in the course of their audit. As the introduction says.
“Our review of the company’s systems of internal control is carried out to help us express an opinion on the accounts of the company as a whole. This work is not primarily directed towards the discovery of weaknesses, the detection of fraud or other irregularities (other than those which would influence us in forming that opinion) and should not, therefore, be relied upon to show that no other weaknesses exist or areas require attention. Accordingly, the comments in this letter refer only to those matters that have come to our attention during the course of our normal audit work and do not attempt to indicate all possible improvements that a special review might develop.
E&Y did not conduct a full technical audit. They were concerned with assessing whether the financial accounts offered a true and fair view of the financial position of the company. Their assessment of internal control was only sufficiently detailed to allow them to form an opinion on the company accounts.
It is, or it should be, monumentally embarrassing for the internal auditors if the external auditors find long-standing control problems. The internal auditors should have the staff, expertise and time to detect these problems and ensure that they are resolved long before external auditors spot them. The external auditors are around for only a few weeks or months, and it is not their primary responsibility to find problems like this. I wrote about this from the perspective of an IT auditor last year (see section “Superusers going ‘off piste'”).
The specific issue in the management letter that rightly attracted most attention in the Horizon Issues’ case was the poor control over user IDs with high privilege levels. Not only did this highlight the need to improve Fujitu’s management of the IT service and the oversight provided by the Post Office, it also pointed to an ineffective internal audit function at the Post Office, and previously the Royal Mail before the Post Office was hived off.
When I was reading throught the E&Y management letter I was struck by how familiar the problems were. When I worked for IBM I spent three years as an information security manager. My background had been in software development, testing and IT audit. The contract on which I was working was winding down and one day my phone rang and I was made an interesting offer. Service Delivery Security wanted another information security manager to work with new outsourced accounts. My background showed I had a grasp of security issues, the ability to run projects, and a track record of working with clients without triggering unseemly brawls or litigation. So I was a plausible candidate. I would rely on the deeply technical experts and make sure that IBM and the client got what they wanted.
The job entailed working with the client right at the start of the outsourcing deal, for a few months either side of the cutover. An important responsibility was reaching agreement with the client about the detail of what IBM would provide.
All the issues relating to privileged access raised by E&Y in their management letter were within my remit. The others, mainly change management, were dealt with by the relevant experts. Each outsourcing contract required us to reach agreement on the full detail of the service by a set date, typically within a few months of the service cutover. In one case we had to reach agreement before service even started. On the service cutover date all staff transferring to IBM were required to continue working to exactly the same processes and standards until they were told to do something new.
I had to set up a series of meetings and workshops with the client and work through the detail of the security service. We would agree all the tedious but vital details; password lengths and formats, the processes required for authorising and reviewing new accounts and access privileges, logging and review of accesses, security incident response actions. It went on and on.
For each item we would document the IBM recommended action or setting. Alongside that we had to record what the client was currently doing. Finally we would agree the client’s requirement for the future service. If the future requirement entailed work by IBM to improve on what the client was currently doing that would entail a charge. If the client wanted something lower than the IBM recommendation then it was important that we had evidence that IBM was required to do something we would usually regard as unsatisfactory. This happened only rarely, and with good reason. The typical reason was that the client’s business meant the risk did not justify the tighter, and more expensive, control.
We also had to ensure that all the mainframe systems and servers were inventoried, and the settings documented. That was a huge job, but I farmed that out to the unenthusiastic platform experts. For all these platforms and settings we also had to agree how they should be configured in future.
The next step, and my final involvement, would be to set up a project plan to make all the changes required to bring the service up to the standard that the client needed. A new project manager would come in to run that boring project.
After three clients I felt I had learned a lot but staying in the job was going to mean endless repetition of very similar assignments. I also had some disagreements about IBM’s approach to outsourcing security services that meant I was unlikely to get promoted. I was doing a very good job at my current level and it was clearly recognised that I would only cause trouble if I were given more power! It’s true. I would have done. So I secured a move back to test management.
I enjoyed those three years because it gave me the chance to work with some very interesting clients. These were big, blue chip names; AstraZeneca, Boots (the UK retailer), and Nokia (when they were utterly dominant in the mobile phone market). I don’t have any qualms about naming these clients because they were all very thorough, professional and responsible.
The contrast with the Post Office and Fujitsu is striking. Fujitsu won the Post Office outsourcing contract [PDF, opens in new tab] in 1996 for an initial eight years. Yet, 15 years later, by which time the contract had been extended twice, E&Y reported that Fujitsu had not set up the control regime IBM demanded we create, with client agreement, at the very start of an outsourcing contract. The problems had still not been fully resolved by 2015.
Getting these basics correct is vital if corporations want to show that they are in control of their systems. If users have high privilege levels without effective authorisation, logging and monitoring then the corporation cannot have confidence in its data, which can be changed without permission and without a record of who took what action. Nobody can have confidence in the integrity of the systems. That has clear implications for the Horizon scandal. The Post Office insisted that Horizon was reliable when the reality was that Fujitsu did not apply the controls to justify that confidence.
Fujitsu may have failed to manage the service properly, but the Post Office is equally culpable. Outsourcing an IT service is not a matter of handing over responsibility then forgetting about it. The service has to be specified precisely then monitored carefully and constantly.
Why were the two corporations so incompetent and so negligent for so long? Why were the Post Office and Fujitsu so much less responsible and careful than IBM, AstraZeneca, Boots and Nokia?
Why did the Royal Mail’s and subsequently the Post Office’s internal auditors not detect problems with the outsourced service and force through an effective response?
When I became an information security manager I was told a major reason we had to tie the service down tightly was in case we ended up in court. We had to be able to demonstrate that we were in control of the systems, that we could prove the integrity of the data and the processing. So why did Fujitsu and the Post Office choose not to act as responsibly?
I was working in a well-trodden field. None of the issues we were dealing with were remotely new. The appropriate responses were very familiar. They were the mundane basics that every company using IT has to get right. Lay observers might think that the outsourcing arrangement was responsible for the failure of management control by distancing user management from the service providers. That would be wrong. The slackness seen at Fujitsu is more likely to occur in an in-house operation that has grown and evolved gradually. An outsourcing agreement should mean that everything is tied down precisely, and that was my experience.
I have worked as an IT auditor, and I have been an information security manager on big outsourcing contracts. I know how these jobs should be done and it amazes me to see that one of our major rivals was able to get away with such shoddy practices at the very time I was in the outsourcing game. Fujitsu still has the Post Office contract. That is astonishing.