The System We Couldn't Patch
Notes on a single sign-on project, sixteen years later
In 2010 I submitted my dissertation to Oxford about a single sign-on platform I’d spent the previous two years building. The project was real: two financial firms had merged in a 50/50 joint venture, and between them they owned several dozen client-facing reporting portals — each with its own URL, its own credential store, its own support group, and its own development team. Clients could tell we hadn’t finished integrating because the legacy firm names were still in the URLs, and sans logo, the legacy branding of each app.
We spent a years replacing that with one portal, one login, and one identity. Thirty-odd engineers, roughly ten thousand pages of documentation, and a 41-hour deployment window that started at 6pm on a Friday and had to be finished before the markets opened Sunday afternoon.
I reread the whole thing recently. Most of the technology in it is dead. Oracle Access Manager 10g, Oracle Virtual Directory, WebSphere 6.1, the custom Tomcat valve we wrote because no connector existed — none of that is anything you’d build on today. The problem it solved has largely become a checkbox. You get OIDC from your identity provider, wire it up in an afternoon, and federation that took us a year of vendor negotiation is now a config file.
What surprised me is how little the hard parts have changed. Three things in particular.
1. We customised the system into a corner
This is the one I’d underline for anyone doing platform integration work today.
The three legacy applications ran on three different stacks — WebSphere, IIS, and Tomcat — and each needed a different integration approach. WebSphere got a Trust Association Interceptor that handed off to an LTPA token. IIS got a header-variable approach where the portal user ID rode in on an HTTP header. Tomcat had no vendor connector at all, so we wrote our own valve to read the SSO cookie and set the principal.
Each of those decisions was correct in isolation. Together they produced a system that could not take a vendor patch.
I wrote at the time that we hadn’t identified this as a risk in the original assessment and that there was no real mitigation for it. That’s still true, and it’s the most expensive sentence in the document. We spent the following years unable to apply security updates to the identity layer of a financial institution without a regression cycle that nobody wanted to fund. My exit plan was to decommission the last legacy application, strip the customisations, and get back to the architecture I’d originally drawn.
The lesson isn’t “don’t customise.” Sometimes there’s no connector and you write the valve. The lesson is that integration customisation should be scored as a recurring liability, not a one-time cost, and it belongs in the vendor risk assessment next to the questions about the vendor’s financial health. Ask what a patch cycle looks like after every integration seam you’re proposing. If nobody can answer, you’ve found the risk.
2. A portal is not an application
We described PORTAL as a single system and it was never a single system. It was an authentication layer, an authorisation layer, a virtual directory stitching numerous incompatible identity stores together, a provisioning workflow engine, a custom administration tool, a landing page that computed per-user link visibility, three bespoke connectors, and an email notification system — all of which had to stay coherent while the underlying applications kept shipping their own changes.
This distinction gets lost constantly, and it gets lost in a predictable direction. From the outside a portal looks like one URL, so it gets estimated, staffed, and governed like one application. The work that actually consumes the time is the seam work: the mapping between the portal ID and every application-specific ID, the account consolidation logic that had to guarantee a chosen username was unique not just in the new directory but in every legacy store behind it, the single logout that had to reach across domains.
None of that lives inside any one application team’s scope. It only exists between them, which means it only gets done if somebody owns the between.
3. The most expensive person on the project was the one who didn’t understand it
The PMO assigned us a project manager with no background in the SDLC, no exposure to identity management, and no prior SSO implementation. The intent was reasonable — process discipline is a real contribution, and subject matter expertise isn’t automatically part of the job.
In practice, the cost of getting one person to the point where they could scope the work was never budgeted, so the work was scoped by someone who couldn’t yet see it. The final cost came in at roughly double the original estimate, and several items from the original statement of work were quietly dropped because the budget tracking hadn’t caught the drift in time. My team absorbed them later.
I argued then that the PM on a technical program must be a subject matter expert. I’d soften that now — what’s actually required is that someone with decision-making authority can read the architecture and tell when an estimate is fictional. That can be the program manager, and in my experience it’s cheapest when it is. But it has to be somebody, and “the engineers know” is not an answer, because the engineers aren’t in the room where the number gets committed.
The part that aged best
Chapter 6 of the dissertation is where I wrote down what went wrong, and it’s the only chapter I’d keep verbatim. The memory leaks we didn’t catch in testing and eventually traced to Spring and Hibernate. The vendors who wouldn’t sit in the same room to triage a problem, which ended with me escalating to the group CIO and vendors flying engineers to Chicago to write a hot fix. The network team changing a proxy setting years later and taking the entire platform down, because by then everything talked to everything else over web services and nobody had modelled that dependency.
That last one is the most modern problem in the document. We’d built a distributed system without calling it one, and the failure arrived through a channel nobody owned.
If I could send one note back to 2009, it wouldn’t be about the technology choice. Oracle versus Sun versus BMC mattered far less than the scorecard suggested — the totals came out 101, 98, and 74, and the gap between the top two was noise dressed up as analysis. The note would say: every integration you customise is a patch you won’t be able to apply. Write that down before you start, and make someone sign it.
The original dissertation, “The Integration of Heterogeneous Web Portals via Single Sign-On,” was submitted for an MSc in Software Engineering at Kellogg College, Oxford, in October 2010. My degree was awarded a year later on January 25th, 2011