The System We Couldn't Patch
Notes on a single sign-on project, sixteen years later
I submitted my dissertation to Oxford in 2010 after 2 years of work on a single sign-on (SSO) project. Two financial firms merged in a 50/50 joint venture and owned many client-facing reporting portals. Each portal had a URL, a credential store, a support team, and a development team. Clients could tell we were still integrating because the legacy firm names were still in the URLs, and each application's legacy branding was without a logo.
It took us a year to replace all of that one by one with a single portal with one login and one identity. Roughly 30 engineers and about 10,000 pages of documentation, and a 41-hour deployment window that started at 6pm on a Friday and couldn’t be completed until the markets opened on Sunday afternoon.
I recently went back and read the dissertation. Most of the technology is already obsolete. We built a custom Tomcat valve because there was no connector. Oracle Access Manager 10g, Oracle Virtual Directory, WebSphere 6.1, the custom connectors we developed and used in that project would not be used today. The problem we solved has become a checkbox. Now, you connect with OIDC using your identity provider and configuring federation used to take 1 year and now takes a few minutes.
The hard core parts of the project were not as innovative as I thought they would be when I first read the dissertation. A few things in particular.
We customized the system into a corner
This is the example I would underscore to demonstrate platform integration work today.
Three legacy systems spanned three distinct stacks: WebSphere, IIS, and Tomcat. Each stack required a different integration approach. WebSphere took a Trust Association Interceptor that passed to an LTPA token. IIS took a header-based approach where the user ID of the portal rode on an HTTP header. Since Tomcat lacked a vendor connector, we created an implementation that read the SSO cookie and set the principal.
Each of those choices was correct in isolation. Collectively, they formed an integration that could not accept a vendor patch.
I said at the time that we did not identify this as a risk during our assessment, and that there was no real workaround for this. That is still true, and it is the most expensive statement in the document. We spent all those years unable to implement security updates to the identity layer of a financial institutional service without a regression cycle and no one wanted to sponsor that. My exit strategy was to remove all legacy apps, cut out the customizations, and return to the original SOL architecture.
The lesson isn’t not to customize. Sometimes there is no connector and we need to write the valve. The lesson is that integration customization should be scored as a recurring liability, not a one-time cost, and should be in the vendor risk assessment along with the questions about the vendor's financial situation. Ask what the integration seam you are proposing takes for a patch cycle. If no one can answer this, then you have found the risk.
2. A portal isn’t an application
We said that PORTAL was a single system. It was never a single system. It was an authentication and authorization system, a virtual directory that integrated incompatible identity stores, a provisioning and custom administration system, a landing page that computed per user link visibility, three custom connectors, an email notification system, and an integration layer. All of which needed to be maintained while the other systems continued to release.
The distinction is lost almost always, and it is almost always lost in a predictable direction. From the outside a portal appears to be a single URL, so it is estimated, staffed, and governed like a single system. The work that consumes the most time is the seam work, the integration of a unique identifier and the consolidation of accounts to ensure that the identifier is unique across all legacy systems.
That falls outside of any application team’s responsibility. That area exists only between teams, therefore, it gets taken care of only if someone claims that area between.
3. The most expensive person on the project was the one who didn't understand it
We got a project manager with zero experience of the SDLC, zero experience with identity management, and zero experience implementing SSO. The Purpose of it was a good one, as process and discipline is a good factor but subject matter expertise isn’t a given in a role.
Practically, the cost of bringing that one person to the level of understanding on how to scope the work, was never factored in, so the work was scoped by someone who couldn't yet see it. The final accounting on this came in at about double what was originally budgeted and several items from the original statement of work were quietly removed because the budget tracking system caught up to this last. My team picked up the slack on it later.
At this time, I would state that the PM on a technical program must be a subject matter expert. I would say now, that this is more like someone has to be in the room when the budget is being committed, and read the architecture and come to the opinion that a number is being stated that is completely unrealistic. That can be the program manager, and from my experience, it is the most financially responsible way to do this set up. It has to be someone. “The engineers know” is not an answer, because the engineers aren’t in the room saying what the budget is.
Chapter 6 of my dissertation was the only chapter I’d copy verbatim. It’s where I documented the issues. There were memory leaks we didn’t catch, which were due to Spring and Hibernate. There were vendors who refused to be in the same room to triage issues, which meant I had to escalate to the group CIO and the vendor engineers had to fly to Chicago to write a hot fix. There was the time a proxy setting was changed, years later, by the network team. That took the entire platform down (because by that time everything talked to everything else over web services and no one had modeled that dependency) It was the most modern problem I detailed in the document.
The failure came through a channel that no one “owned”.
The advice I would give to 2009 me comes down to something other than technology choices. The choice of using Oracle, Sun, or BMC was immaterial. The scores were 101, 98, and 74. The gap between the top two was noise covered in analysis. That advice would be to document that every custom integration is a patch you will not be able to update. Write it before you start. Make them sign it.
This dissertation, “The Integration of Heterogeneous Web Portals via Single Sign-On,” was submitted for the MSc Software Engineering at Kellogg College, Oxford, in October 2010. My degree was awarded on January 25th, 2011.