API Design: Planned, Unplanned, Security and Utter Chaos

I’ve discussed APIs and API Management in previous blog posts. These are among my older blog posts, but the points being made are, generally, still relevant. I’ve never published a blog post exclusively about API design. I have a blog post about “API Gateways and Multiple Consumer Types” where I explore one aspect of API design already. At this point, much has been published on the topic, but there are a few points to be made.

One should always design their API interfaces upfront. This is called a design-first or Top-Down API design methodology. The opposite of this approach where one writes code and then generates an interface (OpenAPI, etc) from the code is called Bottom-Up API Design.

Historically, for projects I’ve been involved in, the Top-Down approach meant using Swagger (or, more recently, OpenAPI) documents to document what the API should look like. This gives us an intentional design for the API resource (path) structure, base URL, use of HTTP verbs, request / response JSON schemas, HTTP Headers, OAuth2 security details, and other aspects of the REST API. The OpenAPI spec also provides the benefit of being supported by most of the major cloud API Gateway services and other major API Gateway vendors. That gives us two types of APIs:

Planned: Designing API interfaces, creating reusable APIs that are called by multiple clients in multiple use cases. This is another name for that whole API design thing mentioned above.

Unplanned: For every user community, every client type, every project, create a new instance of each API version that is needed. Further multiply that by the number of active versions your API Governance paradigm dictates you must have for each API. Fast-forward a few years, and suddenly, you have several 1000s of APIs advertised on your API Gateway. This is the utter chaos part of the title.

I’ve encountered clients (and individuals) who proudly boast of the many 1000s of APIs that they have deployed to their production API Gateway (or ESB). Rather than something to be proud of as an operations marvel, this represents a complete lack (or break down) of governance.

While a lack of planning can lead to this situation, someone may have decided to do this intentionally to minimize dependencies across different API Consumers (browser apps vs mobile apps vs B2B Integration, etc), API versions, User Communities, customers, etc. Balance is needed. If you catch yourself bragging about the 1000s of endpoints that you manually maintain on your API Gateways (this is not a good thing, don’t brag about it) and you are the bottleneck on every project (also, not good) that the entire organization despises dealing with (yet, again, not good), rethink your design choices and career.

The same tendencies existed for SOAP Web Services as well. And, probably all the equivalent in all the technologies that came before it. I suppose we can extend the list of begotten / forgotten technologies to:

• GraphQL allowed the client to dictate the structure of the response (which allows for smaller payloads, better performance, potentially stricter contracts, fewer API calls, reduced bandwidth utilization)
• GRPC made some improvements on weaknesses in REST APIs (high performance for small messages, microservices communication, strict contracts, streaming).
• APIs formalized REST.
• REST fixed the problems in SOAP
• SOAP fixed the problems in EJBs
• EJBs fixed the problems in CORBA
• CORBA improved DCOM
• DCOM was supposed to be better than RPCs
• Remote Procedure Calls (RPCs)

I am curious how many times I’ll update that list (and how long it will get) before I retire (or get run over by a bus). I’ve long suspected my demise will involve a run away bus. We’ll talk more about GraphQL and GRPC in some future post.

Some version of the unplanned service design chaos mentioned earlier has happened with each one of these technologies. To a large extent, the shortcomings of each generation of these technologies has suffered from the same governance shortcomings.

It’s nearly impossible to optimize, monitor, or properly secure that many distinct endpoints. I’ve watch several shops try to do it. There are always 1000+ endpoints that are years out of date. Even if you make it a priority to update the security patterns, updating, testing, and coordinating a 1000+ of anything in production takes a lot of time. Having one API (or at least a much smaller number of) endpoint(s) is easier to maintain, but does introduce a different type of complexity. One must now manage the logic that allows the various authentication / authorization types to coexist together. At every turn this approach has allowed things to go more smoothly. It doesn’t address the coordination aspects of dealing with external API Consumers, which is likely as much about politics as it is about technology, costs, priorities, etc. At the same time, if there are fewer endpoints to call on your side that have been exposed, there is probably less work for them to do as well. Note, there are cracked-enhanced, evolution-without-the-benefit-of-natural-selection approaches to API design and deployment that can render the last point mute.

The point is that effective API Design that is part of an API Governance program leads to cleaner API Gateway deployments that in turn are easier and simpler to maintain and understand. More easy and more simple leads to greater understanding of the whole system, which equals easier to secure and maintain that security assurance in the long-term.

It will also, generally, lead to less staff turnover due to burnout.

An API Gateway that has fewer, planned / designed “enterprise” APIs is going to have a smaller operational footprint. It will be “smaller” to manage, more nimble, and turn-around on requested changes will be faster. Suddenly, the API Management (Integration, whatever you are calling yourselves today) team is not a bottleneck. The original point of one of these teams was to be able to implement integration logic faster than what a traditional development team could. That’s been lost in many shops as the years have gone by.

To summarize:

• Design your APIs before you start coding.
• Reuse API endpoints (across different client types, different user communities, versions, etc) to the greatest extent possible.
• Minimize the number of active versions.
• Put version sunset clauses in legal contracts with paying customers;
• put the same in Terms&Conditions for free-tier users
• Automate everything to the greatest extent possible (this one should be a no brainer at this point, yet, here we are).

You will very likely experience lower operational costs, greater visibility into your technology stack, and a more secure environment.