Reduce the complexity: be sharing and open source

By Franck Pachot

.
I thought that my last post on this blog was the previous one but I blog when it comes to my mind – nothing planned. And two things got me to write this. First, last week we got our 20th dbi xChange where we meet with colleagues. And I love discussing with colleagues about what we see, and foresee, in our jobs. The second one is:

The Law of Conservation of Complexity by Daniel Finkhttps://t.co/XGsvCo8gbf
My🤔's13 years later:
– Microservices, automation, autoscaling, elasticity, cloud, are ok if they reduce complexity. Not if they increase it
– Open Source reduces complexity by documenting how it works pic.twitter.com/TBfvdxqWlt

— Franck Pachot (@FranckPachot) June 27, 2021

The web.archive.org is full of gems like The Law of Conservation of Complexity from Daniel Fink written in 2008. As it is not about one technology, but methodology, it is still relevant today.

The best you can do in IT is avoid to increase the complexity of the system. If this is not your main goal, you may find short-term solutions. or workarounds, or quick wins, but increase the technological debt. More complexity means more bugs, less agility for future evolution, poor performance because of additional layers and latency, and additional costs. If you add a cache on top of a system, do you do it to decrease complexity (it would require more code or hardware to get same performance without the cache) or increase it (the problem is in application design and you add a new component to hide it)? If you move from RDBMS to NoSQL, do you reduce the complexity (you don’t need consistency and access to small items only) or increase it (you will have to write more code to get consistent data sets)? If you split your monolith to microservices, do you reduce complexity (simplify the regression tests and deployment procedures) or increase it (have to stream data between many services and try to get consistent eventually)? Those are just examples. And think about complexity at all levels. Modern full-stack developers require so many skills… think about the hiring complexity as well.

The The Law of Conservation of Complexity mentions that you can decrease the complexity by understanding how system works and by knowledge sharing. I agree on both. My next job is developer advocate: help the users understand how the system works, and help the developers understand how their system is used. Without this role, the systems tend to over-engineer features that may not even been used. And users tend to interact with the system in a sub-optimal way and add workarounds in their code. By system, in my case, I’m referring to the YugabyteDB distributed database, as that’s my next job. SQL can reduce the complexity of code because there are many things that you don’t have to do in the application (like integrity constraints, lock management,…). YugabyteDB is PostgreSQL compatible, well known to provide extensible features while limiting the complexity. And consensus protocols can reduce the complexity of auto-scaling, sharding, HA and DR with a simple definition: replication factor and AZ/region placement. In addition to those, YugabyteDB is Open Source, which also reduces complexity. First because all is documented publicly. And also because opening your code to contribution forces you to making it simple to understand and evolve.

I like Oracle because, even if closed source, I can have a good knowledge on how the system works. Thanks to the many troubleshooting tools and instrumentation, the quality of the documentation, and the great community documenting, for years, how it works. I’m thinking about OakTable, and Oracle ACE there, and many others. This will stay for sure, but the Oracle database is also very complex. The new challenge is in distributed databases, which mimics the NoSQL scalability, but with full SQL support for reliable OLTP. My main goal, as developer advocate at Yugabyte, will be to reduce the complexity by knowledge sharing: help you know, use, and maybe contribute to YugabyteDB. I’ve put all links to follow this journey in the previous post.