In this article, I aim to share personal insights gained over the past year, rather than presenting factual or technological certainties. It is important for readers to recognize that the perspectives I offer are shaped by my own experiences and reflections.
To provide some background: approximately one year ago, I transitioned from a career in software development to a role as a Solution Architect at Red Hat, with a continued focus on application development. At that time, I identified as a full-stack developer and a DevOps engineer. However, I soon came to understand that the concept of a "full-stack developer" is somewhat misleading; in reality, my strengths lay in back-end development, with a keen interest in front-end technologies. While I was capable of initiating projects using Angular or React, my approach to these setups would likely have been criticized by seasoned front-end developers.
DevOps: An illusion?
After one week at Red Hat, an epiphany occurred. I encountered a significant revelation: I started to believe that DevOps is an illusion. It became apparent that individuals tend to possess a natural inclination towards either development or operations, a theme I have previously discussed in my article Why the developer experience is so important. This realization stemmed from an observation that both myself and others within software development often neglect to adequately consider infrastructure or external technologies as solutions to problems, preferring instead to address challenges through coding.
During that period, I harbored a strong aversion to the standardization of our microservices or miniservices architecture, perceiving such uniformity as a potential impediment to flexibility and innovation. I feared that adherence to standardization at the microservice or miniservice level would stifle our ability to explore new technologies, cloud solutions, and dependencies.
However, my perspective has evolved with time. I have come to recognize that standardization and innovation are not mutually exclusive but can coexist harmoniously. This article is dedicated to exploring this duality and sharing the insights I have acquired through my journey.
Common ground
Before delving deeper into how standardization and innovation can coexist harmoniously, it is pertinent to explore why organizations, particularly those in management roles, prioritize standardization. The rationale behind this inclination towards standardization lies in the pursuit of establishing a common framework across various code projects. Without a unified approach, developers face significant challenges when transitioning between projects, such as difficulties in locating bugs, adding new features, and understanding the overall project structure. This lack of consistency not only complicates the development process but also has tangible implications for the business.
From a business perspective, the absence of standardization renders the estimation of development timelines nearly speculative. The inability to leverage past development costs as a benchmark for future projects due to varying setups introduces a level of uncertainty in sales projections. This uncertainty escalates the risk of projects becoming financially untenable. Thus standardization, or the establishment of a common foundation across projects, is essential for enabling the business to make more accurate forecasts and to convey costs more effectively to clients.
Onboarding
As mentioned in the preceding discussion, navigating through the complexity of non-standardized microservices or miniservices setups can be a challenging endeavor. This challenge is not limited to long-standing employees within an organization but is significantly amplified for newcomers. For those just beginning their journey within a company, they may have questions such as how to initiate a new project, where to locate relevant documentation, and which technologies are allowed.
One primary challenge, often referred to as "bookmark hell," arises from the need to manage and navigate through a number of different components integral to the development process. A solution to this problem is the adoption of Red Hat Developer Hub, a tool designed to consolidate all relevant development components into a single, accessible location. Figure 1 shows the Red Hat Developer Hub user interface.
A second challenge involves the uncertainty of how to effectively start a project. This can be addressed through the use of "golden path templates." These templates, a concept integrated within Developer Hub, serve as a standardized starting point for coding a microservice or miniservice. However, the effective implementation of these "golden path templates" necessitates a certain level of standardization.
Therefore, it becomes evident that standardization is beneficial not only from a managerial perspective but also in the context of enhancing the onboarding process and facilitating the initiation of new projects. Even seasoned professionals can encounter difficulties when setting up a new microservice or miniservice, underscoring the value of standardization across the board.
Experience
In earlier sections, we've talked about how standardization can greatly benefit an organization. However, what does this mean for the more experienced members of the team? There's a real risk that confining them solely to standardized technologies could lead to disengagement or a feeling of stagnation. Moreover, shying away from innovation and sticking too rigidly to well-trodden paths could potentially hinder your organization's growth, especially considering the rapid pace of change in the IT industry.
Fortunately, embracing standardization doesn't mean that innovation has to take a back seat. The key lies in finding a balance that allows for the efficiency and predictability of standardization while still fostering an environment where creativity and innovation can flourish. How exactly can this be achieved (in my opinion)? Let’s find out in the next section.
Building innovative solutions around standardized components
In this section, I would like to outline an approach for integrating clean architecture principles within an IT infrastructure, emphasizing the use of a select toolbox comprising Kafka, service mesh, API gateway, Camel, and more, to build microservices or miniservices with a clean, standardized tech stack. By maintaining a balance between standardization and innovation, I advocate for a platform design that simplifies development, encourages continuous improvement, and supports a culture of flexibility and efficiency among developers, thereby enhancing overall productivity and competitiveness, or even think about introducing InnerSource within your organization now.
My view
In order to implement this, I will make use of the Application Foundations toolbox, containing Kafka, service mesh, API gateway, Camel, Red Hat OpenShift Serverless (i.e., Knative), Red Hat build of Quarkus, etc. A microservice or miniservice should be built according to the principles of clean architecture on code level, with a limited tech stack. A limited number of (master/primary) databases should be allowed and interfacing with the services only happens on API level (i.e., REST, GraphQL and maybe gRPC). E.g., Kafka connections should not be built within the code. This microservice or miniservice would represent the core layer of our clean architecture principles based platform. Now, regarding innovation, we would like to integrate Kafka, data lakes, etc. to our services. This is possible, and we keep these solutions in our infrastructure layer of the clean architecture setup. The platform, as a whole, plays then the role from the configuration layer.
In several of my prior blog entries, I've shared my enthusiasm for clean architecture. By applying these principles to the infrastructure layer, we can achieve a harmonious balance between innovation and standardization. For further insights, I recommend exploring two previous articles My advice for transitioning to a clean architecture platform and Knative versus AWS Lambda and Azure Functions.
To realize this vision, I plan to leverage the Application Foundations toolbox, which includes technologies such as Kafka, service mesh, API gateway, Camel, OpenShift Serverless (i.e., Knative), and Quarkus, among others. The construction of a microservice or miniservice should adhere to clean architecture principles at the code level, utilizing a concise tech stack. The approach will involve restricting the number of databases to a select few and ensuring that service interfacing occurs solely at the API level, employing REST, GraphQL, and potentially gRPC. For instance, Kafka connections ought to be external to the codebase. This strategy forms the core layer of our platform, built on the principles of Clean Architecture.
When it comes to integrating innovative technologies like Kafka and data lakes into our services, this is entirely feasible within the infrastructure layer of our Clean Architecture framework. The platform thus serves as the configuration layer, enabling us to blend innovation with standardization seamlessly. Figure 2 illustrates this.
The only thing we now need to find out is how we implement interfaces on infrastructure level. Well, for this, we can make use of Knative, Debezium (i.e., a Kafka connector), and/or Camel. See Figure 3.
This standardized building block now can be used in a more modern, innovative landscape. Figure 4 depicts this (in an example architecture).
Advantages of such a setup
Adopting the approach outlined in the previous section offers numerous benefits, in my view:
- Standardization and simplification: By enforcing standardization and a streamlined tech stack at the microservice or miniservice level, the platform simplifies the development of business logic (i.e., features) for developers, particularly those who are junior or newly joined. This simplification aids in producing more accurate project estimations, thereby enhancing the satisfaction of sales teams and managers.
- Encouragement of innovation: The platform design permits innovation, ensuring that your organization remains competitive. This aspect is crucial for fostering a culture of continuous improvement and adaptation.
- Flexible work assignments: The structure allows individuals to express their interest in or be assigned to work on more complex solutions based on their expertise and career aspirations. For instance, while a junior developer might focus on the standardized aspects of microservices or miniservices, a more seasoned developer could tackle more sophisticated tasks such as Camel integrations. This approach ensures that all team members find their work both comfortable and challenging, contributing to job satisfaction and personal growth.
- Uniformity across artifacts: Implementing a consistent framework across various artifacts can lead to more efficient use of tools like Developer Hub, an internal developer platform. This uniformity is instrumental in boosting developer productivity by providing a common set of tools and methodologies.
- Reduced onboarding time: A standardized and well-documented platform significantly shortens the learning curve for new employees, leading to quicker onboarding and faster time-to-productivity.
In summary, structuring your platform as described not only streamlines development processes and enhances project predictability but also cultivates an environment where innovation thrives alongside standardization. This balanced ecosystem is key to both employee fulfillment and competitive edge.
Conclusion
Once again, be aware that this article is opinionated, but I firmly believe that this way of thinking can result in more efficient software and platform development. Maybe DevOps is in that extent not an illusion at all….
