Resource versioning is a key feature in CI CD platforms that offers developers traceability, reproducibility, and stability in builds. Conveyor CI lacks built-in versioning, which has prevented many teams from taking advantage of these benefits. Implementing a solid versioning scheme allows tracking how a pipeline changes over time and reliably recovering old executions.
In other systems, the implementation varies depending on the purpose and architecture, but the central idea is to distinguish each execution of the same resource by means of a unique identifier. Three common approaches are: Semantic Versioning or SemVer with the MAJOR.MINOR.PATCH format, which facilitates human understanding and is used in many CI CD systems; Commit SHA Pinning in Git environments, which ensures absolute reproducibility by pinning a commit; and Custom revision generation, where a unique string is attached without the need for specific semantics.
In Conveyor CI, which is a minimalist state-based system with a resource-driven architecture, it is preferable to base versioning on internal state rather than on external systems. We are looking for an intuitive and semantic scheme for people. The simplest and most effective proposal is incremental versioning using positive integers, incrementing from zero each time a new version of the resource is created.
The technical implementation may seem straightforward but has important nuances. A naive approach would consist of creating a new record in the database for each resource update and maintaining a pointer to the most recent version. This works but causes exponential growth of records. For example, 1,000 resources with 100 revisions generate 100,000 records; if there are 1,000,000 resources with 100 revisions, the database would reach 100 million records, which can degrade performance and scalability.
Conveyor CI uses etcd as a key-value store, which internally supports Multi Version Concurrency Control MVCC. This means that etcd stores previous versions of a key with incremental revisions and allows reading data at specific revisions. Therefore, instead of creating a new record for each revision, one can rely on etcd's MVCC and its incremental revision numbering to implement resource versioning efficiently.
However, etcd incorporates a compaction mechanism that removes old revisions to avoid key explosion and storage consumption. Automatic compaction can cause the loss of historical revisions if one relies exclusively on etcd. Fortunately, automatic compaction can be disabled and carried out manually by an administrator, which allows designing a controlled and safe compaction strategy.
Controlled compaction strategy proposal Step 1 Enable manual compaction in etcd Step 2 Introduce an External Audit Storage where snapshots of historical revisions are saved Step 3 When running compaction, export and store the relevant revisions in the External Audit Storage and then compact etcd Step 4 When a user requests very old revisions, recover those revisions from the External Audit Storage
With this strategy, etcd maintains an efficient size and avoids key explosion while the External Audit Storage acts as a historical repository and backup. This approach balances performance and historical retention and allows scaling without sacrificing integrity or traceability.
The described solution has proven practical and production-ready in large-scale systems such as Kubernetes and is the proposed path to patch Conveyor CI, unless a more robust or efficient alternative is identified.
At Q2BSTUDIO, a company dedicated to software development and custom applications, we are specialists in designing and implementing solutions like this one. We offer comprehensive services related to custom software, custom applications, artificial intelligence and AI for businesses, AI agents, cybersecurity, AWS and Azure cloud services, and business intelligence services including Power BI. Our experience in critical projects and integrations with infrastructures such as etcd and Kubernetes allows us to implement versioning, auditing, and compaction strategies that ensure performance and compliance.
If your organization needs to implement resource versioning in Conveyor CI or any other platform, Q2BSTUDIO can help design and implement the optimal strategy, integrating artificial intelligence to automate audits, improving the cybersecurity of data stores, and deploying solutions on AWS and Azure cloud services. We have experience in business intelligence services, AI agents, and Power BI to turn historical and version data into actionable information.
Final summary The use of incremental versioning supported by etcd's MVCC combined with a manual compaction strategy and an External Audit Storage provides reproducible traceability, stability, and scalability. It is a pragmatic and proven solution that will improve Conveyor CI and that Q2BSTUDIO can implement for projects requiring custom software, custom applications, artificial intelligence, cybersecurity, and cloud services.
