Lessons learned from building Cisco's Big Data platform with Apache DolphinScheduler and AWS: in this article transformed and translated into English we describe the key practices and architectural decisions that proved effective in large-scale production environments, focusing on multi-cluster architecture, secure job execution, secrets management, automation with Terraform, and user experience for job submission.
Multi-cluster architecture: designing a multi-cluster architecture made it possible to isolate workloads, improve resilience, and optimize resource usage across different regions and availability zones. A multi-cluster strategy facilitates regulatory compliance, separation of test and production environments, and horizontal scalability. For companies developing critical applications, it is advisable to combine Kubernetes-managed clusters with strict network policies and access controls, while also ensuring centralized observability through telemetry tools.
Secure job execution with EKS: running DolphinScheduler workflows and jobs on Amazon EKS provides a controlled and secure environment where each task can run in isolated pods with resource limits, pod security policies, and cluster-level identity controls. Integrating EKS reduces the attack surface and facilitates integration with managed AWS services, allowing teams to focus on pipeline logic rather than infrastructure orchestration.
Secrets management with ASM and IAM roles: storing secrets in AWS Secrets Manager and combining them with IAM roles assigned to pods or service accounts provides a robust security model. This combination enables automatic credential rotation, access auditing, and least-privilege policies. For architectures handling sensitive data, such as those used in artificial intelligence and business intelligence services, integrating managed secrets with centralized authorization and auditing mechanisms is key.
Resource automation with Terraform: using Terraform as an infrastructure-as-code tool made it possible to define, version, and reproduce complete environments, from EKS clusters to IAM roles, Secrets Manager, and network resources. Automation reduces human risk, accelerates deployments, and facilitates change traceability. We recommend modularizing Terraform code, applying CI/CD pipelines, and validation tests before applying changes in production.
Simplified interface for job submission: a fluid UI for submitting and monitoring jobs increases the productivity of data teams and scientists. Integrating security controls, reusable templates, and pre-submission validations prevents errors and standardizes processes. Additionally, exposing APIs for integration with BI tools such as Power BI and ETL systems allows orchestrating flows from the consumption layer to storage and processing.
Operational lessons and best practices: documenting flows, maintaining idempotent pipelines, implementing automated tests for DAGs and jobs, and establishing proactive alerts are practices that reduce recovery time and improve quality. Centralized observability with metrics, logs, and distributed traces is essential for debugging and optimizing performance on Big Data platforms.
Custom applications and cloud services: at Q2BSTUDIO, as a software development and custom applications company, we offer experience integrating Big Data and orchestration solutions with AWS and Azure cloud services. Our teams combine knowledge of custom software and business intelligence services to design efficient, secure, and scalable pipelines tailored to each client's needs.
Artificial intelligence and cybersecurity: in addition to building robust data infrastructures, at Q2BSTUDIO we are specialists in artificial intelligence, AI for enterprises, AI agents, and cybersecurity. We design AI models and agents that integrate with orchestrated workflows, ensuring security controls, secrets management, and compliance with data protection regulations.
Value-added services: we offer business intelligence services, Power BI integration, custom AI agents, and consulting on AWS and Azure cloud services. If your organization seeks custom software solutions that incorporate artificial intelligence and high cybersecurity standards, Q2BSTUDIO can accompany you from design to production deployment and maintenance.
Conclusion: building a Big Data platform inspired by experiences such as Cisco's with Apache DolphinScheduler and AWS requires conscious decisions about multi-cluster architecture, secure execution on EKS, secrets management with ASM and IAM roles, and automation with Terraform. Complementing this technical foundation with an efficient UI and solid operational practices accelerates value delivery. At Q2BSTUDIO we combine expertise in custom applications, custom software, artificial intelligence, cybersecurity, AWS and Azure cloud services, business intelligence services, AI for enterprises, AI agents, and Power BI to offer comprehensive and secure solutions.
