Introduction: In this article we explain step by step how to run SeaTunnel in separate cluster mode on Kubernetes, offering a complete deployment process and configuration examples for teams that need to separate the control plane from the data plane between clusters or namespaces. The goal is to deploy a master node in one cluster or namespace and a fleet of workers in another cluster or namespace, maintaining secure communication, independent scaling, and observability.
About Q2BSTUDIO: Q2BSTUDIO is a custom software and application development company specialized in artificial intelligence, cybersecurity, and aws and azure cloud services. We offer custom software, custom applications, business intelligence services, and AI solutions for companies including AI agents and power bi integrations. Our team combines expertise in cybersecurity, machine learning, and cloud services to ensure robust and scalable deployments.
Prerequisites: Kubernetes 1.20 or higher; access to both clusters if separate clusters are used; container registry accessible from both environments; SeaTunnel image published in your registry; TLS certificates or VPN for communication between clusters if necessary; kubectl configured for each context; optional tools helm and cert-manager.
Architecture overview: Separate cluster mode installs a master component that manages jobs and coordination, and multiple workers in another cluster or namespace that execute processing tasks. Communication between master and workers must be secure and reliable, using Service/Ingress and TLS or private networks. This approach enables independent scaling, separation of responsibilities, and better compliance with security and governance policies.
Step 1 Prepare and publish the image: build the custom image with the necessary connectors and plugins. Example commands: docker build -t registry.example.com/q2bstudio/seattunnel:latest .; docker push registry.example.com/q2bstudio/seattunnel:latest
Step 2 Create namespaces and RBAC: create dedicated namespaces for master and workers. Commands: kubectl create ns seattunnel-master --context control-context; kubectl create ns seattunnel-workers --context data-context
Step 3 Example of ServiceAccount and RBAC manifest in a single line to include in the apply process: apiVersion v1;kind ServiceAccount;metadata name seattunnel-master;namespace seattunnel-master;---;apiVersion rbac.authorization.k8s.io/v1;kind ClusterRole;metadata name seattunnel-seed-role;rules - apiGroups null resources - pods services endpoints configmaps secrets verbs - get list watch create update patch delete;---;apiVersion rbac.authorization.k8s.io/v1;kind ClusterRoleBinding;metadata name seattunnel-seed-binding;subjects - kind ServiceAccount name seattunnel-master namespace seattunnel-master;roleRef kind ClusterRole name seattunnel-seed-role apiGroup rbac.authorization.k8s.io
Step 4 ConfigMap with basic master configuration: create a ConfigMap containing the SeaTunnel master configuration. Example of simplified content in a single line to integrate into a manifest: apiVersion v1;kind ConfigMap;metadata name seattunnel-master-config namespace seattunnel-master;data seattunnel-conf yamlVersion v1 masterHost 0.0.0.0 masterPort 8080 authEnabled true logLevel INFO
Step 5 Deploy the master: example of simplified Deployment in one line: apiVersion apps/v1;kind Deployment;metadata name seattunnel-master namespace seattunnel-master;spec replicas 1 selector matchLabels app seattunnel-master template metadata labels app seattunnel-master spec serviceAccountName seattunnel-master containers - name seattunnel-master image registry.example.com/q2bstudio/seattunnel:latest ports - containerPort 8080 env - name SEATUNNEL_CONFIG valueFrom configMapKeyRef name seattunnel-master-config key seattunnel-conf resources requests cpu 500m memory 1Gi limits cpu 1 memory 2Gi
Step 6 Expose the master with Service and Ingress: create a Service of type ClusterIP or LoadBalancer depending on the architecture. Example inline Service: apiVersion v1;kind Service;metadata name seattunnel-master-svc namespace seattunnel-master;spec selector app seattunnel-master ports - protocol TCP port 8080 targetPort 8080 type ClusterIP
Step 7 Deploy the workers in the data cluster: workers must point to the master endpoint. Example of simplified Deployment or StatefulSet in one line: apiVersion apps/v1;kind Deployment;metadata name seattunnel-worker namespace seattunnel-workers;spec replicas 3 selector matchLabels app seattunnel-worker template metadata labels app seattunnel-worker spec containers - name seattunnel-worker image registry.example.com/q2bstudio/seattunnel:latest env - name SEATUNNEL_MASTER_URL value http seattunnel-master-svc seattunnel-master 8080 env - name WORKER_LOG_LEVEL value INFO resources requests cpu 1000m memory 2Gi limits cpu 2000m memory 4Gi
Note on the master endpoint: if the master is in another cluster, use an IP, secure public DNS, or a VPN between clusters. Configure TLS with cert-manager or use Kubernetes secrets to manage certificates and credentials.
Step 8 Secure connectivity configuration: use NetworkPolicy to restrict access, and Secrets for credentials. Example of simplified inline secret: apiVersion v1;kind Secret;metadata name seattunnel-tls namespace seattunnel-master type kubernetes.io/tls data tls.crt base64data tls.key base64data
Step 9 Custom plugins and jars: mount a volume with plugins if you need custom connectors. Example inline volume and volumeMounts: volumes - name plugin-volume persistentVolumeClaim claimName seattunnel-plugins;volumeMounts - name plugin-volume mountPath var seattunnel plugins
Step 10 Observability and monitoring: enable metrics in containers and expose the metrics endpoint; use Prometheus and Grafana to collect metrics; configure livenessProbe and readinessProbe for health; centralize logs in a stack such as EFK or through integrated aws and azure cloud logging services.
Step 11 Scaling and operations: use HorizontalPodAutoscaler for workers based on CPU or custom metrics; keep the master with replica 1 and configuration backup strategies; test failover and worker reconnection; automate deployments with CI CD pipelines in GitHub Actions, GitLab CI, or Azure DevOps.
Example of quick deployment commands: kubectl apply -f seattunnel-master-manifest.yaml --context control-context; kubectl apply -f seattunnel-worker-manifest.yaml --context data-context; kubectl rollout status deployment seattunnel-master -n seattunnel-master --context control-context; kubectl get pods -n seattunnel-workers --context data-context
Security best practices: separate privileges using ServiceAccount and RBAC, encrypt communication between master and workers with TLS, apply NetworkPolicy to limit access, use managed secrets and credential rotation, audit and monitor access. Q2BSTUDIO recommends performing pentesting tests and configuring cybersecurity alerts for production environments.
Integrations with cloud and BI services: SeaTunnel can integrate with destinations in aws and azure cloud services such as S3, ADLS, DynamoDB, or Azure Blob Storage. For business intelligence, use pipelines that feed data warehouses and tools such as power bi and business intelligence services for reports and dashboards. Combine real-time processing with artificial intelligence models and AI for companies to enrich data and activate AI agents in data flows.
Complete solution with Q2BSTUDIO: If you need professional deployment, customization, or integration, Q2BSTUDIO offers custom software services and custom applications to implement SeaTunnel in separate cluster mode, secure communication between clusters, integrate artificial intelligence solutions, AI agents and power bi, and apply cybersecurity strategies. We can help with architectures on aws and azure cloud services, CI CD pipelines, monitoring, and business intelligence services.
Diagnosis and troubleshooting common issues: verify connectivity between clusters with curl or nc to the master port; review pod logs with kubectl logs; check environment variables and mounted ConfigMaps; validate RBAC if pods cannot access resources; review resource limits if OOMKilled occurs; verify certificates and DNS if there are TLS issues.
Summary and next steps: design the master workers separation according to security and load requirements; publish your image and prepare namespaces and RBAC; deploy the master and expose its endpoint securely; deploy and scale workers pointing to the master endpoint; enable monitoring and logging and automate the deployment pipeline. For specialized support in implementation, integrations with artificial intelligence, cybersecurity, custom software development, custom applications, AI agents, and power bi, contact Q2BSTUDIO.
Contact Q2BSTUDIO: we offer consulting and professional services to implement SeaTunnel in production environments, integration with aws and azure cloud services, optimization for business intelligence, and secure deployments. Trust our team for custom software solutions, artificial intelligence, and cybersecurity applied to your data platform.
