1. 项目概述KubeSphere是一个开源的容器平台基于Kubernetes构建提供了一站式的DevOps解决方案。本文将详细介绍如何实现KubeSphere与DevOps的高可用部署包括环境准备、组件安装、配置优化等全流程实战指南。高可用部署是企业级应用的关键需求通过多节点部署和负载均衡可以确保系统在单点故障时仍能持续提供服务。本指南将采用六脉神剑方法论从六个核心维度构建完整的KubeSphere DevOps高可用环境。2. 环境准备2.1 硬件需求建议生产环境采用以下配置控制节点3台每台8核CPU/16GB内存/100GB存储工作节点3台每台16核CPU/32GB内存/200GB存储存储节点3台每台8核CPU/32GB内存/1TB存储网关节点2台每台4核CPU/8GB内存/50GB存储2.2 软件要求操作系统推荐使用CentOS 7.9或Ubuntu 20.04 LTSKubernetes版本v1.20Docker版本20.10KubeSphere版本v3.3.02.3 网络规划需要预先规划以下网络配置节点间通信网络192.168.0.0/24Pod网络10.233.64.0/18Service网络10.233.0.0/18外部访问VIP192.168.0.1003. KubeSphere高可用部署3.1 使用KubeKey安装KubeKey是KubeSphere官方提供的安装工具支持一键部署高可用集群# 下载KubeKey curl -sfL https://get-kk.kubesphere.io | VERSIONv2.3.0 sh - # 生成配置文件 ./kk create config --with-kubesphere v3.3.0 --with-kubernetes v1.22.10 # 修改配置文件 vi config-sample.yaml配置示例apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata: name: kubesphere-ha spec: hosts: - {name: master1, address: 192.168.0.1, internalAddress: 192.168.0.1, user: root, password: Pssw0rd} - {name: master2, address: 192.168.0.2, internalAddress: 192.168.0.2, user: root, password: Pssw0rd} - {name: master3, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: Pssw0rd} - {name: worker1, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: Pssw0rd} - {name: worker2, address: 192.168.0.5, internalAddress: 192.168.0.5, user: root, password: Pssw0rd} - {name: worker3, address: 192.168.0.6, internalAddress: 192.168.0.6, user: root, password: Pssw0rd} roleGroups: etcd: - master1 - master2 - master3 control-plane: - master1 - master2 - master3 worker: - worker1 - worker2 - worker3 controlPlaneEndpoint: domain: lb.kubesphere.local address: 192.168.0.100 port: 6443 kubernetes: version: v1.22.10 clusterName: cluster.local network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 registry: registryMirrors: [] insecureRegistries: [] addons: []3.2 执行安装# 开始安装 ./kk create cluster -f config-sample.yaml # 检查安装状态 kubectl get nodes kubectl get pods -A3.3 配置负载均衡使用Nginx配置负载均衡示例配置stream { upstream kube-apiserver { server 192.168.0.1:6443 weight5 max_fails3 fail_timeout30s; server 192.168.0.2:6443 weight5 max_fails3 fail_timeout30s; server 192.168.0.3:6443 weight5 max_fails3 fail_timeout30s; } server { listen 6443; proxy_pass kube-apiserver; } }4. DevOps组件高可用部署4.1 启用DevOps组件通过KubeSphere控制台启用DevOps功能以admin用户登录控制台进入平台管理 - 集群管理选择定制资源定义 - 搜索clusterconfiguration编辑ks-installer的YAML配置将devops.enabled设置为truedevops: enabled: true jenkinsMemoryLim: 8Gi jenkinsMemoryReq: 4Gi jenkinsVolumeSize: 32Gi jenkinsJavaOpts_Xms: 3g jenkinsJavaOpts_Xmx: 6g jenkinsJavaOpts_MaxRAM: 8g4.2 验证DevOps组件# 检查DevOps组件状态 kubectl get pods -n kubesphere-devops-system # 预期输出 NAME READY STATUS RESTARTS AGE devops-apiserver-858fdcc978-hwxbf 1/1 Running 0 5m devops-controller-989995d9f-wc6hs 1/1 Running 0 5m devops-jenkins-df4d7cd4b-pkmhn 1/1 Running 0 5m s2ioperator-0 1/1 Running 0 5m4.3 配置Jenkins高可用修改Jenkins Deployment为StatefulSet配置持久化存储设置资源限制和请求示例配置apiVersion: apps/v1 kind: StatefulSet metadata: name: devops-jenkins namespace: kubesphere-devops-system spec: serviceName: devops-jenkins replicas: 2 selector: matchLabels: app: devops-jenkins template: metadata: labels: app: devops-jenkins spec: containers: - name: jenkins image: jenkins/jenkins:2.319.3-lts-jdk11 resources: limits: cpu: 4 memory: 8Gi requests: cpu: 2 memory: 4Gi volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home volumeClaimTemplates: - metadata: name: jenkins-home spec: accessModes: [ ReadWriteOnce ] storageClassName: longhorn resources: requests: storage: 50Gi5. 配置CI/CD流水线5.1 创建凭证代码仓库凭证(Git)镜像仓库凭证(Docker Registry)Kubernetes集群凭证(kubeconfig)5.2 示例流水线pipeline { agent { node { label maven } } parameters { string(name: BRANCH_NAME, defaultValue: master, description: ) string(name: TAG_NAME, defaultValue: , description: ) } environment { DOCKER_CREDENTIAL_ID harbor-creds GITHUB_CREDENTIAL_ID git-creds KUBECONFIG_CREDENTIAL_ID kubeconfig REGISTRY registry.example.com DOCKERHUB_NAMESPACE myproject APP_NAME demo-app } stages { stage(拉取代码) { steps { git(url: https://github.com/myproject/demo-app.git, credentialsId: $GITHUB_CREDENTIAL_ID, branch: $BRANCH_NAME) } } stage(单元测试) { steps { container(maven) { sh mvn clean test } } } stage(编译构建) { steps { container(maven) { sh mvn clean package -DskipTests } } } stage(构建镜像) { steps { container(maven) { withCredentials([usernamePassword( credentialsId: $DOCKER_CREDENTIAL_ID, usernameVariable: DOCKER_USERNAME, passwordVariable: DOCKER_PASSWORD )]) { sh echo $DOCKER_PASSWORD | docker login $REGISTRY -u $DOCKER_USERNAME --password-stdin docker build -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME . docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME } } } } stage(部署到K8S) { steps { input(id: deploy-to-prod, message: 部署到生产环境?) container(maven) { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: KUBECONFIG )]) { sh envsubst deploy/prod.yaml | kubectl apply -f - } } } } } }6. 监控与运维6.1 配置监控告警启用KubeSphere监控组件配置Jenkins性能监控设置流水线执行告警6.2 备份策略Jenkins数据备份# 定期备份Jenkins home目录 kubectl exec -n kubesphere-devops-system devops-jenkins-0 -- tar czf /tmp/jenkins-backup.tar.gz /var/jenkins_home kubectl cp kubesphere-devops-system/devops-jenkins-0:/tmp/jenkins-backup.tar.gz ./jenkins-backup-$(date %Y%m%d).tar.gzetcd数据备份ETCDCTL_API3 etcdctl --endpointshttps://192.168.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /backup/etcd-snapshot-$(date %Y%m%d).db6.3 性能优化建议Jenkins JVM参数调优env: - name: JAVA_OPTS value: -Xms4g -Xmx8g -XX:MaxRAMPercentage70.0 -Djava.awt.headlesstrue配置Pod反亲和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - devops-jenkins topologyKey: kubernetes.io/hostname启用缓存stage(构建) { steps { container(maven) { sh mvn -Dmaven.repo.local/m2repo clean package } } }7. 故障排查7.1 常见问题Jenkins Pod频繁重启检查内存配置是否足够查看日志定位具体错误kubectl logs -n kubesphere-devops-system devops-jenkins-0流水线执行失败检查凭证配置是否正确验证网络连通性查看详细错误日志镜像推送失败检查镜像仓库凭证验证网络策略是否允许访问检查存储空间是否充足7.2 日志收集配置集中式日志收集# 安装EFK日志系统 apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: name: ks-installer namespace: kubesphere-system spec: logging: enabled: true elasticsearchMaster: volumeSize: 100Gi elasticsearchData: volumeSize: 100Gi kibana: resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi8. 最佳实践资源隔离为不同团队创建独立的DevOps项目设置资源配额限制安全加固启用RBAC权限控制定期轮换凭证配置网络策略性能优化使用专用节点运行CI/CD任务配置构建缓存优化Docker构建过程灾备方案定期备份关键数据多集群部署配置自动化故障转移通过以上步骤我们完成了KubeSphere DevOps的高可用部署。这套方案已经在多个生产环境验证能够支撑企业级的CI/CD需求。实际部署时建议根据具体业务场景调整资源配置和架构设计。
1. 项目概述KubeSphere是一个开源的容器平台基于Kubernetes构建提供了一站式的DevOps解决方案。本文将详细介绍如何实现KubeSphere与DevOps的高可用部署包括环境准备、组件安装、配置优化等全流程实战指南。高可用部署是企业级应用的关键需求通过多节点部署和负载均衡可以确保系统在单点故障时仍能持续提供服务。本指南将采用六脉神剑方法论从六个核心维度构建完整的KubeSphere DevOps高可用环境。2. 环境准备2.1 硬件需求建议生产环境采用以下配置控制节点3台每台8核CPU/16GB内存/100GB存储工作节点3台每台16核CPU/32GB内存/200GB存储存储节点3台每台8核CPU/32GB内存/1TB存储网关节点2台每台4核CPU/8GB内存/50GB存储2.2 软件要求操作系统推荐使用CentOS 7.9或Ubuntu 20.04 LTSKubernetes版本v1.20Docker版本20.10KubeSphere版本v3.3.02.3 网络规划需要预先规划以下网络配置节点间通信网络192.168.0.0/24Pod网络10.233.64.0/18Service网络10.233.0.0/18外部访问VIP192.168.0.1003. KubeSphere高可用部署3.1 使用KubeKey安装KubeKey是KubeSphere官方提供的安装工具支持一键部署高可用集群# 下载KubeKey curl -sfL https://get-kk.kubesphere.io | VERSIONv2.3.0 sh - # 生成配置文件 ./kk create config --with-kubesphere v3.3.0 --with-kubernetes v1.22.10 # 修改配置文件 vi config-sample.yaml配置示例apiVersion: kubekey.kubesphere.io/v1alpha2 kind: Cluster metadata: name: kubesphere-ha spec: hosts: - {name: master1, address: 192.168.0.1, internalAddress: 192.168.0.1, user: root, password: Pssw0rd} - {name: master2, address: 192.168.0.2, internalAddress: 192.168.0.2, user: root, password: Pssw0rd} - {name: master3, address: 192.168.0.3, internalAddress: 192.168.0.3, user: root, password: Pssw0rd} - {name: worker1, address: 192.168.0.4, internalAddress: 192.168.0.4, user: root, password: Pssw0rd} - {name: worker2, address: 192.168.0.5, internalAddress: 192.168.0.5, user: root, password: Pssw0rd} - {name: worker3, address: 192.168.0.6, internalAddress: 192.168.0.6, user: root, password: Pssw0rd} roleGroups: etcd: - master1 - master2 - master3 control-plane: - master1 - master2 - master3 worker: - worker1 - worker2 - worker3 controlPlaneEndpoint: domain: lb.kubesphere.local address: 192.168.0.100 port: 6443 kubernetes: version: v1.22.10 clusterName: cluster.local network: plugin: calico kubePodsCIDR: 10.233.64.0/18 kubeServiceCIDR: 10.233.0.0/18 registry: registryMirrors: [] insecureRegistries: [] addons: []3.2 执行安装# 开始安装 ./kk create cluster -f config-sample.yaml # 检查安装状态 kubectl get nodes kubectl get pods -A3.3 配置负载均衡使用Nginx配置负载均衡示例配置stream { upstream kube-apiserver { server 192.168.0.1:6443 weight5 max_fails3 fail_timeout30s; server 192.168.0.2:6443 weight5 max_fails3 fail_timeout30s; server 192.168.0.3:6443 weight5 max_fails3 fail_timeout30s; } server { listen 6443; proxy_pass kube-apiserver; } }4. DevOps组件高可用部署4.1 启用DevOps组件通过KubeSphere控制台启用DevOps功能以admin用户登录控制台进入平台管理 - 集群管理选择定制资源定义 - 搜索clusterconfiguration编辑ks-installer的YAML配置将devops.enabled设置为truedevops: enabled: true jenkinsMemoryLim: 8Gi jenkinsMemoryReq: 4Gi jenkinsVolumeSize: 32Gi jenkinsJavaOpts_Xms: 3g jenkinsJavaOpts_Xmx: 6g jenkinsJavaOpts_MaxRAM: 8g4.2 验证DevOps组件# 检查DevOps组件状态 kubectl get pods -n kubesphere-devops-system # 预期输出 NAME READY STATUS RESTARTS AGE devops-apiserver-858fdcc978-hwxbf 1/1 Running 0 5m devops-controller-989995d9f-wc6hs 1/1 Running 0 5m devops-jenkins-df4d7cd4b-pkmhn 1/1 Running 0 5m s2ioperator-0 1/1 Running 0 5m4.3 配置Jenkins高可用修改Jenkins Deployment为StatefulSet配置持久化存储设置资源限制和请求示例配置apiVersion: apps/v1 kind: StatefulSet metadata: name: devops-jenkins namespace: kubesphere-devops-system spec: serviceName: devops-jenkins replicas: 2 selector: matchLabels: app: devops-jenkins template: metadata: labels: app: devops-jenkins spec: containers: - name: jenkins image: jenkins/jenkins:2.319.3-lts-jdk11 resources: limits: cpu: 4 memory: 8Gi requests: cpu: 2 memory: 4Gi volumeMounts: - name: jenkins-home mountPath: /var/jenkins_home volumeClaimTemplates: - metadata: name: jenkins-home spec: accessModes: [ ReadWriteOnce ] storageClassName: longhorn resources: requests: storage: 50Gi5. 配置CI/CD流水线5.1 创建凭证代码仓库凭证(Git)镜像仓库凭证(Docker Registry)Kubernetes集群凭证(kubeconfig)5.2 示例流水线pipeline { agent { node { label maven } } parameters { string(name: BRANCH_NAME, defaultValue: master, description: ) string(name: TAG_NAME, defaultValue: , description: ) } environment { DOCKER_CREDENTIAL_ID harbor-creds GITHUB_CREDENTIAL_ID git-creds KUBECONFIG_CREDENTIAL_ID kubeconfig REGISTRY registry.example.com DOCKERHUB_NAMESPACE myproject APP_NAME demo-app } stages { stage(拉取代码) { steps { git(url: https://github.com/myproject/demo-app.git, credentialsId: $GITHUB_CREDENTIAL_ID, branch: $BRANCH_NAME) } } stage(单元测试) { steps { container(maven) { sh mvn clean test } } } stage(编译构建) { steps { container(maven) { sh mvn clean package -DskipTests } } } stage(构建镜像) { steps { container(maven) { withCredentials([usernamePassword( credentialsId: $DOCKER_CREDENTIAL_ID, usernameVariable: DOCKER_USERNAME, passwordVariable: DOCKER_PASSWORD )]) { sh echo $DOCKER_PASSWORD | docker login $REGISTRY -u $DOCKER_USERNAME --password-stdin docker build -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME . docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME } } } } stage(部署到K8S) { steps { input(id: deploy-to-prod, message: 部署到生产环境?) container(maven) { withCredentials([ kubeconfigFile( credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: KUBECONFIG )]) { sh envsubst deploy/prod.yaml | kubectl apply -f - } } } } } }6. 监控与运维6.1 配置监控告警启用KubeSphere监控组件配置Jenkins性能监控设置流水线执行告警6.2 备份策略Jenkins数据备份# 定期备份Jenkins home目录 kubectl exec -n kubesphere-devops-system devops-jenkins-0 -- tar czf /tmp/jenkins-backup.tar.gz /var/jenkins_home kubectl cp kubesphere-devops-system/devops-jenkins-0:/tmp/jenkins-backup.tar.gz ./jenkins-backup-$(date %Y%m%d).tar.gzetcd数据备份ETCDCTL_API3 etcdctl --endpointshttps://192.168.0.1:2379 \ --cacert/etc/kubernetes/pki/etcd/ca.crt \ --cert/etc/kubernetes/pki/etcd/server.crt \ --key/etc/kubernetes/pki/etcd/server.key \ snapshot save /backup/etcd-snapshot-$(date %Y%m%d).db6.3 性能优化建议Jenkins JVM参数调优env: - name: JAVA_OPTS value: -Xms4g -Xmx8g -XX:MaxRAMPercentage70.0 -Djava.awt.headlesstrue配置Pod反亲和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - devops-jenkins topologyKey: kubernetes.io/hostname启用缓存stage(构建) { steps { container(maven) { sh mvn -Dmaven.repo.local/m2repo clean package } } }7. 故障排查7.1 常见问题Jenkins Pod频繁重启检查内存配置是否足够查看日志定位具体错误kubectl logs -n kubesphere-devops-system devops-jenkins-0流水线执行失败检查凭证配置是否正确验证网络连通性查看详细错误日志镜像推送失败检查镜像仓库凭证验证网络策略是否允许访问检查存储空间是否充足7.2 日志收集配置集中式日志收集# 安装EFK日志系统 apiVersion: installer.kubesphere.io/v1alpha1 kind: ClusterConfiguration metadata: name: ks-installer namespace: kubesphere-system spec: logging: enabled: true elasticsearchMaster: volumeSize: 100Gi elasticsearchData: volumeSize: 100Gi kibana: resources: requests: cpu: 0.5 memory: 1Gi limits: cpu: 1 memory: 2Gi8. 最佳实践资源隔离为不同团队创建独立的DevOps项目设置资源配额限制安全加固启用RBAC权限控制定期轮换凭证配置网络策略性能优化使用专用节点运行CI/CD任务配置构建缓存优化Docker构建过程灾备方案定期备份关键数据多集群部署配置自动化故障转移通过以上步骤我们完成了KubeSphere DevOps的高可用部署。这套方案已经在多个生产环境验证能够支撑企业级的CI/CD需求。实际部署时建议根据具体业务场景调整资源配置和架构设计。
