起因
开发反馈habor镜像库登陆不了，初步查看是证书过期了。
解决方案 之前Harbor-helm部署镜像库文档可以回顾链接Kubernetes1.13.1集群集成Harbor-helm 1.首先新建新证书的secret
1 [root@elasticsearch01 harbor-helm]# kubectl create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt
2.然后修改harbor-helm的value.yaml，把secretName替换下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 [root@elasticsearch01 harbor-helm]# head -n 20 values.yaml expose: # Set the way how to expose the service. Set the type as "ingress", # "clusterIP" or "nodePort" and fill the information in the corresponding # section type: ingress tls: # Enable the tls or not. Note: if the type is "ingress" and the tls # is disabled, the port must be included in the command when pull/push # images. Refer to https://github.com/goharbor/harbor/issues/5291 # for the detail. enabled: true # Fill the name of secret if you want to use your own TLS certificate # and private key. The secret must contain keys named tls.crt and # tls.key that contain the certificate and private key to use for TLS # The certificate and private key will be generated automatically if # it is not set secretName: "ingress-secret2021" # By default, the Notary service will use the same cert and key as # described above. Fill the name of secret if you want to use a # separated one. Only needed when the type is "ingress".
3.最后使用helm upgrade更新版本
1 [root@elasticsearch01 harbor-helm]# helm upgrade minminmsn . -f values.yaml
到这个时候应该能解决需求，可是事与愿违，不知道哪儿除了问题，这时登陆Harbor证书问题是解决了，但是项目及库访问不了提示内部错误，看Pod的运行状态也都是Running。 最后打算使用helm先delete掉再install，但是这样创建的harbor看起来一切正常，实际上是个初始化环境，是自动生成的新PV并没有原来的数据。此时发现原来的PV还在，下面就开始找PV恢复的方案。
调整PV状态
1.查询此时PV与PVC状态
1 2 3 4 5 6 7 8 9 10 11 12 13 14 [root@elasticsearch01 harbor-helm]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 9h pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Released default/minminmsn-harbor-chartmuseum rbd 417d pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/minminmsn-harbor-jobservice rbd 417d pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Released default/minminmsn-harbor-registry rbd 417d pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/database-data-minminmsn-harbor-database-0 rbd 417d pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/data-minminmsn-harbor-redis-0 rbd 417d [root@elasticsearch01 harbor-helm]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-minminmsn-harbor-redis-0 Bound pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h database-data-minminmsn-harbor-database-0 Bound pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h minminmsn-harbor-chartmuseum Bound pvc-6c903857-c5f0-11ea-9386-52540089b2b6 50Gi RWO rbd 9h minminmsn-harbor-jobservice Bound pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h minminmsn-harbor-registry Bound pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6 500Gi RWO rbd 9h
2.修改PV状态 先把PV的状态由Released改变成 备注：默认创建的PV的回收策略是Delete就是用完就删除，之前特意把RECLAIM POLICY改为了Retain，在线修改PV回收策略可以参考文档在线修改 PV的回收策略 ，否则这里Helm Delete后就会自动删除PV，就没有后来这篇PV数据恢复操作了。 在线编辑PV,需要把其中claimRef这段删除，这样状态就可以变成Available了。
1 2 3 4 5 6 7 8 claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: minminmsn-harbor-chartmuseum namespace: default resourceVersion: "91736092" uid: b31ec8ca-c649-11ea-9386-52540089b2b6 persistentVolumeReclaimPolicy: Retain
具体如下修改
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: ceph.com/rbd rbdProvisionerIdentity: ceph.com/rbd creationTimestamp: "2019-05-24T06:33:55Z" finalizers: - kubernetes.io/pv-protection name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 resourceVersion: "91736100" selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6 spec: accessModes: - ReadWriteOnce capacity: storage: 50Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: minminmsn-harbor-chartmuseum namespace: default resourceVersion: "91736092" uid: b31ec8ca-c649-11ea-9386-52540089b2b6 persistentVolumeReclaimPolicy: Retain rbd: image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905 keyring: /etc/ceph/keyring monitors: - 10.0.4.8:6789 pool: rbd-k8s secretRef: name: ceph-secret namespace: default user: admin storageClassName: rbd volumeMode: Filesystem status: phase: Released
3.其他四个PV同样操作
1 2 3 4 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 [root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6
4.查看效果 现在看PV的STATUS已经变成了Available，然后CLAIM也变空了，这样就可以在后面绑定使用了
1 2 3 4 5 6 7 [root@elasticsearch01 harbor-helm]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Available rbd 417d pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Available rbd 417d pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d
创建PVC
1.先设置好PVC及PV对应关系
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 [root@elasticsearch01 yaml]# cat minminmsn.pvc apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-registry spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 2000Gi volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-jobservice spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: minminmsn-harbor-chartmuseum spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 50Gi volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: database-data-minminmsn-harbor-database-0 spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data-minminmsn-harbor-redis-0 spec: accessModes: - ReadWriteOnce storageClassName: "rbd" resources: requests: storage: 20Gi volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"
2.创建PVC
1 2 3 4 5 6 [root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc persistentvolumeclaim/minminmsn-harbor-registry created persistentvolumeclaim/minminmsn-harbor-jobservice created persistentvolumeclaim/minminmsn-harbor-chartmuseum created persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created persistentvolumeclaim/data-minminmsn-harbor-redis-0 created
3.检查PV与PVC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 [root@elasticsearch01 yaml]# kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Bound default/minminmsn-harbor-chartmuseum rbd 417d pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/minminmsn-harbor-jobservice rbd 417d pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Bound default/minminmsn-harbor-registry rbd 417d pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/database-data-minminmsn-harbor-database-0 rbd 417d pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/data-minminmsn-harbor-redis-0 rbd 417d [root@elasticsearch01 yaml]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 540d data-minminmsn-harbor-redis-0 Pending pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s database-data-minminmsn-harbor-database-0 Pending pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 0 rbd 12s minminmsn-harbor-chartmuseum Pending pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 0 rbd 12s minminmsn-harbor-jobservice Pending pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s minminmsn-harbor-registry Bound pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO rbd 12s [root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry Name: minminmsn-harbor-registry Namespace: default StorageClass: rbd Status: Bound Volume: pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe... pv.kubernetes.io/bind-completed: yes Finalizers: [kubernetes.io/pvc-protection] Capacity: 2000Gi Access Modes: RWO VolumeMode: Filesystem Events: <none> Mounted By: <none>
使用Hlem重新部署Harbor镜像库
1.部署前先删除版本
1 2 3 [root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn helm delete --purge minminmsn release "minminmsn" deleted
2.修改Harbor-helm的values.yaml中PVC相关值 注意existingClaim: “”由空值改成上面生成的PVC名字，注意对应关系，其他不变，具体变更如下
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 persistence: enabled: true # Setting it to "keep" to avoid removing PVCs during a helm delete # operation. Leaving it empty will delete PVCs after the chart deleted resourcePolicy: "keep" persistentVolumeClaim: registry: # Use the existing PVC which must be created manually before bound existingClaim: "minminmsn-harbor-registry" # Specify the "storageClass" used to provision the volume. Or the default # StorageClass will be used(the default). # Set it to "-" to disable dynamic provisioning storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 2000Gi chartmuseum: existingClaim: "minminmsn-harbor-chartmuseum" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 50Gi jobservice: existingClaim: "minminmsn-harbor-jobservice" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi # If external database is used, the following settings for database will # be ignored database: existingClaim: "database-data-minminmsn-harbor-database-0" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi # If external Redis is used, the following settings for Redis will # be ignored redis: existingClaim: "data-minminmsn-harbor-redis-0" storageClass: "rbd" subPath: "" accessMode: ReadWriteOnce size: 20Gi
3.重新部署
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 [root@elasticsearch01 harbor-helm]# helm install . --name minminmsn NAME: minminmsn LAST DEPLOYED: Wed Jul 15 11:18:13 2020 NAMESPACE: default STATUS: DEPLOYED RESOURCES: ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minminmsn-harbor-adminserver ClusterIP 10.254.58.23 <none> 80/TCP 1s minminmsn-harbor-chartmuseum ClusterIP 10.254.154.44 <none> 80/TCP 1s minminmsn-harbor-clair ClusterIP 10.254.25.107 <none> 6060/TCP 1s minminmsn-harbor-core ClusterIP 10.254.56.153 <none> 80/TCP 1s minminmsn-harbor-database ClusterIP 10.254.65.18 <none> 5432/TCP 1s minminmsn-harbor-jobservice ClusterIP 10.254.81.97 <none> 80/TCP 1s minminmsn-harbor-notary-server ClusterIP 10.254.99.90 <none> 4443/TCP 1s minminmsn-harbor-notary-signer ClusterIP 10.254.175.105 <none> 7899/TCP 1s minminmsn-harbor-portal ClusterIP 10.254.242.113 <none> 80/TCP 1s minminmsn-harbor-redis ClusterIP 10.254.127.40 <none> 6379/TCP 1s minminmsn-harbor-registry ClusterIP 10.254.158.222 <none> 5000/TCP,8080/TCP 1s ==> v1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE minminmsn-harbor-adminserver 1 1 1 0 1s minminmsn-harbor-chartmuseum 1 1 1 0 1s minminmsn-harbor-clair 1 0 0 0 1s minminmsn-harbor-core 1 0 0 0 1s minminmsn-harbor-jobservice 1 0 0 0 1s minminmsn-harbor-notary-server 1 0 0 0 1s minminmsn-harbor-notary-signer 1 0 0 0 1s minminmsn-harbor-portal 1 0 0 0 1s minminmsn-harbor-registry 1 0 0 0 1s ==> v1/StatefulSet NAME DESIRED CURRENT AGE minminmsn-harbor-database 1 1 1s minminmsn-harbor-redis 1 1 1s ==> v1beta1/Ingress NAME HOSTS ADDRESS PORTS AGE minminmsn-harbor-ingress core-harbor.minminmsn.com,notary-harbor.minminmsn.com 80, 443 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 1s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 Pending 0 1s minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Pending 0 1s minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Pending 0 1s minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 Pending 0 1s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 Pending 0 1s minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 Pending 0 1s minminmsn-harbor-database-0 0/1 Pending 0 1s minminmsn-harbor-redis-0 0/1 Pending 0 1s ==> v1/Secret NAME TYPE DATA AGE minminmsn-harbor-adminserver Opaque 4 1s minminmsn-harbor-chartmuseum Opaque 1 1s minminmsn-harbor-core Opaque 4 1s minminmsn-harbor-database Opaque 1 1s minminmsn-harbor-jobservice Opaque 1 1s minminmsn-harbor-registry Opaque 1 1s ==> v1/ConfigMap NAME DATA AGE minminmsn-harbor-adminserver 39 1s minminmsn-harbor-chartmuseum 24 1s minminmsn-harbor-clair 1 1s minminmsn-harbor-core 1 1s minminmsn-harbor-jobservice 1 1s minminmsn-harbor-notary-server 5 1s minminmsn-harbor-registry 2 1s NOTES: Please wait for several minutes for Harbor deployment to complete. Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com. For more details, please visit https://github.com/goharbor/harbor. 3.查看新生成Pods的信息 [root@elasticsearch01 harbor-helm]# kubectl get pods NAME READY STATUS RESTARTS AGE minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 9s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 ContainerCreating 0 9s minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Running 0 9s minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Running 0 9s minminmsn-harbor-database-0 0/1 Init:0/1 0 9s minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 ContainerCreating 0 9s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 ContainerCreating 0 9s minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 ContainerCreating 0 9s minminmsn-harbor-portal-64cf8b9b69-xm8nl 0/1 ContainerCreating 0 8s minminmsn-harbor-redis-0 0/1 ContainerCreating 0 9s minminmsn-harbor-registry-755746c5bb-q8m55 0/2 ContainerCreating 0 8s
再等2分钟查看就上恢复了
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 [root@elasticsearch01 harbor-helm]# kubectl get pods NAME READY STATUS RESTARTS AGE jenkins-0 1/1 Running 0 62d rbd-provisioner-67b4857bcd-rjwlg 1/1 Running 0 61d minminmsn-harbor-adminserver-b5d58db8c-wmrbd 1/1 Running 1 2m33s minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 1/1 Running 0 2m33s minminmsn-harbor-clair-54465ff7dd-d7bxx 1/1 Running 1 2m33s minminmsn-harbor-core-587cc5d9b5-2xxl9 1/1 Running 1 2m33s minminmsn-harbor-database-0 1/1 Running 0 2m33s minminmsn-harbor-jobservice-764bb697d-wsxqx 1/1 Running 0 2m33s minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 1/1 Running 0 2m33s minminmsn-harbor-notary-signer-8466d68f5b-klv76 1/1 Running 0 2m33s minminmsn-harbor-portal-64cf8b9b69-xm8nl 1/1 Running 0 2m32s minminmsn-harbor-redis-0 1/1 Running 0 2m33s minminmsn-harbor-registry-755746c5bb-q8m55 2/2 Running 0 2m32s
4.Harbor控制验证 证书更新了项目也恢复了 https://core-harbor.minminmsn.com/harbor/projects
