前言:本文是一篇 kubernetes(下文用 k8s 代替)的入门文章,将会涉及 k8s 的架构、集群搭建、一个 Redis 的例子,以及如何使用 operator-sdk 开发 operator 的教程。在文章过程中,会穿插引出 Pod、Deployment、StatefulSet 等 k8s 的概念,这些概念通过例子引出来,更容易理解和实践。文章参考了很多博客以及资料,放在最后参考资料部分。
一 k8s架构
-
Apiserver:上知天文下知地理,上连其余组件,下接ETCD,提供各类 api 处理、鉴权,和 Node 上的 kubelet 通信等,只有 apiserver 会连接 ETCD。
-
Controller-manager:控制各类 controller,通过控制器模式,致力于将当前状态转变为期望的状态。
-
Scheduler:调度,打分,分配资源。
-
Etcd:整个集群的数据库,也可以不部署在 Master 节点,单独搭建。
-
Docker:具体跑应用的载体。 -
Kube-proxy:主要负责网络的打通,早期利用 iptables,现在使用 ipvs技术。 -
Kubelet:agent,负责管理容器的生命周期。
二 搭建 k8s 集群
-
当我们安装了 Docker Desktop APP 之后,勾选 k8s 支持就能搭建起来。
-
使用 MiniKube 来搭建,社区提供的一键安装脚本。
-
直接在云平台购买,例如阿里云 ack。
-
使用 kubeadmin,这是 k8s 社区推荐的可以部署生产级别 k8s 的工具。
-
使用二进制,下载各组件安装,此教程需要注意,下载的各组件版本要和博客中保持一致,就可以成功。
➜ ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:16:05Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:10:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
三 从需求出发
-
部署一个Redis服务 -
支持高可用 -
提供统一的 EndPoint 访问地址
1 部署单机版
➜ ~ kubectl run redis --image=redis
pod/redis created
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis 1/1 Running 0 5s
➜ ~ kubectl exec -it redis -- bash
root@redis:/data# redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379>
2 Pod 与 Deployment
➜ ~ kubectl create deployment redis-deployment --image=redis
deployment.apps/redis-deployment created
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis 1/1 Running 0 32m
redis-deployment-866c4c6cf9-8z8k5 1/1 Running 0 8s
➜ ~
➜ ~ kubectl delete pod redis redis-deployment-866c4c6cf9-8z8k5
pod "redis" deleted
pod "redis-deployment-866c4c6cf9-8z8k5" deleted
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-deployment-866c4c6cf9-zskkb 1/1 Running 0 10s
➜ ~
3 k8s 使用 yaml 来描述命令
➜ ~ cat pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: redis
spec:
containers:
- name: redis
image: redis
➜ ~ kubectl create -f pod.yaml
pod/redis created
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis 1/1 Running 0 6s
redis-deployment-866c4c6cf9-zskkb 1/1 Running 0 6m32s
四 k8s 组件调用流程
下面我们看下kubectl create deployment redis-deployment --image=redis下发之后,k8s 集群做了什么。
-
首先 controller-manager, scheduler, kubelet 都会和 apiserver 开始进行 List-Watch 模型,List 是拿到当前的状态,Watch 是拿到期望状态,然后 k8s 集群会致力于将当前状态达到达期望状态。
-
kubectl 下发命令到 apiserver,鉴权处理之后将创建信息存入 etcd,Deployment 的实现是使用 ReplicaSet 控制器,当 controller-manager 提前拿到当前的状态(pod=0),接着接收到期望状态,需要创建 ReplicaSet(pod=1),就会开始创建 Pod。
-
然后 scheduler 会进行调度,确认 Pod 被创建在哪一台 Node 上。
-
之后 Node 上的 kubelet 真正拉起一个 docker。
五 部署主从版本
1 StatefulSet
-
拓扑状态:实例的创建顺序和编号是顺序的,会按照 name-index 来编号,比如 redis-0,redis-1 等。
-
存储状态:可以通过声明使用外部存储,例如云盘等,将数据保存,从而 Pod 重启,重新调度等都能读到云盘中的数据。
apiVersion: apps/v1
kind: StatefulSet # 类型为 statefulset
metadata:
name: redis-sfs # app 名称
spec:
serviceName: redis-sfs # 这里的 service 下面解释
replicas: 2 # 定义了两个副本
selector:
matchLabels:
app: redis-sfs
template:
metadata:
labels:
app: redis-sfs
spec:
containers:
- name: redis-sfs
image: redis # 镜像版本
command:
- bash
- "-c"
- |
set -ex
ordinal=`hostname | awk -F '-' '{print $NF}'` # 使用 hostname 获取序列
if [[ $ordinal -eq 0 ]]; then # 如果是 0,作为主
echo > /tmp/redis.conf
else
echo "slaveof redis-sfs-0.redis-sfs 6379" > /tmp/redis.conf # 如果是 1,作为备
fi
redis-server /tmp/redis.conf
➜ ~ kubectl create -f server.yaml
statefulset.apps/redis-sfs created
➜ ~ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis 1/1 Running 0 65m
redis-deployment-866c4c6cf9-zskkb 1/1 Running 0 71m
redis-sfs-0 1/1 Running 0 33s # 按照
redis-sfs-1 1/1 Running 0 28s
➜ ~ kubectl logs -f redis-sfs-1
1:S 05 Nov 2021 08:02:44.243 * Connecting to MASTER redis-sfs-0.redis-sfs:6379
1:S 05 Nov 2021 08:02:50.287 # Unable to connect to MASTER: Resource temporarily unavailable
...
2 Headless Service
-
VIP:访问 VIP 随机返回一个后端的 Pod -
DNS:通过 DNS 解析到后端某个 Pod 上
...svc.cluster.local
apiVersion: v1
kind: Service
metadata:
name: redis-sfs
labels:
app: redis-sfs
spec:
clusterIP: None # 这里的 None 就是 Headless 的意思,表示会主动由 k8s 分配
ports:
- port: 6379
name: redis-sfs
selector:
app: redis-sfs
➜ ~ kubectl create -f service.yaml
service/redis-sfs created
➜ ~ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1
443/TCP 24d redis-sfs ClusterIP None
6379/TCP 33s ➜ ~ kubectl logs -f redis-sfs-1
...
1:S 05 Nov 2021 08:23:31.341 * Connecting to MASTER redis-sfs-0.redis-sfs:6379
1:S 05 Nov 2021 08:23:31.345 * MASTER REPLICA sync started
1:S 05 Nov 2021 08:23:31.345 * Non blocking connect for SYNC fired the event.
1:S 05 Nov 2021 08:23:31.346 * Master replied to PING, replication can continue...
1:S 05 Nov 2021 08:23:31.346 * Partial resynchronization not possible (no cached master)
1:S 05 Nov 2021 08:23:31.348 * Full resync from master: 29d1c03da6ee2af173b8dffbb85b6ad504ccc28f:0
1:S 05 Nov 2021 08:23:31.425 * MASTER REPLICA sync: receiving 175 bytes from master to disk
1:S 05 Nov 2021 08:23:31.426 * MASTER REPLICA sync: Flushing old data
1:S 05 Nov 2021 08:23:31.426 * MASTER REPLICA sync: Loading DB in memory
1:S 05 Nov 2021 08:23:31.431 * Loading RDB produced by version 6.2.6
1:S 05 Nov 2021 08:23:31.431 * RDB age 0 seconds
1:S 05 Nov 2021 08:23:31.431 * RDB memory usage when created 1.83 Mb
1:S 05 Nov 2021 08:23:31.431 # Done loading RDB, keys loaded: 0, keys expired: 0.
1:S 05 Nov 2021 08:23:31.431 * MASTER REPLICA sync: Finished with success
^C
➜ ~ kubectl exec -it redis-sfs-1 -- bash
root@redis-sfs-1:/data# redis-cli -h redis-sfs-0.redis-sfs.default.svc.cluster.local
redis-sfs-0.redis-sfs.default.svc.cluster.local:6379> ping
PONG
redis-sfs-0.redis-sfs.default.svc.cluster.local:6379>
六 Operator
-
定义对象,比如你的集群默认有几个节点,都有啥组件 -
定义对象触发的操作,当创建对象时候要做什么流程,HA 时候要做什么流程等
1 准备工作
-
安装好 go 环境 -
安装 operator-sdk
2 初始化项目
➜ ~ cd $GOPATH/src
➜ src mkdir memcached-operator
➜ src cd memcached-operator
➜ memcached-operator operator-sdk init --domain yangbodong22011 --repo github.com/yangbodong22011/memcached-operator --skip-go-version-check // 这里需要注意 domain 最好是和你在 https://hub.docker.com 的注册名称相同,因为后续会发布 docker 镜像
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.9.2
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ operator-sdk create api
3 创建 API 和 Controller
➜ memcached-operator operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1alpha1/memcached_types.go
controllers/memcached_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.6.1
go get: installing executables with 'go get' in module mode is deprecated.
To adjust and download dependencies of the current module, use 'go get -d'.
To install using requirements of the current module, use 'go install'.
To install ignoring the current module, use 'go install' with a version,
like 'go install example.com/cmd@latest'.
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
...
go get: added sigs.k8s.io/yaml v1.2.0
/Users/yangbodong/go/src/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
➜ memcached-operator
➜ memcached-operator vim api/v1alpha1/memcached_types.go // 修改下面 Memcached 集群的定义
// MemcachedSpec defines the desired state of Memcached
type MemcachedSpec struct {
//+kubebuilder:validation:Minimum=0
// Size is the size of the memcached deployment
Size int32 `json:"size"`
}
// MemcachedStatus defines the observed state of Memcached
type MemcachedStatus struct {
// Nodes are the names of the memcached pods
Nodes []string `json:"nodes"`
}
➜ memcached-operator make generate
/Users/yangbodong/go/src/memcached-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
➜ memcached-operator make manifests
/Users/yangbodong/go/src/memcached-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
➜ memcached-operator
4 实现 Controller
➜ memcached-operator vim controllers/memcached_controller.go
https://raw.githubusercontent.com/operator-framework/operator-sdk/latest/testdata/go/v3/memcached-operator/controllers/memcached_controller.go //
将 example 换成 yangbodong22011,注意,// 注释中的也要换,实际不是注释,而是一种格式
➜ memcached-operator go mod tidy; make manifests
/Users/yangbodong/go/src/memcached-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
5 发布 operator 镜像
➜ memcached-operator vim Makefile
将 -IMG ?= controller:latest 改为 +IMG ?= $(IMAGE_TAG_BASE):$(VERSION)
➜ memcached-operator docker login // 提前登录下 docker
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: yangbodong22011
Password:
WARNING! Your password will be stored unencrypted in /Users/yangbodong/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
➜ memcached-operator sudo make docker-build docker-push
...
=> => writing image sha256:a7313209e321c84368c5cb7ec820fffcec2d6fcb510219d2b41e3b92a2d5545a 0.0s
=> => naming to docker.io/yangbodong22011/memcached-operator:0.0.1 0.0s
fac03a24e25a: Pushed
6d75f23be3dd: Pushed
0.0.1: digest: sha256:242380214f997d98186df8acb9c13db12f61e8d0f921ed507d7087ca4b67ce59 size: 739
6 修改镜像和部署
➜ memcached-operator vim config/manager/manager.yaml
image: controller:latest 修改为 yangbodong22011/memcached-operator:0.0.1
➜ memcached-operator vim config/default/manager_auth_proxy_patch.yaml
因为国内访问不了 gcr.io
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0 修改为 kubesphere/kube-rbac-proxy:v0.8.0
➜ memcached-operator make deploy
...
configmap/memcached-operator-manager-config created
service/memcached-operator-controller-manager-metrics-service created
deployment.apps/memcached-operator-controller-manager created
➜ memcached-operator kubectl get deployment -n memcached-operator-system // ready 说明 operator 已经部署了
NAME READY UP-TO-DATE AVAILABLE AGE
memcached-operator-controller-manager 1/1 1 1 31s
➜ memcached-operator
7 创建 Memcached 集群
➜ memcached-operator cat config/samples/cache_v1alpha1_memcached.yaml
apiVersion: cache.yangbodong22011/v1alpha1
kind: Memcached
metadata:
name: memcached-sample
spec:
size: 1
➜ memcached-operator kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml
memcached.cache.yangbodong22011/memcached-sample created
➜ memcached-operator kubectl get pods
NAME READY STATUS RESTARTS AGE
memcached-sample-6c765df685-xhhjc 1/1 Running 0 104s
redis 1/1 Running 0 177m
redis-deployment-866c4c6cf9-zskkb 1/1 Running 0 3h4m
redis-sfs-0 1/1 Running 0 112m
redis-sfs-1 1/1 Running 0 112m
➜ memcached-operator
➜ ~ kubectl logs -f deployment/memcached-operator-controller-manager -n memcached-operator-system
2021-11-05T09:50:46.042Z INFO controller-runtime.manager.controller.memcached Creating a new Deployment {"reconciler group": "cache.yangbodong22011", "reconciler kind": "Memcached", "name": "memcached-sample", "namespace": "default", "Deployment.Namespace": "default", "Deployment.Name": "memcached-sample"}
七 总结
八 参考资料
[2] 《Kubernetes 权威指南》第五版
[3] 《Large-scale cluster management at Google with Borg》
文章评论