高级软工课程-微服务上云实践

前言

学校课程课设是写一个商城,要求微服务+上云。整体架构是前端vue打到nginx,nginx做负载均衡打到gateway,gateway根据token鉴权+redis缓存权限和路径的映射,后端服务注册到nacos实现服务发现,消息中间件采用rabbitmq。最后使用阿里云k8s部署,并使用hpa实现动态扩容,hrey进行压力测试,并加了skywalking进行监控。
设计图如下:

-w848

k8s配置等

k8s默认使用的配置文件为${HOME}/.kube/config,如果设置了$KUBECONFIG则会使用该文件。

k8s部署过程

首先docker镜像部署到云镜像仓库,使用阿里云,参考如下文章即可。
https://mp.weixin.qq.com/s/jEAA1OcXMq-kORlRIH8kyg

接下来使用kompose将docker-compose转换为k8s的yaml文件:

1
kompose convert

看一下生成的yaml,有service和deployment。deployment通俗理解就是管理pods,比如pods崩了就会重新起来。service可以设置LoadBalancer,可以走云服务商的LB,并设置公网IP。默认的话是Cluster IP,k8s内网中通过Cluster IP通信。按上图架构,只有vue+nginx的机器需要暴露公网ip。

上云的话deployment中的image要先docker push到一个云镜像仓库,然后设置sercret来拉取。注意kubectl create secret生成的secret默认是当前namespace。

1
kubectl create secret docker-registry fdu  --docker-server=registry.cn-hangzhou.aliyuncs.com --docker-username=x --docker-password=x -n ase-ns-07

gateway的配置了skywalking的deployment.yaml如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.26.0 (40646f47)
creationTimestamp: null
labels:
apps: gateway
name: gateway
namespace: ase-ns-07
spec:
selector:
matchLabels:
apps: gateway
strategy: {}
template:
metadata:
labels:
io.kompose.network/mynetwork: "true"
apps: gateway
spec:
volumes:
- name: skywalking-agent
emptyDir: { }
initContainers:
- name: agent-container
image: apache/skywalking-java-agent:8.5.0-jdk8
volumeMounts:
- name: skywalking-agent
mountPath: /agent
command: [ "/bin/sh" ]
args: [ "-c", "cp -R /skywalking/agent /agent/" ]
imagePullSecrets:
- name: fdu
containers:
- image: registry.cn-hangzhou.aliyuncs.com/fdu/gateway:0.1
name: gateway
ports:
- containerPort: 8090
resources:
requests:
cpu: 5m
memory: "200Mi"
volumeMounts:
- name: skywalking-agent
mountPath: /skywalking
env:
- name: JAVA_TOOL_OPTIONS
value: "-javaagent:/skywalking/agent/skywalking-agent.jar"
- name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
value: "oap:11800"
- name: SW_AGENT_NAME
value: "gateway"
restartPolicy: Always
status: {}

skywalking部署过程踩坑

skywalking的实现原理是java agent的premain模式,通过注入agent实现oap对每个instance的监控。instance部分是每个实例的性能状态,trace模块可以查看api经过pods的完整调用路径。

-w1884

-w453

-w1895

skywalking的部署需要依赖elasticsearch实现持久化,需要rbac设置下对k8s api的访问权限,核心分为ui和oap两部分。一个可用的yaml如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
apiVersion: v1
kind: Service
metadata:
labels:
app: elasticsearch-service
name: elasticsearch
namespace: ase-ns-07
spec:
ports:
- name: http
port: 9200
protocol: TCP
targetPort: 9200
- name: tcp
port: 9300
protocol: TCP
targetPort: 9300
selector:
app: elasticsearch
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: elasticsearch
name: elasticsearch
namespace: ase-ns-07
spec:
replicas: 1
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- env:
- name: TZ
value: CST-8
- name: discovery.type
value: single-node
image: elasticsearch:7.0.0
# image: registry.cn-chengdu.aliyuncs.com/lswzw/elasticsearch:6.8.14
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 2
successThreshold: 1
tcpSocket:
port: 9300
timeoutSeconds: 2
name: elasticsearch
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: tcp
protocol: TCP
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 2
successThreshold: 2
tcpSocket:
port: 9300
timeoutSeconds: 2
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 200m
memory: 512Mi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
apiVersion: apps/v1
kind: Deployment
metadata:
name: oap
namespace: ase-ns-07
labels:
app: oap
spec:
replicas: 2
selector:
matchLabels:
app: oap
template:
metadata:
labels:
app: oap
spec:
serviceAccountName: skywalking-oap-sa
containers:
- name: oap
image: apache/skywalking-oap-server:8.5.0-es7
imagePullPolicy: Always
livenessProbe:
tcpSocket:
port: 12800
initialDelaySeconds: 15
periodSeconds: 20
readinessProbe:
tcpSocket:
port: 12800
initialDelaySeconds: 15
periodSeconds: 20
ports:
- containerPort: 11800
name: grpc
- containerPort: 12800
name: rest
resources:
requests:
memory: 2Gi
limits:
memory: 4Gi
env:
- name: JAVA_OPTS
value: "-Xmx2g -Xms2g"
- name: SW_CLUSTER
value: standalone
- name: SKYWALKING_COLLECTOR_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: SW_STORAGE
value: elasticsearch7
- name: SW_STORAGE_ES_CLUSTER_NODES
value: elasticsearch:9200
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: v1
kind: ServiceAccount
metadata:
name: skywalking-oap-sa
namespace: ase-ns-07

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: skywalking-clusterrolebinding
subjects:
- kind: Group
name: system:serviceaccounts:skywalking
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: skywalking-clusterrole
apiGroup: rbac.authorization.k8s.io
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: skywalking-clusterrole
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"]

To learn

k8s网络
hpa算法
statefulset使用
k8s鉴权细节

Proudly powered by Hexo and Theme by Hacker
© 2021 LFY