一、前提条件：
Kubernetes > v1.16
kubesphere > v3.0.0 参考文档 https://v3-0.docs.kubesphere.io/docs/quick-start/all-in-one-on-linux/
1 master + 3 node
二、在kubesphere部署
备注：步骤二仍适用于仅在K8S环境部署jmeter集群
在kubesphere后台的master节点创建以下文件 :
jmeter_cluster_create.sh jmeter_master_configmap.yaml jmeter_grafana_deploy.yaml
jmeter_master_deploy.yaml start_test.sh jmeter_grafana_svc.yaml
jmeter_influxdb_configmap.yaml jmeter_slaves_deploy.yaml
1、部署组件
执行jmeter_cluster_create.sh,并输入一个唯一的 namespace， 如 jmeter
./jmeter_cluster_create.sh
查看pods安装情况
[root@master ~]# kubectl get pod -n jmeter NAME READY STATUS RESTARTS AGE influxdb-jmeter-5f7dd64975-xkgqc 1/1 Running 0 21h jmeter-grafana-5856f7b855-rw4lp 1/1 Running 0 21h jmeter-master-75c7d64449-46pjm 1/1 Running 0 21h jmeter-slaves-d69c647d-dd6sb 1/1 Running 0 21h jmeter-slaves-d69c647d-glbjn 1/1 Running 0 21h jmeter-slaves-d69c647d-nsfgf 1/1 Running 0 21h
2、部署清单
2.1 执行脚本
jmeter_cluster_create.sh【用于创建命名空间和所有组件（jmeter master，slaves，influxdb 和 grafana）】:
#!/usr/bin/env bash #Create multiple Jmeter namespaces on an existing kuberntes cluster working_dir=`pwd` echo "checking if kubectl is present" if ! hash kubectl 2>/dev/null then echo "‘kubectl‘ was not found in PATH" echo "Kindly ensure that you can acces an existing kubernetes cluster via kubectl" exit fi kubectl version --short echo "Current list of namespaces on the kubernetes cluster:" echo kubectl get namespaces | grep -v NAME | awk ‘{print $1}‘ echo echo "Enter the name of the new tenant unique name, this will be used to create the namespace" read tenant echo #Check If namespace exists kubectl get namespace $tenant > /dev/null 2>&1 if [ $? -eq 0 ] then echo "Namespace $tenant already exists, please select a unique name" echo "Current list of namespaces on the kubernetes cluster" sleep 2 kubectl get namespaces | grep -v NAME | awk ‘{print $1}‘ exit 1 fi echo echo "Creating Namespace: $tenant" kubectl create namespace $tenant echo "Namspace $tenant has been created" echo echo "Creating Jmeter slave nodes" nodes=`kubectl get no | egrep -v "master|NAME" | wc -l` echo echo "Number of worker nodes on this cluster is " $nodes echo #echo "Creating $nodes Jmeter slave replicas and service" echo kubectl create -n $tenant -f $working_dir/jmeter_slaves_deploy.yaml kubectl create -n $tenant -f $working_dir/jmeter_slaves_svc.yaml echo "Creating Jmeter Master" kubectl create -n $tenant -f $working_dir/jmeter_master_configmap.yaml kubectl create -n $tenant -f $working_dir/jmeter_master_deploy.yaml echo "Creating Influxdb and the service" kubectl create -n $tenant -f $working_dir/jmeter_influxdb_configmap.yaml kubectl create -n $tenant -f $working_dir/jmeter_influxdb_deploy.yaml kubectl create -n $tenant -f $working_dir/jmeter_influxdb_svc.yaml echo "Creating Grafana Deployment" kubectl create -n $tenant -f $working_dir/jmeter_grafana_deploy.yaml kubectl create -n $tenant -f $working_dir/jmeter_grafana_svc.yaml echo "Printout Of the $tenant Objects" echo kubectl get -n $tenant all echo namespace = $tenant > $working_dir/tenant_export
2.2 jmeter-slave
jmeter_slaves_deploy.yaml（Jmeter slave 的部署文件）：
apiVersion: apps/v1 kind: Deployment metadata: name: jmeter-slaves labels: jmeter_mode: slave spec: replicas: 3 selector: matchLabels: jmeter_mode: slave template: metadata: labels: jmeter_mode: slave spec: containers: - name: jmslave image: wenxinxin/jmeter-slave:latest imagePullPolicy: IfNotPresent ports: - containerPort: 1099 - containerPort: 50000
- containerPort: 50001
resources:
limits:
cpu: 4000m
memory: 4Gi
requests:
cpu: 500m
memory: 512Mi
jmeter_slaves_svc.yaml（ Jmeter slave 服务的部署文件）：
apiVersion: v1 kind: Service metadata: name: jmeter-slaves-svc labels: jmeter_mode: slave spec: clusterIP: None ports: - port: 1099 name: first targetPort: 1099 - port: 50000 name: second targetPort: 50000 - port: 50001 name: third targetPort: 50001
2.3、jmeter_master
jmeter_master_configmap.yaml（jmeter_master 应用配置）：
apiVersion: v1 kind: ConfigMap metadata: name: jmeter-load-test labels: app: influxdb-jmeter data: load_test: | #!/bin/bash #Script created to invoke jmeter test script with the slave POD IP addresses #Script should be run like: ./load_test "path to the test script in jmx format" /jmeter/apache-jmeter-*/bin/jmeter -n -t $1 `getent ahostsv4 jmeter-slaves-svc | cut -d‘ ‘ -f1 | sort -u | awk -v ORS=, ‘{print $1}‘ | sed ‘s/,$//‘`
jmeter_master_deploy.yaml（jmeter_master 部署文件）：
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: jmeter-master labels: jmeter_mode: master spec: replicas: 1 selector: matchLabels: jmeter_mode: master template: metadata: labels: jmeter_mode: master spec: containers: - name: jmmaster image: wenxinxin/jmeter-master:latest imagePullPolicy: IfNotPresent command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumeMounts: - name: loadtest mountPath: /load_test subPath: "load_test" ports: - containerPort: 60000 resources: limits: cpu: 4000m memory: 4Gi requests: cpu: 500m memory: 512Mi volumes: - name: loadtest configMap: name: jmeter-load-test
2.4、influxdb
jmeter_influxdb_configmap.yaml（influxdb 的应用配置）：
apiVersion: v1 kind: ConfigMap metadata: name: influxdb-config labels: app: influxdb-jmeter data: influxdb.conf: | [meta] dir = "/var/lib/influxdb/meta" [data] dir = "/var/lib/influxdb/data" engine = "tsm1" wal-dir = "/var/lib/influxdb/wal" # Configure the graphite api [[graphite]] enabled = true bind-address = ":2003" # If not set, is actually set to bind-address. database = "jmeter" # store graphite data in this database
jmeter_influxdb_deploy.yaml（influxdb 部署文件）：
apiVersion: apps/v1 kind: Deployment metadata: name: influxdb-jmeter labels: app: influxdb-jmeter spec: replicas: 1 selector: matchLabels: app: influxdb-jmeter template: metadata: labels: app: influxdb-jmeter spec: containers: - image: influxdb:1.8.4-alpine imagePullPolicy: IfNotPresent name: influxdb volumeMounts: - name: config-volume mountPath: /etc/influxdb ports: - containerPort: 8083 name: influx - containerPort: 8086 name: api - containerPort: 2003 name: graphite volumes: - name: config-volume configMap: name: influxdb-config
jmeter_influxdb_svc.yaml（influxdb 服务的部署文件）：
apiVersion: v1 kind: Service metadata: name: jmeter-influxdb labels: app: influxdb-jmeter spec: ports: - port: 8083 name: http targetPort: 8083 - port: 8086 name: api targetPort: 8086 - port: 2003 name: graphite targetPort: 2003 selector: app: influxdb-jmeter
2.5、grafana
jmeter_grafana_deploy.yaml（grafana 部署文件）：
apiVersion: apps/v1 kind: Deployment metadata: name: jmeter-grafana labels: app: jmeter-grafana spec: replicas: 1 selector: matchLabels: app: jmeter-grafana template: metadata: labels: app: jmeter-grafana spec: containers: - name: grafana image: grafana/grafana:5.2.0 imagePullPolicy: IfNotPresent ports: - containerPort: 3000 protocol: TCP env: - name: GF_AUTH_BASIC_ENABLED value: "true" - name: GF_USERS_ALLOW_ORG_CREATE value: "true" - name: GF_AUTH_ANONYMOUS_ENABLED value: "true" - name: GF_AUTH_ANONYMOUS_ORG_ROLE value: Admin - name: GF_SERVER_ROOT_URL # If you‘re only using the API Server proxy, set this value instead: # value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy value: /
jmeter_grafana_svc.yaml（grafana 服务的部署文件）：
apiVersion: v1 kind: Service metadata: name: jmeter-grafana labels: app: jmeter-grafana spec: ports: - port: 3000 targetPort: 3000 selector: app: jmeter-grafana type: NodePort --- apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/service-weight: ‘jmeter-grafana: 100‘ name: jmeter-grafana-ingress spec: rules: # 配置七层域名 - host: grafana-jmeter.com http: paths: # 配置Context Path - path: / backend: serviceName: jmeter-grafana servicePort: 3000
3、 初始化dashbord
3.1 启动dashword脚本
$ ./dashboard.sh
检查 service 部署情况：
[root@master ~]# kubectl get svc -n jmeter NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jmeter-grafana NodePort 10.233.22.208 <none> 3000:30157/TCP 22h jmeter-influxdb ClusterIP 10.233.26.95 <none> 8083/TCP,8086/TCP,2003/TCP 22h jmeter-slaves-svc ClusterIP None <none> 1099/TCP,50000/TCP,50001/TCP 22h
通过 http:// 节点IP:30157 即可访问 grafana
在granafa导入dashbord模版(5496)，选择数据库即可。
3.2 部署文件
dashboard.sh 该脚本用于自动创建以下内容：
- （1）influxdb pod 中的一个 influxdb 数据库（Jmeter）
- （2）grafana 中的数据源（jmeterdb）
#!/usr/bin/env bash working_dir=`pwd` #Get namesapce variable tenant=`awk ‘{print $NF}‘ $working_dir/tenant_export` ## Create jmeter database automatically in Influxdb echo "Creating Influxdb jmeter Database" ##Wait until Influxdb Deployment is up and running ##influxdb_status=`kubectl get po -n $tenant | grep influxdb-jmeter | awk ‘{print $2}‘ | grep Running influxdb_pod=`kubectl get po -n $tenant | grep influxdb-jmeter | awk ‘{print $1}‘` kubectl exec -ti -n $tenant $influxdb_pod -- influx -execute ‘CREATE DATABASE jmeter‘ ## Create the influxdb datasource in Grafana echo "Creating the Influxdb data source" grafana_pod=`kubectl get po -n $tenant | grep jmeter-grafana | awk ‘{print $1}‘` ## Make load test script in Jmeter master pod executable #Get Master pod details master_pod=`kubectl get po -n $tenant | grep jmeter-master | awk ‘{print $1}‘` kubectl exec -ti -n $tenant $master_pod -- cp -r /load_test /![]()jmeter/load_test kubectl exec -ti -n $tenant $master_pod -- chmod 755 /jmeter/load_test ##kubectl cp $working_dir/influxdb-jmeter-datasource.json -n $tenant $grafana_pod:/influxdb-jmeter-datasource.json kubectl exec -ti -n $tenant $grafana_pod -- curl ‘http://admin:admin@127.0.0.1:3000/api/datasources‘ -X POST -H ‘Content-Type: application/json;charset=UTF-8‘ --data-binary ‘{"name":"jmeterdb","type":"influxdb","url":"http://jmeter-influxdb:8086","access":"proxy","isDefault":true,"database":"jmeter","user":"admin","password":"admin"}‘
4、 启动测试
4.1 执行脚本
$ ./start_test.sh
需要一个jmeter测试脚本
4.2 脚本说明
start_test.sh（此脚本用于自动运行 Jmeter 测试脚本，而无需手动登录 Jmeter 主 shell，它将询问 Jmeter 测试脚本的位置，然后将其复制到 Jmeter master pod 并启动自动对 Jmeter slave 进行测试）:
#!/usr/bin/env bash #Script created to launch Jmeter tests directly from the current terminal without accessing the jmeter master pod. #It requires that you supply the path to the jmx file #After execution, test script jmx file may be deleted from the pod itself but not locally. #直接从当前终端启动 Jmeter 测试而创建的脚本，无需访问 Jmeter master pod。 #要求提供 jmx 文件的路径 #执行后，测试脚本 jmx 文件不会从 pod 本身删除，也不会在本地删除。 working_dir="`pwd`" # 获取 namesapce 变量 tenant=`awk ‘{print $NF}‘ "$working_dir/tenant_export"` jmx="$1" [ -n "$jmx" ] || read -p ‘Enter path to the jmx file ‘ jmx if [ ! -f "$jmx" ]; then echo "Test script file was not found in PATH" echo "Kindly check and input the correct file path" exit fi test_name="$(basename "$jmx")" # 获取 master pod 详细信息 master_pod=`kubectl get po -n $tenant | grep jmeter-master | awk ‘{print $1}‘` kubectl cp "$jmx" -n $tenant "$master_pod:/$test_name" ## 启动 Jmeter 压测 kubectl exec -ti -n $tenant $master_pod -- /bin/bash /load_test "$test_name" kubectl exec -ti -n $tenant $master_pod -- /bin/bash /load_test "$test_name"
jmeter_stop.sh（停止测试）:
#!/usr/bin/env bash #Script writtent to stop a running jmeter master test #Kindly ensure you have the necessary kubeconfig #编写脚本来停止运行的 jmeter master 测试 #请确保你有必要的 kubeconfig working_dir=`pwd` #获取 namesapce 变量 tenant=`awk ‘{print $NF}‘ $working_dir/tenant_export` master_pod=`kubectl get po -n $tenant | grep jmeter-master | awk ‘{print $1}‘` kubectl -n $tenant exec -it $master_pod -- bash -c "./jmeter/apache-jmeter-5.2.1/bin/stoptest.sh"
三、使用kubesphere管理jmeter集群：
1、新建企业空间
2、进入集群管理-->项目管理-->用户项目
将项目jmeter分配到企业空间wx
3、通过工作台-->企业空间wx-->项目jmeter，查看和管理jmeter集群。
参考资料：
- [1]：https://github.com/kubernauts/jmeter-kubernetes