概述

阿里云服务网格ASM支持Sidecar模式和Sidecarless模式。Sidecar模式，即在每个服务实例旁运行一个代理，是当前最主流、稳定的方案，但这种架构引入了显著的延迟和资源开销。为了解决Sidecar模式所带来的延迟和资源消耗，近些年出现了不同的Sidecarless模式方案，例如Istio Ambient，Istio Ambient 通过在每个节点部署ztunnel，对节点上运行的Pod进行四层流量代理，并引入了waypoint对七层流量进行代理。虽然Sidecarless模式能够降低延迟和资源消耗，但其稳定性以及功能的完善程度仍有待提升。

阿里云服务网格ASM当前支持不同的Sidecarless模式，例如Istio Ambient模式，ACMG模式以及Kmesh等。Kmesh（具体参考https://kmesh.net/）是一款基于ebpf+可编程内核实现的高性能服务网格数据面软件。通过将流量治理下沉到内核，实现网格内服务通信无需经过代理软件，大大缩减了流量转发路径，有效提升了服务访问的转发性能。

- Kmesh简介 -

Kmesh的双引擎模式（dual-engine）模式使用eBPF在内核空间拦截流量，部署Waypoint Proxy来处理复杂的L7流量管理，从而将L4和L7的治理在内核空间（eBPF）和用户空间（Waypoint）之间分离。相比Istio的Ambient Mesh，延迟减少了30%。相较于内核原生模式，双引擎模式无需对内核进行增强，具有更广泛的适用性。

图丨Kmesh双引擎模式架构图

当前阿里云服务网格ASM支持使用Kmesh双引擎模式作为服务网格的数据面之一，从而实现更加高效的服务治理。具体来说, 可以使用ASM作为控制面，并在ACK Kubernetes集群中部署Kmesh作为数据面。

在阿里云ACK集群中

部署Kmesh并连接ASM控制面

- 使用前提 -

参考阿里云服务网格ASM的官方文档, 创建一个ASM集群实例与ACK Kubernetes集群, 并将ACK Kubernetes集群添加到ASM集群实例中进行管理。具体操作步骤, 可以参考文档：

https://help.aliyun.com/zh/asm/getting-started/add-a-cluster-to-an-asm-instance-1?spm=a2c4g.11186623.help-menu-147365.d_1_2.436356ccEXYKBU

- 安装Kmesh -

执行以下命令，将Kmesh项目下载到本地。

git clone https://github.com/kmesh-net/kmesh.git && cd kmesh

查看ASM控制面Service信息

下载完成后，您首先需要执行以下命令，查看当前ASM控制面在集群中的Service名称，以配置Kmesh与ASM控制面的连接。

kubectl get svc -n istio-system | grep istiod# istiod-1-22-6 ClusterIP None <none> 15012/TCP 2d

使用kubectl安装Kmesh

您可以使用kubectl在ACK Kubernetes集群中安装 Kmesh，但在安装前，请为Kmesh DaemonSet添加ClusterId和xdsAddress环境变量，用于Kmesh和ASM控制面认证与连接。ClusterId为您部署Kmesh的ACK Kubernetes集群Id，xdsAddress为ASM控制面的Service信息。

apiVersion: apps/v1kind: DaemonSetmetadata: name: kmesh labels: app: kmesh namespace: kmesh-systemspec: spec: containers: - env: # 修改 xdsAddress 为 ASM 控制面 Service - name: XDS_ADDRESS value: "istiod-1-22-6.istio-system.svc:15012" # 添加阿里云 ACK 集群 id - name: CLUSTER_ID value: "cluster-id" ...

修改以下命令中的Kmesh Daemonset环境变量后，执行以部署Kmesh。

kubectl apply -f -<<EOFapiVersion: apps/v1kind: DaemonSetmetadata: name: kmesh labels: app: kmesh namespace: kmesh-systemspec: selector: matchLabels: app: kmesh template: metadata: labels: app: kmesh annotations: prometheus.io/path: "status/metric" prometheus.io/port: "15020" prometheus.io/scrape: "true" spec: tolerations: - effect: NoSchedule operator: Exists - key: CriticalAddonsOnly operator: Exists - effect: NoExecute operator: Exists volumes: # use cgroup requires - name: mnt hostPath: path: /mnt # for eBPF program into the host machine - name: sys-fs-bpf hostPath: path: /sys/fs/bpf # required for compiling and building ko - name: lib-modules hostPath: path: /lib/modules # k8s default cni conflist path - name: cni hostPath: path: /etc/cni/net.d # k8s default cni path - name: kmesh-cni-install-path hostPath: path: /opt/cni/bin - name: host-procfs hostPath: path: /proc type: Directory - name: istiod-ca-cert configMap: defaultMode: 420 name: istio-ca-root-cert - name: istio-token projected: defaultMode: 420 sources: - serviceAccountToken: audience: istio-ca expirationSeconds: 43200 path: istio-token # 修改 xdsAddress 为 ASM 控制面 Service - name: XDS_ADDRESS value: "istiod-1-22-6.istio-system.svc:15012" # 添加阿里云 ACK 集群 id - name: CLUSTER_ID value: "cluster-id" containers: - name: kmesh image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/kmesh:latest imagePullPolicy: IfNotPresent command: ["/bin/sh", "-c"] args: [ "./start_kmesh.sh --mode=dual-engine --enable-bypass=false --enable-bpf-log=true", ] securityContext: privileged: true capabilities: add: ["all"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: INSTANCE_IP valueFrom: fieldRef: fieldPath: status.podIP - name: XDS_ADDRESS value: "istiod.istio-system.svc:15012" - name: SERVICE_ACCOUNT valueFrom: fieldRef: fieldPath: spec.serviceAccountName - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: mnt mountPath: /mnt readOnly: false - name: sys-fs-bpf mountPath: /sys/fs/bpf readOnly: false - name: lib-modules mountPath: /lib/modules readOnly: false # k8s default cni conflist path - name: cni mountPath: /etc/cni/net.d readOnly: false # k8s default cni path - name: kmesh-cni-install-path mountPath: /opt/cni/bin readOnly: false - mountPath: /host/proc name: host-procfs readOnly: true - name: istiod-ca-cert mountPath: /var/run/secrets/istio - name: istio-token mountPath: /var/run/secrets/tokens resources: limits: # image online-compile needs 800Mi, or only 200Mi memory: "800Mi" cpu: "1" priorityClassName: system-node-critical serviceAccountName: kmesh---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: name: kmesh labels: app: kmeshrules:- apiGroups: [""] resources: ["pods","services","namespaces"] verbs: ["get", "update", "patch", "list", "watch"]- apiGroups: ["apps"] resources: ["daemonsets"] verbs: ["get"]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: kmesh labels: app: kmeshroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: kmeshsubjects:- kind: ServiceAccount name: kmesh namespace: kmesh-system---apiVersion: v1kind: Namespacemetadata: name: kmesh-system---apiVersion: networking.istio.io/v1alpha3kind: EnvoyFiltermetadata: name: add-listener-filter namespace: istio-system labels: asm-system: 'true' provider: asmspec: workloadSelector: labels: gateway.istio.io/managed: istio.io-mesh-controller configPatches: - applyTo: LISTENER match: proxy: proxyVersion: .* patch: operation: ADD value: name: kmesh-listener address: socket_address: protocol: TCP address: 0.0.0.0 port_value: 15019 additional_addresses: - address: socket_address: protocol: TCP address: "::" port_value: 15019 default_filter_chain: filters: - name: envoy.filters.network.tcp_proxy typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: kmesh cluster: main_internal filter_chains: - filter_chain_match: application_protocols: - "http/1.1" - "h2c" filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: kmesh route_config: name: default virtual_hosts: - name: default domains: - '*' routes: - match: prefix: "/" route: cluster: main_internal http_filters: - name: waypoint_downstream_peer_metadata typed_config: "@type": type.googleapis.com/udpa.type.v1.TypedStruct type_url: type.googleapis.com/io.istio.http.peer_metadata.Config value: downstream_discovery: - workload_discovery: {} shared_with_upstream: true - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router listener_filters: - name: "envoy.listener.kmesh_tlv" typed_config: "@type": "type.googleapis.com/udpa.type.v1.TypedStruct" "type_url": "type.googleapis.com/envoy.listener.kmesh_tlv.config.KmeshTlv" - name: "envoy.filters.listener.http_inspector" typed_config: "@type": "type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector"---apiVersion: networking.istio.io/v1alpha3kind: EnvoyFiltermetadata: name: skip-tunneling namespace: istio-system labels: asm-system: 'true' provider: asmspec: workloadSelector: labels: gateway.istio.io/managed: istio.io-mesh-controller configPatches: - applyTo: NETWORK_FILTER match: proxy: proxyVersion: .* listener: filterChain: filter: name: envoy.filters.network.tcp_proxy patch: operation: REPLACE value: name: envoy.filters.network.tcp_proxy typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: kmesh_original_dst_cluster cluster: kmesh_original_dst_cluster---apiVersion: networking.istio.io/v1alpha3kind: EnvoyFiltermetadata: name: add-original-dst-cluster namespace: istio-system labels: asm-system: 'true' provider: asmspec: workloadSelector: labels: gateway.istio.io/managed: istio.io-mesh-controller configPatches: - applyTo: CLUSTER match: proxy: proxyVersion: .* context: SIDECAR_INBOUND patch: operation: ADD value: name: "kmesh_original_dst_cluster" type: ORIGINAL_DST connect_timeout: 2s lb_policy: CLUSTER_PROVIDED---apiVersion: v1kind: ServiceAccountmetadata: name: kmesh namespace: kmesh-systemEOF

- 检查Kmesh服务状态 -

安装完毕后，执行以下命令查看Kmesh服务启动状态。

kubectl get pods -A | grep kmesh# kmesh-system kmesh-l5z2j 1/1 Running 0 117m

执行以下命令查看Kmesh服务运行状态。

kubectl logs -f -n kmesh-system kmesh-l5z2j# time="2024-02-19T10:16:52Z" level=info msg="service node sidecar~192.168.11.53~kmesh-system.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/envoy# time="2024-02-19T10:16:52Z" level=info msg="options InitDaemonConfig successful" subsys=manager# time="2024-02-19T10:16:53Z" level=info msg="bpf Start successful" subsys=manager# time="2024-02-19T10:16:53Z" level=info msg="controller Start successful" subsys=manager# time="2024-02-19T10:16:53Z" level=info msg="command StartServer successful" subsys=manager# time="2024-02-19T10:16:53Z" level=info msg="start write CNI config\n" subsys="cni installer"# time="2024-02-19T10:16:53Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"# time="2024-02-19T10:16:54Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"# time="2024-02-19T10:16:54Z" level=info msg="kubeconfig either does not exist or is out of date, writing a new one" subsys="cni installer"# time="2024-02-19T10:16:54Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"# time="2024-02-19T10:16:54Z" level=info msg="command Start cni successful" subsys=manager

您可以通过执行以下命令来对指定命名空间启用Kmesh。

kubectl label namespace default istio.io/dataplane-mode=Kmesh

- 流量调度示例 -

部署示例应用和流量调度规则

在为default命名空间启用Kmesh后，执行以下命令安装示例应用。

kubectl apply -f samples/fortio/fortio-route.yamlkubectl apply -f samples/fortio/netutils.yaml

执行以下命令查看实例应用运行状态。

kubectl get pod # NAME READY STATUS RESTARTS AGE# fortio-v1-596b55cb8b-sfktr 1/1 Running 0 57m# fortio-v2-76997f99f4-qjsmd 1/1 Running 0 57m# netutils-575f5c569-lr98z 1/1 Running 0 67mkubectl describe pod netutils-575f5c569-lr98z | grep Annotations# Annotations: kmesh.net/redirection: enabled

当您看到应用Pod具有kmesh.net/redirection: enabled时，代表Kmesh转发已经对该Pod启用。

执行以下命令查看当前定义的流量调度规则，可以看出，此时定义了90%的流量流向v1版本的fortio，10%的流量流向v2版本的fortio。

kubectl get virtualservices -o yaml# apiVersion: v1# items:# - apiVersion: networking.istio.io/v1beta1# kind: VirtualService# metadata:# annotations:# kubectl.kubernetes.io/last-applied-configuration: |# {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"fortio","namespace":"default"},"spec":{"hosts":["fortio"],"http":[{"route":[{"destination":{"host":"fortio","subset":"v1"},"weight":90},{"destination":{"host":"fortio","subset":"v2"},"weight":10}]}]}}# creationTimestamp: "2024-07-09T09:00:36Z"# generation: 1# name: fortio# namespace: default# resourceVersion: "11166"# uid: 0a07f283-ac26-4d86-b3bd-ce6aa07dc628# spec:# hosts:# - fortio# http:# - route:# - destination:# host: fortio# subset: v1# weight: 90# - destination:# host: fortio# subset: v2# weight: 10# kind: List# metadata:# resourceVersion: ""

发起测试流量

您可以通过执行以下命令发起测试流量，您应该可以看到，此时只有大约10%的流量流向v2版本的fortio。

for i in {1..20}; do kubectl exec -it $(kubectl get pod | grep netutils | awk '{print $1}') -- curl -v $(kubectl get svc -owide | grep fortio | awk '{print $3}'):80 | grep "Server:"; done# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 2# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 1# < Server: 2# < Server: 1# < Server: 1

/ END /