K8s 集群调度 WASM 实验

Webassembly（WASM）是 Next-Generation 的软件分发方式，具有跨语言、跨平台、自带运行时的无敌属性¹。知晓了它的美妙之处，我们自然会想拥有次世代的体验，本文描述尝试将轻量级 WASM 用类 OCI 打包方式调度到 K8s 集群中运行的过程，以及其中的艰辛之处。

WASM 在当前阶段还很尴尬，WASI（WebAssembly System Interface）还处于早期阶段，WASM 运行时项目 Wasmer 与 wasmtime 对于多语言的支持还不足，云厂商例如 Cloudflare 与 AWS Lambda@Edge 都不是原生支持 WASM，WASM 的执行性能也可能比不上 Google 的 v8 JavaScript 引擎。

当我们也看到了一些希望：Go 官方也许会支持 WASI、APISIX 准备使用 Wamser 作为 WASM 运行时。

我当前在进行的另一个项目也通过在浏览器中加载 WASM 实现 X.509 证书解析、签名与 ACME 申请证书，该项目还会使用 Serverless 技术进行流量代理，不过这个坑何时能填完就没有人知道了 💀。

1. Krustlet 调度 WASM

Krustlet 介绍：使用 Rust 开发的 kubelet，用于运行 WASM 工作负载

概述：Krustlet 的工作方式类似于 Kubelet，是在 K8s 集群工作节点上运行的后台程序，负责上报节点状态以及运行工作负载（WASM）。

目前不支持 CNI 网络。

1.1. 部署安装

1.1.1. 测试环境

阿里云测试集群 - 192.168.33.161 机器（2c 8g）

1.1.2. 安装 Krustlet

#!/bin/bash

wget https://krustlet.blob.core.windows.net/releases/krustlet-v0.7.0-linux-amd64.tar.gz -O /tmp/krustlet.tar.gz
tar zxvf /tmp/krustlet.tar.gz && mv krustlet-wasi /usr/local/bin/

1.1.3. 节点初始化

根据官方文档进行节点初始化: quickstart。

步骤：

节点上安装 Kubectl 并配置有在 kube-system 下创建 CSR 资源的凭证

执行初始化脚本

bash <(curl https://raw.githubusercontent.com/deislabs/krustlet/master/docs/howto/assets/bootstrap.sh)

创建 Systemd 文件 /etc/systemd/system/krustlet.service

[Unit]
Description=Krustlet, a kubelet implementation for running WASM

[Service]
Restart=on-failure
RestartSec=5s
Environment=KUBECONFIG=/root/.kube/config
ExecStart=/usr/local/bin/krustlet-wasi
User=root
Group=root

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable krustlet.service
systemctl start krustlet.service

审核 CSR 申请

kubectl certificate approve ${HOSTNAME}-tls

1.1.4. 节点状态

等待手动 Approve

程序正常启动

Krustlet 节点已加入集群

Krustlet 节点详情 kubectl describe node ${HOSTNAME}

Name:               izuf6cc5ecqwtuhm1p2rcaz
Roles:              <none>
Labels:             beta.kubernetes.io/arch=wasm32-wasi
                    beta.kubernetes.io/os=wasm32-wasi
                    kubernetes.io/arch=wasm32-wasi
                    kubernetes.io/hostname=iZuf6cc5ecqwtuhm1p2rcaZ
                    kubernetes.io/os=wasm32-wasi
                    type=krustlet
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 08 Jul 2021 14:45:26 +0800
Taints:             kubernetes.io/arch=wasm32-wasi:NoExecute
                    kubernetes.io/arch=wasm32-wasi:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  izuf6cc5ecqwtuhm1p2rcaz
  AcquireTime:     Thu, 08 Jul 2021 14:49:37 +0800
  RenewTime:       Thu, 08 Jul 2021 14:49:37 +0800
Conditions:
  Type        Status  LastHeartbeatTime                 LastTransitionTime                Reason                     Message
  ----        ------  -----------------                 ------------------                ------                     -------
  Ready       True    Thu, 08 Jul 2021 14:49:37 +0800   Thu, 08 Jul 2021 14:45:26 +0800   KubeletReady               kubelet is posting ready status
  OutOfDisk   False   Thu, 08 Jul 2021 14:45:26 +0800   Thu, 08 Jul 2021 14:45:26 +0800   KubeletHasSufficientDisk   kubelet has sufficient disk space available
Addresses:
  InternalIP:  192.168.33.161
  Hostname:    iZuf6cc5ecqwtuhm1p2rcaZ
Capacity:
  cpu:                4
  ephemeral-storage:  61255492Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4032800Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  61255492Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4032800Ki
  pods:               110
System Info:
  Machine ID:
  System UUID:
  Boot ID:
  Kernel Version:
  OS Image:
  Operating System:           linux
  Architecture:               wasm-wasi
  Container Runtime Version:  mvp
  Kubelet Version:            0.7.0
  Kube-Proxy Version:         v1.17.0
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (3 in total)
  Namespace                   Name                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                   ------------  ----------  ---------------  -------------  ---
  kruise-system               kruise-daemon-v9nfh    0 (0%)        50m (1%)    0 (0%)           64Mi (1%)      4m17s
  kube-system                 calico-node-m4d67      150m (3%)     300m (7%)   64M (1%)         500M (12%)     4m17s
  kube-system                 nodelocaldns-k5kd2     100m (2%)     0 (0%)      70Mi (1%)        170Mi (4%)     4m17s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests        Limits
  --------           --------        ------
  cpu                250m (6%)       350m (8%)
  memory             137400320 (3%)  745366784 (18%)
  ephemeral-storage  0 (0%)          0 (0%)
  hugepages-1Gi      0 (0%)          0 (0%)
  hugepages-2Mi      0 (0%)          0 (0%)
Events:              <none>

节点带有两个污点：

Taints:             kubernetes.io/arch=wasm32-wasi:NoExecute
                    kubernetes.io/arch=wasm32-wasi:NoSchedule

1.2. WASM OCI

Krustlet 使用 wasm-to-oci 打包 WASM 编译后的二进制文件到镜像仓库。

1.2.1. 推送镜像

安装 wasm-to-oci CLI

wget https://github.com/engineerd/wasm-to-oci/releases/download/v0.1.2/linux-amd64-wasm-to-oci -O /usr/local/bin/wasm-to-oci \
&& chmod +x /usr/local/bin/wasm-to-oci

wasm-to-oci Usage

Usage:
  wasm-to-oci [command]

Available Commands:
  help        Help about any command
  pull        Pulls a WASM module from an OCI registry
  push        Pushes a WASM module from an OCI registry

Flags:
  -d, --dir string         Directory where the trust data is persisted to (default "/root/.wasm-to-oci")
  -h, --help               help for wasm-to-oci
      --insecure           allow connections to SSL registry without certs
      --log string         Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --server string      The trust server used (default "https://notary.docker.io")
  -t, --timeout string     Timeout for the trust server (default "5s")
      --tlscacert string   Trust certs signed only by this CA
      --use-http           use plain http and not https

打包示例 WASM（二进制编译后的文件）

wget https://github.com/engineerd/wasm-to-oci/raw/master/testdata/hello.wasm
wasm-to-oci push hello.wasm harbor.domain.dev/bifrost/hello-wasm:v1

Docker 需要有 Registry 的凭证 ~/.docker/config.json

Push 输出：

INFO[0001] Pushed: harbor.domain.dev/bifrost/hello-wasm:v1
INFO[0001] Size: 1624962
INFO[0001] Digest: sha256:a01f32cc647abe49bb34727cc2c520e6e304e3049d669f53e2d30d49ee2ed9c7

wasm-oci 使用了自定义的 OCI Layer，属于非标准类型。

Securely distributing and signing WebAssembly modules using OCI and TUF

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.wasm.config.v1+json",
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "size": 2
  },
  "layers": [
    {
      "mediaType": "application/vnd.wasm.content.layer.v1+wasm",
      "digest": "sha256:4c7915b4c1f9b0c13f962998e4199ceb00db39a4a7fa4554f40ae0bed83d9510",
      "size": 1624962
    }
  ]
}

1.2.2. 拉取镜像

由于是非标准格式的 OCI 镜像，直接使用 Docker CLI 拉取会出错：

v1: Pulling from bifrost/hello-wasm
4c7915b4c1f9: Pulling fs layer
invalid rootfs in image configuration

1.2.3. WASM module 与容器区别

Krustlet-tutorial pod get stuck in init:regitered status · Issue #624 · deislabs/krustlet

WASM OCI 不是 Docker 容器
WASM OCI 不能使用 Docker pull

1.3. WASM Pod 调度

WASM 调度的最小单元是 Pod，Pod 定义中必须包含 WASM 节点的 tolerations。

1.3.1. Pod 调度

示例 Pod：

apiVersion: v1
kind: Pod
metadata:
  name: wasm-hello
spec:
  containers:
    - name: wasm-hello
      image: harbor.domain.dev/bifrost/hello-wasm:v1
  tolerations:
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wasi"
      effect: "NoExecute"
    - key: "kubernetes.io/arch"
      operator: "Equal"
      value: "wasm32-wasi"
      effect: "NoSchedule"

> k apply -f wasm-hello.yaml
pod/wasm-hello created

小插曲

试验过程中发现使用 wasm-to-oci 推送镜像到自建的 Harbor 镜像仓库后，拉取镜像时会报错。

Krustlet-tutorial pod get stuck in init:regitered status · Issue #624 · deislabs/krustlet

关联到一个 Issue，部分 Registry 会无法拉取镜像，报错：

krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG hyper::proto::h1::conn] incoming body is content-length (950 bytes)
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG hyper::proto::h1::conn] incoming body completed
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG hyper::client::pool] pooling idle connection for ("https", harbor....)
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG reqwest::async_impl::client] response '200 OK' for https://harbor..../service/token?scope=repository%3Abifrost%2Fhello-wasm%3Apull&service=harbor-registry
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG oci_distribution::client] Received response from auth request: {"token":"...","access_token":"","expires_in":1800,"issued_at":"2021-07-08T08:31:46Z"}
krustlet-wasi[20706]: [2021-07-08T08:31:46Z ERROR kubelet::state::common::image_pull] Failed to decode registry token from auth request
krustlet-wasi[20706]:
krustlet-wasi[20706]:     Caused by:
krustlet-wasi[20706]:         duplicate field `token` at line 1 column 1129
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG krator::state] State::status
krustlet-wasi[20706]: [2021-07-08T08:31:46Z DEBUG krator::state] Applying status patch to object. name=wasm-hello patch={"metadata":{"resourceVersion":""},"status":{"phase":"Pending","message":"ImagePullBackoff","reason":"ImagePullBackoff"}}

将镜像替换为 webassembly.azurecr.io/hello-world-wasi-rust:v0.1.0 可正常运行。

1.3.2. Pod 运行时

查看 Pod 运行状态：

> k get pod -n ns-msp | grep wasm
wasm-hello        0/1     ExitCode:0         0          13m

Pod 被调度后立刻被执行，返回 ExitCode 0。

> k logs -n ns-msp -f wasm-hello
hello from stdout!
hello from stderr!
Args are: []

kubectl 查看与运行日志。

Pod YAML

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2021-07-08T08:37:39Z"
  name: wasm-hello
  namespace: ns-msp
  resourceVersion: "543645433"
  selfLink: /api/v1/namespaces/ns-msp/pods/wasm-hello
  uid: 09f74071-fd90-45be-be7d-466ef211043a
spec:
  containers:
  - image: webassembly.azurecr.io/hello-world-wasi-rust:v0.1.0
    imagePullPolicy: IfNotPresent
    name: wasm-hello
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-h8q2z
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: izuf6cc5ecqwtuhm1p2rcaz
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: kubernetes.io/arch
    operator: Equal
    value: wasm32-wasi
  - effect: NoSchedule
    key: kubernetes.io/arch
    operator: Equal
    value: wasm32-wasi
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-h8q2z
    secret:
      defaultMode: 420
      secretName: default-token-h8q2z
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-07-08T08:37:39Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: ""
    imageID: ""
    lastState: {}
    name: wasm-hello
    ready: false
    restartCount: 0
    started: true
    state:
      terminated:
        exitCode: 0
        finishedAt: "2021-07-08T08:37:52Z"
        message: Module run completed
        startedAt: null
  message: Completed
  phase: Succeeded
  qosClass: BestEffort
  reason: Completed

2. Containerd 插件调度 WASM

containerd-wasm 可以让 containerd 支持 WASM container，并且可以利用 Kubernetes 集群管理和调度 WASM container。

注：这个项目更多是概念验证，进程管理、资源限制，性能优化等的细节并没未完整实现。

“container-shim-wasm-v1” 作为 Containerd 的插件，利用 wasmer 作为 WASM 应用运行时环境，同时将其注册为 K8s 的一个 RuntimeClass ，利用 K8s 调度 WASM。

2.1. 部署安装

使用项目：https://github.com/dippynark/containerd-shim-wasm

验证需要一个使用 Containerd 的 K8s 集群环境，使用 Kind 部署集群。

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: config/containerd.toml
    containerPath: /etc/containerd/config.toml
  - hostPath: bin/containerd-shim-wasm-v1
    containerPath: /usr/local/bin/containerd-shim-wasm-v1
  - hostPath: bin/wasmer
    containerPath: /usr/local/bin/wasmer

Containerd 的配置中配置 wasm Runtime：

# Custom configuration
# https://github.com/containerd/cri/blob/master/docs/config.md
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasm]
  runtime_type = "io.containerd.wasm.v1"

参考文章：理解 RuntimeClass 与使用多容器运行时

Containerd 的 Runtime 会对应到 K8s 的 RuntimeClass。

apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: wasm
handler: wasm

2.2. WASM Runtime 调度

2.2.1. 编译 WASM Module

使用 target 为 wasi/wasm 进行编译，最终会使用 wasm32-wasi-clang 进行编译。

FROM --platform=$BUILDPLATFORM tonistiigi/xx:llvm AS builder
ARG TARGETPLATFORM
WORKDIR /src
COPY hello-wasm/main.c .
RUN clang -static -o /hello-wasm main.c

FROM scratch
COPY --from=builder /hello-wasm .
CMD ["/hello-wasm"]

虽然 Dockerfile 里写了 CMD ，但是如果直接在操作系统上执行命令，会报错：

> ./bin/hello-wasm
zsh: exec format error: ./bin/hello-wasm

推测 github.com/dmcgowan/containerd-wasm 库实际上会解析 Container 文件系统，提取出要执行的 WASM module，最后调用运行时（wasmtime/wasmer）运行 WASM。

我们实际使用任意的 WASM module 打包进 Docker 容器中：

FROM scratch
COPY ./hello.wasm .
CMD ["/hello.wasm"]

2.2.2. 调度 Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-wasm
spec:
  selector:
    matchLabels:
      app: hello-wasm
  template:
    metadata:
      labels:
        app: hello-wasm
    spec:
      runtimeClassName: wasm
      containers:
      - name: hello-wasm
        ## WASM 编译打包的 Docker 镜像
        image: harbor.domain.dev/bifrost/hello-wasm2:v1
        imagePullPolicy: Always

Pod 运行状态：

default  hello-wasm-86db957848-cgvdj     0/1     Completed     0      7s

Pod 输出日志：

> kubectl logs hello-wasm-86db957848-cgvdj
Hello from WebAssembly!

2.3. 对比 Krustlet

One difficulty with this shim implementation is that the shim API assumes a container runtime (as that's what it was designed for), but this doens't align as well with running WebAssmebly modules (for example currently you can't exec into a WebAssmebly module as you would a container). The Krustlet project implements a Kubelet replacement that treats wasi/wasm modules as first class citizens.

作为 Containerd 插件的 WASM Runtime 实现起来较为简单，同时代码中有大量的 TODO。

但是参照其实现的模式，公司内部也可自行实行一套 WASM 模块分发机制，运行时只需加载对应模块，调用（wasmtime/wasmer）运行即可。

Webassembly 在各类文章中被捧上天，但是截至 2021 年它的使用场景还十分局限。 ↩