Istio的流量管理(实操二)(istio 系列四)

涵盖官方文档Traffic Management章节中的inrgess部分。

Istio的流量管理(实操二)(istio 系列四)

Ingress网关

在kubernetes环境中，kubernetes ingress资源用于指定暴露到集群外的服务。在istio服务网格中，使用了一种不同的配置模型，称为istio网关。一个网关允许将istio的特性，如镜像和路由规则应用到进入集群的流量上。

本节描述了如何使用istio网关将一个服务暴露到服务网格外。

环境准备

使用如下命令创建httpbin服务

$ kubectl apply -f samples/httpbin/httpbin.yaml

确定ingress的IP和端口

由于本环境中没有配置对外的负载均衡，因此此处的EXTERNAL-IP为空，使用node port进行访问

# kubectl get svc istio-ingressgateway -n istio-system

NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP  PORT(S)    AGE

istio-ingressgateway   LoadBalancer   10.84.93.45   <pending>     ...        11d

获取ingressgateway service的http2和https对应的端口

$ export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

$ export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')

$ export TCP_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="tcp")].nodePort}')

下面是istio-system命名空间的istio-ingressgateway service中的一部分端口信息，可以看到http2和https的nodeport分别为31194和31785，对应上面的INGRESS_PORT和SECURE_INGRESS_PORT

{

    "name": "http2",

    "nodePort": 31194,

    "port": 80,

    "protocol": "TCP",

    "targetPort": 80

},

{

    "name": "https",

    "nodePort": 31785,

    "port": 443,

    "protocol": "TCP",

    "targetPort": 443

},

获取istio-system命名空间中ingressgateway pod 的hostIP

$ export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')

使用istio网关配置ingress

一个ingress网关描述了在网格边缘用于接收入站HTTP/TCP连接的负载均衡，配置了暴露的端口，协议等。kubernetes ingress资源不包括任何流量路由配置。ingress 流量的路由使用istio路由规则，与内部服务请求相同：

创建istio Gateway，将来自httpbin.example.com的流量导入网格的80端口(即默认的ingressgatewaypod)

kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: httpbin-gateway

spec:

  selector:

    istio: ingressgateway # use Istio default gateway implementation

  servers:

  - port:

      number: 80

      name: http

      protocol: HTTP

    hosts:

    - "httpbin.example.com"

EOF

通过Gateway配置进入的流量路由，将来自httpbin.example.com，且目的地为/status或/delay的请求分发到httpbin服务的8000端口，其他请求会返回404响应。
```
kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "httpbin.example.com"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /status

    - uri:

        prefix: /delay

    route:

    - destination:

        port:

          number: 8000

        host: httpbin

EOF
```
可以看到流量被导入了httpbin service暴露的8000端口上
```
$ oc get svc |grep httpbin

httpbin       ClusterIP      10.84.222.69   <none>    8000/TCP   19h
```
来自网格内部其他服务的请求则不受此规则的约束，会使用默认的轮询路由进行请求分发。为了限制内部的调用规则，可以将特定的值mesh添加到gateways列表中。由于内部服务的主机名可能与外部不同，因此需要将主机名添加到hosts列表中。

使用curl命令访问httpbin服务，此时通过-H选项修改了HTTP请求首部的Host字段，使用http2的nodeport方式访问：

$ curl -s -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/status/200

HTTP/1.1 200 OK

server: istio-envoy

date: Thu, 21 May 2020 03:22:50 GMT

content-type: text/html; charset=utf-8

access-control-allow-origin: *

access-control-allow-credentials: true

content-length: 0

x-envoy-upstream-service-time: 21

访问其他URL路径则返回404错误

$ curl -s -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/headers

HTTP/1.1 404 Not Found

date: Thu, 21 May 2020 03:25:20 GMT

server: istio-envoy

transfer-encoding: chunked

使用浏览器访问ingress服务

由于无法像使用curl一样修改请求的Host首部字段，因此无法使用浏览器访问httpbin服务。为了解决这个问题，可以在Gateway和VirtualService中的host字段使用通配符*。

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: httpbin-gateway

spec:

  selector:

    istio: ingressgateway # use Istio default gateway implementation

  servers:

  - port:

      number: 80

      name: http

      protocol: HTTP

    hosts:

    - "*"  #指定通配符，不限制外部流量的地址

---

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "*"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /headers

    route:

    - destination:

        port:

          number: 8000

        host: httpbin

EOF

问题定位

检查环境变量INGRESS_HOST和INGRESS_PORT，保证这两个值是有效的

$ kubectl get svc -n istio-system

$ echo INGRESS_HOST=$INGRESS_HOST, INGRESS_PORT=$INGRESS_PORT

校验相同的端口上没有其他istio ingress网格
```
$ kubectl get gateway --all-namespaces
```
校验没有在相同的IP和端口上定义kubernetes ingress资源
```
$ kubectl get ingress --all-namespaces
```
如果没有负载均衡，可以参照上面步骤使用node port方式

卸载

$ kubectl delete gateway httpbin-gateway

$ kubectl delete virtualservice httpbin

$ kubectl delete --ignore-not-found=true -f samples/httpbin/httpbin.yaml

Ingress(kubernetes)

执行ingress流量控制中的Before you begin 的Before you begin 和Determining the ingress IP and ports小节的操作，部署httpbin服务。本节介绍如何通过kubernetes的Ingress(非istio的gateway)进行访问。

下面展示如何配置一个80端口的Ingress，用于HTTP流量：

创建一个istio Gateway，将来自httpbin.example.com:80/status/*的流量分发到service httpbin的8000端口

$ kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

  annotations:

    kubernetes.io/ingress.class: istio

  name: ingress

spec:

  rules:

  - host: httpbin.example.com

    http:

      paths:

      - path: /status/*

        backend:

          serviceName: httpbin

          servicePort: 8000

EOF

注意需要使用 kubernetes.io/ingress.class annotation来告诉istio网关控制器处理该ingress，否则会忽略该ingress。

使用curl命令访问httpbin服务。Ingress的流量也需要经过istio ingressgateway。

$ curl -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/status/200

HTTP/1.1 200 OK

server: istio-envoy

date: Fri, 22 May 2020 06:12:56 GMT

content-type: text/html; charset=utf-8

access-control-allow-origin: *

access-control-allow-credentials: true

content-length: 0

x-envoy-upstream-service-time: 20

httpbin服务的发现是通过EDS实现的，使用如下命令查看：

$ istioctl proxy-config cluster istio-ingressgateway-569669bb67-b6p5r|grep 8000

httpbin.default.svc.cluster.local                   8000    -    outbound      EDS

outbound_.8000_._.httpbin.default.svc.cluster.local   -     -       -          EDS

访问其他未暴露的路径，返回HTTP 404错误：

$ curl -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/headers

HTTP/1.1 404 Not Found

date: Fri, 22 May 2020 06:24:30 GMT

server: istio-envoy

transfer-encoding: chunked

下一步

TLS

Ingress支持TLS设置。Istio也支持TLS设置，但相关的secret必须存在istio-ingressgateway deployment所在的命名空间中。可以使用cert-manager生成这些证书。

指定路径类型

默认情况下，Istio会将路径视为完全匹配，如果路径使用/*或*结尾，则该路径为前缀匹配。不支持其他正则匹配。

kubernetes 1.18中新增了一个字段pathType，允许声明为Exact或Prefix。

指定`IngressClass`

kubernetes 1.18中新增了一个资源IngressClass，替换了Ingress资源的 kubernetes.io/ingress.class annotation。如果使用该资源，则需要将controller设置为istio.io/ingress-controller：

apiVersion: networking.k8s.io/v1beta1

kind: IngressClass

metadata:

  name: istio

spec:

  controller: istio.io/ingress-controller

---

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

  name: ingress

spec:

  ingressClassName: istio

  ...

卸载

$ kubectl delete ingress ingress

$ kubectl delete --ignore-not-found=true -f samples/httpbin/httpbin.yaml

安全网关

本节讲述如何使用simple或mutual TLS暴露安全的HTTPS服务。证书是通过SDS进行密钥发现的。

TLS需要的私钥，服务端证书，根证书是使用基于文件装载的方法配置的。

执行ingress流量控制中的Before you begin 的Before you begin 和Determining the ingress IP and ports小节的操作，部署httpbin服务，并获取 INGRESS_HOST 和SECURE_INGRESS_PORT变量。

生成服务端证书和私钥

下面使用openssl生成需要的证书和密钥

生成一个根证书和一个私钥，用于签名服务的证书

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

为 httpbin.example.com生成一个证书和私钥

$ openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt

单主机配置TLS ingress网关

为ingree网关创建一个secret

secret的名字不能以istio或prometheus开头，且secret不应该包含token字段

$ kubectl create -n istio-system secret tls httpbin-credential --key=httpbin.example.com.key --cert=httpbin.example.com.crt

在server部分定义一个443端口的Gateway，将credentialName指定为httpbin-credential，与secret的名字相同，TLS的mode为SIMPLE。

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: httpbin-credential # must be the same as secret

    hosts:

    - httpbin.example.com

EOF

配置进入Gateway的流量路由。与上一节中的VirtualService相同

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "httpbin.example.com"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /status

    - uri:

        prefix: /delay

    route:

    - destination:

        port:

          number: 8000  #可以看到tls只是这是在gateway上的，当进入网格之后就不需要了

        host: httpbin

EOF

使用curl向SECURE_INGRESS_PORT发送HTTPS请求访问httpbin服务，请求中携带了公钥example.com.crt 。--resolve标记可以在使用curl访问TLS的网关IP时，在SNI中支持httpbin.example.com。--cacert选择支持使用生成的证书校验服务。

-HHost:httpbin.example.com 选项仅在SECURE_INGRESS_PORT不同于实际的网关端口(443)时才会需要，例如，通过映射的NodePort方式访问服务时。

通过将请求发送到/status/418 URL路径时，可以看到httpbin确实被访问了，httpbin服务会返回418 I’m a Teapot代码。.

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

*       subject: O=httpbin organization,CN=httpbin.example.com

*       start date: May 22 09:03:01 2020 GMT

*       expire date: May 22 09:03:01 2021 GMT

*       common name: httpbin.example.com

*       issuer: CN=example.com,O=example Inc.

> GET /status/418 HTTP/1.1

> User-Agent: curl/7.29.0

> Accept: */*

> Host:httpbin.example.com

>

< HTTP/1.1 418 Unknown

< server: istio-envoy

< date: Fri, 22 May 2020 09:08:29 GMT

< x-more-info: http://tools.ietf.org/html/rfc2324

< access-control-allow-origin: *

< access-control-allow-credentials: true

< content-length: 135

< x-envoy-upstream-service-time: 2

<

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

查看curl输出中的Server certificate中的信息，上述返回值的最后有一个茶壶的图片，说明运行成功。

删除老的网关secret，创建一个新的secret，并使用该secret修改ingress网关的凭据

$ kubectl -n istio-system delete secret httpbin-credential

$ mkdir new_certificates

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout new_certificates/example.com.key -out new_certificates/example.com.crt

$ openssl req -out new_certificates/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout new_certificates/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"

$ openssl x509 -req -days 365 -CA new_certificates/example.com.crt -CAkey new_certificates/example.com.key -set_serial 0 -in new_certificates/httpbin.example.com.csr -out new_certificates/httpbin.example.com.crt

$ kubectl create -n istio-system secret tls httpbin-credential \

--key=new_certificates/httpbin.example.com.key \

--cert=new_certificates/httpbin.example.com.crt

使用新的证书链访问httpbin服务

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert new_certificates/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

如果使用老的证书访问，则返回错误

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* Server certificate:

*       subject: O=httpbin organization,CN=httpbin.example.com

*       start date: May 22 09:24:07 2020 GMT

*       expire date: May 22 09:24:07 2021 GMT

*       common name: httpbin.example.com

*       issuer: CN=example.com,O=example Inc.

* NSS error -8182 (SEC_ERROR_BAD_SIGNATURE)

* Peer's certificate has an invalid signature.

* Closing connection 0

curl: (60) Peer's certificate has an invalid signature.

多主机配置TLS ingress网关

本节会为多个主机(httpbin.example.com和helloworld-v1.example.com)配置一个ingress网关。ingress网关会在credentialName中查找唯一的凭据。

删除之前创建的secret并为httpbin重建凭据

$ kubectl -n istio-system delete secret httpbin-credential

$ kubectl create -n istio-system secret tls httpbin-credential \

--key=httpbin.example.com.key \

--cert=httpbin.example.com.crt

启动helloworld-v1

$ cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: Service

metadata:

  name: helloworld-v1

  labels:

    app: helloworld-v1

spec:

  ports:

  - name: http

    port: 5000

  selector:

    app: helloworld-v1

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: helloworld-v1

spec:

  replicas: 1

  selector:

    matchLabels:

      app: helloworld-v1

      version: v1

  template:

    metadata:

      labels:

        app: helloworld-v1

        version: v1

    spec:

      containers:

      - name: helloworld

        image: istio/examples-helloworld-v1

        resources:

          requests:

            cpu: "100m"

        imagePullPolicy: IfNotPresent #Always

        ports:

        - containerPort: 5000

EOF

为helloworld-v1.example.com创建证书和私钥

$ openssl req -out helloworld-v1.example.com.csr -newkey rsa:2048 -nodes -keyout helloworld-v1.example.com.key -subj "/CN=helloworld-v1.example.com/O=helloworld organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in helloworld-v1.example.com.csr -out helloworld-v1.example.com.crt

为helloworld-credential创建secret

$ kubectl create -n istio-system secret tls helloworld-credential --key=helloworld-v1.example.com.key --cert=helloworld-v1.example.com.crt

定义两个网关，网关端口为443。在credentialName字段分别设置httpbin-credential和helloworld-credential，TLS模式为SIMPLE。

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https-httpbin #httpbin的gateway配置

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: httpbin-credential

    hosts:

    - httpbin.example.com

  - port:

      number: 443

      name: https-helloworld #https-helloword的gateway配置

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: helloworld-credential

    hosts:

    - helloworld-v1.example.com

EOF

配置gateway的流量路由，为新应用添加对应的virtual service

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: helloworld-v1

spec:

  hosts:

  - helloworld-v1.example.com

  gateways:

  - mygateway

  http:

  - match:

    - uri:

        exact: /hello

    route:

    - destination:

        host: helloworld-v1

        port:

          number: 5000

EOF

向helloworld-v1.example.com发送HTTPS请求

$ curl -v -HHost:helloworld-v1.example.com --resolve "helloworld-v1.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://helloworld-v1.example.com:$SECURE_INGRESS_PORT/hello"

...

< HTTP/1.1 200 OK

< content-type: text/html; charset=utf-8

< content-length: 60

< server: istio-envoy

< date: Sat, 23 May 2020 07:38:40 GMT

< x-envoy-upstream-service-time: 143

<

Hello version: v1, instance: helloworld-v1-5dfcf5d5cd-2l44c

* Connection #0 to host helloworld-v1.example.com left intact

向 httpbin.example.com发送请求

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

 ...

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

配置一个mutual TLS ingress网关

删除之前的secreting创建一个新的secret，server会使用该CA证书校验client，使用cacert保存CA证书。

$ kubectl -n istio-system delete secret httpbin-credential

$ kubectl create -n istio-system secret generic httpbin-credential --from-file=tls.key=httpbin.example.com.key \

--from-file=tls.crt=httpbin.example.com.crt --from-file=ca.crt=example.com.crt

将gateway的TLS模式设置为MUTUAL

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

 name: mygateway

spec:

 selector:

   istio: ingressgateway # use istio default ingress gateway

 servers:

 - port:

     number: 443

     name: https

     protocol: HTTPS

   tls:

     mode: MUTUAL #TLS模式设置为MUTUAL

     credentialName: httpbin-credential # must be the same as secret

   hosts:

   - httpbin.example.com

EOF

使用先前的方式发送HTTPS请求，可以看到访问失败

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* NSS: client certificate not found (nickname not specified)

* NSS error -12227 (SSL_ERROR_HANDSHAKE_FAILURE_ALERT)

* SSL peer was unable to negotiate an acceptable set of security parameters.

* Closing connection 0

curl: (35) NSS: client certificate not found (nickname not specified)

生成client的证书和私钥。在curl中传入客户端的证书和私钥，使用--cert传入客户端证书，使用--key传入私钥

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert example.com.crt --cert ./client.example.com.crt --key ./client.example.com.key "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

...

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

istio支持几种不同的Secret格式，来支持与多种工具的集成，如cert-manager:

一个TLS Secret使用tls.key和tls.crt；对于mutual TLS，会用到ca.crt
一个generic Secret会用到key和cert；对于mutual TLS，会用到cacert
一个generic Secret会用到key和cert；对于mutual TLS，会用到一个单独的名为 <secret>-cacert的generic Secret，以及一个cacert key。例如httpbin-credential 包含key 和cert，httpbin-credential-cacert 包含cacert

问题定位

检查INGRESS_HOST和SECURE_INGRESS_PORT环境变量

$ kubectl get svc -n istio-system

$ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT

检查istio-ingressgateway控制器的错误日志

$ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \

-n istio-system -o jsonpath='{.items[0].metadata.name}')"

校验istio-system命名空间中成功创建了secret。上例中应该存在httpbin-credential和helloworld-credential
```
$ kubectl -n istio-system get secrets
```

校验ingress网关agent将密钥/证书对上传到了ingress网关

$ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \

-n istio-system -o jsonpath='{.items[0].metadata.name}')"

定位mutul TLS问题

校验CA加载到了 istio-ingressgateway pod中，查看是否存在example.com.crt

$ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-ca-certs

如果创建了istio-ingressgateway-ca-certs secret，但没有加载CA证书，删除ingress网关pod，强制加载该证书
```
$ kubectl delete pod -n istio-system -l istio=ingressgateway
```
校验CA证书的Subject字段是否正确
```
$ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-ca-certs/example.com.crt | openssl x509 -text -noout | grep 'Subject:'

        Subject: O=example Inc., CN=example.com
```
log中可以看到添加了httpbin-credential secret。如果使用mutual TLS，那么也会出现 httpbin-credential-cacert secret。校验log中显示了网关agent从ingress网关接收到了SDS请求，资源的名称为 httpbin-credential，且ingress网关获取到了密钥/证书对。如果使用了mutual TLS，日志应该显示将密钥/证书发送到ingress网关，网关agent接收到了带 httpbin-credential-cacert 资源名称的SDS请求，并回去到了根证书。

卸载

删除Gateway配置，VirtualService和secret

$ kubectl delete gateway mygateway

$ kubectl delete virtualservice httpbin

$ kubectl delete --ignore-not-found=true -n istio-system secret httpbin-credential \

helloworld-credential

$ kubectl delete --ignore-not-found=true virtualservice helloworld-v1

删除证书目录

$ rm -rf example.com.crt example.com.key httpbin.example.com.crt httpbin.example.com.key httpbin.example.com.csr helloworld-v1.example.com.crt helloworld-v1.example.com.key helloworld-v1.example.com.csr client.example.com.crt client.example.com.csr client.example.com.key ./new_certificates

停止httpbin和helloworld-v1 服务：

$ kubectl delete deployment --ignore-not-found=true httpbin helloworld-v1

$ kubectl delete service --ignore-not-found=true httpbin helloworld-v1

不终止TLS的ingress网关

上一节中描述了如何配置HTTPS ingree来访问一个HTTP服务。本节中描述如何配置HTTPS ingrss来访问HTTPS服务等。通过配置ingress网关来执行SNI方式的访问，而不会在请求进入ingress时终止TLS。

本例中使用一个NGINX服务器作为HTTPS服务。

生成客户端和服务端的证书和密钥

生成一个根证书和私钥，用于签名服务

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

为 nginx.example.com创建证书和私钥

$ openssl req -out nginx.example.com.csr -newkey rsa:2048 -nodes -keyout nginx.example.com.key -subj "/CN=nginx.example.com/O=some organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in nginx.example.com.csr -out nginx.example.com.crt

部署NGINX服务

创建kubernetes Secret保存服务的证书

$ kubectl create secret tls nginx-server-certs --key nginx.example.com.key --cert nginx.example.com.crt

为NGINX服务创建配置文件

$ cat <<EOF > ./nginx.conf

events {

}

http {

  log_format main '$remote_addr - $remote_user [$time_local]  $status '

  '"$request" $body_bytes_sent "$http_referer" '

  '"$http_user_agent" "$http_x_forwarded_for"';

  access_log /var/log/nginx/access.log main;

  error_log  /var/log/nginx/error.log;

  server {

    listen 443 ssl;

    root /usr/share/nginx/html;

    index index.html;

    server_name nginx.example.com;

    ssl_certificate /etc/nginx-server-certs/tls.crt;

    ssl_certificate_key /etc/nginx-server-certs/tls.key;

  }

}

EOF

为NGINX服务创建kubernetes configmap

$ kubectl create configmap nginx-configmap --from-file=nginx.conf=./nginx.conf

部署NGINX服务

$ cat <<EOF | istioctl kube-inject -f - | kubectl apply -f -

apiVersion: v1

kind: Service

metadata:

  name: my-nginx

  labels:

    run: my-nginx

spec:

  ports:

  - port: 443

    protocol: TCP

  selector:

    run: my-nginx

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: my-nginx

spec:

  selector:

    matchLabels:

      run: my-nginx

  replicas: 1

  template:

    metadata:

      labels:

        run: my-nginx

    spec:

      containers:

      - name: my-nginx

        image: nginx

        ports:

        - containerPort: 443

        volumeMounts:

        - name: nginx-config

          mountPath: /etc/nginx

          readOnly: true

        - name: nginx-server-certs

          mountPath: /etc/nginx-server-certs

          readOnly: true

      volumes:

      - name: nginx-config

        configMap:

          name: nginx-configmap

      - name: nginx-server-certs

        secret:

          secretName: nginx-server-certs #保存了NGINX服务的证书和私钥

EOF

为了测试NGINX服务部署成功，向服务发送不使用证书的方式请求，并校验打印信息是否正确：

$ kubectl exec -it $(kubectl get pod  -l run=my-nginx -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl -v -k --resolve

...

* Server certificate:

*  subject: CN=nginx.example.com; O=some organization

*  start date: May 25 02:09:02 2020 GMT

*  expire date: May 25 02:09:02 2021 GMT

*  issuer: O=example Inc.; CN=example.com

*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

...

配置一个ingress gateway

定义一个gateway，端口为443.注意TLS的模式为PASSTHROUGH，表示gateway会放行ingress流量，不会终止TLS

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https

      protocol: HTTPS

    tls:

      mode: PASSTHROUGH #不终止TLS

    hosts:

    - nginx.example.com

EOF

配置经过Gateway的流量路由

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: nginx

spec:

  hosts:

  - nginx.example.com

  gateways:

  - mygateway

  tls:

  - match:

    - port: 443 #将gateway的流量导入kubernetes的my-nginx service

      sniHosts:

      - nginx.example.com

    route:

    - destination:

        host: my-nginx

        port:

          number: 443

EOF

根据指导配置SECURE_INGRESS_PORT和INGRESS_HOST环境变量

通过ingress访问nginx，可以看到访问成功

$ curl -v --resolve nginx.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt https://nginx.example.com:$SECURE_INGRESS_PORT

...

* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

* Server certificate:

*       subject: O=some organization,CN=nginx.example.com

*       start date: May 25 02:09:02 2020 GMT

*       expire date: May 25 02:09:02 2021 GMT

*       common name: nginx.example.com

*       issuer: CN=example.com,O=example Inc.

...

<title>Welcome to nginx!</title>

...

卸载

移除kubernetes资源

$ kubectl delete secret nginx-server-certs

$ kubectl delete configmap nginx-configmap

$ kubectl delete service my-nginx

$ kubectl delete deployment my-nginx

$ kubectl delete gateway mygateway

$ kubectl delete virtualservice nginx

删除证书和密钥

$ rm example.com.crt example.com.key nginx.example.com.crt nginx.example.com.key nginx.example.com.csr

删除生成的配置文件
```
$ rm ./nginx.conf
```