Grpc client connect error with grpc server

I had two GRPC docker images built by Golang. Client grpc and server grpc. I tried kubernetes only without Istio, it works well. When I integrated with Istio, I got an error from client grpc:

Could not greet: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp …: connect: connection refused”

Istiogateway

apiVersion: v1
kind: Service
metadata:
  labels:
    app: istio-ingressgateway
    chart: gateways-1.0.0
    heritage: Tiller
    istio: ingressgateway
    release: RELEASE-NAME
  name: istio-ingressgateway
  namespace: istio-system
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http2
    nodePort: 31380
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    nodePort: 31390
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http2-golang-client-api
    nodePort: 31753
    port: 15040
    protocol: TCP
    targetPort: 15040
  - name: http2-golang-server
    nodePort: 31754
    port: 15041
    protocol: TCP
    targetPort: 15041
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  type: LoadBalancer
  externalIPs:
  - 172.17.0.17

Golang Server

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: golang-server-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 15041
      name: http2-golang-server
      protocol: HTTP2
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: golangserver
spec:
  hosts:
  - "*"
  gateways:
  - golang-server-gateway
  http:
  - match:
    - port: 15041
    route:
    - destination:
        host: golang-server-service
        port:
          number: 31001
---
apiVersion: v1
kind: Service
metadata:
  name: golang-server-service
  labels:
    app: golang-server-service
spec:
  ports:
  - port: 31001
    name: grpc
    targetPort: 30000
  selector:
    app: golangserver-deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: golangserver-deployment-v1
  labels:
    app: golangserver-deployment
    version: v1
spec:
  selector:
    matchLabels:
      app: golangserver-deployment
      version: v1
  replicas: 2
  template:
    metadata:
      labels:
        app: golangserver-deployment
        version: v1
    spec:
      containers:
      - name: golangserver-demo
        image: private-docker-registry.com/golang-server:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 30000

Golang Client

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: golang-client-api-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 15040
      name: http2-golang-client-api
      protocol: HTTP2
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: golang-client-api
spec:
  hosts:
  - "*"
  gateways:
  - golang-client-api-gateway
  http:
  - match:
    - port: 15040
    route:
    - destination:
        host: golang-client-api-service
        port:
          number: 31000
---
apiVersion: v1
kind: Service
metadata:
  name: golang-client-api-service
  labels:
    app: golang-client-api-service
spec:
  ports:
  - port: 31000
    name: http
    targetPort: 8080
  selector:
    app: golangclient-api-deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: golangclient-api-deployment-v1
  labels:
    app: golangclient-api-deployment
    version: v1
spec:
  selector:
    matchLabels:
      app: golangclient-api-deployment
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: golangclient-api-deployment
        version: v1
    spec:
      containers:
      - name: golang-api-client-demo
        image: private-docker-registry.com/golang-api-client:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        env:
        - name: SERVER_URL
          value: golang-server-service.default.svc.cluster.local
        - name: SERVER_PORT
          value: "30000"

If grpc client connects to golang server service with port 31001. It doesn’t connect.

Problem

The grpc client produced the error when connect to server from grpc client REST API endpoint.

Could not greet: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp …: connect: connection refused”