Hi,
Any help will highly be appreciated if someone has face the similar issue or knows the answer let me know.
I am getting 503 error when I do curl to my micro service, below is my curl:
curl -vvv https://api-gateway.cap-staging.saturn.xyz.net/v1/health?authkey=password
* Trying 44.235.129.53...
* TCP_NODELAY set
* Connected to api-gateway.cap-staging.saturn.xyz.net (44.235.129.53) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=*.cap-staging.xyz.net
* start date: Sep 2 00:00:00 2020 GMT
* expire date: Oct 2 12:00:00 2021 GMT
* subjectAltName: host "api-gateway.cap-staging.xyz.net" matched cert's "*.cap-staging.xyz.net"
* issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
* SSL certificate verify ok.
> GET /v1/health?authkey=password HTTP/1.1
> Host: api-gateway.cap-staging.saturn.xyz.net
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-type: text/plain
< date: Wed, 07 Apr 2021 23:22:50 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 57
< Content-Length: 95
< Connection: keep-alive
<
* Connection #0 to host api-gateway.cap-staging.saturn.xyz.net left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination
From istio-ingressgateway the error is:
[2021-04-07T23:14:03.320Z] "GET /v1/health?authkey=password HTTP/1.1" 503 URX via_upstream - "-" 0 95 64 64 "12.107.176.9,100.127.250.64" "curl/7.58.0" "b30b497a-5403-9b52-a83a-38703526636f" "api-gateway.cap-staging.saturn.xyz.net" "100.104.126.149:8081" outbound|80||api-gateway.cap-staging.svc.cluster.local 100.97.92.11:34544 100.97.92.11:8443 100.127.250.64:28986 - -
on the micro service the error is:
[2021-04-07T23:23:02.850Z] "GET /v1/health?authkey=password HTTP/1.1" 503 UC upstream_reset_before_response_started{connection_termination} - "-" 0 95 6 - "72.52.80.4,172.20.49.60" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15" "26410976-8e52-9cc7-973a-4d13970b245c" "api-gateway.cap-staging.saturnxyz.net" "127.0.0.1:8081" inbound|8081|| 127.0.0.1:45442 100.104.126.149:8081 172.20.49.60:0 - default
my config:
installed istio from istioctl command: istioctl install --set profile=demo -y
istio version:
istioctl version --remote
client version: 1.9.1
control plane version: 1.9.1
data plane version: 1.9.1 (47 proxies)
kubernetes version:
kubectl version --short
Client Version: v1.18.0
Server Version: v1.17.0
logs from istio: all are SYNCED except for below: istioctl proxy-status
istio-egressgateway-5888c588c8-cb27f.istio-system SYNCED SYNCED SYNCED NOT SENT istiod-5f89f959fd-gn9dr 1.9.1
below is the service:
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: istio-ingressgateway
release: istio
name: istio-ingressgateway-service-account
namespace: istio-system
---
apiVersion: v1
kind: Service
metadata:
name: istio-ingressgateway
namespace: istio-system
annotations:
#service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
#service.beta.kubernetes.io/aws-load-balancer-type: alb
#service.beta.kubernetes.io/subnets: "us-west-2b","us-west-2a","us-west-2c"
#service.beta.kubernetes.io/scheme: "internet-facing"
#service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: true
#service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: 60
#service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: s3-logs-elb-logs-kube-staging
#service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: saturn
#service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "s3-logs-elb-logs-kube-staging"
#service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "saturn"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:us-west-*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
labels:
app: istio-ingressgateway
release: istio
istio: ingressgateway
spec:
#type: NodePort
type: LoadBalancer
selector:
app: istio-ingressgateway
ports:
- name: http2
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
---
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: ingressgateway
namespace: istio-system
labels:
release: istio
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: grpc-ingressgateway
namespace: istio-system
labels:
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
spec:
minAvailable: 1
selector:
matchLabels:
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
---
apiVersion: v1
kind: Service
metadata:
name: grpc-ingressgateway
namespace: istio-system
annotations:
labels:
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
spec:
type: LoadBalancer
selector:
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
ports:
- name: http2
port: 80
#targetPort: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8443
- name: tcp
port: 31400
targetPort: 31400
- name: tls
port: 15443
targetPort: 15443
- name: httpscr
port: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: grpc-ingressgateway
namespace: istio-system
labels:
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
spec:
selector:
matchLabels:
app: grpc-ingressgateway
istio: grpc-ingressgateway
template:
metadata:
labels:
chart: gateways
heritage: Tiller
release: istio
app: grpc-ingressgateway
istio: grpc-ingressgateway
annotations:
sidecar.istio.io/inject: "false"
#scheduler.alpha.kubernetes.io/critical-pod: ""
spec:
volumes:
- name: istio-certs
secret:
optional: true
secretName: istio.istio-ingressgateway-service-account
- name: istio-envoy
emptyDir: {}
- name: istiod-ca-cert
configMap:
name: istio-ca-root-cert
defaultMode: 420
- name: podinfo
downwardAPI:
items:
- path: labels
fieldRef:
apiVersion: v1
fieldPath: metadata.labels
- path: annotations
fieldRef:
apiVersion: v1
fieldPath: metadata.annotations
defaultMode: 420
- name: config-volume
configMap:
name: istio
defaultMode: 420
optional: true
- name: ingressgateway-grpc-certs
secret:
optional: true
secretName: istio-grpcingressgateway-certs
- name: ingressgateway-ca-certs
secret:
optional: true
secretName: istio-ingressgateway-ca-certs
serviceAccountName: grpc-ingressgateway-service-account
containers:
- name: istio-proxy
image: 'docker.io/istio/proxyv2:1.9.1'
args:
- proxy
- router
- --domain
- $(POD_NAMESPACE).svc.cluster.local
- --proxyLogLevel=warning
- --proxyComponentLogLevel=misc:error
- --log_output_level=default:info
- --drainDuration
- 45s
- --parentShutdownDuration
- 1m0s
- --connectTimeout
- 10s
- --serviceCluster
- grpc-ingressgateway
- --zipkinAddress
- zipkin.istio-system:9411
- --proxyAdminPort
- "15000"
- --statusPort
- "15021"
- --controlPlaneAuthPolicy
- MUTUAL_TLS
- --discoveryAddress
- istio-pilot.istio-system:15011
- '--trust-domain=cluster.local'
ports:
- containerPort: 15021
- containerPort: 8080
- containerPort: 8443
- containerPort: 31400
- containerPort: 15090
- containerPort: 80
- containerPort: 443
- containerPort: 853
- containerPort: 31400
- containerPort: 15011
- containerPort: 15012
- containerPort: 8060
- containerPort: 15029
- containerPort: 15030
- containerPort: 15031
- containerPort: 15032
- containerPort: 15443
- containerPort: 15020
- containerPort: 15090
protocol: TCP
name: http-envoy-prom
env:
- name: SDS_ENABLED
value: "false"
- name: JWT_POLICY
value: first-party-jwt
- name: PILOT_CERT_PROVIDER
value: istiod
- name: CA_ADDR
value: 'istiod.istio-system.svc:15012'
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
- name: SERVICE_ACCOUNT
valueFrom:
fieldRef:
fieldPath: spec.serviceAccountName
- name: ISTIO_META_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: ISTIO_META_CONFIG_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: ISTIO_META_WORKLOAD_NAME
value: grpc-ingressgateway
- name: ISTIO_META_OWNER
value: kubernetes://api/apps/v1/namespaces/istio-system/deployments/grpc-ingressgateway
- name: ISTIO_META_ROUTER_MODE
value: sni-dnat
- name: ISTIO_META_MESH_ID
value: cluster.local
- name: ISTIO_METAJSON_LABELS
value: |
{"app":"grpc-ingressgateway","grpc-ingressgateway"}
- name: ISTIO_META_CLUSTER_ID
value: Kubernetes
resources:
limits:
cpu: '2'
memory: 1Gi
requests:
cpu: 10m
memory: 40Mi
volumeMounts:
- name: istio-certs
mountPath: /etc/certs
readOnly: true
- name: config-volume
mountPath: /etc/istio/config
- name: istiod-ca-cert
mountPath: /var/run/secrets/istio
- name: podinfo
mountPath: /etc/istio/pod
- name: ingressgateway-grpc-certs
mountPath: /etc/istio/ingressgateway-certs
- name: ingressgateway-ca-certs
mountPath: /etc/istio/ingressgateway-ca-certs
- name: istio-envoy
mountPath: /etc/istio/proxy
readinessProbe:
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
timeoutSeconds: 1
periodSeconds: 2
successThreshold: 1
failureThreshold: 30
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: grpc-ingressgateway-service-account
serviceAccount: grpc-ingressgateway-service-account
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- ppc64le
- s390x
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 2
preference:
matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- weight: 2
preference:
matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- ppc64le
- weight: 2
preference:
matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- s390x
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 100%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
---
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
labels:
app: grpc-ingressgateway
istio: grpc-ingressgateway
release: istio
name: grpc-ingressgateway
namespace: istio-system
spec:
maxReplicas: 5
metrics:
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: grpc-ingressgateway
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: grpc-ingressgateway-service-account
namespace: istio-system
labels:
app: grpc-ingressgateway
release: istio
---
apiVersion: v1
kind: Service
metadata:
name: istio-egressgateway
namespace: istio-system
annotations:
labels:
app: istio-egressgateway
release: istio
istio: egressgateway
spec:
type: ClusterIP
selector:
app: istio-egressgateway
ports:
-
name: http2
port: 80
-
name: https
port: 443
-
name: tcp
port: 31400
-
name: tls
port: 15443
targetPort: 15443
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-multicluster-egressgateway
namespace: istio-system
labels:
app: istio-egressgateway
release: istio
spec:
selector:
istio: egressgateway
servers:
- hosts:
- "*.global"
port:
name: tls
number: 15443
protocol: TLS
tls: {}
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: istio-multicluster-destinationrule
namespace: istio-system
labels:
app: istio-egressgateway
release: istio
spec:
host: "*.global"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: istio-multicluster-egressgateway
namespace: istio-system
labels:
app: istio-egressgateway
release: istio
spec:
gateways:
- istio-multicluster-egressgateway
hosts:
- "*.global"
tls:
- match:
- port: 15443
sniHosts:
- "*.global"
route:
- destination:
host: istio-multicluster-egressgateway.istio-system.svc.cluster.local
port:
number: 15443
weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: istio-egressgateway-envoy
namespace: istio-system
labels:
app: istio-egressgateway
release: istio
spec:
workloadLabels:
istio: egressgateway
filters:
- listenerMatch:
portNumber: 15443
listenerType: GATEWAY
filterName: envoy.filters.network.sni_cluster
filterType: NETWORK
filterConfig: {}
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "api-server"
namespace: istio-system
labels:
release: istio
spec:
host: "kubernetes.default.svc.cluster.local"
trafficPolicy:
tls:
mode: DISABLE
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "default"
namespace: istio-system
labels:
release: istio
spec:
host: "*.local"
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
labels:
release: istio
spec:
mtls:
mode: PERMISSIVE
---
apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
name: default
namespace: istio-system
labels:
release: istio
spec:
egress:
- hosts:
- "*/*"
---
--------------------------------
Micro service YML FILE Gateway and virtual service:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: api-gateway
# annotations:
# ingress.kubernetes.io/force-ssl-redirect: "true"
# kubernetes.io/ingress.class: "istio-gateway"
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- api-gateway.cap-xyz.net
- api-gateway.xyz.net
tls:
httpsRedirect: true # sends 301 redirect for http requests
- port:
number: 443
name: https-443
protocol: HTTP
hosts:
- api-gateway.cap-xyz.net
- api-gateway.xyz.net
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: api-gateway
spec:
hosts:
- "api-gateway.cap-xyz.net"
- "api-gateway.xyz.net"
# - "*"
gateways:
- api-gateway
http:
- route:
- destination:
host: api-gateway
port:
number: 80
---