Strange behaviour for accessing external service

Policies and Telemetry Kiali
1

One of my apps is accessing cve.cicl.lu over http. I am seeing the following log in mixer for this access. Note the following entries: “requestedServerName”:“cve.circl.lu”, “destinationServiceHost”:“speech.googleapis.com”, “protocol”:“tcp”. Not sure why protocol is “tcp” and why the destinationServiceHost is speech.googleapis.com ! Any clues/help is appreciated.

{
“level”:“info”,
“time”:“2019-09-30T20:36:57.648062Z”,
“instance”:“tcpaccesslog.instance.istio-system”,
“connectionDuration”:“25.029441s”,
“connectionEvent”:“close”,
“connection_security_policy”:“unknown”,
“destinationApp”:"",
“destinationIp”:“149.13.33.14”,
“destinationName”:“unknown”,
“destinationNamespace”:“default”,
“destinationOwner”:“unknown”,
“destinationPrincipal”:"",
“destinationServiceHost”:“speech.googleapis.com”,
“destinationWorkload”:“unknown”,
“protocol”:“tcp”,
“receivedBytes”:31,
“reporter”:“source”,
“requestedServerName”:“cve.circl.lu”,
“responseFlags”:"",
“sentBytes”:0,
“sourceApp”:“billing”,
“sourceIp”:“192.169.150.113”,
“sourceName”:“billing-v1-7456896688-42g9w”,
“sourceNamespace”:“bookinfo”,
“sourceOwner”:“kubernetes://apis/extensions/v1beta1/namespaces/bookinfo/deployments/billing-v1”,
“sourcePrincipal”:"",
“sourceWorkload”:“billing-v1”,
“totalReceivedBytes”:1120,
“totalSentBytes”:1540930
}

2

This can happen if you have VirtualService defined which is routing traffic destined for host cve.circl.lu to speech.googleapis.com. Additionally, if you application is using HTTPS to talk to cve.circl.lu, then sidecar proxy will only TLS SNI based routing and can only report TCP (not HTTP) metrics.

3

Thanks for replying. I have the following virtual services defined. Do you see any issue with this?

Name: appguard-cve
Namespace: default
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:“networking.istio.io/v1alpha3”,“kind”:“VirtualService”,“metadata”:{“annotations”:{},“name”:“appguard-cve”,“namespace”:“default”},"spec":{…
API Version: networking.istio.io/v1alpha3
Kind: VirtualService
Metadata:
Cluster Name:
Creation Timestamp: 2019-09-17T20:46:56Z
Generation: 1
Resource Version: 3290050
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/appguard-cve
UID: 49700395-d98c-11e9-af11-0025b5ee0000
Spec:
Hosts:
cve.circl.lu
Http:
Route:
Destination:
Host: cve.circl.lu
Timeout: 10s
Events:

4

Also, I have the following virtual service for speech API:

Name: bookinfo-google-speech-apis-virtualservice
Namespace: istio-system
Labels: ag/app_uid=app_Zx7SVDPkvp5E
ag/component_uid=pc_X8CGoroYBN6
ag/policy_uid=po_MnsJGbG90D5
Annotations:
API Version: networking.istio.io/v1alpha3
Kind: VirtualService
Metadata:
Cluster Name:
Creation Timestamp: 2019-09-18T18:58:07Z
Generation: 1
Resource Version: 3461281
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/virtualservices/bookinfo-google-speech-apis-virtualservice
UID: 40667aee-da46-11e9-af11-0025b5ee0000
Spec:
Hosts:
speech.googleapis.com
Http:
Match:
Port: 80
Route:
Destination:
Host: speech.googleapis.com
Port:
Number: 443
Events:

5

Also, below are the serviceentries:

Name: appguard-cve
Namespace: default
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:“networking.istio.io/v1alpha3”,“kind”:“ServiceEntry”,“metadata”:{“annotations”:{},“name”:“appguard-cve”,“namespace”:“default”},“spec”:{"h…
API Version: networking.istio.io/v1alpha3
Kind: ServiceEntry
Metadata:
Cluster Name:
Creation Timestamp: 2019-09-18T18:45:22Z
Generation: 1
Resource Version: 3459681
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/default/serviceentries/appguard-cve
UID: 7839e1e1-da44-11e9-af11-0025b5ee0000
Spec:
Hosts:
cve.circl.lu
Location: MESH_EXTERNAL
Ports:
Name: cve
Number: 80
Protocol: HTTP
Resolution: NONE
Events:

Name: bookinfo-google-speech-api-serviceentry
Namespace: istio-system
Labels: ag/app_uid=app_YGFBkZNk074
ag/component_uid=pc_Vb7CeloDlPnx
ag/policy_uid=po_awIjlPZlw3b
Annotations:
API Version: networking.istio.io/v1alpha3
Kind: ServiceEntry
Metadata:
Cluster Name:
Creation Timestamp: 2019-08-29T18:53:43Z
Generation: 1
Resource Version: 1106946
Self Link: /apis/networking.istio.io/v1alpha3/namespaces/istio-system/serviceentries/bookinfo-google-speech-api-serviceentry
UID: 52d29850-ca8e-11e9-af11-0025b5ee0000
Spec:
Hosts:
speech.googleapis.com
Location: MESH_EXTERNAL
Ports:
Name: https
Number: 443
Protocol: HTTPS
Resolution: DNS
Events:

6

It’s little difficult to trace through the examples without formatting. Can you add yaml tags to your comments and then I can see if there’s any issue with your configuration. Thanks!

7

How to I add yaml tags?

8

I was able to resolve the destinationServiceHost issue by adding an entry for port 443 in the ServiceEntry as below (since the access is via HTTPS). BUT I still see the log reporting protocol as tcp as opposed to https. How do I fix this?

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"networking.istio.io/v1alpha3","kind":"ServiceEntry","metadata":{"annotations":{},"name":"appguard-cve","namespace":"default"},"spec":{"hosts":["cve.circl.lu"],"location":"MESH_EXTERNAL","ports":[{"name":"cve","number":80,"protocol":"HTTP"}],"resolution":"NONE"}}
    clusterName: ""
   creationTimestamp: 2019-09-12T18:58:23Z
   generation: 1
   name: appguard-cve
   namespace: default
   resourceVersion: "4319139"
   selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/serviceentries/appguard-cve
   uid: 4bad5b3c-d58f-11e9-b3da-0025b5ee0200
spec:
    hosts:
   - cve.circl.lu
   location: MESH_EXTERNAL
   ports:
   - name: cve
     number: 80
     protocol: HTTP
   - name: cves
     number: 443
     protocol: HTTPS
   resolution: NONE
9

So here is how the log looks now. Notice the protocol is still “tcp”

{
“level”:“info”,
“time”:“2019-10-02T20:44:50.736139Z”,
“instance”:“tcpaccesslog.instance.istio-system”,
“connectionDuration”:“463.777514ms”,
“connectionEvent”:“close”,
“connection_security_policy”:“unknown”,
“destinationApp”:"",
“destinationIp”:“149.13.33.14”,
“destinationName”:“unknown”,
“destinationNamespace”:“default”,
“destinationOwner”:“unknown”,
“destinationPrincipal”:"",
“destinationServiceHost”:“cve.circl.lu”,
“destinationWorkload”:“unknown”,
“protocol”:“tcp”,
“receivedBytes”:265,
“reporter”:“source”,
“requestedServerName”:“cve.circl.lu”,
“responseFlags”:"",
“sentBytes”:392,
“sourceApp”:“billing”,
“sourceIp”:“192.169.242.154”,
“sourceName”:“billing-v1-657b5fcc76-9jn7h”,
“sourceNamespace”:“bookinfo”,
“sourceOwner”:“kubernetes://apis/extensions/v1beta1/namespaces/bookinfo/deployments/billing-v1”,
“sourcePrincipal”:"",
“sourceWorkload”:“billing-v1”,
“totalReceivedBytes”:265,
“totalSentBytes”:392
}

10

Because you’re using HTTPS initiated from your application, the sidecar proxy does TCP proxying via SNI header in the TLS handshake. If you want visibility into external services, you should have the application talk HTTP to the external service and have the sidecar proxy do TLS origination. This means the following:

Application -----HTTP—> Sidecar Proxy ----HTTPS-----> External service

If you configure it like this, the logs and all reported metrics will be HTTP. Here’s an example of how to do this.

11

Thanks, appreciate your help!