Hello I have my microservices deploy args as follows:
spec:
containers:
- args:
- -c
- echo ‘10.206.117.116 api.ecommerce.com’ >> /etc/hosts; echo -n | openssl s_client -servername api.ecommerce.com -connect api.ecommerce.com:443 | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > /tmp/ssca.crt;keytool -import -trustcacerts -keystore /etc/ssl/certs/java/cacerts -storepass changeit -noprompt -alias mycert -file /tmp/ssca.crt; java -Djava.security.egd=file:/dev/./urandom…
These arguments run fine as soon as I remove the envoy.
When I add the envoy – echo command to update “/etc/hosts” fails; “echo -n” with “openssl” command also fails to run. Only “java” command runs. Can anyone take a guess what might be happening here ?
If I create a file at a different location such as /work/test.txt and do an echo to this file. it works fine. Somethign breaks down when system files are in picture.
UPDATE:
I ran all commands through their own script and captured the error. On “openssl” command, I am gettign following error when envoy is there in pod:
140412274599296:error:0200206F:system library:connect:Connection refused:…/crypto/bio/b_sock2.c:108:
140412274599296:error:2008A067:BIO routines:BIO_connect:connect error:…/crypto/bio/b_sock2.c:109:
140412274599296:error:02002065:system library:connect:Network is unreachable:…/crypto/bio/b_sock2.c:108:
140412274599296:error:2008A067:BIO routines:BIO_connect:connect error:…/crypto/bio/b_sock2.c:109:
140412274599296:error:02002065:system library:connect:Network is unreachable:…/crypto/bio/b_sock2.c:108:
140412274599296:error:2008A067:BIO routines:BIO_connect:connect error:…/crypto/bio/b_sock2.c:109:
connect:errno=101
Can someone tell me why envoy showing such behaviour
Two things.
First, the sidecar may not come up in the first second. Try again with “sleep 15s;” at the beginning of your command.
Second, depending on your install options you’ll need to say that you use an external service for api.ecommerce.com:443 and that the connection is HTTP.
With 1.1 it has become much easier to control HTTPS external traffic. See https://github.com/istio/istio/blob/c386865bda31bd3d6580520e496e31ef787d04a8/samples/external/pypi.yaml for an example. With 1.0.6 you need a ServiceEntry and a VirtualService with explicit SNI tags.
When I go into the pod after it is up, I can run all the commands that were missed in startup without any problem. I tried adding a 30s dealy but it didnt work.
Initial commands work if I use commands on non-system files.
Also, I used pod’s lifecycle event “postStart” with same commands, that failed with same error too.
I already have serviceentry for " api.ecommerce.com:443" and its working fine.
UPDATE:
I used same commands in deploy args and postStart, i got following error after around 600 restarts:
keytool error: java.lang.Exception: Certificate not imported, alias already exists
So the command in args was compelted successfully which made command in postStart to fail with this error.
Hey @ed.snible
I was able to move forward with the help of your comment HERE . Thanks a lot for that.
Today I installed 1.1RC4 to test the functionality in 1.1 too but right now “openssl” command is failing even from inside the pod after all containers are up. This error has something to do with 1.1 version installation:
root@auth-29-7b755d75d6-fnmbp:/# openssl s_client -servername api.ecommerce.com -connect api.ecommerce.com:443 -debug -state -showcerts
CONNECTED(00000003)
SSL_connect:before SSL initialization
write to 0x56191538e860 [0x56191539fa20] (212 bytes => 212 (0xD4))
0000 - 16 00 01 00 cf 01 00 00-cb 03 03 00 8c 15 11 c5 …2…
0010 - 00 00 f4 fb 52 1c a1 1b-ad 2f 03 82 f0 1a 84 98 …R…/…
0020 - e5 4b f1 64 b4 6b a2 76-3a 70 4d 00 00 38 c0 2c .K.d.k.v:pM…8.,
0030 - c0 30 00 9f cc a9 cc a8-cc aa c0 2b c0 00 00 9e .0…+./…
0040 - c0 24 00 28 00 00 c0 23-c0 27 00 67 c0 00 c0 14 .$.(.k.#.'.g…
0050 - 00 39 c0 09 c0 00 00 33-00 9d 00 9c 00 3d 00 3c .9…3…=.<
0060 - 00 35 00 2f 00 ff 01 00-00 6a 00 00 00 20 00 1e .5./…j… …
0070 - 00 00 1b 61 65 31 37 2d-61 70 69 2e 6b 6f 68 6c …api.
0080 - 73 65 63 6f 6d 6d 65 72-63 65 2e 63 6f 6d 00 0b ecommerce.com…
0090 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 1d 00 17 …
00a0 - 00 19 00 00 00 23 00 00-00 16 00 00 00 17 00 00 …#…
00b0 - 00 0d 00 20 00 1e 06 01-06 02 06 03 05 01 05 02 … …
00c0 - 05 00 04 01 04 02 04 03-03 01 03 02 03 03 02 01 …
00d0 - 02 02 02 03 …
SSL_connect:SSLv3/TLS write client hello
read from 0x56191538e860 [0x561915396803] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
SSL_connect:error in SSLv3/TLS write client hello
write:errno=104
Can you help me here, how to resolve this and why this is happening in 1.1 and not in 1.0.6
Some changes were made in 1.1 to provide good defaults for ServiceEntry SNI and an implicit VirtualService for the ServiceEntry.
Can you share your api.ecommerce.com ServiceEntry and also let us know if you have other ServiceEntries or K8s Services listening on port 443.
I deleted the istio-system namespace when I was removing istio 1.0.6 but gateway, serviceentry, virtualservice was still there.
After 1.1rc4 installation I realised this and deleted all these object, thinking its better to create them again.
I didnt created any serviceentries after that as I read somewhere that all outgoing traffic is allowed in 1.1 by default, unlike previous versions.
I am creating the serviceentry for api.ecommerce.com. Will let you know the results.
Hey @ed.snible
In my setup I have a headless service which has endpoint of an Internal load balancer. Config is like this:
in /etc/hosts file of our hosts we have “headless service IP” mapped to “api.ecommerce.com”. I am directly able to access api.ecommerce.com but not through headless service. I have serviceentry for “api.ecommerce.com” in place too. here it is:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
creationTimestamp: 2019-03-14T00:12:11Z
generation: 1
name: openapi-ext
namespace: r19-1-mui-qa
resourceVersion: “12968684”
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/r19-1-mui-qa/serviceentries/openapi-ext
uid: d06d6fdb-45ed-11e9-8ff7-42010a8000f7
spec:
hosts:
- ‘*.ecommerce.com’
- ‘*.debian.org’
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
I tried following serviceentry config for a free flow of traffic from pods to headless service but none is working
Config 1:
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egproxy-headless-service-fqdn-ext
namespace: r19-1-mui-qa
spec:
hosts:
- egproxy.egproxy-lle.svc.cluster.local
addresses:
- 10.206.117.116/32
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
Config 2: (Static resolution and added loadbalancer IP as endpoint)(Tried with "MESH_INTERNAL too)
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egproxy-headless-service-fqdn-ext
namespace: r19-1-mui-qa
spec:
hosts:
- egproxy.egproxy-lle.svc.cluster.local
addresses:
- 10.206.117.116/32
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
resolution: STATIC
endpoints:
- address: 10.207.132.8
Config 3: (static resolution and service IP as endpoint)(Tried with "MESH_INTERNAL too):
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: egproxy-headless-service-fqdn-ext
namespace: r19-1-mui-qa
spec:
hosts:
- egproxy.egproxy-lle.svc.cluster.local
addresses:
- 10.206.117.116/32
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
resolution: STATIC
endpoints:
- address: 10.206.117.116
Here is the full openssl error:
root@resources-11-7dbc7f884-qr86d:/# openssl s_client -servername api.ecommerce.com -connect api.ecommerce.com:443 -debug -state
CONNECTED(00000003)
SSL_connect:before SSL initialization
write to 0x5648b80c6400 [0x5648b80d75c0] (212 bytes => 212 (0xD4))
0000 - 16 03 01 00 cf 01 00 00-cb 03 03 a3 46 9e 45 1e …F.E.
0010 - 00 36 00 5c 38 4f 00 7b-d6 d1 70 cc cf 52 49 7c u6Q\8O.{…p…RI|
0020 - 8b 13 3d a8 71 90 11 61-4a b5 c4 00 00 38 c0 2c …=.q…aJ…8.,
0030 - c0 30 00 9f cc a9 cc a8-cc aa c0 2b c0 2f 00 9e .0…+./…
0040 - c0 24 c0 28 00 6b c0 23-c0 27 00 67 c0 0a c0 14 .$.(.k.#.'.g…
0050 - 00 00 c0 00 13 00 33-00 9d 00 9c 00 3d 00 3c .9…3…=.<
0060 - 00 35 00 2f 00 ff 00 00-00 6a 00 00 00 20 00 1e .5./…j… …
0070 - 00 00 1b 61 65 31 37 2d-61 70 69 2e 6b 6f 68 6c …api.
0080 - 73 65 63 6f 6d 6d 65 72-63 65 2e 63 6f 6d 00 0b ecommerce.com…
0090 - 00 04 03 00 01 02 00 0a-00 0a 00 08 00 1d 00 17 …
00a0 - 00 19 00 18 00 00 00-00 0d 00 20 00 1e 06 01 …#… …
00b0 - 06 02 06 00 05 01 05 00-05 03 04 01 04 02 04 03 …
00c0 - 03 01 03 00 03 00 02 01-02 02 02 03 00 16 00 00 …
00d0 - 00 17 …
00d4 - <SPACES/NULS>
SSL_connect:SSLv3/TLS write client hello
read from 0x5648b80c6400 [0x5648b80ce3a3] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
SSL_connect:error in SSLv3/TLS write client hello
write:errno=104
—
no peer certificate available
—
No client certificate CA names sent
—
SSL handshake has read 0 bytes and written 212 bytes
Verification: OK
—
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
- Protocol : TLSv1.2*
- Cipher : 0000*
- Session-ID:*
- Session-ID-ctx:*
- Master-Key:*
- PSK identity: None*
- PSK identity hint: None*
- SRP username: None*
- Start Time: 1552588671*
- Timeout : 7200 (sec)*
- Verify return code: 0 (ok)*
- Extended master secret: no*
—
root@resources-11-7dbc7f884-qr86d:/#
This is not an area of Istio I know well. Perhaps your problem is related to https://github.com/istio/istio/issues/7495 . You may wish to ask your question upon that thread.
Thanks for your reply @ed.snible.
For a minute consider I don’t have headless service and I am sending data directly to loadbalancer IP. (the endpoint of headless service). Is that possible ? and what would be the serviceentry config for that ?
I dont have a hostname for loadbalancer. just IP(with subnet) and open ports. Its a Google cloud internal load balancer.
I used following config but got the error:
Allow registering headless kubernetes service using "istioctl register" · Issue #8883 · istio/istio · GitHub - This also tries to address this issue and is “closed” but I am not able to wrap my head around what should be the final config for my case.
If that is possible I can move ahead and start our QA and Performance tests on the setup and can work on a resolution/workaround for headless service on the side. Please let me know if I can do this.
Also If I need a virtualservice too to make it work, please let me know about that too.
Thanks,
Sourabh Wadhwa
