Today, we upgraded the test environment to istio 1.6. After deploying the project, we found the following problems:
Deploy a service: basemq.
After a few minutes of deployment, it can be seen that the pod status changes from running to CrashLoopBackOff:
[root@node0 dep]# kubectl get pods
NAME READY STATUS RESTARTS AGE
basemq-575c7cc4f6-8pxcv 0/2 Init:CrashLoopBackOff 2 5m50s
[root@node0 dep]# kubectl describe pod basemq-575c7cc4f6-8pxcv
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned baoli-test/basemq-575c7cc4f6-8pxcv to node2
Normal Pulling 9m52s kubelet, node2 Pulling image "reg.domain.com:5000/basemq:baoli-test-0.0.3"
Normal Pulled 9m52s kubelet, node2 Successfully pulled image "reg.domain.com:5000/basemq:baoli-test-0.0.3"
Normal Pulling 9m51s kubelet, node2 Pulling image "docker.io/istio/proxyv2:1.6.0"
Normal Started 9m51s kubelet, node2 Started container basemq
Normal Created 9m51s kubelet, node2 Created container basemq
Normal Started 9m36s kubelet, node2 Started container istio-proxy
Normal Pulled 9m36s kubelet, node2 Successfully pulled image "docker.io/istio/proxyv2:1.6.0"
Normal Created 9m36s kubelet, node2 Created container istio-proxy
Warning Unhealthy 9m34s kubelet, node2 Readiness probe failed: Get http://10.244.104.2:15021/healthz/ready: dial tcp 10.244.104.2:15021: connect: connection refused
Normal Started 7m46s (x3 over 9m53s) kubelet, node2 Started container istio-init
Normal Pulling 7m34s (x4 over 9m59s) kubelet, node2 Pulling image "docker.io/istio/proxyv2:1.6.0"
Normal Pulled 7m19s (x4 over 9m53s) kubelet, node2 Successfully pulled image "docker.io/istio/proxyv2:1.6.0"
Normal Created 7m19s (x4 over 9m53s) kubelet, node2 Created container istio-init
Warning BackOff 4m53s (x10 over 7m45s) kubelet, node2 Back-off restarting failed container
But check the pod log:
[root@node0 dep]# kubectl logs basemq-575c7cc4f6-8pxcv -f -c basemq
Picked up JAVA_TOOL_OPTIONS: -XX:+UseContainerSupport
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.7.RELEASE)
2020-05-22 08:56:29.465 INFO 1 --- [ main] com.kinglandgroup.BaseMQApplication : Starting BaseMQApplication vbaoli-0.0.3-SNAPSHOT on basemq-575c7cc4f6-8pxcv with PID 1 (/app.jar started by root in /)
2020-05-22 08:56:29.469 INFO 1 --- [ main] com.kinglandgroup.BaseMQApplication : The following profiles are active: test
2020-05-22 08:56:30.582 INFO 1 --- [ main] ptablePropertiesBeanFactoryPostProcessor : Post-processing PropertySource instances
2020-05-22 08:56:30.653 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource configurationProperties [org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource] to AOP Proxy
2020-05-22 08:56:30.654 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource servletConfigInitParams [org.springframework.core.env.PropertySource$StubPropertySource] to EncryptablePropertySourceWrapper
2020-05-22 08:56:30.655 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource servletContextInitParams [org.springframework.core.env.PropertySource$StubPropertySource] to EncryptablePropertySourceWrapper
2020-05-22 08:56:30.655 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper
2020-05-22 08:56:30.656 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableMapPropertySourceWrapper
2020-05-22 08:56:30.656 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper
2020-05-22 08:56:30.656 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource applicationConfig: [classpath:/application-test.properties] [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper
2020-05-22 08:56:30.656 INFO 1 --- [ main] c.u.j.EncryptablePropertySourceConverter : Converting PropertySource applicationConfig: [classpath:/application.properties] [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper
2020-05-22 08:56:30.826 INFO 1 --- [ main] c.u.j.r.DefaultLazyPropertyResolver : Property Resolver custom Bean not found with name 'encryptablePropertyResolver'. Initializing Default Property Resolver
2020-05-22 08:56:30.829 INFO 1 --- [ main] c.u.j.d.DefaultLazyPropertyDetector : Property Detector custom Bean not found with name 'encryptablePropertyDetector'. Initializing Default Property Detector
2020-05-22 08:56:31.080 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-05-22 08:56:31.094 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-05-22 08:56:31.095 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.34]
2020-05-22 08:56:31.187 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-05-22 08:56:31.187 INFO 1 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 1649 ms
2020-05-22 08:56:32.016 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-05-22 08:56:32.291 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-05-22 08:56:32.295 INFO 1 --- [ main] com.kinglandgroup.BaseMQApplication : Started BaseMQApplication in 3.412 seconds (JVM running for 4.098)
The above log shows that the service is started without any problem.
What’s the matter? The above services are OK under istio 1.5.4.