springboot-how to solve 'Can't read configMap with name: in namespace. Ignoring configmaps is forbidden' when using spring cloud config client with kubernetes(k8s)

Problem

When we are using spring cloud with kubernetes , sometimes, we get this error:


  .   ____          _            __ _ _ 
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \ 
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ 
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) ) 
  '  |____| .__|_| |_|_| |_\__, | / / / / 
 =========|_|==============|___/=/_/_/_/ 
 :: Spring Boot ::        (v2.3.2.RELEASE) 
 
2020-11-28 10:14:43.597  WARN 1 --- [           main] o.s.c.k.config.ConfigMapPropertySource   : Can't read configMap with name: [app6] in namespace:[ns-bswen]. Ignoring. 
 
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/api/v1/namespaces/ns-bswen/configmaps/app6. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "app6" is forbidden: User "system:serviceaccount:ns-bswen:default" cannot get resource "configmaps" in API group "" in the namespace "ns-bswen". 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:503) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:440) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:406) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:810) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:218) ~[kubernetes-client-4.4.1.jar:na] 
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[kubernetes-client-4.4.1.jar:na] 
	at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.getData(ConfigMapPropertySource.java:97) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE] 
	at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.<init>(ConfigMapPropertySource.java:78) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE] 
	at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.getMapPropertySourceForSingleConfigMap(ConfigMapPropertySourceLocator.java:96) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE] 
	at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:79) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE] 
	at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_212] 
	at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:78) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE] 
	at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE] 
	at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE] 
	at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE] 
	at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] 
	at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] 
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] 
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] 
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE] 
	at com.bswen.app6.Main.main(Main.java:9) ~[app6/:na] 
 
2020-11-28 10:14:43.606  INFO 1 --- [           main] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.app6.ns-bswen'}] 
2020-11-28 10:14:43.614  INFO 1 --- [           main] com.bswen.app6.Main                      : The following profiles are active: dev 
2020-11-28 10:14:44.809  INFO 1 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=6a3c4ecd-2134-3834-bb64-3a51025a6037 
2020-11-28 10:14:45.410  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8082 (http) 
2020-11-28 10:14:45.431  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat] 
2020-11-28 10:14:45.431  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.37] 
2020-11-28 10:14:45.848  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext 
2020-11-28 10:14:45.849  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2209 ms 
2020-11-28 10:14:46.536  INFO 1 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor' 
2020-11-28 10:14:46.983  WARN 1 --- [//10.43.0.1/...] i.f.k.c.d.i.WatchConnectionManager       : Exec Failure: HTTP 403, Status: 403 - configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen" 
 
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden' 
	at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) ~[okhttp-3.12.0.jar:na] 
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) ~[okhttp-3.12.0.jar:na] 
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) [okhttp-3.12.0.jar:na] 
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [okhttp-3.12.0.jar:na] 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] 
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] 
 
2020-11-28 10:14:46.988 ERROR 1 --- [           main] .r.EventBasedConfigurationChangeDetector : Error while establishing a connection to watch config maps: configuration may remain stale 
 
io.fabric8.kubernetes.client.KubernetesClientException: configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen" 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:203) ~[kubernetes-client-4.4.1.jar:na] 
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[okhttp-3.12.0.jar:na] 
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[okhttp-3.12.0.jar:na] 
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) ~[okhttp-3.12.0.jar:na] 
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[okhttp-3.12.0.jar:na] 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212] 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212] 
	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212] 
 
Exception in thread "OkHttp Dispatcher" java.util.concurrent.RejectedExecutionException: Task java.[email protected]56b95013 rejected from [email protected][Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] 
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) 
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) 
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) 
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) 
	at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632) 
	at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:305) 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:48) 
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:218) 
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) 
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) 
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) 
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
	at java.lang.Thread.run(Thread.java:748) 
2020-11-28 10:14:46.997  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator' 
2020-11-28 10:14:47.070  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8082 (http) with context path '' 
2020-11-28 10:14:47.090  INFO 1 --- [           main] com.bswen.app6.Main                      : Started Main in 8.203 seconds (JVM running for 11.489) 

The core error is :

configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen" 

Environment

  • SpringBoot 2.3
  • Spring Cloud Config Server 2.2.3.RELEASE
  • SpringCloudVersion Hoxton.SR6
  • Kubernetes 1.19

Reason

We are using kubernetes configmap as the config property source , according to this document:

You should check the security configuration section. To access config maps from inside a pod you need to have the correct Kubernetes service accounts, roles and role bindings.

If you don’t specify the service account name in your kubernetes deployment, then you are using the ‘default’ service account, but the ‘default’ service account can not ‘watch’ the configmap api without proper authorizations.

image-20201129161500362

So the reason is that our account does not have the permission to watch the configmap in kubernetes.

Solution

We should create RBAC role/rolebinding to specified service account for our app.

# create the service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-reader
  namespace: ns-bswen
---
# create the role to grant access to configmaps
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ns-bswen
  name: role-api-reader
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources: ["pods","configmaps"]
    verbs: ["get", "watch", "list"]
---
# bind the role and the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rolebinding-api-reader
  namespace: ns-bswen
subjects:
  - kind: ServiceAccount
    name: api-reader # Name is case sensitive
    namespace: ns-bswen
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: role-api-reader # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

Then we can use the service account in our kubernetes deployment yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deployment-app6
  namespace: ns-bswen
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app6
  template:
    metadata:
      labels:
        app: app6
    spec:
      serviceAccountName: api-reader  # here is the key point
      imagePullSecrets:
        - name: secret-harbor
      containers:
        - image: app6:latest
          name: app6
          ports:
            - containerPort: 8082
              name: app6-port

Then everything runs ok.

-->