springboot-how to solve 'Can't read configMap with name: in namespace. Ignoring configmaps is forbidden' when using spring cloud config client with kubernetes(k8s)
Problem
When we are using spring cloud with kubernetes , sometimes, we get this error:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.2.RELEASE)
2020-11-28 10:14:43.597 WARN 1 --- [ main] o.s.c.k.config.ConfigMapPropertySource : Can't read configMap with name: [app6] in namespace:[ns-bswen]. Ignoring.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/api/v1/namespaces/ns-bswen/configmaps/app6. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps "app6" is forbidden: User "system:serviceaccount:ns-bswen:default" cannot get resource "configmaps" in API group "" in the namespace "ns-bswen".
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:503) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:440) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:406) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:365) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:330) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:311) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:810) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:218) ~[kubernetes-client-4.4.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:162) ~[kubernetes-client-4.4.1.jar:na]
at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.getData(ConfigMapPropertySource.java:97) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE]
at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.<init>(ConfigMapPropertySource.java:78) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE]
at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.getMapPropertySourceForSingleConfigMap(ConfigMapPropertySourceLocator.java:96) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE]
at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:79) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE]
at java.util.ArrayList.forEach(ArrayList.java:1257) ~[na:1.8.0_212]
at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:78) [spring-cloud-kubernetes-config-1.1.3.RELEASE.jar:1.1.3.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:52) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceLocator.locateCollection(PropertySourceLocator.java:47) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:98) ~[spring-cloud-context-2.2.3.RELEASE.jar:2.2.3.RELEASE]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.2.RELEASE.jar:2.3.2.RELEASE]
at com.bswen.app6.Main.main(Main.java:9) ~[app6/:na]
2020-11-28 10:14:43.606 INFO 1 --- [ main] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.app6.ns-bswen'}]
2020-11-28 10:14:43.614 INFO 1 --- [ main] com.bswen.app6.Main : The following profiles are active: dev
2020-11-28 10:14:44.809 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=6a3c4ecd-2134-3834-bb64-3a51025a6037
2020-11-28 10:14:45.410 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8082 (http)
2020-11-28 10:14:45.431 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-11-28 10:14:45.431 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.37]
2020-11-28 10:14:45.848 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-11-28 10:14:45.849 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2209 ms
2020-11-28 10:14:46.536 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-11-28 10:14:46.983 WARN 1 --- [//10.43.0.1/...] i.f.k.c.d.i.WatchConnectionManager : Exec Failure: HTTP 403, Status: 403 - configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen"
java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229) ~[okhttp-3.12.0.jar:na]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196) ~[okhttp-3.12.0.jar:na]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) [okhttp-3.12.0.jar:na]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [okhttp-3.12.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2020-11-28 10:14:46.988 ERROR 1 --- [ main] .r.EventBasedConfigurationChangeDetector : Error while establishing a connection to watch config maps: configuration may remain stale
io.fabric8.kubernetes.client.KubernetesClientException: configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen"
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:203) ~[kubernetes-client-4.4.1.jar:na]
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[okhttp-3.12.0.jar:na]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198) ~[okhttp-3.12.0.jar:na]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206) ~[okhttp-3.12.0.jar:na]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[okhttp-3.12.0.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
Exception in thread "OkHttp Dispatcher" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@56b95013 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@74cdbf97[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:305)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:48)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:218)
at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:198)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-11-28 10:14:46.997 INFO 1 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-11-28 10:14:47.070 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8082 (http) with context path ''
2020-11-28 10:14:47.090 INFO 1 --- [ main] com.bswen.app6.Main : Started Main in 8.203 seconds (JVM running for 11.489)
The core error is :
configmaps is forbidden: User "system:serviceaccount:ns-bswen:default" cannot watch resource "configmaps" in API group "" in the namespace "ns-bswen"
Environment
- SpringBoot 2.3
- Spring Cloud Config Server 2.2.3.RELEASE
- SpringCloudVersion Hoxton.SR6
- Kubernetes 1.19
Reason
We are using kubernetes configmap as the config property source , according to this document:
You should check the security configuration section. To access config maps from inside a pod you need to have the correct Kubernetes service accounts, roles and role bindings.
If you don’t specify the service account name in your kubernetes deployment, then you are using the ‘default’ service account, but the ‘default’ service account can not ‘watch’ the configmap api without proper authorizations.
So the reason is that our account does not have the permission to watch the configmap in kubernetes.
Solution
We should create RBAC role/rolebinding to specified service account for our app.
# create the service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-reader
namespace: ns-bswen
---
# create the role to grant access to configmaps
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: ns-bswen
name: role-api-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods","configmaps"]
verbs: ["get", "watch", "list"]
---
# bind the role and the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rolebinding-api-reader
namespace: ns-bswen
subjects:
- kind: ServiceAccount
name: api-reader # Name is case sensitive
namespace: ns-bswen
roleRef:
kind: Role #this must be Role or ClusterRole
name: role-api-reader # this must match the name of the Role or ClusterRole you wish to bind to
apiGroup: rbac.authorization.k8s.io
Then we can use the service account in our kubernetes deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-app6
namespace: ns-bswen
spec:
replicas: 1
selector:
matchLabels:
app: app6
template:
metadata:
labels:
app: app6
spec:
serviceAccountName: api-reader # here is the key point
imagePullSecrets:
- name: secret-harbor
containers:
- image: app6:latest
name: app6
ports:
- containerPort: 8082
name: app6-port
Then everything runs ok.