Multi-cluster services discovery & communication
Demo Architecture
For demonstration purposes we will be creating 4 Kubernetes clusters and high-level architecture will look something like the below:
As a convention and for this demo we will be creating a separate stand-alone cluster to serve as a control plane cluster, but that isn’t strictly required as a separate cluster and it could be one of any existing cluster.
Pre-requisites
kubectx
: for switching between multiplekubeconfig contexts
(clusters)k3d
: for creating multiplek3s
clusters locally using containers- FSM CLI: for deploying
FSM
docker
: required to runk3d
- Have
fsm
CLI available for managing the service mesh. - FSM version >= v1.2.0.
Demo clusters & environment setup
In this demo, we will be using k3d a lightweight wrapper to run k3s (Rancher Lab’s minimal Kubernetes distribution) in docker, to create 4 separate clusters named control-plane
, cluster-1
, cluster-2
, and cluster-3
respectively.
We will be using the HOST machine IP address and separate ports during the installation, for us to easily access the individual clusters. My demo host machine IP address is 192.168.1.110
(it might be different for your machine).
cluster | cluster ip | api-server port | LB external-port | description |
---|---|---|---|---|
control-plane | HOST_IP(192.168.1.110) | 6444 | N/A | control-plane cluster |
cluster-1 | HOST_IP(192.168.1.110) | 6445 | 81 | application-cluster |
cluster-2 | HOST_IP(192.168.1.110) | 6446 | 82 | Application Cluster |
cluster-3 | HOST_IP(192.168.1.110) | 6447 | 83 | Application Cluster |
Network
Creates a docker bridge
type network named multi-clusters
, which run all containers.
Find your machine host IP address, mine is 192.168.1.110
, and export that as an environment variable to be used later.
Cluster creation
We are going to use k3d
to create 4 clusters.
Install FSM
Install the service mesh FSM to the clusters cluster-1
, cluster-2
, and cluster-3
. The control plane does not handle application traffic and does not need to be installed.
We have our clusters ready, now we need to federate them together, but before we do that, let’s first understand the mechanics on how FSM is configured.
Federate clusters
We will enroll clusters cluster-1
, cluster-2
, and cluster-3
into the management of control-plane
cluster.
Deploy Demo application
Deploying mesh-managed applications
Deploy the httpbin
application under the httpbin
namespace of clusters cluster-1
and cluster-3
(which are managed by the mesh and will inject sidecar). Here the httpbin
application is implemented by Pipy and will return the current cluster name.
Deploy the curl
application under the namespace curl
in cluster cluster-2
, which is managed by the mesh.
Export Service
Let’s export services in cluster-1
and cluster-3
After exporting the services, FSM will automatically create Ingress rules for them, and with the rules, you can access these services through Ingress.
To view one of the ServiceExports
resources.
The exported services can be imported into other managed clusters. For example, if we look at the cluster cluster-2
, we can have multiple services imported.
Testing
Staying in the cluster-2
cluster (kubectx k3d-cluster-2
), we test if we can access these imported services from the curl
application in the mesh.
Get the pod of the curl
application, from which we will later launch requests to simulate service access.
At this point you will find that it is not accessible.
Note that this is normal, by default no other cluster instance will be used to respond to requests, which means no calls to other clusters will be made by default. So how to access it, then we need to be clear about the global traffic policy GlobalTrafficPolicy
.
Global Traffic Policy
Note that all global traffic policies are set on the user’s side, so this demo is about setting global traffic policies on the cluster cluster-2
side. So before you start, switch to cluster cluster-2
: kubectx k3d-cluster-2
.
The global traffic policy is set via CRD GlobalTrafficPolicy
.
Global load balancing types .spec.lbType
There are three types.
Locality
: uses only the services of this cluster, and is also the default type. This is why accessing thehttpbin
application fails when we don’t provide any global policy, because there is no such service in clustercluster-2
.FailOver
: proxies to other clusters only when access to this cluster fails, which is often referred to as failover, similar to primary backup.ActiveActive
: Proxy to other clusters under normal conditions, similar to multi-live.
The FailOver
and ActiveActive
policies are used with the targets field to specify the id of the standby cluster, which is the cluster that can be routed to in case of failure or load balancing. ** For example, if you look at the import service httpbin/httpbin
in cluster cluster-2
, you can see that it has two endpoints
for the outer cluster, note that endpoints
here is a different concept than the native endpoints.v1
and will contain more information. In addition, there is the cluster id clusterKey
.
Routing Type - Locality
The default routing type is Locality
, and as tested above, traffic cannot be dispatched to other clusters.
Routing Type - FailOver
Since setting a global traffic policy for causes access failure, we start by enabling FailOver
mode. Note that the global policy traffic, to be consistent with the target service name and namespace. For example, if we want to access http://httpbin.httpbin:8080/
, we need to create GlobalTrafficPolicy
resource named httpbin
under the namespace httpbin
.
After setting the policy, let’s try it again by requesting.
The request is successful and the request is proxied to the service in cluster cluster-1
. Another request is made, and it is proxied to cluster cluster-3
, as expected for load balancing.
What will happen if we deploy the application httpbin
in the namespace httpbin
of the cluster cluster-2
?
After the application is running normally, this time we send the request to test again. From the results, it looks like the request is processed in the current cluster.
Even if the request is repeated multiple times, it will always return Hi, I am from cluster-2!
, which indicates that the services of same cluster are used in preference to the services imported from other clusters.
In some cases, we also want other clusters to participate in the service as well, because the resources of other clusters are wasted if only the services of this cluster are used. This is where the ActiveActive
routing type comes into play.
Routing Type - ActiveActive
Moving on from the status above, let’s test the ActiveActive
type by modifying the policy created earlier and updating it to ActiveActive
: `ActiveActive
Multiple requests will show that httpbin
from all three clusters will participate in the service. This indicates that the load is being proxied to multiple clusters in a balanced manner.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.