Deploy Management cluster

This guide describes how to setup a management cluster which you will use to deploy one or more instances of Kubeflow.

The management cluster is used to run Cloud Config Connector. Cloud Config Connector is a Kubernetes addon that allows you to manage Google Cloud resources through Kubernetes.

While the management cluster can be deployed in the same project as your Kubeflow cluster, typically you will want to deploy it in a separate project used for administering one or more Kubeflow instances, because it will run with escalated permissions to create Google Cloud resources in the managed projects.

Optionally, the cluster can be configured with Anthos Config Management to manage Google Cloud infrastructure using GitOps.

Deployment steps

Install the required tools

1. gcloud components

   gcloud components install kubectl kustomize kpt anthoscli beta
   gcloud components update
   # If the output said the Cloud SDK component manager is disabled for installation, copy the command from output and run it.
   

   You can install specific version of kubectl by following instruction (Example: Install kubectl on Linux. Latest patch version of kubectl from v1.17 to v1.19 works well too.

   Note: Starting from Kubeflow 1.4, it requires kpt v1.0.0-beta.6 or above to operate in kubeflow/gcp-blueprints repository. gcloud hasn’t caught up with this kpt version yet, install kpt separately from https://github.com/GoogleContainerTools/kpt/tags for now. Note that kpt requires docker to be installed.

Fetch kubeflow/gcp-blueprints package

The management cluster manifests live in GitHub repository kubeflow/gcp-blueprints, use the following commands to pull Kubeflow manifests:

1. Clone the GitHub repository and check out the v1.4.0 tag:

   git clone https://github.com/kubeflow/gcp-blueprints.git 
   cd gcp-blueprints
   git checkout tags/v1.4.0 -b v1.4.0
   

   Alternatively, you can get the package by using kpt:

   # Check out Kubeflow v1.4.0 blueprints
   kpt pkg get https://github.com/kubeflow/gcp-blueprints.git@v1.4.0 gcp-blueprints
   cd gcp-blueprints
   

2. Go to gcp-blueprints/management directory for Management cluster configurations.

   cd management
   

Configure Environment Variables

Fill in environment variables in gcp-blueprints/management/env.sh as followed:

MGMT_PROJECT=<the project where you deploy your management cluster>
MGMT_DIR=<path to your management cluster configuration directory>
MGMT_NAME=<name of your management cluster>
LOCATION=<location of your management cluster>

And run:

source env.sh

This guide assumes the following convention:

• The ${MGMT_PROJECT} environment variable contains the Google Cloud project ID where management cluster is deployed to.

• The ${MGMT_DIR} environment variable contains the path to your management directory, which holds your management cluster configuration files. For example, ~/gcp-blueprints/management/. You can choose any path you would like for the directory ${MGMT_DIR}.

  To continuously manage the management cluster, you are recommended to check the management configuration directory into source control.

• ${MGMT_NAME} is the cluster name of your management cluster and the prefix for other Google Cloud resources created in the deployment process. Management cluster should be a different cluster from your Kubeflow cluster.

  Note, ${MGMT_NAME} should

  • start with a lowercase letter
  • only contain lowercase letters, numbers and -
  • end with a number or a letter
  • contain no more than 18 characters

• The ${LOCATION} environment variable contains the location of your management cluster. you can choose between regional or zonal, see Available regions and zones.

Configure kpt setter values

Use kpt to set values for the name, project, and location of your management cluster. Run the following command:

bash kpt-set.sh

Note, you can find out which setters exist in a package and what their current values are by running the following command:

kpt fn eval -i list-setters:v0.1 ./manifests

Deploy Management Cluster

1. Deploy the management cluster by applying cluster resources:

   make apply-cluster
   

   Optionally, you can verify the management cluster spec before applying it by:

   make hydrate-cluster
   

   and look into ./manifests/build/cluster folder.

2. Create a kubectl context for the management cluster, it will be named ${MGMT_NAME}:

   make create-context
   

3. Install the Cloud Config Connector:

   make apply-kcc
   

   This step:

   • Installs Config Connector in your cluster, and
   • Creates the Google Cloud service account ${MGMT_NAME}-cnrm-system@${MGMT_PROJECT}.iam.gserviceaccount.com.

   Optionally, you can verify the Config Connector installation before applying it by:

   make hydrate-kcc
   

   and check ./manifests/build/cnrm-install-* folders.

Understanding the deployment process

This section gives you more details about the configuration and deployment process, so that you can customize your management cluster if necessary.

Management cluster folder layout

Your management cluster directory contains the following files and directories:

• Makefile is a file that defines rules to automate deployment process. You can refer to GNU make documentation for more introduction. The Makefile we provide is designed to be user maintainable. You are encouraged to read, edit and maintain it to suit your own deployment customization needs.

• manifests/cluster is a kustomize package defining all the Google Cloud resources needed using gcloud beta anthos apply. It can apply some Google Cloud resource types using the same resource definition for Config Connector. You can edit this kustomize package in order to customize the Google Cloud resources for your purposes.

• manifests/cnrm-install-* folders contain kustomize packages for Google Cloud services, iam policies and Kubernetes resources to install Config Connector following Advanced installation options for Config Connector.

• manifests/build is a directory that will contain the hydrated manifests outputted by the make rules.

Customizing the installation

Once you understand the folder layout, you can create Kustomize overlays folder in corresponding directory, for example manifests/cnrm-install/iam, so you can define patches in overlays folder. Then use overlays in kustomization.yaml file.

The following tips help you customize, verify and re-apply them to your cluster.

Some useful links for Kustomize:

• patchesStrategicMerge let you patch any fields of an existing resource using a partial resource definition.
• Reference for all Kustomize features: https://kubectl.docs.kubernetes.io/references/kustomize/.

To get detailed reference for Google Cloud resources, refer to Config Connector resources reference.

To verify whether hydrated resources match your expectation:

make hydrate-cluster
# or
make hydrate-kcc

and refer to hydrated resources in ./manifests/build/*.

To apply your customizations:

make apply-cluster
# or
make apply-kcc

Note that, some fields in some resources may be immutable. You may need to manually delete them before applying again.

FAQs

• Where is kfctl?

  kfctl is no longer being used to apply resources for Google Cloud, because required functionalities are now supported by generic tools including Make, Kustomize, kpt, and Cloud Config Connector.

• Why do we use an extra management cluster to manage Google Cloud resources?

  The management cluster is very lightweight cluster that runs Cloud Config Connector. Cloud Config Connector makes it easier to configure Google Cloud resources using YAML and Kustomize.

For a more detailed explanation of the drastic changes happened in Kubeflow v1.1 on Google Cloud, read kubeflow/gcp-blueprints #123.

Next steps

• Deploy Kubeflow using kubectl, kustomize and kpt.