PKS uses the PKS CLI to manage clusters; another management option: register PKS with VMware Admiral, for example, built into vRealize Automation and manage clusters from it. However, Admiral will not allow automating work with clusters, this is just a GUI. What if you need the ability to make a single process, combining the creation of a cluster, registering it in DNS, setting monitoring in vROps, creating a project in Harbor and adding rights to users?
As one solution, you can use the vRA-PKS integration pack from the following article: Provision and Manage Kubernetes Clusters as-a-Service using vRealize Automation and Pivotal Container Service . The author offers a set of basic processes for managing kubernetes clusters on PKS. What is implemented:
- To work with kubernetes clusters, PKS CLI commands are called from a remote Linux machine via SSH;
- Created data type DynamicTypes:PKS.PKSCluster and developed two processes for it: search for all clusters and search for a cluster by ID;
- Cluster data is stored in vRO configuration items (to reduce the number of server calls);
- Added processes for calling PKS commands to create, modify and delete a cluster;
- The PKSCluster type is registered in vRA (Custom Resources), an XaaS blueprint for creating a cluster and two Resources Actions are added: changing the number of worker nodes and deleting a cluster.
However, we did not succeed in using this package “as is” due to insufficient functionality, so we had to take its ideas as a basis and write an extended version …
VRA-PKS Integration Pack
Changes in the processes of calling PKS commands:
- Separate process: “Run PKS command”. Executing each command is registering with the PKS and then invoking the command. For convenience, this sequence is implemented as a separate process;
- Running PKS CLI commands on the local vRO server. You can choose where to run the command, on a local or remote server, the way of launching is set by the runLocal flag in the server configuration item. Allowing operating system commands to run from orchestrator processes ;
- Account password protection. In case of a process failure, the values of the variables are written to the logs, including the pks login command along with the password. Added clearing of the cmdlogin variable after registration on the server. You can also use a separate script for registration by placing it on the server with the PKS CLI, but then you lose the flexibility of configuring the PKS account (see “Run PKS command” -> “Local command”);
- Waiting for commands to execute. Added “-wait” key to create-cluster, delete-cluster, resize calls to wait for command execution in VRA interface;
- “Improved” expectation. An alternative process for running PKS commands with waiting for execution through improve sleep has been developed: “Run PKS command and wait”. It can be used instead of calling the “Run PKS command” in time-consuming processes (for connoisseurs of CPU time).
Changes in the implementation and parameters of the main processes for working with K8S clusters:
- Choosing a deployment plan when creating a cluster. The list of deployment plans is saved in the server configuration (pksplans attribute) and can be obtained by calling the “Get PKS server info” process;
- Checking the input of the number of working nodes in the “Scale a K8S cluster” process. For PKS CLI versions 1.1.x and 1.2.x, only an increase in the number of nodes is available, starting from version 1.3.0 it is possible to reduce the number of nodes;
- Additional confirmation when calling to delete a cluster;
- Registering a cluster on a DNS server. Calls to create and delete DNS records for clusters. (Since DNS workflows may vary, the package ships with stub processes for DNS operations, be sure to replace them with your own.)
Refinement of DynamicTypes
Extensions to work with DynamicTypes and refinement of the necessary processes:
- Additional PKSCluster attributes. Provides extended information about the cluster. Parameter values are stored in the cluster configuration item.
- Added a new data type PKSServer, the main characteristics of the server are loaded into the object properties;
- Communication of data types. The connection of PKSServer-PKSCluster objects has been created and the necessary processes “Find PKSObject by ID”, “Find relation PKSObjects” and “Has PKSObject children” are implemented.
Implementation of additional processes for developers and administrators:
- PKS server information. Obtaining a list of deployment plans and the current version of the PKS CLI, the result is saved in a configuration item for use in the work of processes;
- Cluster information. Added process for getting information about K8S cluster in JSON format;
- An additional process for registering previously created kubernetes clusters to vRA is “Register K8S cluster”;
- Getting kubeconfig. Creation of the kubernetes configuration file, distributes the archive with the configuration file to the specified user e-mail addresses (this process is registered as Resource Action on vRA).
Version 1.1 candidates:
- Store cluster parameters in a configuration item in json-string format;
- Add the description field (description of the cluster) to the properties of the PKSCluster object and to the cluster creation form;
- Move the path of the directory with scripts ( /opt/vco/tmp/ ) to the attribute of the pksconfig configuration element;
- Improvement of the “Get PKS server info” process to initialize all the necessary data in pksconfig before starting work;
- Split the “Get K8S credentials” process into two separate processes: receiving and sending kubeconfig.
Candidates for the next version:
- Adding an intermediate PKSManager object between PKSServer-PKSCluster to enable multiple users with pks.clusters.manage roles to work on one PKS server.
Installing and preparing the package for work
Installing the package in vRealize Orchestrator:
- ATTENTION! You DO NOT need to import the vRO-PKS.package via the Packages tab toolkit. The package contains DynamicTypes settings that will not be applied on import;
- Go to the ‘Workflows’ tab open Library-> Dynamic Types-> Configuration and run ‘Import Configuration From Package’ and select ‘vRO-PKS.package’ from the ‘vRA-PKS-Integration.zip’ archive;
- Check that all items have been successfully imported: processes, actions, config items, new DynamicTypes.
To import the package into vRealize Automation:
- Connecting to vRA from vRealize CloudClient:
$ vra login userpass --user email@example.com --tenant zabedu --server https://vra.zabedu.ru
- Importing a package into vRA:
$ vra content import --path /home/as/vRA-PKS.zip --resolution OVERWRITE --verbose
- Configure access to blueprints, enable the actions “Delete K8S cluster”, “Scale a K8S cluster” and “Get K8S credentials”.
Configuring components in vRO:
- Decide how to register on PKS in the “Run PKS command” process, the “Login command” component: send the pks login command directly or use an additional script on the server side;
- Select the method of launching PKS commands: locally from the vRO server or from the SSH server (the runLocal flag in the pksconfig configuration item), set the parameters for running commands via SSH: ipaddr, port, sshUser, sshPassword (pksconfig configuration item);
- Provide your PKS server and credentials: pksapi, apiuser, apipassword (pksconfig config item);
- Set the dnszone attribute, it is used to form the value of the “—external-hostname” key of the pks create-cluster command (pksconfig configuration item);
- Install PKS CLI on the selected server and check its operation;
- Copy the scripts from the archive to the server with the PKS CLI, correct the paths used and access rights to the scripts, the processes call scripts from /opt/vco/tmp/;
- Enter the names of the used deployment plans and the version of the PKS CLI: pksplans, pkscliversion or run the “Get PKS server info” process (the pksconfig configuration item);
- Replace the empty registration and deletion of DNS records in the processes of creating and deleting clusters (can be done later);
- Configure the parameters of the mail server for the “Get K8S credentials” process or replace the call to the “E-mail notification with attachment” process in it with your own analogous process (you can do it later);
- Remove demo data (attributes) from PKS-> Clusters-> pksconfig.clusters config item (can be done later).
Files for the article
vRA-PKS-Integration.zip – integration package;
PKS.pdf – documentation for package processes.
Translated by Google Translate