tibco software inc. tibco enterprise message …...tibco software inc. global headquarters 3307...

TIBCO Software Inc.

Global Headquarters

3307 Hillview Avenue

Palo Alto, CA 94304

Tel: +1 650-846-1000

Toll Free: 1 800-420-8450

Fax: +1 650-846-1005

www.tibco.com

TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.

TIBCO Enterprise Message Service™ on EKS This document describes how to run TIBCO Enterprise Message Service servers in an Amazon Elastic Container Service for Kubernetes (Amazon EKS).

Version 1.2 February 2020 Document updated for EMS 8.5

©2020 TIBCO Software Inc. All Rights Reserved. 2

Copyright Notice COPYRIGHT© 2020 TIBCO Software Inc. All rights reserved.

Trademarks TIBCO, the TIBCO logo, TIBCO Enterprise Message Service, TIBCO FTL, Rendezvous, and SmartSockets are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA


Table of Contents

1 Overview ................................................................................................................................ 5 1.1 Supported Versions ........................................................................................................................ 5 1.2 Excluded TIBCO EMS Features ........................................................................................................ 5 1.3 Prerequisites .................................................................................................................................. 6 1.4 Prepare Local Environment ............................................................................................................. 6 1.5 Prepare Preliminary AWS Account and Kubernetes Configuration .................................................. 6

2 Fault Tolerance and Shared Folder Setup ................................................................................ 8 2.1 Shared Storage ............................................................................................................................... 8 2.2 Create the EFS File System .............................................................................................................. 8

3 The EMS Docker image ......................................................................................................... 11 3.1 Creating the Base Docker Image ................................................................................................... 11 3.2 Extending the Base Docker Image ................................................................................................. 12

3.2.1 Provisioning FTL Client Libraries to Use the Corresponding Transports ...................................... 12 3.2.2 Provisioning Custom JAAS Authentication or JACI authorization Modules .................................. 12

3.3 Hosting the Docker Image ............................................................................................................. 13 3.3.1 Configure the AWS Container Registry ........................................................................................ 13 3.3.2 Push the EMS Docker Image to ECR............................................................................................. 13

4 AWS EKS Setup ..................................................................................................................... 14 4.1 Create a New Elastic Container Service for Kubernetes (EKS) ........................................................ 14 4.2 Configuring the Security Groups for EFS ........................................................................................ 16 4.3 Configuring the Elastic Kubernetes Service with Kubectl ............................................................... 16

4.3.1 Create a New Namespace ......................................................................................................... 16 4.3.2 Creating the Cluster Role for EFS ............................................................................................... 16 4.3.3 Provisioning the EFS File System ............................................................................................... 17

4.4 EMS Server Template ................................................................................................................... 19 4.4.1 Adjusting the Services NodePort Range (Optional) .................................................................... 19 4.4.2 TIBEMSD Template ................................................................................................................... 19 4.4.3 Creating a Deployment and Service .......................................................................................... 22 4.4.4 Stopping or Deleting an EMS Server .......................................................................................... 22 4.4.5 EMS Server Configuration ......................................................................................................... 23 4.4.6 Connecting to the EMS Server Container ................................................................................... 23

4.5 Central Administration Server Template ....................................................................................... 23

5 Accessing the EMS Server ..................................................................................................... 24 5.1 Security Group Inbound Security Rules ......................................................................................... 24 5.2 Modifying the Connection Factories in the EMS Server ................................................................. 24

Appendix A: TLS Configuration .................................................................................................. 26 A.1 Creating a Secret .......................................................................................................................... 26 A.2 Adjusting the Template................................................................................................................. 27 A.3 Adjusting the tibemscreateimage EMS Docker image build script ........................................ 28 A.4 Applying the Adjustments ............................................................................................................. 28


Table of Figures FIGURE 1 – CONFIGURE FILE SYSTEM ACCESS ........................................................................................................................ 9 FIGURE 2 - EFS FILE SYSTEM CREATION.............................................................................................................................. 10 FIGURE 3 - CREATE ECR REPOSITORY ................................................................................................................................ 13 FIGURE 4 - GET ECR LOGIN ............................................................................................................................................. 13 FIGURE 5 - TAG EMS DOCKER IMAGE ................................................................................................................................ 13 FIGURE 6 - EKSCTL INPUTS ............................................................................................................................................... 14 FIGURE 7 – EKS CLUSTER CREATION USING EKSCTL ................................................................................................................ 15 FIGURE 8 - KUBECTL RESULTS ........................................................................................................................................... 15 FIGURE 9 - CREATE TIBEMS NAMESPACE ............................................................................................................................. 16 FIGURE 10 - SUCCESSFUL CREATION OF PV AND PVC ............................................................................................................ 19 FIGURE 11 - TIBEMS TEMPLATE EXAMPLE ......................................................................................................................... 21 FIGURE 12 - CHECK DEPLOYMENT RESULTS ......................................................................................................................... 22 FIGURE 13 - TO STOP, START, AND DELETE THE DEPLOYMENT ................................................................................................. 23 FIGURE 14 - APPLY THE EMSCA TEMPLATE ........................................................................................................................ 23


1 Overview

Running TIBCO Enterprise Message Service (EMS) on Amazon EKS involves:

• Configuring the Amazon Elastic Container Service for Kubernetes (EKS) for TIBCO Enterprise Message Service (EMS).

• Configuring the Amazon Elastic File System (EFS) for the EMS shared storage • Configuring the Amazon Elastic Container Registry (ECR) for the Docker® image registry • Creating a Docker® image embedding EMS and hosting it in ECR • Configuring and creating EMS Kubernetes containers based on the EMS Docker image

1.1 Supported Versions

The steps described in this document are supported for the following versions of the products and components involved:

• TIBCO EMS 8.5.1 • TIBCO FTL 5.4 and later (static TCP transports only) • Docker Community/Enterprise Edition should be most recent version. V19.03.5 was used

in conjunction with this document • Amazon Linux 2 for EKS • Kubernetes 1.13 or newer

1.2 Excluded TIBCO EMS Features

As of March 2020, TIBCO EMS on EKS supports all EMS features, with the following exceptions:

• Excludes transports for TIBCO Rendezvous® • Excludes transports for TIBCO SmartSockets® • Excludes stores of type dbstore • Excludes stores of type mstores


1.3 Prerequisites

The reader of this document must be familiar with:

• Docker concepts • Amazon AWS console and the AWS CLI • Kubernetes installation and administration • TIBCO EMS configuration • NFSv4 (EFS)

1.4 Prepare Local Environment

The following infrastructure should already be in place:

• A Linux or MacOS machine equipped for building Docker images • JRE installation package (.tar.gz) • TIBCO EMS 8.5.1 installer for Linux (.zip) • TIBCO Enterprise Message Service 8.5.1 for Linux downloaded and installed to access the

/opt/tibco/ems/8.5/samples/openshift directory and files • The tibems_eks_files.zip has been downloaded from

https://community.tibco.com/wiki/tibcor-messaging-article-links-quick-access • Tibems_eks_files.zip (EMS Kubernetes yaml files)

Create a directory, place tibems_eks_files.zip in the directory. Unzip tibems_eks_files.zip. Place the TIBCO EMS installer zip, and the JRE tar file in the newly created tibems_eks_files directory.

1.5 Prepare Preliminary AWS Account and Kubernetes Configuration

Use the following to prepare the preliminary environment to install EMS on EKS. For more details on preparing AWS, see the Getting started documentation for EKS.

• An AWS account is required. If necessary, create one at http://aws.amazon.com and follow the on-screen instructions.

• Use the region selector in the navigation bar to choose the AWS Region to deploy EMS in EKS. The documentation will refer to us-east-1. Note: Currently, not all AWS regions and Availability Zones (AZ) support EKS. As of February 2020, the following AWS regions support EKS: US EU AP N. Virginia (us-east-1) Ireland (eu-west-1) Singapore (ap-southeast-1) Ohio (us-east-2) Frankfurt (eu-central-1) Tokyo (ap-northeast-1) Oregon (us-west-2) Stockholm (eu-north-1) Sydney (ap-southeast-2) Montreal (ca-central-1) London (eu-west-2) Seoul (ap-northeast-2) Sao Paulo (sa-east-1) Paris (eu-west-3) Mumbai (ap-south-1) Bahrain (me-south-1) Hong Kong (ap-east-1)


• Install and configure Amazon AWS CLI on the workstation used. Note: After creating the AWS account, ensure the AWS credential and config files are created, and contain the appropriate AWS key, secret key, profile, and region. Export the following environmental variable: export AWS_SDK_LOAD_CONFIG=1 when the default AIM Role is not used.

• Install Docker on the workstation to build the TIBCO EMS images.

• Install the kubectl command-line tool do manage and deploy applications to Kubernetes in AWS from a workstation.


2 Fault Tolerance and Shared Folder Setup

2.1 Shared Storage

A traditional EMS server configured for fault tolerance relies on its state being shared by a primary and a secondary instance, one being in the active state while the other is in standby, ready to take over. The shared state relies on the server store and configuration files to be located on a shared storage device. The fault tolerance model used by EMS on Kubernetes/EKS is different in that it relies on Kubernetes restart mechanisms. Only one EMS server instance is running and, in case of a server failure, will be restarted inside its container. In case of a failure of the container or of the corresponding cluster node, the cluster will recreate the container, possibly on a different AWS EC2 node, and restart the EMS server there. Within the container, the health of the EMS server can be monitored via the probe port. See the EMS 8.5.1 documentation for more information on the probe port. In any case, the server still needs its state to be shared. In AWS, this can only be accomplished with the AWS Elastic File System (EFS).

2.2 Create the EFS File System

This section outlines creating the EFS file system. Though the EFS file system can be created through the AWS CLI, the following will create the EFS file system through the AWS Console.

• From the AWS console, click on Services • Under Storage, click on EFS • Click on Create file system

o Select the appropriate VPC, and select the available mount targets in the availability zones where the EKS cluster will be configured.

o Click on next step.


Figure 1 – Configure File System Access

o Under Configure file system settings • Choose throughput mode: Bursting or Provisioned. Bursting throughput mode

should be used for most EMS shared file systems, such development or testing. Use Provisioned throughput mode for applications that require more throughput than provided by Bursting throughput. Only production should utilize provisioned throughput. If provisioned is used, select the Throughput in MiB/s. A valid range of 1 – 1024 MiB/s is available. Note: Provisioned can be expensive at $6.00 for 1MiB/s, and $6144.00 for 1024 MiB/s per month. 100 MiB/s is recommended, if used.

• Choose performance mode: General Purpose or MAX I/O. No measurable performance gains were seen with max i/o over GP. GP is recommended for all environments.

• Click on Enable encryption, if desired. Note: There will be some read/write performance degradation if selected.

• Click on Next Step o Under File system policy

• Select any policy, if required. No policies are necessary for EMS shared storage. o No Access points are required for EMS o Review and Create File System o Wait for a “Successful” creation

• Once the new EFS File System is created: o Should still be on the File systems page in the AWS console

• Refresh the page


• Select the newly created file system. Note the DNS name and that the “Life cycle state” is “Available” for all zones.

• Verify the correct VPC and Security Group were used

Figure 2 - EFS File System Creation


3 The EMS Docker image

3.1 Creating the Base Docker Image

The content of the container that will run on EKS derives from a Docker image that first needs to be created and then hosted in the AWS Elastic Container Registry (ECR). To create the Docker images, use the following: To create an EMS Docker image, use the tibemscreateimage script on a machine equipped for building Docker images.

Use the following steps to prepare the environment:

• Change directory to the tibems_eks_files directory previously created. • Ensure the following have been copied to tibems_eks_files:

o EMS 8.5 installation package o Optional EMS hotfixes o Optional Java package

Once all files are located to tibems_eks_files, tibemscreateimage script can be used to create the EMS Docker image. The script will also let you choose whether to save the image as an archive and create a user and group set to the required uid and gid values. For example: > tibemscreateimage TIB_ems_8.5.1_linux_x86_64.zip \ -j <JRE installation package>.tar.gz \ -u 1000 \ -g 1000

This example creates a Docker image based on the EMS 8.5.1 Linux installation package, adding a JVM, the 1000 uid and the 1000 gid. If you are curious to run this image stand-alone: > docker run -p 7222:7222 -v `pwd`:/shared ems:8.5.1 tibemsd

creates a sample EMS server folder hierarchy and configuration in the current directory and starts the corresponding server. > docker run -p 8080:8080 -v `pwd`:/shared ems:8.5.1 tibemsca

creates a sample Central Administration server folder hierarchy and configuration in the current directory and starts the corresponding server. You can override the creation and use of the sample configuration with your own setup:

> docker run -p 7222:7222 -v <path to shared location>:/shared \ ems:8.5.0 tibemsd -config /shared/<your server config file>

starts an EMS server using the <path to shared location>/<your server config file> configuration.


The tibemscreateimage script can be modified to meet your specific needs.

3.2 Extending the Base Docker Image

The base Docker image can be extended to include FTL client libraries and custom JAAS authentication and JACI authorization modules.

3.2.1 Provisioning FTL Client Libraries to Use the Corresponding Transports

1. Copy the FTL client library files to a temporary folder. 2. From the temporary folder, use a Dockerfile based on the example given below to copy

these files into the base Docker image: FROM ems:8.5.1 COPY --chown=tibuser:tibgroup . /opt/tibco/ems/docker/ftl

> docker build -t ems:8.5.1_ftl .

3. Upon customizing your EMS configuration, make sure to include /opt/tibco/ems/docker/ftl in the Module Path property.

3.2.2 Provisioning Custom JAAS Authentication or JACI authorization Modules

1. Copy your custom JAAS or JACI plugin files, including the static configuration files they may rely on, to a temporary folder.

2. From the temporary folder, use a Dockerfile based on the example given below to copy these files into the base Docker image: FROM ems:8.5.1 COPY --chown=tibuser:tibgroup . /opt/tibco/ems/docker/security

> docker build -t ems:8.5.1_security .

3. Upon customizing your EMS configuration, make sure to include the relevant paths to those files in the Security Classpath property. Note: The other required files are in their usual location: /opt/tibco/ems/<version>/bin and /opt/tibco/ems/<version>/lib For example: /opt/tibco/ems/docker/security/user_jaas_plugin.jar:/opt/tibco/ems/8.5/bin/tibemsd_jaas.jar:/opt/tibco/ems/8.5/lib/tibjmsadmin.jar, etc.


3.3 Hosting the Docker Image

3.3.1 Configure the AWS Container Registry A new ECR repository must be created to host the EMS Docker image.

• Create a new ECR repository named tibco/tibems in AWS. The repository can be created via the AWS CLI or via the console. Please note the URL of your ECR repository (e.g. 123456789012.dkr.ecr.us-east-1.amazonaws.com).

> aws ecr create-repository --repository-name tibco/tibems --region us-east-1

Figure 3 - Create ECR Repository

• Retrieve the login command to use to authenticate your Docker client with AWS registry. Adjust AWS region and access keys created in the previous step. > aws ecr get-login --no-include-email --region us-east-1

Figure 4 - Get ECR login

• Execute Docker login command returned in the previous step.

3.3.2 Push the EMS Docker Image to ECR • Tag the EMS Docker image to the ECR repository using the URL of the appropriate

repository account ID instead of 123456789012 and region.

> docker tag ems:latest 123456789012.dkr.ecr.us-east-1.amazonaws.com/tibco/tibems:latest

Figure 5 - Tag EMS Docker image

• Push the EMS Docker image to ECR. Replace 123456789012 with the appropriate repository account ID and region. > docker push 123456789012.dkr.ecr.us-east-1.amazonaws.com/tibco/tibems:latest


4 AWS EKS Setup

4.1 Create a New Elastic Container Service for Kubernetes (EKS)

A new Kubernetes cluster must be created in EKS. There are now just a few steps necessary to create the Kubernetes cluster in EKS. Follow the Getting Started EKS User Guide to create a new Kubernetes cluster. Cluster can be created via the AWS console or with eksctl. Note: Creating the EKS cluster via eksctl is highly recommended. This following will be based on using eksctl. Note: If AWS account has multiple roles or there are multiple clusters, it is recommended that the AWS CLI be used, and the –profile=XXX is used to ensure the correct permissions/roles are used.

• Install eksctl following the AWS documentation, if not already installed. • Create the new EKS cluster. The following example shows the minimum inputs. The

names, number of nodes, zones, EC2 instance size can all be adjusted to meet the requirements of the environment. It will take several minutes for the cluster to be created.

Note: Read through the eksctl documentation. There are other parameters which can be used, such as if the nodes should be “managed”.

eksctl create cluster \ -n emseks \ (1) -r us-east-1 \ (2) --zones us-east-1a,us-east-1b,us-east-1c \ (3) --nodegroup-name ems-workers \ (4) -t t3.medium \ (5) --nodes 3 \ (6) --nodes-min 2 \ (7) --nodes-max 3 \ (8) --node-volume-size 30 \ (9)

--vpc-cidr x.x.x.x/x (10) Figure 6 - eksctl inputs

(1): The name of the EKS Cluster (2): The AWS region where the EKS cluster will be built (3): The AWS Zones. This can be increased to fit requirements. Verify the AZ has the resources first. Also ensure that these zones were selected when creating the EFS storage (4): The Node Group name for the nodes (workers) (5): The EC2 instance size. A larger instance can be used to fit the requirements. Note: If the default size for the instance is 4 GB of RAM and 2 vCPUs. The instance size can be larger, if desired. (6): The number of cluster nodes. Three (3) is recommended.


(7): The minimum number of nodes. Two (2) are required. (8): The maximum number of nodes. No more than three are required. (9): The volume size (GB) of each node. Not used for persisted data. Can be small. (10): The CIDR of the VPC to use. If not included, eksctl will choose.

Figure 7 – EKS Cluster creation using eksctl

• To check on the status of the creation of the cluster, use: > aws eks describe-cluster --name <EKS Cluster> --region <AWS Region> --query cluster.status

• Test the configuration before continuing. Use kubectl get svc and kubectl get nodes. The results should be similar to the following. Cluster-IP can be different:

Figure 8 - Kubectl results

If a Kubernetes cluster is not shown, resolve any issues before continuing. If there are issues, it is usually a Kubernetes/AWS permissions/IAM role issue.


4.2 Configuring the Security Groups for EFS

After the EKS cluster is configured, and the worker nodes have been created, a new AWS Security Group will be created. The new security group will have the Kubernetes Cluster name as part of the security group name. This security group must now be associated with the EFS file system created previously. Note: If this step is not followed, it will not be possible for Kubernetes to connect to the EFS file system.

• Use the AWS Console, and navigate to the EFS service. • Click on the EFS file system previously created, and click on Actions, and then on Manage

file system access, • All of the mount targets will appear with the current available security groups. • Modify the security groups to include the new security group created for the EKS cluster.

4.3 Configuring the Elastic Kubernetes Service with Kubectl

With EKS, the Kubernetes command line tool, kubectl, is used to configure the Kubernetes cluster for EMS on EKS.

4.3.1 Create a New Namespace Create a new namespace in Kubernetes, if desired. This is optional. The default namespace can be used. Note: If the namespace tibems is not used, ensure the provided yaml files are modified to use the correct namespace or default. The examples shown below will use the tibems namespace.

> kubectl create namespace tibems

Figure 9 - Create tibems namespace

4.3.2 Creating the Cluster Role for EFS The EFS file system previously created, must now be configured (provisioned) to be accessible in EKS. This section will outline configuring EFS in EKS. With EKS, the Kubernetes command line tool, kubectl, is used to configure the Kubernetes cluster for EMS on EKS. Before the EFS file system can be provisioned, the Cluster Roles must be created for EFS. This will provide the roles and binding for EKS/Kubernetes to access EFS. Adjust ems-efs-clusterrole.yaml for namespace only, if not using tibems. Note: Do not modify any other part of the file!

Apply the file: > kubectl apply -f ems-efs-clusterrole.yaml -n tibems


4.3.3 Provisioning the EFS File System Provisioning the EFS storage resource in Kubernetes on EKS differs significantly from other file system mounts on AWS or in a generic Kubernetes environment. A separate deployment must be created that deploys the necessary Storage Class, Persistent Volume Claim (PVC), and Persistent Volume (PV) to mount the file system in the container. Once created, the file system will automatically be mounted on the active EMS Docker container in EKS. The following will create the deployment, Storage Class, and the PVC. The EFS provisioner will then create the PV based on the other resources.

Note: Ensure section 4.2 has been completed! Adjust provided file ems-efs-provisioner.yaml to your particular setup:

--- apiVersion: v1 kind: ConfigMap metadata: name: efs-provisioner data: file.system.id: fs-12d12345 (1) aws.region: us-east-1 (2) provisioner.name: example.com/aws-efs dns.name: "" --- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: efs-provisioner spec: replicas: 1 strategy: type: Recreate template: metadata: labels: app: efs-provisioner spec: containers: - name: efs-provisioner image: quay.io/external_storage/efs-provisioner:latest env: - name: FILE_SYSTEM_ID valueFrom: configMapKeyRef: name: efs-provisioner key: file.system.id - name: AWS_REGION valueFrom: configMapKeyRef: name: efs-provisioner key: aws.region - name: DNS_NAME valueFrom: configMapKeyRef:


name: efs-provisioner key: dns.name optional: true - name: PROVISIONER_NAME valueFrom: configMapKeyRef: name: efs-provisioner key: provisioner.name volumeMounts: - name: pv-volume mountPath: /persistentvolumes volumes: - name: pv-volume nfs: server: fs-12d12345.efs.us-east-1.amazonaws.com (3) path: / --- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: aws-efs provisioner: example.com/aws-efs reclaimPolicy: Retain --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: efs annotations: volume.beta.kubernetes.io/storage-class: "aws-efs" spec: accessModes: - ReadWriteMany resources: requests: storage: 1Mi ---

(1): AWS EFS file system ID (2): AWS Region, such us-east-1 (3): The complete AWS EFS DNS name

Note: DO NOT make any other changes to the file! Create the EFS provisioner > kubectl apply –f ems-efs-provisioner.yaml –n tibems

Check the results with the following. It may take a couple of minutes for the provisioner to create the PV. > kubectl get pv,pvc –n tibems


Figure 10 - Successful creation of PV and PVC

Note that the same PV/PVC can be used by multiple pods within the same namespace. Note: Creating the PV/PVC is done once for the lifetime of the namespace. So, when EMS creates the configuration file and data stores, these will never be deleted.

4.4 EMS Server Template

EMS server containers are created in a Kubernetes cluster through the provided tibemsd-template.yaml sample template. A deployment and a service will be created. This template includes sections that define a limited set of parameters, ports, and names for the deployment and the service, which can be changed to meet the environment. Note: The template creates the EMS server with TCP access. This maybe suitable for development and testing environments. However, for production environments, a TLS configuration is highly recommended. See Appendix A: for details for configuring the EMS server with TLS in Kubernetes on EKS.

4.4.1 Adjusting the Services NodePort Range (Optional) Services of type NodePort or LoadBalancer are used to expose EMS server listen ports outside the cluster. The range of allowed values defaults to 30000-32767. If you intend to use port numbers outside this range for the EMS server or Central Administration server, you can alter the range by using a load balancer in Kubernetes. See the Kubernetes documentation for details.

4.4.2 TIBEMSD Template The tibemsd-template.yaml has several sections that may need modification. The deployment section includes the names of the Kubernetes deployment, ports, and location/name of the ECR repository. The service section contains the port numbers. The default values in the example below can all be used, except for the name and location of the ECR repository, the runAsUser, and the trusted IP range. These must be updated for the environment. Only modify the values marked. Changes other values may prevent the TIBEMS deployment/service from being created or running. apiVersion: apps/v1 kind: Deployment metadata: labels: name: emstest01(1) name: emstest01(1) spec: replicas: 1


selector: matchLabels: name: emstest01(1) strategy: type: Recreate template: metadata: labels: name: emstest01 (1) name: emstest01 (1) spec: containers: - name: tibemsd-container image: "<Your AWS ECR>" (2) imagePullPolicy: Always (3) env: - name: EMS_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: EMS_PUBLIC_PORT value: "30722"(4) - name: EMS_SERVICE_NAME value: emstest01(1) - name: EMS_PROBE_PORT value: "7220"(5) args: - tibemsd livenessProbe: httpGet: path: /isLive port: probe-tcp initialDelaySeconds: 1(8) timeoutSeconds: 5 periodSeconds: 6 readinessProbe: httpGet: path: /isReady port: probe-tcp initialDelaySeconds: 1(8) timeoutSeconds: 5 periodSeconds: 6 ports: - containerPort: 7222(6) name: tibemsd-tcp protocol: TCP - containerPort: 7220(5) name: probe-tcp protocol: TCP resources: {} securityContext: runAsUser: 1000(7) terminationMessagePath: /dev/termination-log


terminationMessagePolicy: File volumeMounts: - mountPath: /shared name: tibemsd-volume dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - name: tibemsd-volume persistentVolumeClaim: claimName: efs --- apiVersion: v1 kind: Service metadata: labels: name: emslb(9) name: emslb(9) namespace: tibems(10) spec: externalTrafficPolicy: Cluster ports: - nodePort: 30722(4) port: 30722(4) protocol: TCP targetPort: 7222(6) selector: name: emstest01 sessionAffinity: None type: LoadBalancer loadBalancerSourceRanges: - <your trusted IP range>(11) status: loadBalancer: {}

Figure 11 - TIBEMS Template example

(1): The name of the EMS deployment and service. If modifying, change ALL locations. (2): The name and location of the Elastic Container Repository (ECR) where the TIBCO EMS

Docker image is located. Ensure the proper permissions are set. The image maybe something different for lastest, depending on how it was tagged in Docker.

(3): Determines if the EMS Docker image should be pulled from the ECR prior to starting the container. Use Always to download the Docker image every time, or Never to use the existing image.

(4): Throughout the template, 30722 is used for the external EMS port. If 30722 is not used, change the value accordingly.

(5): Throughout the template, 7220 is used for the EMS Probe port. If 7220 is not used, change the value accordingly.


(6): Throughout the template, 7222 is used for the internal EMS port. If 7222 is not used, change the value accordingly.

(7): The uid the container will run as. The default is 1000. Change runAsUser to the uid the EMS server container must run as with respect to accessing NFS. Note: The uid provided here must match that used when creating the EMS Docker image.

This constraint should be removed in a future version of Kubernetes. (8): The initialDelaySeconds determines the delay before the probes become active. This is set

for 1 second. (9): The name of the EMS Load Balancer service. If modifying, change all locations. (10): The name of the Kubernetes namespace used in the cluster. If no namespace is used, change

to default. (11): The load balancer source range. This is the trusted IP range to connect to the load balancer.

4.4.3 Creating a Deployment and Service Create a deployment and service with an EMS server using the corresponding template. For example: > kubectl apply –f tibemsd-template.yaml –n tibems

The kubectl operation transforms the tibemsd-template.yaml template into a list of resources using the default and overridden parameter values. That list is then passed on to create process for creation. In this particular case, it results in the creation of a deployment, a ReplicaSet, a pod and a service. Three of the four objects can be selected through the emstest01 label. The service will have the label emslb. The service exposes itself as emslb inside the cluster and maps internal port 7222 to port 30722 for both inside and outside the cluster.

Check the results using the following: > kubectl –n tibems get --selector name=emstest01 all > kubectl –n tibems describe deploy/emstest01 > kubectl –n tibems describe svc/emslb

Figure 12 - Check Deployment Results

or in the Kubernetes Web Console (not documented).

4.4.4 Stopping or Deleting an EMS Server To stop an EMS server without deleting it, use the kubectl scale operation to set its number of replicas to 0.

For example: > kubectl scale --replicas=0 deploy emstest01 –n tibems

To start it again, set its number of replicas back to 1: > kubectl scale --replicas=1 deploy emstest01 –n tibems

To delete an EMS server deployment and service entirely, use the kubectl delete operation. For example:


> kubectl delete --selector name=emstest01 deploy,svc –n tibems

Figure 13 - To Stop, Start, and Delete the Deployment

The corresponding pod and ReplicaSet will also be deleted. The PC and PV will not be deleted. These must be deleted separately.

4.4.5 EMS Server Configuration As mentioned in Section 3.1, running a container off of the EMS Docker image creates a default EMS server folder hierarchy and configuration. In an EKS cluster, the configuration will be created under /shared/ems/config/<EMS_SERVICE_NAME>.json. The Central Administration server works in a similar way. This is handled by the tibems.sh script embedded in tibemscreateimage and is invoked through the Docker image ENTRYPOINT. It can be overridden by altering the args entry in the template and is provided only for illustration purposes. Feel free to alter tibems.sh or to directly provision your own configuration files to suit your needs.

4.4.6 Connecting to the EMS Server Container The EMS server logs and configuration file can be accessed directly through the following command. Substitute the name of the running EMS pod for <Pod name>. This can be useful for viewing the logs or configuration file. Accessing the pod will be necessary to modify the EMS Connection Factories discussed in section 5. > kubectl -n tibems exec -it <Pod Name> /bin/bash

4.5 Central Administration Server Template

A Central Administration server container is created in the Kubernetes cluster through the tibemsca-template.yaml sample template. The structure of this template is almost identical to that of the EMS server template. Most of the concepts described in the previous section also apply to the Central Administration server. Note: Ensure to update the Docker image location from the ECR, and the external port number. The default is 30088. Note: The Central Administrator is not secure. The following is for example only! JAAS must be implemented in both the EMS server and CA before use.

Example of deployment and service creation with a Central Administration server: > kubectl apply -f tibemsca-template.yaml –n tibems

Figure 14 - Apply the EMSCA Template


5 Accessing the EMS Server

The EMS server is configured so that access from an EMS client can be internal to AWS, or accessed externally. Access is the same. The section will outline how to connect to the EMS Server.

5.1 Security Group Inbound Security Rules

Note: The Kubernetes load balancer(s) created in AWS by the EMS and EMSCA services have inbound security rules limited to the trusted IP range. Any IP address in this range can access the load balancer(s). The EMS Server running in the Docker Container on the node in AWS is accessed via the External-IP address and Port created via the emslb service. To get the load balancer External-IP address and Port, use: > kubectl -n tibems get svc emslb To test access to the EMS Server running the Docker Container on the node in AWS use the EMS tibemsadmin CLI. In the following example, use the EXTERNAL-IP of the load balancer, and 30722 is the external port. Note: If there are issues connecting to the EMS Server, this will be due to a Security Group rule, usually not having inbound/outbound rules with access to/from the external load balancer. > cd /opt/tibco/ems/8.5/bin > ./tibemsadmin -server tcp://a1a2142d71234561088108-1150113189.us-east-1.elb.amazonaws.com:30722

5.2 Modifying the Connection Factories in the EMS Server

The EMS server running in the Docker Container on the node in AWS must be modified for the Connection Factory URLs. The EMS Connection Factories must be updated from the Node Name to the load balancer External-IP address. If not done, the EMS clients will not be able to reconnect to EMS after a fail-over of the EMS Server deployed in EKS. The approaches to use to modify the tibemsd json configuration file being used in the container are:

• The tibemsadmin CLI for another machine (can be external) o Use the command shown above to access the EMS Server o Use the setprop option and modify all Connection Factories for the URL o Commit your changes o Exit out tibemsadmin Note: a script can be created and used to make the modifications via tibemsadmin

• Logging into the Kubernetes node, and modifying the configuration file. o Use kubectl exec –it <pod> -n tibems -- /bin/bash


o Edit ems/config/emstest01.json and modify all Connection Factories for the URL o Exit out of the Kubernetes node o Stop and re-start the deployment as defined in Section 4.4.4

• The EMS Central Administrator, if so configured. o Easiest approach. o Must ensure the Central Administrator is secured. o Modify the Connection Factories for the URL in EMSCA o Save and deploy the new configuration


Appendix A: TLS Configuration

This appendix takes you through the steps of modifying the EMS server template and Docker image build script so that EMS clients can connect to the server through TLS (formerly SSL). Whether an EMS listen port is configured for TCP or TLS makes no difference in terms of exposing it through a service, either in the AWS VPC or from a remote location. However, you need to decide how to provision the corresponding certificate files. While these could be placed in the EFS shared folder or embedded in the EMS Docker image, the standard practice in the Kubernetes world consists of using secret objects. These are meant to decouple sensitive information from the pods and can be mounted into containers as volumes populated with files to be accessed by programs. In this example, we will assume that we want the EMS server to be authenticated by EMS clients. This involves providing the server with its certificate, private key and the corresponding password, which we will store inside a secret. We will mount that secret into the container, point the EMS server configuration to the certificate and private key files and pass the corresponding password to the server through its -ssl_password command-line option. Based on the sample certificates that ship with EMS, the files will eventually be made available inside the container as follows: /etc/secret/server.cert.pem /etc/secret/server.key.pem /etc/secret/ssl_password

A.1 Creating a Secret

To store the server certificate, private key and the corresponding password in a secret, based on the sample certificates available in the EMS package under ems/<version>/samples/certs: > cd …/ems/<version>/samples > kubectl create secret generic tibemsd-secret \ --from-file=server.cert.pem=certs/server.cert.pem \ --from-file=server.key.pem=certs/server.key.pem \ --from-literal=ssl_password=password –n tibems

You can check the result this way: > kubectl describe secret tibemsd-secret –n tibems


A.2 Adjusting the Template

The tibemsd-template.yaml template needs to be adjusted to mount the secret as a volume. This involves adding one new entry to the volumes section and another one to the volumeMounts sections. kind: Deployment … spec: … template: … spec: containers: - name: tibemsd-container … volumeMounts: - mountPath: /shared name: tibemsd-volume - mountPath: /etc/secret name: tibemsd-secret-volume readOnly: true … volumes: - name: tibemsd-volume persistentVolumeClaim: claimName: ${{EMS_PVC}} - name: tibemsd-secret-volume secret: secretName: tibemsd-secret


A.3 Adjusting the tibemscreateimage EMS Docker image build script

In the tibemsd-configbase.json section: Modify the primary_listen to use ssl: "primary_listens":[ { "url":"ssl://7222" } ],

Add an ssl section pointing to the certificate files: "tibemsd":{ … "ssl":{ "ssl_server_identity":"/etc/secret/server.cert.pem", "ssl_server_key":"/etc/secret/server.key.pem" },

In the tibems.sh section: The tibemsd_run() function needs to be modified to launch the EMS server with the proper value for its -ssl_password command-line option: … if [[ \$# -ge 1 ]]; then PARAMS=\$* else tibemsd_seed PARAMS="-config /shared/ems/config/\$EMS_SERVICE_NAME.json -ssl_password \`cat /etc/secret/ssl_password\`" fi …

A.4 Applying the Adjustments

• Regenerate the EMS Docker image, tag it and push it to the Registry (see section 3.1). • Create a new deployment and service (see section 4.4.3).

Check the results by connecting to the EMS server with one of the EMS TLS sample clients: > java tibjmsSSL -server ssl://13.118.90.27:30722 \ -ssl_trusted ../certs/server_root.cert.pem \ -ssl_hostname server

tibco software inc. tibco enterprise message …...tibco software inc. global headquarters 3307...

Documents