An Encryption-as-a-service Architecture on CloudNative Platform

Bo YangComputer Science and Technology

Nanjing Tech UniversityNanjing, China

201961220032@njtech.edu.cn

Fan ZhangIBM Watson Group

IBM Massachusetts LabLittleton, MA

fzhang@us.ibm.com

Samee U. KhanElectrical and Computer Eng

Mississippi State UniversityMississippi State, MSskhan@ece.msstate.edu

Abstract—The Kubernetes is currently one of the most popularand powerful open-source cloud-native container orchestrationplatforms. Security at the container level of a Kubernetes clusteris known to be lacking. In this paper, we study an encryption-as-a-service architecture for Kubernetes pods, which provides aunified, flexible and universal method for encrypting the dataflow internally and externally between containers. We inject anHAProxy container on the fly to the application pod to performthe encryption. We use Elasticsearch-based microservices as acase study to implement our architecture. In the experiments,we tested the overhead and performance degradation with thisframework using various experimental settings. The results showthat our proposed solution achieves communication encryptionat the cost of only 7% of the write performance degradation.The query/read performance hardly drops.

Index Terms—Kubernetes, HAProxy, Pod, Encrypt, Elastic-search

I. INTRODUCTION

With the continuous evolution of cloud computing, con-tainer technology, and microservice architecture [1], cloudcomputing platforms are facing various threats [2]. Securityand privacy issues in the cloud have become prevailing incloud services. As cloud-native Kubernetes [3] being themainstream of container cloud, its security is even moreimportant. A survey from StackRox shows that 94% of or-ganizations have encountered serious security problems in theKubernetes container environment [4]. Loosely speaking, thetransition from servers and virtual machines to cloud-nativeenvironments with containers and Kubernetes did not eliminatethe threat or impact of attacks, nor did it shift responsibility tocloud providers. Instead, it brings a series of new and differentsecurity challenges [5].

A secured Kubernetes cluster is composed of multi-dimensional levels that must be handled from different angles.As shown in Fig. 1, the security breach of a Kubernetes clustercan be divided into four aspects: Infrastructure, Kubernetes,containers, and applications.

Security at the infrastructure level, or level four, lays thefoundation of the pyramid [7]. Infrastructure security needsto address issues ranging from network, storage, host oper-ating system, and host-level access management. Level threedefines the computing platform ingredients, where Kubernetesprovides configurable policies, e. g. Identity and Access Man-

Fig. 1: The security breach of a Kubernetes cluster [6]

agement(IAM), Role-Based Access Control(RBAC), SecurityContext Control(SCC) and Pod Security Policies (PSP) [8].to secure upper-level deployments. The second security levelruns at the core of the running pods or containers [9] [10],guarding the data securely flow among the containers. Thetop-level security lies at the applications themselves, providingfunctionalities that tie to applications, e. g., authentication andauthorization.

As shown in Fig. 1, security at the pod or containerlevel is mainly caused by data being eavesdropped from anunauthorized or untrust-worthy third party. There are two maintypes of container-related data, one being a request from auser to an application within a Kubernetes cluster using HTTP,the other one being data flow between application pods in aKubernetes cluster using Transport protocols.

In this paper, we mainly focus on proposing an encryption-as-a-service architecture, tackling the security issues at leveltwo in Fig. 1. There are a few reasons we focus on this level.First, the security at the application level is usually default

at the application itself, and the security at the platform andinfrastructure level are well provided/upgraded as a given ina Kubernetes cluster. Second, the security at the containerlevel, specifically, the IPC communication data flow betweencontainers, is way less provided. End user needs a simple,pluggable and performant option to encrypt the data flow.

Therefore, we propose an encryption-as-a-service architec-ture design solution tailored for micro-services running onKubernetes pods. This novel architecture provides enterprisesand users with an effective model to safeguard the internaland external data on the cloud. It injects encryption proxy asa third-party service without extra hardware cost. Furthermore,this is a unified, flexible and universal Kubernetes Podscommunication encryption method, suitable for applicationsof cloud-native platforms.

II. RELATED WORK

Traditionally, for data flow encryption of microservicesrunning on Kubernetes, researchers have proposed servicemesh [11]. The service grid is a dedicated infrastructure layerthat can be used to handle service-to-service communication.Similar to the container abstracting the operating system fromthe application, the service mesh abstracts the communicationbetween processes. The service mesh assists in the implemen-tation of keys, certificates, and TLS configuration managementfor continuous encryption. The service grid provides policy-based authentication, which can establish a two-way TLS con-figuration between two services to achieve secure encryptedcommunication between services (service-to-service) and enduser authentication.

Sarada Prasad et al. proposed a new lightweight decentral-ized encrypted cloud storage architecture called Yugula, whichuses blockchain to maintain file confidentiality, eliminate cen-tralized data deduplication and improve file integrity [12].

Anton Vedeshin et al. proposed a safe and reliable in-frastructure and architecture. Their solution incorporates thephysical limitations of the computing process into the defensestrategy, thereby making distributed file storage and transmis-sion highly secure. The main idea is to replace asymmetricor public key encryption functions with cryptographic hashfunctions of non-key, conflict, second pre-image and antigenicimage [13].

Luigi Coppolino et al. proposed a virtual security zone(VISE), a method that combines Intel SGX and homomor-phic encryption. VISE moves the execution of sensitive HEprimitives (such as encryption) to the cloud in the remotelyproven SGX secure area, and then uses all available memoryresources in the secure area to perform sensitive processingon the secure data (outside the secure area). This method canprotect the data in use in the cloud from the challenge ofprivileged attackers [14].

Different from all the above work, We propose aKubernetes-based encryption-as-a-service architecture solu-tion, which doesn’t require any third-party service, and itprovides enterprises and end users with a simple, pluggableand performant data encryption solution.

III. ENCRYPTION-AS-A-SERVICE ARCHITECTURE

A. Encryption at the pod level

The smallest building block of an application in a Kuber-netes cluster is called pod [15]. A pod is a group of one ormore containers. They share the container’s IP address andits port space. In practice, microservice-based applications arecontainerized and deployed as pods on Kubernetes clusters[16]. By default, the data flow between pods deployed onKubernetes is unencrypted. Therefore, the data of the podsor containers are easily compromised after they have beendeployed. We need to add enhanced data security at thedevelopment stage at the very beginning, that is, implementingencryption at the pods or containers level should start initially,when configuring the pods’ YAML resource files [17].

Fig. 2: Data traffic related to pods

Since the security concern at the pods level is protectingthe data stream that flows securely between pods, we need toconsider encrypting the data stream from external to the pod,or from within the pods. Therefore there are two types of datastreams associated with pods:

(1) Inter-pod data flow can be either in the same node orbetween different nodes:

• Inter-pod data flow in the same node: As shown inFig. 2, each pod on one node has its own IP address.Every pod has a plain Ethernet device eth0, throughwhich network requests/responses go through. Kuber-netes disguised it as a virtual Ethernet connection. Theeth0 device of each pod is actually connected to thevirtual Ethernet device in the node. A virtual Ethernetdevice is a tunnel that connects the pod’s network andnodes. This device has two different names but serves onthe same purpose. It is named eth0 on the pod side andvethX on the node side.When a pod sends data to a pod of another node, it willdo so through its eth0 interface. This is tunneled to the

corresponding virtual vethX interface of the node. In thesame node, the default routes of pod1 and pod2 both usecbr0. Since they are associated on the same cbr0 bridgeand have the same address network segment. The requestpacket of pod1 reaches to pod2 on the same node as it.The response packet will follow exactly the same routeas the incoming packet.

• Inter-pod data flow in different nodes: As shownin Fig. 2, if the pods are not on the same node, thecbr0 bridge will fall back to the default gateway. Thiswill go up to the cluster level and IP address look-up is kicked in. At the cluster level, there is a tablethat maps the IP address to each node. Pods on thesenodes will be assigned on IP addresses from these ranges.The communication between pod1 in node1 and pod1 innode2 must meet two conditions: First, pods’ IP addressescannot conflict. Second, associate the pods IP with the IPof the node where they are located in, and data can flowbetween pods through this association.

(2) Data flow from/to the outside the cluster: Since thepod’s IP of the flannel-based Kubernetes cluster is an internaladdress, pod cannot be accessed directly from the outside ofthe cluster. We need to define external resources or deployplugins through Kube-proxy to access the pods, e. g., byconfiguring pod’s hostNetwork to true, defining LoadBalanceron pod’s service or deploy Ingress controller, etc.

In order to achieve security at the pod level, we need toat least encrypt the data flow relevant to pods communicationfrom all the above case scenarios. At the same time, we needto consider the availability and performance by doing so.

B. Encryption-as-a-service architecture

Fig. 3: Encryption-as-a-service architecture framework

As shown in Fig. 3, We depict a simple yet usefulencryption-as-a-service architecture framework [18], whichhas the following characteristics:

1) An encrypting tooling serving as a proxy that encryptsdata flow runs within a container in each pod the main serviceresides in.

2) The encrypting tooling not only encrypts HTTP data fromthe external (Kubernetes Service), but also data between podsusing Transport protocol (Kubernetes Headless Service).

2) Within the pod, the container of the encryption toolingand the container of the main service uses two-way datacommunication.

The detail of the architecture is depicted as follows:1. Encrypting tooling

The core of this architecture scheme is a built-in toolingfor encrypting data streams in the pods. Such encryptingtoolings generally have proxy functions based on TCP andHTTP applications and can provide high-performance SSLTermination, which can encrypt and decrypt traffic. We usethis type of encrypting tooling as the communication agent ofthe main service and then turn on the encryption function toencrypt the data traffic of the main service. In this experiment,we use HAProxy as a tooling for encrypting pod data streams.Of course, we can also use other encrypting tooling, such asStunnel, Pound, and Nginx, etc.2. Container injection

We use AdmissionWebhook to inject encrypting tooling as athird-party service. Admission webhooks are HTTP callbacks,which receive admission requests and perform correspondingactions per these requests. There are two types of Admission-Webhooks, ValidatingAdmissionWebhook and MutatingAd-missionWebhook. MutaingAdmissionWebhook can interceptKubernetes API requests like an interceptor to modify thetarget resource. So we use MutatingAdmissionWebhook toimplement the injection of encrypting tooling services.3. Experimental case

In order to demonstrate the feasibility of the solution,we use microservice-based Elasticsearch as an experimentalcase below. It is a multi-node distributed search engine [19]using an HTTP web interface. Each of its main services isdeployed on a Kubernetes cluster in the form of the containerwithin a pod. In the experiment, we assign two ports to theElasticsearch container: 9200 and 9300. Port 9200 is usedfor communicating with the external requests, and is theRESTful endpoint of the incoming HTTP protocol. The podthat receives a client request doesn’t typically handle it aloneand usually needs to transfer it on to other pods for furtherprocessing. Port 9300 is used for such kind of communicationbetween Elasticseach pods by using a Transport protocol.

(a) HTTP request encrypted by HAProxy

(b) Transport request encrypted by HAProxy

Fig. 4: HAProxy encryption principle

As shown in Fig. 4(a), HAProxy serves as a proxy serverrunning as a server-side interface that redirects HTTP/HTTPSrequests to the destination [20]. To be more specific, a user

Fig. 5: Encryption-as-a-service implementation dataflow chart

connects to an HAProxy via HTTPS in order to accessthe Elasticsearch endpoint container. Elasticsearch will thenrespond by sending the result to HAProxy first, and HAProxySSL Termination will encrypt the response and send it backto the user who initially originated the request, as also shownin Fig. 4(b).

C. Encryption-as-a-service implementationAs shown in Fig. 5, it is a structural diagram of encryption-

as-a-service. HAProxy containers are built into Elasticsearchpods. HAProxy plays the role of Elasticsearch front-end.We configure the HAProxy container’s 9200 port to proxyElasticsearh’s HTTP port to accept user requests. We configureport 9300 of the HAProxy container to proxy Elasticsearch’sTransport port for forwarding to the other Elasticsearch nodes.

Fig. 6: HAProxy automatic injection process

We use Kubernetes’ MutatingAdmissionWebhook to au-tomatically inject the HAProxy container into the Elastic-search pod. Fig. 6 shows the process of such an automaticalinjection. The configuration files of HAProxy and Elastic-search are stored in two Configmaps - HAProxy-injector andElasticsearch, which contains the required Go template. TheHAProxy automatic injection process needs to follow the stepsbelow:

• Kubernetes needs to accept the information about theElasticsearch cluster and its configuration that theHAProxy to be injected is connected to.

• Kubernetes needs to accept the configuration informationof the HAProxy container itself to be injected.

• Kubernetes fills in HAProxy configuration parametersaccording to the HAProxy injection template and theabove configuration, and injects the above configurationinto the side of the application container.

apiVersion: admissionregistration.k8s.io/v1beta1kind: MutatingWebhookConfigurationmetadata:name: haproxy-injector

webhooks:- name:matchPolicy: EquivalentobjectSelector:

matchLabels:elasticsearch: "yes"

rules:- operations: ["CREATE"]

apiGroups: [""]apiVersions: ["v1"]resources: ["pods"]scope: "Namespaced"

clientConfig:service:

name: haproxy-injector-servicenamespace: elasticpath: "/add-haproxy"port: 443

caBundle: ""failurePolicy: Ignore

Fig. 7: An example of MutatingAdmissionWebhook configu-ration

MutatingAdmissionWebhook is an administration controllerin the form of a plug-in, which can edit requests according tocustom configurations. Fig. 7 shows the configuration of ourMutatingAdmissionWebhook. During the HAProxy automaticinjection phrase, the webhook follows the below steps tocomplete the process:

• Parse Webhook REST request and deserialize the rawdata of AdmissionReview.

• Parse the pod and deserialize the AdmissionRequest inthe AdmissionReview.

• Use pod to render HAProxy configuration template.

• Use pod and rendered template to create Json Patch.• Construct AdmissionResponse.• Construct AdmissionReview and send it to apiserver.

IV. PERFORMANCE TESTING

In this section, we extensively test the performance of Elas-ticsearch using benchmark toolings. We compare the perfor-mance of Elasticsearch before and after our encryption-as-a-service solution to discuss the overhead as to the performancedegradation after applying encryption. We use an officialElastic launch, stress testing tool Esrally for performanceevaluation.

A. Experiment Benchmark Setup

Fig. 8 shows a schematic of our experimental setup. Elastic-search and Esrally are deployed on the cluster. We configuredService for Elasticsearch. Esrally connects to the Elasticsearchcluster through the ClusterIP assigned by this Service.

Fig. 8: Experiment setup

Our experimental platform is a 6-node Kubernetes cluster ofversion 1.14.6. The cluster has three worker nodes and threemaster nodes. The specification of each node server is 16-coreCPU and 32GB Memory. We set up a 3-node Elasticsearchcluster, which is deployed on each worker node. We have alsoimposed CPU and Memory restrictions on the Elasticsearchcontainer, so that the injection of the HAProxy container hasenough CPU and Memory. As shown in TABLE I, it is theElasticsearch cluster related configuration.

TABLE I: Elasticsearch related parameter settingsParameter Value

image elasticsearch:7.5.2esJavaOpts -Xmx1g -Xms1g

replicas 3storage 30Gi

resources.requests.cpu 100mresources.requests.memory 1200Mi

resources.limits.cpu 1000mresources.limits.memory 2Gi

We used Esrally’s official dataset Geonames to thoroughlyevaluate the performance. To enrich the experimental results,

we modified the task definition of Geonames track in order toserve our experimental purpose. We modified the bulk size inthe track task definition and conducted multiple experimentsto test the write performance of Elasticsearch. Furthermore,we defined a variety of search operation types in the tracktask to test the query performance.

In order to make the experimental results more accurate,we repeated each set of experiments three times. The finalexperimental result is the average of the three experimentalresults.

B. Results and Analysis

Fig. 9: Bulk operation test write performance

We sequentially modified the bulk size value in the tracktask definition to 1000, 5000, 10000, and tested them re-spectively. Fig. 9 shows the mean throughput(docs/s) of theElasticsearch cluster when the bulk size is varied when theElasticsearch cluster performs the index-append bulk opera-tion. In the figure, we conclude that the throughput of theencrypted Elasticsearch cluster has decreased, with a rangeof about 1000-1500doc/s. That translates to merely 7% writeperformance degradation when applying the encryption frame-work, which is totally acceptable.

As shown in Fig. 10, we demonstrate the service time bytaking the unencrypted and encrypted Elasticsearch to processseven types of query task requests. We found out that the aver-age service time used to perform query operations before andafter the Elasticsearch communication is encrypted is almostthe same. After encryption, Elasticsearch takes slightly longerservice time for match all query, term query, phrase query,country agg uncached query, country agg cached query, ran-dom function score query and random script score query,and the error range is within 0-200ms. From this observation,we conclude that the query performance of Elasticsearch hashardly decreased after encryption.

Summarized above, we conclude that this encryption-as-a-service architecture design has very limited overhead onthe write and query performance of an Elasticsearch clusterrunning on Kubernetes platform.

(a) Comparison of service time of two Elasticsearch execution match all querytasks

(b) Comparison of service time of two Elasticsearch execution term query tasks

(c) Comparison of service time of two Elasticsearch execution phrase querytasks

V. CONCLUSIONS AND FUTURE WORK

In this paper, we propose a Kubernetes-based encryption-as-a-service architecture design scheme, which aims at encryptingthe data flow within the Kubernetes pods. We propose to useHAProxy container to the main service pods by injecting iton the fly via a webhook interface. The monitoring and portforward of the communication is realized through HAProxy.Also served as an SSL proxy, HAProxy encrypts the dataflowing into the Kubernetes cluster, or the data that flows inbetween the cluster. This is a light-weight, flexible, and low-overhead Kubernetes encryption method running at the podlevel.

To discuss the feasibility of the solution, we use Elastic-search based on microservices as a case study. Our experi-mental results show that the HAProxy container effectivelyimplements the encryption of communication between pod

(d) Comparison of service time of two kinds of Elasticsearch executing coun-try agg uncached query tasks

(e) Comparison of service time of two kinds of Elasticsearch executing coun-try agg cached query tasks

(f) Comparison of service time of two kinds of Elasticsearch executing ran-dom function score query tasks

(g) Comparison of service time of two kinds of Elasticsearch executing ran-dom script score query tasks

Fig. 10: Query operation test query performance

nodes in the Elasticsearch cluster and the encryption ofaccess outside the Elasticsearch cluster. In addition, throughthe performance test of Elasticsearch, we found that afterusing HAProxy encryption, Elasticsearch’s write performancedropped by about 7%, and query performance hardly sees anydegradation. The verified the effectiveness of this method.

In future, we will continue along the path by exploringfurther Kubernetes pods communication encryption methodswith more datastore benchmark applications, such as Redisor GlusterFS. We would like to conduct more performanceevaluations on varied workload patterns. Meanwhile, since wewill also consider encryption for the stored data by using dm-crypt 2.0k [21] or use LUKS [22] to achieve that. We believethat our solution will benefit both engineers and researchersthat are working enthusiasm on cloud-native ream.

REFERENCES

[1] Newman, Sam. Building microservices: designing fine-grained systems.” O’Reilly Media, Inc.”, 2015.

[2] Sabahi, Farzad. ”Cloud computing security threats and responses.” 2011IEEE 3rd International Conference on Communication Software andNetworks. IEEE, 2011.

[3] Burns, Brendan, Joe Beda, and Kelsey Hightower. Kubernetes: up andrunning: dive into the future of infrastructure. O’Reilly Media, 2019.

[4] StackRox. Kubernetes-native Security[Online]. Available:https://security.stackrox.com/kubernetes-native-security-what-is-it-why-it-matters.html?Source=Website&LSource=Website. 2020.

[5] Yarygina, Tetiana, and Anya Helene Bagge. ”Overcoming securitychallenges in microservice architectures.” 2018 IEEE Symposium onService-Oriented System Engineering (SOSE). IEEE, 2018.

[6] Arvind Gupta. A Security Checklist for Cloud Native Kubernetes En-vironments. https://thenewstack.io/a-security-checklist-for-cloud-native-kubernetes-environments. 2020.

[7] Habbal, Nadin. ”Enhancing Availability of Microservice Architecture:A Case Study on Kubernetes Security Configurations.” (2020).

[8] Baier, Jonathan, and Jesse White. Getting Started with Kubernetes:Extend your containerization strategy by orchestrating and managinglarge-scale container deployments. Packt Publishing Ltd, 2018.

[9] Gao, Xing, et al. ”ContainerLeaks: Emerging security threats of in-formation leakages in container clouds.” 2017 47th Annual IEEE/IFIPInternational Conference on Dependable Systems and Networks (DSN).IEEE, 2017.

[10] Marmol, Victor, Rohit Jnagal, and Tim Hockin. ”Networking in contain-ers and container clusters.” Proceedings of netdev 0.1, February (2015).

[11] Li, Wubin, et al. ”Service mesh: Challenges, state of the art, and futureresearch opportunities.” 2019 IEEE International Conference on Service-Oriented System Engineering (SOSE). IEEE, 2019.

[12] Gochhayat, Sarada Prasad, et al. ”Yugala: Blockchain Based EncryptedCloud Storage for IoT Data.” 2019 IEEE International Conference onBlockchain (Blockchain). IEEE, 2019.

[13] Vedeshin, Anton, et al. ”A secure data infrastructure for personalmanufacturing based on a novel key-less, byte-less encryption method.”IEEE Access 8 (2019): 40039-40056.

[14] Coppolino, Luigi, et al. ”VISE: Combining Intel SGX and HomomorphicEncryption for Cloud Industrial Control Systems.” IEEE Transactions onComputers (2020).

[15] Vayghan, Leila Abdollahi, et al. ”Microservice based architecture: To-wards high-availability for stateful applications with Kubernetes.” 2019IEEE 19th International Conference on Software Quality, Reliability andSecurity (QRS). IEEE, 2019.

[16] Burns, Brendan, et al. ”Borg, Omega, and Kubernetes: Lessons learnedfrom three container-management systems over a decade.” Queue 14.1(2016): 70-93.

[17] Shamim, Md Shazibul Islam, Farzana Ahamed Bhuiyan, and AkondRahman. ”XI Commandments of Kubernetes Security: A Systematiza-tion of Knowledge Related to Kubernetes Security Practices.” 2020 IEEESecure Development (SecDev). IEEE, 2020.

[18] Rahmani, Hossein, et al. ”Encryption as a Service (EaaS) as a Solutionfor Cryptography in Cloud.” Procedia Technology 11 (2013): 1202-1210.

[19] Bajer, Marcin. ”Building an IoT data hub with Elasticsearch, Logstashand Kibana.” 2017 5th International Conference on Future Internet ofThings and Cloud Workshops (FiCloudW). IEEE, 2017.

[20] Tarreau, Willy. ”HAProxy-the reliable, high-performance TCP/HTTPload balancer.” 2011-8)[2013-4]. http://haproxy. lwt. eu (2012).

[21] Broz, Milan, et al. ”dm-crypt: Linux kernel device-mapper cryptotarget.” URL: https://gitlab. com/cryptsetup/cryptsetup/wikis/DMCrypt(2015).

[22] Bossi, Simone, and Andrea Visconti. ”What users should know aboutfull disk encryption based on LUKS.” International Conference onCryptology and Network Security. Springer, Cham, 2015.