Carlo Alberto Ferraris, Ronak Banka | Rakuten, Inc.

Everyday life with CFin a big organization

2

5 years of Cloud Foundry at Rakuten

https://www.youtube.com/watch?v=CwJJyQQUsV4

Integrating with company systems

Porting existing applications

Turning users into advocates

Integrating with company systems

5

RPaaS API and plugins

• API for Rakuten-specific tasks– Automated organization creation– Billing system integration

• Operates with admin privileges on the CF API on behalf of regular users

• Runs as Cloud Foundry application

6

RPaaS API and plugins

• User-facing features exposed via CF CLI plugins– Org administration (including demo orgs)

• Sign up can be done fully via CLI– Billing report

• Report resource usage– Manifest generation

• Rakuten-specifics aware• Helps new users onboarding

7

RPaaS API and plugins

• Benefits– Vanilla CF API– Our API is outside the critical path– Easy/low risk to experiment with

• Limitations– Can’t be used for “policy enforcement”

8

Multiple envs and the “stack hack”

• Rakuten has multiple networks (e.g. prod/non-prod)• Small team delivering a prod-level platform using the

open-source version of CF–Minimizing human operation work is important

• Placement pools Elastic clusters Isolation segments Rainbows and unicorns were (and still are) not ready

9

Multiple envs and the “stack hack”

• Solution: using the CF stack mechanism to create different zones

–Use the standard cflinuxfs2 stack but give it different names on different “zones”

–Concourse pipeline patch the buildpacks to disable the stack name check

–Plugin helps users select buildpack and stack name

10

Multiple envs and the “stack hack”

http://slides.com/cafxx/the-stack-hack

Porting existing applications

12

Can I use NFS?• Why

– Lots of legacy apps depends on NFS for data exchange

• Possible solution– Using FUSE NFS with cf apps

• Challenges– Security over NFS mounts– Customizations required to support system calls during app

startup– Reliability from production application point of view

13

How can I know what my application is doing?

• Why– Metrics which are provided on cli output are not enough to

understand system behavior– Many system utils can’t be used with default user on container– Metrics like latency, I/O, swap, RPS per instance are not

available for users.

• Possible solution– Something which can correlate data between routers & app

instances and stream them on logging pipeline

14

Can I restrict some of app operations in my space?

• Why– RBACLs too coarse, space developers can do all the

operations– L1 support don’t need the ability to push application but may

need to restart an instance– Configurations (credentials) are visible to all space users

• Possible solution– Support for operation based role creation (e.g. RPaaS API)

15

My application is not able to access a file?

• Why– Hardcoded paths can create issues because of the way

buildpacks configure the app directory

– Hardcoded configurations are again a big issue, when porting applications to different PaaS environments.

• Possible solution– Symlinks can only do so much– Go with docker images, lose part of the “PaaS experience”

16

Can I run my app with PHP 5.4?• Issue

– There are lots of applications out there running on unsupported versions of runtimes

– Custom buildpacks and docker images make this pretty painful– As a operator I want to have visibility of runtimes which people

are using from security perspective

• Possible solutions– Version check on cloud controller can help with hardening– Give cf files-like access to an auditing system

Turning users into advocates

18

Users and advocates

Rakuten doesn’t centrally mandate the technology to use+

In a company with a “long” history many ways of doing things are deeply ingrained in people

=Without a corporate champion for the platform getting new

users turns into a house-by-house battle

19

Supporting our users

Users and advocates

How we spend time in our team

Extending the platform Operating the platform

20

Users and advocates

You don’t need to convince users that the platform isbetter than what they have now

You need to convince them that it isSO FRIGGIN’ AWESOME THAT

OMG I HAVE TO TELL MY BUDDIES IN OTHER TEAMS

21

Users and advocates

Keep all channels openBe transparent

Be (with) the user

22

Users and advocates

https://www.youtube.com/watch?v=1o3LcxkAuNM

23

Users and advocates

ScreencastsIntroduction sessionsArchitectural supportOperational support

DocumentationSamples

24

Allies

CF summits and cf-dev are great for exchanging ideas and solutions

(with some caveats)

25

Allies

Holding sessions with other “private” CF operators is very fertile ground for inspiration and knowledge sharing

26

Q&A

Now or during the networking section after the talks

(BTW, we’re hiring!)