Ellipse 8 Patching Guide

Ellipse 8 Patching Guide

ContentsEllipse 8 Patching Guide 2

Commercial In Confidence 3

Purpose 4

Scope 5

Update Notes 6

Patching the Installation 7

Overview 7

Accessing the Patch 7

Patching non-appliance based installations of Ellipse 8.6 or greater 9

Patching Appliance based installations of Ellipse 8 or greater 9

Load MOR 9

Upgrade the Infrastructure 10

Upgrade the Environment 10

Refresh the Browser 10

Patching the Appliance Operating System - Automatic 11

Overview 11

1. Override Public Yum Server (optional) 11

2. Add Proxy Server (optional) 11

3. Issue the Security Update 11

Patching the Appliance Operating System - Manual 13

Overview 13

1. Configure Yum 13

1.1 Ensure the Oracle Public Yum Server is accessible from the Appliance 13

1.2 Install the Security plugin for Yum 13

1.3 Verify Yum configuration 14

2. Apply patches 14

2.1 Stop Puppet service 14

2.2 Patch a specific errata 14

2.3 Start the Puppet service 14

2.4 Verify patch application 14

3. Roll-back patches 15

3.1 Roll-back the last patch 15

3.2 Verify roll-back 15

4. Patching Command and Control guest 15

4.1 Reboot the Command and Control guest 15

4.2 Verify Command and Control services 15

Patching the Pentaho / ETL System 16

Automated Maintenance Patches 16

Manual Maintenance Patches 16

Commercial In ConfidenceCopyright 2016 ABB

All Rights Reserved

Confidential and Proprietary

Legal Disclaimer

The product described in this documentation may be connected to, and/or communicate information and data via, a networkinterface, which should be connected to a secure network. It is your sole responsibility to ensure a secure connection to thenetwork and to establish and maintain appropriate measures (such as but not limited to the installation of firewalls, applicationof authentication measures, encryption of data, installation of antivirus programs, etc.) to protect the product, the network,your systems, and the interface against any kind of security breach, unauthorised access, interference, intrusion, leakage,damage, or corruption or theft of data. We are not liable for damages or losses related to any such security breach,unauthorised access, interference, intrusion, leakage, damage, or corruption or theft of data.

PurposeThis document describes the process for applying patches to Ellipse 8.

ScopeThis document has sections covering:

• Patching the Installation

• Patching Non-appliance based installations of Ellipse

• Patching Appliance based installations of Ellipse

• Patching the Appliance Operating System

• Configure Yum

• Apply patches for CVEs

• Rollback patches

Update NotesPlease refer to the Ellipse 'Update Notes' and check if there is information that applies.

Patching the Installation

OverviewTo install a patch, access and download the MOR file and then follow the relevant patching instructions. The patch name iscommunicated to the customer by ABB and referred to here as Ellipse-<patch_number>, for example Ellipse 8.6.1 MOR.

Figure: overview of install to upgrade

Each release comes with an uploaded MOR and release notes, that can be found in the Workspace for that product/version.Ensure that the MOR file is placed in the directory and that the MOR release selected corresponds to the version indicated inthe release notes.

The Oracle Linux Disk (for example 6.6 for Ellipse 8.6) needs to be present in the drive during any update/patch. For examplefrom 8.6.1 to 8.6.3.

NoteThe MOR file for each release also contains an updated list of OS security advisories (referred to as ELSA, Enterprise LinuxSecurity Advisory). Application of these security patches is currently an optional step separate from the applianceinfrastructure upgrade. It is envisaged that in a future release, application of OS security patches will be a mandatory stepin the infrastructure upgrade process. This is due to several reasons: the security patches include critical kernel bug fixes;ABB certifies and tests on patched appliances; and customers face issues when patching is ignored and the appliance OSbecomes out of date. It is highly recommended that OS security patches are applied after each appliance infrastructureupgrade as a matter of policy.

There are two sets of instructions, one for each type of installation. Refer to the section that applies.

They are:

• Non-appliance based installations of Ellipse 8.6 or greater

• Appliance based installations of Ellipse 8.6 or greater

Accessing the PatchThis section describes the process by which Ellipse 8 patches are downloaded.

1. Download the MOR file (example Ellipse 8.6.6 MOR) from the "ABB Customer Portal"

a. Login to the Customer Portal (https://enterprisesoftware.force.com/customerportal/login)

b. Select "Workspaces" to display "My Workspaces"

c. Select the "Workspace Name" for the Product Version required

Figure: Customer Product Portal - Select the Ellipse version

d.1. Select the Icon next to the file name to display the Open window, then (2) Select Open

Figure: Customer Product Portal - Select the Icon/File to Open

e. Select the Save action to save the MOR zip file.

Figure: Customer Product Portal - Select Save to download the zip file

NoteIf a customer does not have a login to the ABB Customer Portal they should contact their account manager andrequest one.

For 8.6 Appliance Manager installations:

2. Place the file in the appliance host:

/appliance/data/dist

Patching non-appliance based installations of Ellipse 8.6 orgreaterUse these instructions when patching Ellipse 8.6 versions that have been installed using the Ellipse 8 Manual Installation Guide.

NoteUse the Ellipse 8 Manual Installation guide to assist in installing the patch.

1. Follow the instructions for downloading MOR (Accessing the Patch above).

2. Update the properties files to ensure that the ELLIPSEEAR.BASELINE.VERSION refers to the baseline indicated in theRelease Notes.

3. Remove the application servers that need to be replaced using the instructions Un-install Ellipse 8 Components fromthe E8_Install_Manual.pdf.

4. Install the latest version of MOR by following the Steps to install MOR section from the Ellipse 8 Manual InstallationGuide (E8_Install_Manual.pdf). Use the MOR associated with the release notes.

5. Install the application servers that need to be updated, using the instructions from the Ellipse 8 Manual Installation Guide.

Patching Appliance based installations of Ellipse 8 or greaterUse these instructions when patching Ellipse 8.6 versions that have been installed using the Enterprise Appliance Manager(VEAM), also called Appliance Manager.

The steps for a release upgrade include:

1. Upload the offline software repository or MOR (see Load MOR below)

2. Upgrade the Appliance Infrastructure which is part of the VEAM (see Upgrade the Infrastructure below)

3. Upgrade the Appliance Environment(s), that is each environment hosted in that appliance (see Upgrade the Environmentbelow)

NoteDo not attempt to upgrade an environment release/version unless the appliance infrastructure upgrade is first performed.

Load MORAssumption:

• The new MOR file (.mor) has been uploaded to the directory (/appliance/data/dist) on the Appliance server that is tobe upgraded

1. Access the Appliance Manager at the following URL

http://cmdctl.(fully_qualified_hostname)

2. Click on Upload MOR from the Operations drop down list

3. Enter the file name (no path) of the MOR (.mor) file

4. Click on Execute to load the MOR file into the Appliance Manager

The version shall now be selectable from the version (Add) and new version (Upgrade/Downgrade) drop down lists. Thismeans that the uploaded MOR can be used for a new environment or an upgrade to an environment

Upgrade the Infrastructure

1. Access the Appliance Manager at the following URL

http://cmdctl.(fully_qualified_hostname)

2. Click on Upgrade Infrastructure from the Help link located in the top left of the window

3. Select the target Appliance Infrastructure release from the dropdown list

4. Click on OK to perform the upgrade of the infrastructure.

NoteUpgrades to the infrastructure will not impact deployed environments.

Upgrade the Environment

1. Go to the Manage Environments tab and select the environment to be upgraded.

2. Upgrade the environment (Upgrade/Downgrade button)

NoteThe properties should not need to be changed.

Refresh the BrowserFollowing any Environment upgrade, users will be required to do one of the following to ensure that updated versions ofbrowser side components related to the upgraded application are being used:

• Use F5 to reload the login screen page; or

• Close and reopen the browser tab or window and reload the login screen pageThere should be no need for end users to clear their browser cache or manipulate cache settings in any way.

Patching the Appliance Operating System - Automatic

OverviewEach MOR file installed in the system contains a list of security advisories (referred to as ELSA, Enterprise Linux SecurityAdvisory).

Related software patches addressing these security advisories are tested by ABB before shipping the MOR, thus supported forinstallation on a targeted appliance. Once the Infrastructure is upgraded with contents from a specific MOR, the operation"Security Update" operation can be selected in VEAM to install the patches.

Patches will be downloaded and installed on the Appliance and the Command and Control guest only by the standard utility'yum'.

The amount of time required to apply these patches may vary and depends on factors such as network speed and number ofpatches already installed on the system.

1. Override Public Yum Server (optional)By default, the Security Update operation tries to access Oracle's Public Yum repositories, http://public-yum.oracle.comCustomer may set up a local mirror and override Oracle's address by adding a new property to/appliance/data/conf/etc/appliance.properties

#------------------------------------------------------------------------------### LINUX UPDATE SERVER##------------------------------------------------------------------------------#

# Linux Update Server## * DEFAULT: public-yum.oracle.com# * Allows the overriding of the Public Yum Oracle repository used to download# * security patches for the appliance operating system.# *## [yum_server] = Yum server containing a Yum - compatible OEL distribution tree## Example:# linux.update.server=[yum_server]

NoteAny change to appliance.properties will require restarting services 'puppetmaster' (on the Appliance) and 'puppet' (on theAppliance and Command and Control).

2. Add Proxy Server (optional)To configure a HTTP proxy for use by Yum, edit /etc/yum.conf:

proxy=http://<host>:<port>proxy_username=<username>proxy_password=<password>

NoteChanges to /etc/yum.conf must be done on both the Appliance and the Command and Control Virtual Server.

3. Issue the Security UpdateAssumption:

• The new MOR file (.mor) has been uploaded to the directory (/appliance/data/dist) on the Appliance server that is tobe upgraded

• The Infrastructure level has been upgraded to the level supplied by the uploaded MOR

1. Access the Appliance Manager at the following URL

http://cmdctl.(fully_qualified_hostname)

2. Click on 'Security Update' from the Operations drop down list. Type the word 'reboot' in the text field if an appliancereboot is to be performed right after the patches are applied. Check progress report in'/appliance/data/dist/sec_update.log' and '/appliance/data/dist/sec_update.err'.

NoteGiven that ELSAs may affect several Operating System components, it is almost impossible to predict when a reboot isrequired as a result of the application of a patch. ABB recommend rebooting as soon as it's practicable after performing theupdate. This is especially true when certain components are involved (i.e. Linux kernels, stdlib, glib, ssh).

Patching the Appliance Operating System - Manual

NotePatching an appliance with this manual system may lead to a system running an untested OS configuration and is thereforeno longer supported by ABB. This section is intentionally left in the document to help the customer installing manualpatches if directed to do so by ABB.

This sections covers:

• Configure Yum

• Apply patches for CVEs

• Rollback patches

OverviewThis section will outline the procedures required to patch an Ellipse 8.6 3rd generation Appliance based on Oracle Linux 7.2 OSfor Common Vulnerabilities and Exposures (CVE).

The use of the term "patch" in this document represents the change of existing OS packages to address a specific CVE, thestandard identifier as defined by http://cve.mitre.org.

There will be no new OS functionality introduced as part of this procedure and the OS major and minor version will remainconstant.

1. Configure Yum

1.1 Ensure the Oracle Public Yum Server is accessible from the ApplianceThe two specific Yum repositories required are "ol7_latest" and "ol7_UEK_latest".

Using the Public Yum Server, define the two repositories in the following file /etc/yum.repos.d/public-yum-ol7.repo:

[ol7_latest]name=Oracle Linux $releasever Latest ($basearch)baseurl=http://public-yum.oracle.com/repo/OracleLinux/ol7/latest/$basearch/gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oraclegpgcheck=1enabled=0

[ol7_UEK_latest]name=Latest Unbreakable Enterprise Kernel for Oracle Linux $releasever ($basearch)baseurl=http://public-yum.oracle.com/repo/OracleLinux/ol7/UEK/latest/$basearch/gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oraclegpgcheck=1enabled=0

NoteDisabling both Yum repositories with "enabled=0" is critical as Puppet will utilise available repositories in order to ensurethe latest version of packages it administers. Another option is to specify the Yum config file at execution time with the"--config" flag, this may be a URL (http://public-yum.oracle.com/public-yum-ol7.repo) or a file system path outside of"/etc/yum.repos.d/".

To configure a HTTP proxy for use by Yum, edit /etc/yum.conf:

proxy=http://<host>:<port>proxy_username=<username>proxy_password=<password>

1.2 Install the Security plugin for YumInstall the following plug-in for Yum which will provide security specific package listing and information options.

yum -y install yum-plugin-security.noarch

1.3 Verify Yum configurationDetermine the specific security errata that are applicable using the previously defined Yum configuration:

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list

The Yum query returns a list of errata identifiers, severity and package version.

2. Apply patchesIt is recommended issued patches only be applied no later that one month prior to the specific Deployment Infrastructurerelease date. For example the release date of di-3.10.6 was 2015-10-14, the last patch issued priort to 2015-10-01 wasELSA-2015-1840 - issued 2015-09-29.

Issue date can be determined by inspecting the detailed information for the patch:

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest updateinfo info --advisory ELSA-2013-2576Loaded plugins: security

=============================================================================== unbreakable enterprise kernel security update=============================================================================== Update ID : ELSA-2013-2576 Release : Oracle Linux 7 Type : security Status : final Issued : 2013-10-18 CVEs : CVE-2013-4299Description : [2.6.39-400.209.2] : - dm snapshot: fix data corruption (Mikulas : Patocka) [Orabug: 17618492] {CVE-2013-4299} Severity : Moderateupdateinfo info done

2.1 Stop Puppet serviceStop the Puppet service on the Appliance to ensure there are no conflicts with Yum:

service puppet stop

2.2 Patch a specific errataDetermine the list of available errata, sort most recent to oldest and start from the last recommended patch:yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list | cut -f 1 -d ' ' | egrep ^ELSA-[0-9]*-[0-9]* | sort -r | uniq

Apply the last recommeded patch, for an Ellipse 8.6.1 Appliance running di-3.10.6 this would be "ELSA-2015-1840".

yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest -y update --advisory ELSA-2015-1840

Notethe Yum transaction id, this will be required in the event of a roll-back:

yum history info | egrep '^Transaction ID' | cut -f 4 -d ' '

For kernel patching, specifically the "kernel-uek", ensure the Grub configuration is updated and an outage schedule to rebootthe Appliance. Unless Ksplice is configured, "/etc/grub.conf" will need to be manually copied to "/boot/grub/grub.conf" for thenew kernel to take effect at boot.

2.3 Start the Puppet serviceStart the Puppet service by compiling the Puppet catalog for the Appliance:

puppet agent -t

There should be no errors returned from the catalog compile, else a roll-back will be required (return code 0 or 2).

2.4 Verify patch applicationTo verify the patches have been applied, execute the same query to list the security errata for the current OS packages. This isrecommended after each patch application to determine the next errata to apply:yum --disablerepo=* --enablerepo=ol7_latest,ol7_UEK_latest --security updateinfo list | cut -f 1 -d ' ' | egrep ^ELSA-[0-9]*-[0-9]* | sort -r | uniq

This list will be significantly smaller than the initial list prior to installing the patch.

3. Roll-back patchesIn the event the patch application is unsuccessful, a roll-back is required in order to reset the versions of OS packages. This isachieved using Yum history and transaction rewinding.

3.1 Roll-back the last patchUsing the transaction id determined in the previous section roll-back the changes using Yum history.

service puppet stopyum history undo <transaction_id>puppet agent -t

If there have been no Yum transactions since the patch application, the keyword 'last' maybe substituted:

service puppet stopyum history undo lastpuppet agent -t

Again the manual Puppet catalog compile from both examples should complete without error.

3.2 Verify roll-backIn addition to the clean execution of a manual Puppet catalog compile, query the list of security errata for the newly reset OSpackage versions:

yum --disablerepo=* --enablerepo=el5_latest,ol5_UEK_latest --security updateinfo list

The list returned will represent the patches available for the current versions of OS packages.

4. Patching Command and Control guestTo patch the Command and Control guest, the same procedures for the Appliance may be applied. Ensure an appropriateoutage has been scheduled as this will directly impact accessibility of deployed environments.

In the event a patch introduces a new kernel, the reboot process is obviously different with a guest than the physical Appliance:

4.1 Reboot the Command and Control guestIdentify the OpenNebula id which will be the integer in the first column of the output:

su -c "onevm list | grep cmdctl" oneadmin

Append the id to the string "one-" and reset the guest using "virsh":

virsh reset one-<id>

Optionally connect to the guest console to view the boot process:

virsh console one-<id>

4.2 Verify Command and Control servicesOnce the guest has finished booting, ensure Puppet, Veam and HTTPD services have started without error by inspecting therespective logs:

/var/log/messages/opt/veam/current/server.log/var/log/httpd/error_log

In addition to this, navigate to the Command and Control URL to exercise HTTPD and Veam.

Patching the Pentaho / ETL System

Automated Maintenance PatchesThis process will handle the Standard Maintenance Patches applied on the Appliance System. When this is done the appliancewill be rebuilt with the standard installation. After this has completed these additional steps will need to be done

1. JINDI Database Connection Configuration (refer to Ellipse Operations and Configuration Guilde > Pentaho Server >JINDI...)

2. CRON Job to Schedule /opt/datamart/pentaho/data-integration/load_all_for_SITE1.sh

3. Changes(If any) that you have made in "opt/datamart/pentaho/datamart/Star" will need to be made again.

Manual Maintenance PatchesManual Releases and Patches will be applied using this approach.

The Maintenance Patch and information will exist within these components

• Patch.tar.bz2

• This contains the Pentaho Star Schema

• This file will be used to update the Pentaho ETL Logic• ReleaseNotes.docx

• This will detail the Changes included in this Maintenance Patch• PatchNotes.docx

• This will detail any other steps required in the installationThis process will cover every action required unless more information is contained in the PatchNotes.docx document. Thisdocument should be read prior to starting this step.

1. Copy Tar file to /opt/datamart/pentaho/datamart/Stars

2. Unwind this tar file like : tar jxvf Patch.tar.gz

3. Apply and Steps contained in PatchNotes.docx

4. Run the Monthly Update

• The update ETL layer will make the required changes to the Datamart Schema and Data the first time it is run