with ibm corp.€¦ · kafka connect. check the configuration of tnpm-bde services in ambari...

34
IBM ® Tivoli Netcool Performance Manager Big Data Extension1.4.3 Document Revision R2E1 Troubleshooting Big Data Extension IBM

Upload: others

Post on 02-Aug-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

IBM® Tivoli Netcool Performance Manager Big DataExtension1.4.3Document Revision R2E1

Troubleshooting Big Data Extension

IBM

Page 2: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

NoteBefore using this information and the product it supports, read the information in “Notices” on page 23.

This edition applies to version 1.4.3 of Tivoli Netcool Performance Manager Big Data Extension and to allsubsequent releases and modifications until otherwise indicated in new editions.

© Copyright IBM Corporation 2017.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contractwith IBM Corp.

Page 3: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Contents

Chapter 1. Troubleshooting and support 1

Chapter 2. Log files in Big DataExtension . . . . . . . . . . . . . . 3Log message format . . . . . . . . . . . . 3

Chapter 3. Messages . . . . . . . . . 7Error messages produced by Tivoli NetcoolPerformance Manager Collector Service . . . . . 7

Chapter 4. Troubleshooting installationand uninstallation . . . . . . . . . . 13Installation and uninstallation issues . . . . . . 13

Kafka Connect is unable to recover from a closedconnection. . . . . . . . . . . . . . 13Do not restart all Kafka Services when youremove Kafka Schema Registry from one of thenodes . . . . . . . . . . . . . . . 13Some services do not start automatically onAmbari server restart . . . . . . . . . . 14Ignore the error in some Big Data Extensionservices log files . . . . . . . . . . . . 14Snappy java.lang.UnsatisfiedLinkError error inthe Storage Service log file . . . . . . . . 14Controlling maximum number of processesavailable for a user . . . . . . . . . . . 14A newly added ZooKeeper instance is notupdated in the yarn.resourcemanager.zk-addressconfiguration parameter . . . . . . . . . 15

Tivoli Netcool Performance Manager CollectorService does not shut down gracefully . . . . 15

Chapter 5. Troubleshooting Ambariserver . . . . . . . . . . . . . . . 17Problem in decommissioning DataNodes . . . . 17Ambari HDFS Metric showing huge value forUnder Replicated Blocks in a single nodeenvironment . . . . . . . . . . . . . . 18Ambari Metrics configurations warning keepsappearing . . . . . . . . . . . . . . . 19Timezone changes are not reflected for monitoringBig Data Extension metrics on Ambari by usingFirefox ESR . . . . . . . . . . . . . . 19You might notice an OversizedPayloadExceptionerror in Storage Service log with Metric API calls. . 19

Chapter 6. Troubleshooting Big DataExtension REST APIs . . . . . . . . 21Use the entity names in REST API queries instead ofparent names . . . . . . . . . . . . . . 21

Notices . . . . . . . . . . . . . . 23Trademarks . . . . . . . . . . . . . . 25Terms and conditions for product documentation. . 26

© Copyright IBM Corp. 2017 iii

Page 4: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

iv Troubleshooting Big Data Extension

Page 5: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 1. Troubleshooting and support

You can use this troubleshooting and support information to determine whysomething does not work as expected and how to resolve the problem with BigData Extension.

© Copyright IBM Corp. 2017 1

Page 6: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

2 Troubleshooting Big Data Extension

Page 7: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 2. Log files in Big Data Extension

Log files are created during installation of Big Data Extension. Log files can beused to examine processing results and problems that are associated with differentservices.

Log files for different services:

Service Location

Ambari server /var/log/ambari-server

Ambari agent /var/log/ambari-agent

Ambari Metric Collector /var/log/ambari-metrics-collector

Ambari Metric Monitor /var/log/ambari-metrics-monitor

MapReduce /var/log/hadoop-mapreduce/mapred

Hadoop /var/log/hadoop/hdfs

Kafka /var/log/kafka

YARN components

v Node Manager

v Timeline server

v YARN

/var/log/hadoop-yarn

ZooKeeper /var/log/zookeeper

Manager /opt/IBM/tnpm_bde/tnpm_bde-manager/logs

Storage /opt/IBM/tnpm_bde/tnpm_bde-storage/logs

UI /opt/IBM/tnpm_bde/tnpm_bde-ui/logs

Tivoli® Netcool® Performance ManagerCollector

/opt/IBM/tnpm_bde/tnpm_bde-tnpm-collector/logs

Entity Analytics /opt/IBM/tnpm_bde/tnpm_bde-entity-analytics/logs

For information about Configuring logging, see Configuring TivoliNetcool PerformanceManager - BigData Extension

Log message formatTypically, each log message indicates the log level, time stamp, component, thread,error code, and event description.

An example log message:[INFO] [2017-03-06 01:02:40.455][akka.tcp://[email protected]:2553/user/storage/singleton/collagen/storeopt/localhost.TNPM-BDE.ENTITY_METRIC.RAW.MV001][tnpm-bde-storage.optimizer.dispatcher-5309]GYMSC3003I: Optimization complete on localhost.TNPM-BDE.ENTITY_METRIC.RAW.MV001

Log message elements:

© Copyright IBM Corp. 2017 3

Page 8: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Controlling the services from Ambari administration interfaceStop all IBM Open Platform with Apache Spark and Apache Hadoop services byeither using the Ambari administration interface or commands that start AmbariREST APIs.

Procedure

Stopping the servicesv Click Actions > Stop All from the Ambari web interface.

Then, wait for all of the services to stop.Orv Run the following command to stop various services in your Ambari agent hosts

on the Ambari server agent:

#!/bin/bash#Replace the following variables with values#specific to your cluster

ambari_server=<Ambari_Server_host>ambari_port=8080ambari_user=adminambari_password=adminambari_cluster=<MyCluster>

#MAKE SURE CODE APPEARS ON ONE LINE

services=$(curl --silent -u ${ambari_user}:${ambari_password}-X GET http://${ambari_server}:${ambari_port}/api/v1/clusters/${ambari_cluster}/services | grepservice_name | sed -e ’s,.*:.*"\(.*\)".*,\1,g’)

for serv in $servicesdocurl -u ${ambari_user}:${ambari_password}-H ’X-Requested-By: ambari’-X PUT -d

’{"RequestInfo": {"context" :"Stop service"},"Body": {"ServiceInfo": {"state":"INSTALLED"}}}’ http://${ambari_server}:${ambari_port}/api/v1/clusters/${ambari_cluster}/services/$serv

done

v Optional: Follow this sequence to stop the services on Ambari web interface:The order in which to stop the services:1. Big Data Extension2. MapReduce2

4 Troubleshooting Big Data Extension

Page 9: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

3. YARN4. HDFS5. KAFKA6. Ambari Metrics7. ZooKeeper

Starting the servicesv Click Actions > Start All from the Ambari web interface.

Then, wait for all of the services to start.Orv Run the following command to start various services in your Ambari agent hosts

on the Ambari server agent:

#!/bin/bash#Replace the following variables with values#specific to your cluster

ambari_server=<Ambari_Server_host>ambari_port=8080ambari_user=adminambari_password=adminambari_cluster=<MyCluster>

#MAKE SURE CODE APPEARS ON ONE LINE

services=$(curl --silent -u ${ambari_user}:${ambari_password}-X GET http://${ambari_server}:${ambari_port}/api/v1/clusters/${ambari_cluster}/services | grepservice_name | sed -e ’s,.*:.*"\(.*\)".*,\1,g’)

for serv in $servicesdocurl -u ${ambari_user}:${ambari_password}-H ’X-Requested-By: ambari’-X PUT -d

’{"RequestInfo": {"context" :"Start service"},"Body": {"ServiceInfo": {"state":"STARTED"}}}’ http://${ambari_server}:${ambari_port}/api/v1/clusters/${ambari_cluster}/services/$serv

done

v Optional: Follow this sequence to start the services on Ambari web interface:The order in which to start the services:1. ZooKeeper2. Ambari Metrics3. KAFKA4. HDFS5. YARN6. MapReduce27. Big Data Extension

Chapter 2. Log files in Big Data Extension 5

Page 10: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

6 Troubleshooting Big Data Extension

Page 11: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 3. Messages

A list of error and operational messages generated by the Tivoli NetcoolPerformance Manager Big Data Extension components.

The error codes are identified by a common product code (GYM), component orservice code (CT), message number, and severity. The severity level can be Error,Warning, or Information that are identified by the first alphabet.

For example for GYMCT0001I:v Product code = GYM for Tivoli Netcool Performance Managerv Service or component code = CTv Message number = 0001v Severity level = I

Error messages produced by Tivoli Netcool Performance ManagerCollector Service

List of error messages that are produced by Tivoli Netcool Performance ManagerCollector Service. Whenever possible, explanations are offered, as well as remedialactions.

All the errors that are thrown by Tivoli Netcool Performance Manager CollectorService are identified by CT

GYMCT0001I Number of files to fetch

Explanation: Shows the number of files that can befetched for this polling interval.

GYMCT0003E Failed to retrieve file

Explanation: Unable to ftp a file from the configuredlocation or there is a parsing error,

GYMCT0005I Starting to list files

Explanation: Starting to list files from the path byusing specified ftp settings.

GYMCT0006I Successfully list

Explanation: The number of files from the specifiedlocation in “GYMCT0005I” message are successfullylisted and the time taken to list the files.

© Copyright IBM Corp. 2017 7

Page 12: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

GYMCT0007E Failed to list file

Explanation: Unable to list file by using the ftpcommand.

GYMCT0009I Loaded entities

Explanation: The entity or resource information isloaded successfully from the Storage Service.

GYMCT0010E FailedToLoadEntities

Explanation: The entities or resources cannot beloaded from the Storage Service. The BOF file collectiondoes not start until the next retry is successful.

GYMCT0011I LoadedMetrics

Explanation: Metric metadata is loaded successfullyfrom the database.

GYMCT0012E FailedToLoadMetrics

Explanation: Failed to load metrics from the StorageService. The BOF files collection does not start until thenext retry is successful.

GYMCT0013I List of available FTEs

Explanation: The list of all available FTEs asconfigured in the topology editor in the Tivoli NetcoolPerformance Manager system. The list contains theactive FTEs that contain data in the Tivoli NetcoolPerformance Manager DataChannel done directory.

GYMCT0014I List of configured FTEs

GYMCT0015E Failed to poll Metric Data location

Explanation: Failed to poll FTE location from TivoliNetcool Performance Manager database.

GYMCT0016I Subscribing to topic

Explanation: The Tivoli Netcool Performance ManagerCollector is subscribing to messages from the listedKafka topic.

GYMCT0017E Service pre-check failed

Explanation: Test connection to Tivoli NetcoolPerformance Manager database failed.

GYMCT0007E • GYMCT0017E

8 Troubleshooting Big Data Extension

Page 13: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

GYMCT0018E Kafka Connect connector configurefailed

Explanation: Failed to configure the connector inKafka Connect. Check the configuration of TNPM-BDEservices in Ambari configuration screen.

GYMCT0019I File Path and Records

Explanation: Shows the BOF file path on the TivoliNetcool Performance Manager server and the numberof records that are read from the file.

GYMCT0020I Drops records

Explanation: BOF file records dropped due to nomatching metric and/or parent ID

GYMCT0021E Snapshot Failure inMetricDataManager metadata load

Explanation: Failed to save state of Kafka thatpertains to the metadata information.

GYMCT0022I Actor MetricDataManager is running

Explanation: : The MetricDataManager process, whichcollects ftp location, FTE directory information, andhost-related metadata from Tivoli Netcool PerformanceManager system is running.

GYMCT0023I Actor MetricNameLoader is running

Explanation: The MetricNameLoader process, whichcollects metric ID and name-related information fromTivoli Netcool Performance Manager system is running.

GYMCT0024I Actor EntityLoader is running

Explanation: The EntityLoader process, which collectsresource ID and name-related information from TivoliNetcool Performance Manager system is running.

GYMCT0025I Actor EntityPropertyLoader is running

Explanation: The EntityPropertyLoader process,which collects property ID and name-relatedinformation from Tivoli Netcool Performance Managersystem is running.

GYMCT0018E • GYMCT0025I

Chapter 3. Messages 9

Page 14: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

GYMCT0026I Actor MetricFileImporter is running

Explanation: The MetricFileImporter process, whichloads metric-related BOF data from BOF Files collectedfrom the remote Tivoli Netcool Performance Managersystem is running.

GYMCT0026I

10 Troubleshooting Big Data Extension

Page 15: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

GYMCT0027E Snapshot Failure inTnpmMetricFileImporter load

Explanation: Failed to save state to Kafka.

GYMCT0027E

Chapter 3. Messages 11

Page 16: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

12 Troubleshooting Big Data Extension

Page 17: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 4. Troubleshooting installation and uninstallation

Problems that might occur during installation or uninstallation and how to resolvethem.

About this task

Monitor the log files to examine the processing results and problems that areassociated with installation, configuration, and functioning of Big Data Extensionmicroservices.

Installation and uninstallation issuesProblems that might occur during installation or uninstallation and how to resolvethem.

About this task

Monitor the log files to examine the processing results and problems that areassociated with installation, configuration, and functioning of Big Data Extensionmicroservices.Related concepts:Chapter 2, “Log files in Big Data Extension,” on page 3Log files are created during installation of Big Data Extension. Log files can beused to examine processing results and problems that are associated with differentservices.

Kafka Connect is unable to recover from a closed connectionSymptomsKafka Connect is unable to recover after the connection is closed and restarted.You might see the following error is logged repeatedly in /usr/iop/current/kafka-broker/logs/connect.log file.[2016-12-22 14:59:27,054] ERROR Failed to run query for tableTimestampIncrementingTableQuerier{name=’null’, query=’select * from ...... }:java.sql.SQLRecoverableException: Closed Connection (io.confluent.connect.jdbc.JdbcSourceTask:232)

Resolving the problemTo resolve this issue, restart the Kafka Connect Service from Ambari.

Important: If you restart the Tivoli Netcool Performance Manager Wirelinecomponent database for any reason, monitor the logs and make sure to restart theKafka Connect as well.

Do not restart all Kafka Services when you remove KafkaSchema Registry from one of the nodes

SymptomsWhen you remove Kafka Schema Registry from one of the nodes in your cluster,Ambari might prompt you to restart all your Kafka Brokers. If you follow theprompt, some of your services might not work correctly.

Resolving the problem

© Copyright IBM Corp. 2017 13

Page 18: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

To resolve this issue, restart all the services from Ambari.

Some services do not start automatically on Ambari serverrestart

SymptomsYou might see the following message, when Ambari Server is restarted:[root@c7201 ~]# systemctl status ambari-server ambari-server.service Loaded:not-found (Reason: No such file or directory) Active: inactive (dead)

CausesThis issue occurs because the systemd does not work on ambari-server on RHEL7.2.

Resolving the problemTo resolve this issue, run the following command in a single line on the AmbariServer host:unlink /etc/rc.d/init.d/ambari-server && cp -a /usr/sbin/ambari-server/etc/rc.d/init.d/ambari-server && systemctl daemon-reload

Ignore the error in some Big Data Extension services log filesSymptomsWhen you see the following error in some Big Data Extension services log files:[ERROR] [2016-11-14 00:38:13.569] [akka://npi/user/StreamSupervisor-0/flow-14774-0-unknown-operation][npi-akka.actor.default-dispatcher-72] Error in stage[One2OneBidi]: Inner stream finished before inputs completed.Outputs might have been truncated.akka.http.impl.util.One2OneBidiFlow$OutputTruncationException$:Inner stream finished before inputs completed. Outputs might have been truncated.

Ignore the error as it does not have a functional impact.

Snappy java.lang.UnsatisfiedLinkError error in the StorageService log file

SymptomsYou might encounter the following error in the Storage Service log file after theinstallation of Big Data Extension:java.lang.UnsatisfiedLinkError: /tmp/snappy-1.0.4.1-libsnappyjava.so

Resolving the problemTo resolve this issue, enable execution permission for /tmp folder by using thefollowing command:sudo mount -o remount,exec /tmp

Related concepts

Chapter 2, “Log files in Big Data Extension,” on page 3Log files are created during installation of Big Data Extension. Log files can beused to examine processing results and problems that are associated withdifferent services.

Controlling maximum number of processes available for auser

If you encounter Resource temporarily unavailable error message, increase thesoft and hard limits for the maximum number of processes for the user.

14 Troubleshooting Big Data Extension

Page 19: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

A newly added ZooKeeper instance is not updated in theyarn.resourcemanager.zk-address configuration parameter

SymptomsWhen you add a ZooKeeper instance to a node in your cluster, the hostname withport number might not be available in the comma-separated list in theyarn.resourcemanager.zk-address parameter in Ambari.

The yarn.resourcemanager.zk-address parameter is available from Services >YARN > Advanced > Fault Tolerance section in the Ambari server web interface.

Resolving the problemManually add the host name to the list to the yarn.resourcemanager.zk-addressparameter.

Tivoli Netcool Performance Manager Collector Service doesnot shut down gracefully

SymptomsWhen the Tivoli Netcool Performance Manager Collector Service is stopped orrestarted while RAW data is being written to the Storage Service, it does not startor shut down gracefully.

Typically, it must prevent new processes from starting and gracefully wait forongoing storage loading process to complete before the shut down. There is noimpact to functionality as the collector loads the file again on startup.

Chapter 4. Troubleshooting installation and uninstallation 15

Page 20: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

16 Troubleshooting Big Data Extension

Page 21: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 5. Troubleshooting Ambari server

Use this information to troubleshoot problems when you use Ambari server.

Problem in decommissioning DataNodesWhen you decommission some nodes from the cluster, HDFS replicates the blocksthat belong to decommissioning DataNodes to other live DataNodes to reach thereplication factor that you specified in the dfs.replication setting. You have thesame setting in Ambari HDFS configuration as Block replication.

Before you begin

If you encounter the following error:

Identify the files that are under replicated by using these steps:1. Log in to the host where the HDFS NameNode is installed as hdfs user.

Or, set the HADOOP_USER_NAME environment variable as follows:export HADOOP_USER_NAME=hdfs

About this task

The dfs.replication is an HDFS global setting in hdfs-site.xml.

If you do not have enough live DataNodes to reach the replication factor,decommission process might hang until more DataNodes become available. Forexample, if you have 3 DataNodes in your cluster with dfs.replication is set to 3and you are trying to decommission 1 DataNode out of 3, decommission processhangs until you add another DataNode to the cluster.

Procedure1. Log in to the host where the HDFS NameNode is installed as hdfs user.

Or, set the HADOOP_USER_NAME environment variable as follows:export HADOOP_USER_NAME=hdfs

2. Run the hadoop fs –setrep command as follows:hadoop fs -setrep [-R] [-w] <numReplicas> <path>

Where:v -w flag requests that the command wait for the replication to complete. This

step can potentially take a long time.v -R flag is accepted for backwards compatibility. It has no effect.v <numReplicas>

v <path>

For example, hadoop fs -setrep -w 2 /This command changes the replication factor of a file. If path is a directory,then the command recursively changes the replication factor of all files underthe directory tree rooted at path.

© Copyright IBM Corp. 2017 17

Page 22: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Important: Bye default, the HDFS number of replication (numReplicas) is 3. Ifyou have less than 3 live HDFS DataNodes, set numReplicas to total remainingnumber of live HDFS DataNodes in your cluster.

Related information:

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep

Ambari HDFS Metric showing huge value for Under Replicated Blocksin a single node environment

SymptomsAmbari Big Data Extension HDFS metrics value is highlighted as red and showinga huge value for under replicated blocks in the Ambari server web interface in asingle node environment.

CausesThe HDFS status summary in Ambari server web interface shows the missing andunder replicated blocks.

Some files in your HDFS file system are corrupted either by losing its last blockreplica or just being under replicated.

When a new datanode is added, HDFS replicates these blocks. Even if thereplication factor is set to 1, the HDFS stills report these blocks as under-replicated,as it is not fault tolerant.

This behavior is expected.

Resolving the problem

To work around this behavior, you can opt to follow the suggestions that areprovided:1. You can clear the threshold values from the Ambari server UI from the

following steps:a. Select Edit from the HDFS metrics Under Replicated Block widget.b. Select Edit Shared from the Warning screen.c. Clear the thresholds values. For example, empty the Thresholds fields,

WARNING and CRITICAL.d. Click Next > Save

2. The following are some suggestions to avoid this problem depending on yourdata blocks.a. To get the full details of the files, which are causing the problem, run the

following command by using root user.$ hdfs fsck / -files -blocks -locations

The output identifies the replication factor set on your corrupted files.b. The following list some methods to fix the missing and under-replicated

blocks.v This condition might be temporal; if you have a data under-replicated it

should automatically replicate the blocks to other data nodes to match thereplication factor.

v If it is not replicating on its own, run a balancer manually.

18 Troubleshooting Big Data Extension

Page 23: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Important: Do not run the HDFS balancer if you are using HBase.v If it is not replicating on its own, you can manually set replication on a

specific file that is under replicated to a value higher than it currently setto. This setting makes the cluster to create more replicas.– The recommended default replication factor is to set at 3. If you then

add a datanode, the block is replicated.v If it is just a temporary file, which is created when running the job and

your speculative execution tasks are high, set the speculative executiontasks to match the replication factor.

c.

CAUTION:Run the following command only when you are sure about the corruptedfiles.If you are sure that these files are not needed and would like to eliminatethe error, you can run the following command to automatically delete thecorrupted files:hdfs fsck / -delete

Ambari Metrics configurations warning keeps appearingSymptomsThe Ambari Metrics service configurations warning at times keeps appearingdespite having the correct recommended value.

Resolving the problem

Ensure that your configurations value is according to the requirements or therecommended value. From the Ambari Metrics Warning UI, click Proceed Anywayto proceed.

It is a known limitation.Related information

Recommended Ambari Metric configurations warning keeps appearing

Timezone changes are not reflected for monitoring Big Data Extensionmetrics on Ambari by using Firefox ESR

SymptomsWhen you use Firefox ESR to monitor Big Data Extension metrics on Ambari, thetime zone changes are not reflected correctly.

Resolving the problemIt is a known limitation.

Related information

Unable to change timezone when using Firefox ESR 31.8.0

You might notice an OversizedPayloadException error in StorageService log with Metric API calls

Symptoms

Chapter 5. Troubleshooting Ambari server 19

Page 24: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

When the rows fetched are large in Metric API calls, you might see the followingerror message:OversizedPayloadException: Discarding oversized payload sent to Actor

Resolving the problemTo resolve this issue, follow these steps:1. Open a browser and access the Ambari server dashboard. admin.

Use the following default URL:http://<myserver.ibm.com>:8080The default user name is admin, and the default password is admin.

2. Click Services > TNPM BDE > Configs > Advanced.3. Expand Advanced tnpm_bde-env pane and add the following lines in

tnpm_bde-env template text area:ui.entity.metrics.batchsize = <value>ui.entity.timeseries.batchsize = <value>

The default value is 500 rows. If you want to reduce the payload size, enter alower value.

4. Restart the system.

For more information, see Metric APIs in TivoliNetcool Performance Manager -BigData Extension References.

20 Troubleshooting Big Data Extension

Page 25: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Chapter 6. Troubleshooting Big Data Extension REST APIs

Use this information to troubleshoot problems when you use Big Data Extension.

Use the entity names in REST API queries instead of parent namesSymptomsWhen you change the parent (element) name in Tivoli Netcool PerformanceManager, it is not mapped to its entity (subelement) name and entity nameremains mapped to the old parent name.

You might not be able to use the new parent name in REST API queries.

Resolving the problemUse the entity names in REST API queries instead of parent names.

© Copyright IBM Corp. 2017 21

Page 26: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

22 Troubleshooting Big Data Extension

Page 27: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Notices

This information was developed for products and services offered in the US. Thismaterial might be available from IBM in other languages. However, you may berequired to own a copy of the product or product version in that language in orderto access it.

IBM may not offer the products, services, or features discussed in this document inother countries. Consult your local IBM representative for information on theproducts and services currently available in your area. Any reference to an IBMproduct, program, or service is not intended to state or imply that only that IBMproduct, program, or service may be used. Any functionally equivalent product,program, or service that does not infringe any IBM intellectual property right maybe used instead. However, it is the user's responsibility to evaluate and verify theoperation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matterdescribed in this document. The furnishing of this document does not grant youany license to these patents. You can send license inquiries, in writing, to:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

For license inquiries regarding double-byte character set (DBCS) information,contact the IBM Intellectual Property Department in your country or sendinquiries, in writing, to:

Intellectual Property LicensingLegal and Intellectual Property LawIBM Japan Ltd.19-21, Nihonbashi-Hakozakicho, Chuo-kuTokyo 103-8510, Japan

INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THISPUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHEREXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESSFOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer ofexpress or implied warranties in certain transactions, therefore, this statement maynot apply to you.

This information could include technical inaccuracies or typographical errors.Changes are periodically made to the information herein; these changes will beincorporated in new editions of the publication. IBM may make improvementsand/or changes in the product(s) and/or the program(s) described in thispublication at any time without notice.

Any references in this information to non-IBM websites are provided forconvenience only and do not in any manner serve as an endorsement of those

© Copyright IBM Corp. 2017 23

Page 28: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

websites. The materials at those websites are not part of the materials for this IBMproduct and use of those websites is at your own risk.

IBM may use or distribute any of the information you provide in any way itbelieves appropriate without incurring any obligation to you.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact:

IBM Director of LicensingIBM CorporationNorth Castle Drive, MD-NC119Armonk, NY 10504-1785US

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The licensed program described in this document and all licensed materialavailable for it are provided by IBM under terms of the IBM Customer Agreement,IBM International Program License Agreement or any equivalent agreementbetween us.

The performance data discussed herein is presented as derived under specificoperating conditions. Actual results may vary.

The client examples cited are presented for illustrative purposes only. Actualperformance results may vary depending on specific configurations and operatingconditions.

Information concerning non-IBM products was obtained from the suppliers ofthose products, their published announcements or other publicly available sources.IBM has not tested those products and cannot confirm the accuracy ofperformance, compatibility or any other claims related to non-IBM products.Questions on the capabilities of non-IBM products should be addressed to thesuppliers of those products.

Statements regarding IBM's future direction or intent are subject to change orwithdrawal without notice, and represent goals and objectives only.

All IBM prices shown are IBM's suggested retail prices, are current and are subjectto change without notice. Dealer prices may vary.

This information is for planning purposes only. The information herein is subject tochange before the products described become available.

This information contains examples of data and reports used in daily businessoperations. To illustrate them as completely as possible, the examples include thenames of individuals, companies, brands, and products. All of these names arefictitious and any similarity to actual people or business enterprises is entirelycoincidental.

COPYRIGHT LICENSE:

24 Troubleshooting Big Data Extension

Page 29: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

This information contains sample application programs in source language, whichillustrate programming techniques on various operating platforms. You may copy,modify, and distribute these sample programs in any form without payment toIBM, for the purposes of developing, using, marketing or distributing applicationprograms conforming to the application programming interface for the operatingplatform for which the sample programs are written. These examples have notbeen thoroughly tested under all conditions. IBM, therefore, cannot guarantee orimply reliability, serviceability, or function of these programs. The sampleprograms are provided "AS IS", without warranty of any kind. IBM shall not beliable for any damages arising out of your use of the sample programs.

Each copy or any portion of these sample programs or any derivative work mustinclude a copyright notice as follows:

© (your company name) (year).Portions of this code are derived from IBM Corp. Sample Programs.© Copyright IBM Corp. _enter the year or years_.

TrademarksIBM, the IBM logo, and ibm.com are trademarks or registered trademarks ofInternational Business Machines Corp., registered in many jurisdictions worldwide.Other product and service names might be trademarks of IBM or other companies.A current list of IBM trademarks is available on the web at "Copyright andtrademark information" at www.ibm.com/legal/copytrade.shtml.

Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registeredtrademarks or trademarks of Adobe Systems Incorporated in the United States,other countries, or both.

IT Infrastructure Library is a registered trademark of the Central Computer andTelecommunications Agency which is now part of the Office of GovernmentCommerce.

Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo,Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks orregistered trademarks of Intel Corporation or its subsidiaries in the United Statesand other countries.

Linux is a registered trademark of Linus Torvalds in the United States, othercountries, or both

Microsoft and Windows are trademarks of Microsoft Corporation in the UnitedStates, other countries, or both.

ITIL is a registered trademark, and a registered community trademark of TheMinister for the Cabinet Office, and is registered in the U.S. Patent and TrademarkOffice.

UNIX is a registered trademark of The Open Group in the United States and othercountries.

Notices 25

Page 30: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

Java and all Java-based trademarks and logosare trademarks or registered trademarks ofOracle and/or its affiliates.

Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in theUnited States, other countries, or both and is used under license therefrom.

Linear Tape-Open, LTO, the LTO Logo, Ultrium, and the Ultrium logo aretrademarks of HP, IBM Corp. and Quantum in the U.S. and other countries.

Terms and conditions for product documentationPermissions for the use of these publications are granted subject to the followingterms and conditions.

Applicability

These terms and conditions are in addition to any terms of use for the IBMwebsite.

Personal use

You may reproduce these publications for your personal, noncommercial useprovided that all proprietary notices are preserved. You may not distribute, displayor make derivative work of these publications, or any portion thereof, without theexpress consent of IBM.

Commercial use

You may reproduce, distribute and display these publications solely within yourenterprise provided that all proprietary notices are preserved. You may not makederivative works of these publications, or reproduce, distribute or display thesepublications or any portion thereof outside your enterprise, without the expressconsent of IBM.

Rights

Except as expressly granted in this permission, no other permissions, licenses orrights are granted, either express or implied, to the publications or anyinformation, data, software or other intellectual property contained therein.

IBM reserves the right to withdraw the permissions granted herein whenever, in itsdiscretion, the use of the publications is detrimental to its interest or, asdetermined by IBM, the above instructions are not being properly followed.

You may not download, export or re-export this information except in fullcompliance with all applicable laws and regulations, including all United Statesexport laws and regulations.

26 Troubleshooting Big Data Extension

Page 31: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESEPUBLICATIONS. THE PUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUTWARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDINGBUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY,NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE.

Notices 27

Page 32: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

28 Troubleshooting Big Data Extension

Page 33: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file
Page 34: with IBM Corp.€¦ · Kafka Connect. Check the configuration of TNPM-BDE services in Ambari configuration scr een. GYMCT0019I File Path and Records Explanation: Shows the BOF file

IBM®

Printed in USA