Pivotal Knowledge Base

Follow

How to Collect YARN job logs

Environment

Product Version
 Pivotal HD / HDP  3.x / 2.3,2.4
 YARN  

Purpose

How to collect logs associated with MapReduce, Tez, Hive jobs or any other jobs running in a YARN cluster. These logs will usually be requested by Pivotal Support for troubleshooting purposes. 

Procedure

Identify the job or application ID: This can be achieved by either,

CLI, by Running:

yarn application -list -appStates FINISHED,FAILED,KILLED


Or,
Reviewing the YARN ResourceManager UI available via Ambari GUI / YARN / Quick Links / ResourceManager UI: 

Run the following command to fetch and compress the logs:

yarn logs -applicationId <app_id> -appOwner <user> |& gzip -c > <app_id>.log.gz

The option -appOwner can be omitted if the current user is the same user who ran the job.

 

 

Comments

Powered by Zendesk