Redirecting Talend Console logs

Redirecting Talend Console logs to file

Talend Open studio for data integration does not have any component to redirect console trace to a file. This blog post provides the solution for Redirecting Talend Console logs to file and is very easy to follow and extend. In previous article logging-using-talend we have seen how to customize Talend logs. This article shows how to redirect Talend console logs to file. In talend exchange, there is a custom component to perform similar job.

Solution



Redirecting Talend console logs is required by almost all Talend projects. This is very easy to implement in talend. Use tJava component at the start of any job and enter following lines into it.

Redirecting Talend Console logs

Redirecting Talend Console logs

java.io.File folder = new java.io.File(“_logs/”);
if (!folder.exists()) {
if (folder.mkdir()) {
System.out.println(“Directory is created!”);

}
else {
System.out.println(“Failed to create directory!”);
}
}
java.io.File logFile = new java.io.File(“_logs/”+projectName+”.log”);
java.io.PrintStream ps = new java.io.PrintStream(new java.io.FileOutputStream(logFile));
System.setOut(ps);
System.setErr(ps);

After executing the job, above code creates a flat file with “projectName.log” and starts writing data which appears on the console. You can create a new file at each execution by post/pre fixing with the time stamp in the file name or use following code. Attached image shows

java.io.File file = new java.io.File(projectName+”_”+TalendDate.getDate(“CCYYMMDDhhmmss”)+”.log”);

projectName is the variable name provided by the Talend and you can get the postfix date stamp by another variable TalendDate.getDate(“CCYY-MM-DD hh:mm:ss”). Remove all the special characters in the time stamp. ProjectName can also be replaced by another global variable jobName.

After successfully executing this job, you can see ProjectName_TimeStamp.log  file is created in the root folder of Talend. This file contains complete console stack trace.

Drawback



 Drawback of implementing this strategy is that, it will not print or display anything on the console while using IDE or independent execution. This becomes difficult when developer is developing or debugging the job. Because, it is an extra effort for a developer to open a file every time to find out whether the job is executed successfully and generated error or not.

Logging using Talend

Logging using Talend

Introduction: In this article, we will discuss about different methods of logging in Talend Open Studio. Talend is one of the widely used open source data integration tool in the market. Talend mainly uses three types of logging

  1. Statistics – Job execution statistics at component and job level
  2. Error – Job level errors, warning and exceptions
  3. Meter logging – Data flow details inside job

Best approach for logging is at project level. To enable project level logging, In Talend Open Studio, go to File, Project properties and enable or disable check boxes to start default logging at project level. See the screen shot below.

talend logging 1

If you enable the logging at project level, then every new job created will inherit these settings. There are more settings and options to do if you enable project level logging. See below screenshot.

talend logging 2

You can decide it to log the information to Console/File/Database. In case if you select any of File/Database or both options, then need to set few more default parameters like

talend logging 3

For file names, you can pre-or post fixes the file name with Talend Date Time stamp function. Or else it will write into the same file for every execution and flush out earlier data. In case of databases, you can have existing created database. This scenario does not work when you don’t have any database on the target server this scenario fails. For general JDBC, you need to provide above parameters, else if you select any database such as MySQL, then provide username, password and other required parameter values.

If you enable project level logging, then there is no need to separately use all these components say, tLogCatcher to Log the errors, tFlowMeter to catch the data flow and tStatCatcher to catch the Errors, flow and statistics log. Talend throws the errors or exceptions whenever it occurs and displays its complete trace on the console. tLogCatcher if used with the help of tDie or tWarn, would catch those messages and can be redirected to the required database / file based on the requirement. In order to do this we need to take care of all above components, implement and test the job.

Advantage with this approach is that you get the brief error information in logs table which is automatically created by Talend. In addition to this information Talend also prints its error log trace on console. And the negative side of this is that the console trace is not stored in the log table.

Problem: In both above approaches, Talend does not store or redirect its console trace to database or file.