Passing Context Parameters from Command Line to Talend Job

Passing Context Parameters from Command Line to Talend Job

Some times we have to pass context parameter value from commandline while executing talend job (.bat file or .sh file) which gets generated when we export job. We can make use of “–context_param” command in commandline  to pass the context parameter values while executing talend job from command line.

Here is a simple job for passing the context parameters from commandline.

Place tjava component on working area, and under the component tab write a code to print the context parameter values as shown in below screen shot:

bg1

 

Declare the context Parameters and assign a defualt values under the Contexts tab for this job.

Here I have declared two context parameters as param1 and param2 and assigned default values as shown in the below screen shot:

bg2

Now if we execute this job from studio it will display these default values on the console as shown in the below screen shot.

bg

If we execute this job from command line without providing the context parameters it will display the default values as follows:

bg3

Now execute the job by providing context variables in command line prompt. It will display these values as the output instead of default values as shown in the below output.

Here I passed ‘a1’ and ‘a2’ as the context parameter values for param1 and param2 context variables respectively.

bg4

Thanks and Regards,

Lalitha

Talend Administration Center Configuration

Talend Administration Center Configuration

Talend Administration Center is a Web-based administration application that allows Talend Studio project managers to administrate users and projects and manage access to the remote repository.

If you want to refer to my previous post on Talend Enterprise Studio for DI installation using Installer then click here.

Configuring the web application access:

The recommended way to configure the connection to the database and to the shared repository (SVN) is through the Web interface of Talend Administration Center.

  1. Start the application server (Tomcat or JBoss), then open a Web browser and type in the following URL: http://localhost:8080/<ApplicationPath>.
  2. Replace <localhost> with the IP address or the hostname of the Web server if the Web browser IP is different from the machine you are on and<ApplicationPath> with the Talend Administration Center Web application path.

For example, http://localhost:8080/org.talend.administrator.

  1. When connecting for the first time, a dialog box is displayed, asking you the password to access the Database Configuration page:

 

Type in the default admin password.

  1. H2 database connection parameters display and some automatic checks are performed on driver, url, connection, version information.

Note:

The administration database (storing users, rights, etc.) being an H2 embedded database, its access information is automatically set. However, if you do not want to use the embedded H2 database, you can set up a database server (MySQL, MSSQL or Oracle) in which case, set the required parameters.

  1. Click Set new license.
  2. Browse your system to the License file you received from Talend and click Upload. A final License check is performed.
  3. Click Go to Login.
  4. On the Login page, type in the default connection login for your first access:
  5. Default login: [email protected],
  6. Default password: admin.
  7. Those credentials correspond to the default user of the Web application. You can create a new one via the Users menu in Talend Administration Center, and then delete the [email protected] user after you are connected using the newly created user login.

Note:

If your Web access is restricted, you may need to click Validate your license manually to perform the validation of your license key. Follow the instructions on screen.

  1. Once the license is validated, the navigation bar of Talend Administration Center opens with all the pages accessible for the default administrator user account. For more information on which pages of Talend Administration Center an administrator user can access, see the Talend Administration Center User Guide.

You can now configure the link to the shared repository (SVN):

  1. Click Configuration to access the setting page of Talend Administration Center.
  2. Change the following parameters for the SVN module using the parameters you have set during the installation process of the SVN server.

– Server Location URL: your <repotis> SVN URL.

– Username: your <repotis> user.

– Password: your <repotis> SVN password.

The link to the SVN is now established, you can thus create a new project in order for the Talend clients to have at least one project in their workspace. To do so, you have to:

1. Create one or more users from the Users page;

2. Create a new, remote, collaborative project from the Projects page;

3. Associate the user(s) with the project from the Project authorizations page.

 

Accessing the Administration Center:

To access the Administration Center using the default Administrator account provided, do the following:

  1. In the address bar of your Internet browser, type in or paste the URL address corresponding to the location of Talend Administration Center.

For example, http://localhost:8080/org.talend.administrator

  1. Press Enter to display Talend Administration Center as shown in the below pic.

tac_login_page

Use below credentials to login into TAC.

Login: [email protected]

Password: admin

 

Database Configuration page:

  1. When you access Talend Administration Center for the first time, the Database Configuration page appears. Type in the administrator password (by default, it is admin) and click OK.

database_confi

2.Upon validation of your password, Talend Administration Center runs a series of checks. If no license or an invalid license is found, you will be prompted to specify a license. Click the Set new license button.

db_config_validation

3.  Click Browse to browse to your license file and click Upload.

license_browse

4.  Upon validation of your license, Talend Administration Center runs a series of checks again, and displays the following options:

    • Set new license: allows you to set a new license by repeating the previous step.
    • Validate your license manually: allow you to validate the loaded license.

 Go to login page: allows to you to open the login page to log in to the Talend Administration Center.

  1. Click Go to login page and type in the login and password provided by Talend in the [Login] dialog box.

talend_admin_login

  1. Click the Login button.

Talend Administration Center opens up on a welcome page.

 

Accessing the Configuration page:

The Configuration page offers accurate details on application parameters and their related values including database connection. All these parameters are grouped by module. This helps understand and troubleshoot module-related problems.

To access the Configuration page, click Configuration in the Menu tree view. The figure below illustrates an example.

tac_config_window

The second column in the configuration tabular list indicates whether a parameter is correctly set up  or empty or wrong .

The same icons appear as well in front of the group titles. They indicate whether the corresponding module is up and running and well configured  or if it contains any wrong parameter or is simply down .

 

Configuring TAC parameters according to our system:

Setting up Audit parameters:

audit_params

Setting up command line parameters:

              cmdline_param

Setting up monitoring (AMC) parameters:

amc_config

Setting up Job Conductor parameters:

job_conducter_config

Setting up Log4j Parameters:

log4j_config

Setting up Software Update parameters:

Before configuring this section, we need to start the artifact repository service depends upon the system environment like whether it is 32 bit or 64 bit.

Launching Archiva artifact repository:

For 32-bit Windows users:

  1. From a commandline window, browse to the bin directory of Artifact-Repository-Archiva-VA.B.C.

 

Ex: <Talend_installation_dir>/tac/Artifact-Repository-Archiva-V1.4.10.2/bin

 

  1. On 32-bit Windows, run the archiva.bat console command to run it interactively

For 64-bit Windows users:

Due to licensing constraints, The Archiva artifact repository distribution does not include Tanuki wrapper for 64 bit Java on Windows.

  1. Check that you are running 64-bit Java.
  2. Download 32-bit Java to a separate directory.
  3. Configure a batch file to set the JAVA_HOME variable to the 32-bit JDK,

Then run the archiva.bat console command from within the same batch file so that the JDK is pointing to the correct location.

After creating the new batch file add below parameters to it like below.

 

Archiva_32bit.bat file:

@echo off

set JAVA_HOME=E:\Program_Files\Java_32bit\jdk_1.7\bin\java

call archiva.bat console

This will provide a 32-Java environment just for the Archiva artifact repository, while the rest of your applications run safely in 64-bit mode.

 

Artifact Repository configuration

To configure Talend Software Update, open your Web browser and type in the URL to your instance of the Archiva artifact repository, http://localhost:8082/archiva for example, then proceed as follows:

  1. Log in with the following credentials:

Username: admin

Password: 1q2w3e4r

Once we have launched and configured the Software Update repository, go to the Configuration page of Talend Administration Center and fill in the following information in the Software Update group:

  • Repository URL: Type in the location URL to your software update repository, http://localhost:8082/archiva for example.
  • Admin username: Type in the name of the repository user with Manager role. By default, it is SoftwareUpdateAdmin.
  • Admin password: Type in the password of the repository user with Manager role. By default, it is SoftwareUpdateAdmin.
  • Reader username: Type in the name of the repository user with Observer role. By default, it is SoftwareUpdateReader.
  • Reader password: Type in the password of the repository user with Observer role. By default, it is SoftwareUpdateReader.

                    software_update_config

Setting up SVN parameters:

                    svn_config

Launching the Talend products for the first time:

We installed all the talend products via Talend Installer, so we are nothing to do as it has been unzipped in the Installer installation directory (called Talend by default) with its license file.

Below are the sub-folders available in the Talend installation directory and we can launch by starting their services.

— cmdline

— jobserver

— logserv

— studio

— tac

Launch studio with Remote connection:

Go to talend installation directory path <talend installation path>/studio and launch the appropriate executable file according to your system.

studio_svn

Go to connection section and click on the configuration button (…) to create/edit a connection from the drop down menu.

remote_connection_studio

Click on Ok and select the appropriate project from the svn repository to launch the studio.

Managing users:

Adding a user:

On the Users page, click Add to create a new account. You can create a Viewer, Designer, Operation Manager, Administrator or a user with several of these roles.

add_user

Managing projects:

Adding a project:

To add a project which is stored in SVN

  1. In the Menu tree-view, select Projects to open the project list page.
  2. From the toolbar, click Add. Mandatory fields in the Project panel to the right are followed by .

                         add_project

Managing project authorizations:

The Project Authorizations page in Talend Administration Center allows you to manage user authorizations. From this page, you can:

  • give a user the right to access one or more projects,
  • Specify if the access right is read-only or read-write.

 

Authorizing users:

To give a user the right to access one or more projects, do the following.

Note that the assigned users have to re-launch the Studio to take this authorization into account.

 

  1. In the Menu tree-view, select Project authorizations to display the authorization list.
  2. In the Project panel, select the project to which you want to assign a user.
  3. In the Right column of the User panel, give read or read and write permissions to a user by clicking the corresponding icons.

The Read and Read write icons show on the projects on which the user can work by default. If a user cannot work on a project, a prohibition icon indicates that you cannot give rights to this user: it depends on the Type that is defined during the user creation in the Users page.

The number of users that have read access as well as the number of users that have read and write access to a project are updated in the corresponding columns of the Project panel.

project_authorize_page

 

 

Executing Jobs from Jobs Conductor:

In the Job Conductor page of Talend Administration Center, an execution task gathers the script generation, deployment and execution phases of data integration Jobs. You can launch this task, from this single web-based application, using a simple or a CRON trigger.

Working with Job execution tasks:

A Job execution task represents a set of actions that you can configure in Talend Administration Center in order to execute the Jobs designed in Talend Studio directly from the Administration Center. You can also define different types of triggers to launch an execution task.

To access the execution task list, do the following:

In the Menu tree view, click Job Conductor to display the list of scheduled tasks that will deploy and execute Jobs on a remote server.

                         job_conductor_execution

Adding an execution task

The Talend Administration Center allows you to add execution tasks on Jobs that are either based on the SVN repository, or pre-generated by the Studio as zip files. According to your needs, read one of the procedures below.

Note that once the task is added, you still can switch from one mode to the other.

Adding an execution task on a Job based on SVN repository

From the toolbar on the Job Conductor page, click Add task to clear the Execution task configuration panel.

execution_task

Click on save button will create a execution task.

Once the execution task is created then we will able to deploy and execute the task from the same section by clicking on generate, deploy and run buttons accordingly.

 

Thanks,

Sayagoud Ravelly

How to Export and Import items in Talend

Exporting items from Talend Studio
Whenever we need to take backup of our project, or if there any job migration or product upgrade then we need to export all the jobs, contexts, routines, metadata of our project from Talend Studio. 

So here are the steps to export items from talend:
 
1) Right click on Job Designs in Repository view and select Export items option.
If you want to export only particular job then you can right click on that job name and then select export items option.
e1
2) Browse a path for root directory or archive file. If you want to export all the dependency items like routines, contexts, metadata of your jobs, you can select Export Dependencies option as shown in below screen shot.
e2
3) Click on Finish button to complete the export.
 
Importing items into Talend Studio
 
You can also import the items into Talend which you have exported as back up as earlier.
 
Here are the steps to follow:
 
1) Right click on Job Designs in the Repository view. Select the Import items option.
i1
2) Browse the path where you have stored the back up of your project. A list of items will be displayed as shown in the below screen shot. now select the items you want to import. Choose the overwrite existing items option if the items already exist in your repository and you want to overwrite them.
i2
3) Click on Finish button to complete import process.
 
4) If the newly imported items do not appear then please click on Refresh button which is at the top of Repository view

i3

Thanks and Regards,

S D Lalitha Velamuri

Session Management in Spring Security

Before we go to spring security session management let first understand what is session and why session is required.

Whenever we connect to any web server through URL, we use browser as client which uses a HTTP protocol for communication between browser and server. HTTP is a “stateless” protocol which means each time client receives a web page, the client opens a separate connection to web server and server automatically does not keep any record of previous client request that means for web server every request is a new request to process and they can’t identify if it’s coming from client that has been sending request previously. But sometime in web applications, we should know who client is and process the request accordingly.

Session is a conversional state between client and server and it consists of multiple request and response between client and server. Since HTTP protocol and web server both are “stateless” the only way to maintain a session is when some unique information about session like session id is passed between server and client in every request and response. In other words session is unique Id created by container to identify the user from whom the request in coming in

spring security.

Now we jump to how is session management takes place in spring security.

When is session created?
We can control exactly when our sessions get created and how spring security interact with it.
1. always – a session will always be created if one already does not exist
2. ifRequired – a session is created if required (Default).
3. never – a framework will never create a session itself. But it will use one if it already exists.
4. Stateless – no session is created or used by spring security.

<http create-session=”ifRequired”>….</http>

It is very important to understand that this configuration only controls what spring security does – not the entire application. Spring security may not create the session in we instruct it not to, but our application may!

By default spring security will create session when it needs one. This is ”ifRequired”
For a more “stateless” application, the “never” option will ensure that spring security itself will not create any session; however if application creates one, the spring security makes use of it.

Finally, the strictest session creation option – “stateless” – is guarantee that the application will not create any session at all.

Concurrent Session Control
When a user that is already authenticated tries to authenticate again, the application can deal with that event in one of a few ways. It can either invalidate the active session of the user and authenticate the user again with a new session, or allow both sessions to exist concurrently.

First step in enabling the concurrent session-control support is to add the following listener in the web.xml:

<listener>

<listener-class>

org.springframework.security.web.session.HttpSessionEventPublisher

</listener-class>

</listener>

 

This is essential to make sure that the Spring Security session registry is notified when the session is destroyed.

To enable the scenario which allows multiple concurrent sessions for the same user the element should be used in the XML configuration:

<http …>

<session-management>

<concurrency-control max-sessions=”3″ />

</session-management>

</http>
Session Timeout

After the session has timed out, if the user sends a request with an expired session id, they will be redirected to an URL configurable via the namespace:
<session-management>

<concurrency-control expired-url=”/sessionExpired.html” … />

</session-management>

Similarly, if the user sends a request with a session id which is not expired, but entirely invalid, they will also be redirected to a configurable URL:
<session-management invalid-session-url=”/invalidSession.html”>

</session-management>

Talend Enterprise Studio for DI installation using Installer

Talend Enterprise Studio for DI installation using Installer

This post provides the information about the following topics on Talend Enterprise Studio for Data Integration r.

  1. Installation Requirements
  2. Introduction to the Talend products
  3. Talend Enterprise Studio Installation (Using Installer)

1. Installation Requirements:

To make the most out of the Talend products, please consider the following hardware and software requirements.

Hardware Requirements:

Memory usage heavily depends on the size and nature of your Talend projects. However, in summary, if your Jobs include many transformation components, you should consider upgrading the total amount of memory allocated to your servers, based on the following recommendations.

Table 1.1. Memory usage

Product Client/Server Recommended alloc. memory
Talend Administration Center + Talend Activity Monitoring Console Web application Server 4GB minimum, 8GB recommended
CommandLine Server 2GB minimum, 5 GB recommended
JobServer Server 1GB minimum, more recommended1
Studio Client 3GB minimum, 4 GB recommended
Talend Runtime Server 2GB minimum, 4 GB recommended1

1. Memory requirements depend on the executed processes.

The same requirements also apply for disk usage. It also depends on your projects but can be summarized as:

Table 1.2. Disk usage

Product Client/Server Required disk space for installation Required disk space for use
Talend Administration Center with Nexus and Archiva artifact repositories + Talend Activity Monitoring Console Web application Server 800MB + more than 50MB for Talend Activity Monitoring Console Web application 800MB minimum + project size = 20GB+ recommended
CommandLine Server 3GB 2GB minimum + project size = 20 GB+ recommended
JobServer Server 20MB 2GB minimum + Jobs deployed = 20+ GB recommended
Studio Client 3GB 3+ GB
Talend Runtime Server 400MB 400+ MB

 

Software requirements:

The following software and modules are properly set up on the station where we are trying to install the Talend modules:

  • Operating System Linux 32/64 bits, Windows 32/64 bits or Mac OS X
  • JDK 1.6.0 or higher must be installed

Note: Talend Installer automatically checks for JDK version 1.6 or higher version. If no instance of JDK is found, the installer will shut down.

  • .NET version 2 to version 3.5 is required by Talend Installer to install services on Windows;
  • (Optional) a mail server (to send notifications by email).

 

Environment variable configuration: on Windows

Prior to installing your Talend solutions, you have to set the JAVA_HOME Environment variable:

  1. Define your JAVA_HOME environment variable so that it points to the JDK directory.
  2. Example: if the JDK path is C:\Java\JDKx.x.x\bin, you must set the JAVA_HOME environment variable to point to:C:\Java\JDKx.x.x.

 

  1. If you use Talend Installer, you also have to set the Path system variable:
  2. Add the previously defined JAVA_HOME variable to the Path environment variable.
  3. Example: <PathVariable>;%JAVA_HOME%\bin.

 

2. Introduction to the Talend products:

The present section lists all the elements required for using the Talend products. To ease their management, we recommend you to centralize all the server modules on one single system.

  1. An application server (Apache Tomcat server + CommandLine) that hosts Talend Administration Center.
  2. A database server storing the administration metadata of Talend Administration Center (by default, an embedded H2 database is used).
  3. An SVN server for Project metadata.
  4. A Web browser to access Web application:
  5. Talend Administration Center where projects, users and processes can be managed and administrated.
  6. An artifact Repository (based on Apache Archiva) for software updates.
  7. Execution servers (JobServers) or Talend Runtime execution containers (based on Apache Karaf) to deploy and execute processes.
  8. A Studio API to carry out technical processes

Talend Activity Monitoring Console, providing detailed monitoring capabilities that can be used to consolidate the collected log information.

3. Talend Enterprise DI installation using Installer:

After downloading Talend-Tools-Installer-rYYYYY-VA.B.C-installer.zip, we need to extract it to see a dist and four executable files under same directory like below.

Talend-Tools-Installer-rYYYY-VA.B.C-windows-installer.exe for 32-bit and 64-bit Windows users,

Talend-Tools-Installer-rYYYY-VA.B.C-linux-installer.run and Talend-Tools-Installer-rYYYYY-VA.B.C-linux64-installer.runfor 32-bit or 64-bit Linux users respectively,

Talend-Tools-Installer-rYYYYY-VA.B.C-osx-installer.app in the .tar file called Talend-Installer-Tools-rYYYYY-VA.B.C-osx-installer.app.tar for MacOS users

Note: YYYY is the revision number and A.B.C is the revision level (Major.Minor.Patch).

From the above executable files select the Talend-Tools-Installer-rYYYY-VA.B.C-windows-installer.exe and launch the installation process by double clicking on it.

tac_setup1

Click on next button will pop up a license agreement window like below and accept it.

accept_agreement

Installation types and styles

Three installation types are available:

  1. The Server type: allows you to install all Talend server components with default configuration.
  2. The Client type: allows you to install the Talend Studio.
  3. The Custom type: allows you to select and configure the Talend solutions that will be installed.

By default, the Easy install option is selected and will install the Talend solutions with default configuration (default local host, default ports and so on) without asking you for any information. Before performing this installation, make sure the default ports are not used by other applications.

Advanced Installer will give the option to customize the individual talend product configuration according to the requirement.

Setting up the installer based on the Easy Install:

installation_type

Select the valid license file path and click on next will validate the license file.

tac_licence_validation

After validating the license file, it will show all the list of talend products and their confirmation.

easy_setup

Now the setup is ready to begin installing based on the default configuration.

tac_install_ready1

Click on next will install all the talend products and click on finish once the installation  is done.

Setting up the installer based on the Advanced Install:

advanced_install

Select the talend products from the list and click on next button will show  a new window with email server configuration.

select_components

 

Talend Administration Center (TAC) Configuration:

Configure the admin user details for the TAC web application login through the browser.

tac_config1

tac_setup_with_email

Email Notifications Configuration:

If we want to get the email notifications from the TAC then we need to enable the Setup Email Notifications option and configure the required email server details like below.

email_notify_config

Log Server Configuration:

Below are the two types of configurations available.

  • Use TAC Tomcat
  • Use Customer Tomcat (6/7)

Use TAC Tomcat: Log server will also use the same tomcat which is configured for the TAC.

tac_log_server_config

Use an existing tomcat server:

We need to specify the existing tomcat directory location here.

existing_tomcat_config

Command line configuration:

command_config

Job Server Configuration:

jobserver_config

Subversion Repository configuration:

svn_select

Select the type of svn you want to install from the drop down and below are the available options.

  • I will configure my SVN server later.
  • Use an existing svn server
  • Install an svn server locally (Linux only)

Use an existing svn server:

Configure all the svn server details as shown in the picture .

existing_svn_config

 

Services Installation:

This option will give use to setup the Talend components as a system service. The application will automatically start at system startup when we install a component as a service.

services_install

Now the setup is ready to begin installing all the selected Talend products on our computer.

tac_install_ready1

We are done.!!! Talend  Enterprise studio is successfully installed now.Just go to the installation directory and launch Talend studio by clicking on the appropriate launcher depends upon the operating system.

If you are looking for my next post on Talend Administration Center Configuration click here

Thanks,

Sayagoud Ravelly

How to Send mail using gmail with Talend

How to Send mail using gmail with Talend

This is a very common requirement by every talend user when it is required to send mail in different scenarios. These scenarios are

  • Successful completion of a job
  • Failure of job or subjobs
  • On particular event or based on condition or specific variable value

In order to send mail, mail server details are required. It may be possible that the client may provide server details, but often they are not provided. In this case, you can create an account on gmail with specific user name like [email protected] To send mail using gmail, talend consists of a component tSendMail.

Following settings are required inside tSendMail component to send mail using newly created gmail account. They are as follows

SMTP Host – “smtp.gmail.com”

SMTP Port – 465

SSL Support – Enabled

Need authentication – Enabled

username – “[email protected]

password – “yourpassword”

Use above settings to send mail using your gmail account.

Conditional mail Sending

When you want to send mail when particular condition arises after completion of your subjob, then you can connect tSendMail component using if clause and fill the condition variable with required condition say

context.variable >= 10

When you want to send mail when failure of particular subjob, then connect the subjob to tSendMail component and use system variable for that subjob where value is 1. Condition could be

((Integer)globalMap.get(“tRunJob_1_CHILD_RETURN_CODE”)) ==1

When you want to send mail on success, then connect another tSendMail component with following condition

((Integer)globalMap.get(“tRunJob_1_CHILD_RETURN_CODE”)) == 0

 

Vaibhav Waghmare

Helical IT Solutions

Transpose rows to columns using Talend Open Studio

Transpose rows to columns using Talend Open Studio

To transpose rows to columns using Talend open studio could be a very challenging task which is required during data migration activities from flat files to other flat files. Major challenge in migrating this data to database table is that, the user is not aware of how many columns are going to be transposed in a row. So, it is not possible to have a metadata of the columns. In this case, it is always suggested to keep all the transposed column data into the single column separated by specific field separator. I have encountered multiple projects in which this task is required and due to this transpose activity, customer is not able to process and analyze the data from the source files. Many times if the data is not represented in the required format, data becomes meaningless.

Sample Data

In the data given below, it is expected to pivot column Week from rows to columns. Following data is the number of vehicles sold by particular user in a week for two different user accounts.

Account;Week;Sale

334234;Week_1;23;

334234;Week_2;72

334234;Week_3;14

116675;Week_1;33

116675;Week_3;78

Expected output

Based on above data it is expected that the data is distributed like sale for each week for respective Account.

Account;Week_1;Week_2;Week_3

334234;23;72;14

116675;33;;78

Way out – Talend Job

Talend Open Studio for data integration can help you to read the data from the file and convert it to column data for particular key. Talend consists of a component tPivotToColumnsDelimited. This component can convert rows to columns based on the three different conditions.

1) Pivot Column – In order to convert rows to columns, we need to identify a one column which need to be converted to multiple columns based on Aggregate column. In above case, Week is the Pivot Column.

2) Aggregate column  – Aggregation column is the column from source data on which aggregation is to be applied with specific function. In above case, Sale is the Aggregate Column.

3)Aggregation function – which type of aggregation is to be applied on input data. If no aggregation function is applied, then you can select “First”. Aggregation functions available are Sum, Count, Min, Max, First, Last. In above case, First is the Aggregate function.

4) Group by – You need to provide a group by column name, this is the column based on which pivot columns are created. In above case, Account is the group by Column.

Once you execute the job, the pivoted data is written into a flat file at the given location with specific file name. You can change this path to dynamic path using context variables and time stamp in the file name for the file. This file uses specific field separator and row separator. You can customize it depending on your need. If you want to create directory, if it does not exists, then enable the check box to create it when executed. In advance properties, you can configure CSV options to include field enclosure, advance separator for numbers and even check box for not to create a blank file when there is no data.

Converting rows to columns

Output Data

Following table shows the data in table format which is converted from input.

Account Week_1 Week_2 Week_3
334234 23 72 14
116675 33 78

Following is the data from output file generated by the component.

Account;Week_1;Week_2;Week_3

334234;23;72;14

116675;33;;78

Above data shows null value if there is no sale by particular account in respective week. If you want to further process this file, then you can read the file and create a metadata and do further analysis of Sale by particular account holder.

 

Vaibhav Waghmare

Helical IT Solutions

ORGANIZATION ADMIN PERMISSION IN LIFERAY

In liferay, Administrator of any Organization is able to delete, update or change user details of any other organization which is not secure to organizations. User Details of any other organization should not display to any other organization admin.

To avoid the above scenario, it is required to change permissions and role for organizations admin.

Role Keywords

Administrator:   Administrator can create organization, assign or add users to organization, update or remove user, organization. It means Poweruser has all the authority of liferay portal.

Organization Administrator: Organisaton Administrator is able to see organization pages, he can edit organization page, but he is not able to edit his private and public pages.

 

How can we change Role of organization admin

  1. Login as Admin (Administrator role).
  2. Go to Control Panel.
  3. Select User and Organization menu from portal panel of left side to window.

Liferay user and role management

 

4. Select Organization in which you want to change the role of admin.
E.g. Selecting New York Organization

Liferay user and role managment

Liferay user and role managment

We can see there are 2 user in New York Organization, NYC (Admin) and NY1USER (organization User). Here NYC is Administrator for this organization.

5.Click Actions (NYC in this example) -> Edit

liferay user and role management

liferay user and role management

6. Select Roles tab from User Information under Admin menu [NYC Admin] which is a right side of window

liferay 4

7. We can see Current role of this user in Regular Roles. (Current is Administrator)

liferay 5

8. Remove Administrator from Regular Role and assign New Role from “+Select” option. When we click on Select Hyperlink, pop-up window appears. From this window, we can see 2 Roles- Administrator and Power User. [It is not that only above mentioned roles can be seen, there are chances of seeing other regular roles too]

liferay 6

9. Select Power User.

 

10. Now, from Organization Roles view. Click on “Select” Hyperlink and select organizations role from pop-up window.

liferay 7

11. Select Organization Administrator.

liferay 8

Now, we can see we have two roles for Organization Administrator, Power User and Organization Administrator

12. Click Save.

Now, when same organization admin logins into his/her account user can’t get access to any other organization details from control panel.

 

Before applying changes to organization administrator role.

liferay 9

After applying changes to organization administrator role.

liferay 10

He is not able to see any organization except his organization.

Finally we have secured Organization’s details from any other organization’s admin.

How to Optimize Tmap Component in Talend

How to Optimize Tmap Component in Talend

                                                                                                                                                                   

Optimize Tmap component in Talend

If Lookup table has few rows:

  • Open tmap select lookup model as Load Once .This will load lookup data one time before mail flow starts.
  • This Lookup data will be stored in memory then main flow execution is very fast with comparing with lookup data in memory.

IF lookup table data is very large:

Talend cannont store the lookup table data in memory. you will get java heap space Execption. To resolve this issue follow the below steps

  • open tmap
  • go to lookup table
  • click on tmap settings
  • select the value for store temp data property to True
  • click on ok
  • In Tmap properties basic settings  set Temp Data Directory path by browsing folder
  • Go to Advance settings
  • set  max buffer size(nb of row) to some value based on  lookup condition data type

Exception in thread “main” java.lang.OutOfMemoryError: GC overhead limit exceeded

Different causes for OOM

Every one in java development face java.lang.OutOfMemoryError now and then, OutOfMemoryError in Java is one problem which is may be due to

  • Improper memory management by the Java code
  • Insufficient hardware memory available to process the code
  • Programming mistakes
  • Memory leak

Types of OOM

Any of the above reason can result into OutOfMemoryError. Following three different types of OOM errors are common

  1. Java.lang.OutOfMemoryError: Java heap space
  2. Java.lang.OutOfMemoryError: PermGen space
  3. Java.lang.OutOfMemoryError: GC overhead limit exceeded

OOM in Talend

From above the last error I observed during flat file data migration to database using Talend Open Studio. File sizes were in the range of 10 MB-500MB in size. Initially job worked well, but when started with larger files of around 100MB, this error popped up. Hardware memory (RAM) available was 16 GB. But default memory allocated by Talend was Xmx1024m (1GB). There was a possibility to increase Xmx to 10240m which could have solved the issue, but this GC overhead limit exceeded was related to garbage collection. After searching for the issue, I came across a very descriptive article related to garbage collection at http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors

Resolution

Following workaround solved the problem in Talend without increasing the memory limit to higher figures.

Add new option to Windows–>Preferences–>Talend–>Run/Debug  – XX:-UseGCOverheadLimit