Python – Get data from CSV and create chart

Python – Get data from CSV and create chart


Hello everyone, this blog shows how we can import data from csv and plot as mesg grid chart in python libraries, through the help of Pycharm IDE (Community Version)


Prerequisite :



The following are the steps for creating mesh-grid in pycharm :
1. Installing NumPy,Pandas and Matplotlib libraries in Pycharm IDE :

  • 1. Create a scratch file in Pycharm.
  • 2. Click on “File” and goto settings.
  • 3. Goto “Project Interpreter” and click on ‘+’ symbol to add new libraries.
  • NumPy

2. Writing python code for importing csv data and creating meshgrid :

  • 1. Create a csv file containing these data “-5,5,0.01”.
  • 2. Use this python code in your scratch file .
  • import csv
    import numpy as np
    import matplotlib.pyplot as plt

    point = ""
    with open('C:/Users/Helical/Desktop/Blog/Python/MeshGrid.csv','r') as csvfile:
    meshreader = csv.reader(csvfile, delimiter=',')
    for row in meshreader:
    point = point + ', '.join(row)
    print(point)
    meshpoints = point.split(',')
    points = np.arange(float(meshpoints[0]),float(meshpoints[1]),float(meshpoints[2]))
    dx, dy = np.meshgrid(points,points)
    z = (np.sin(dx)+np.sin(dy))
    plt.imshow(z)
    plt.colorbar()
    plt.title('plot for sin(x)+sin(y)')
    plt.show()

  • If you are able to sucessfully run the code, then you will generate mesh grid in an new Pycharm window.
  • GraphMatplotLib

How to install NTP Client on Ubuntu 16.04

How to install NTP Client on Ubuntu 16.04

To install NTP Client on Ubuntu 16.04 follow below steps :

Note :To install NTP Client it have some dependency ,we need to install libopts25 package before installing NTP Client.

1.Download or use below attached libopts25 package(.deb) which supports Ubuntu 16.04
 
libopts25_5.18.10-4_amd64
 

2.Keep package file at specific location and execute below command to install ibopts25 package
 
sudo dpkg -i libopts25_5.18.10-4_amd64.deb
 
The package will automatically get installed.

> You can check whether package is installed or not using below command :
 
dpkg –get-selections | grep libopts25
 
3.Next download or use below attached ntpdate(ntp client) .deb package to install NTP Client.
 
ntpdate_4.2.8p4+dfsg-3ubuntu5.3_amd64

 
sudo dpkg -i ntpdate_4.2.8p4+dfsg-3ubuntu5.3_amd64.deb
 
The package will automatically get installed.
> You can check whether package is installed or not using below command :
 
dpkg –get-selections | grep ntpdate
 
4. Now we need to configure the NTP client to which NTP server NTP Client can access , goto server at location (/etc/default/) where we installed NTP client.
 
5.Open the file “ntpdate” present at location “/etc/default/” and add the ntp server which you are using mentioned below :
 
Server ntp-server-host-name
 
For Example : Server 172.20.1.200
 
6.Save file and check whether you are able to connect to NTP Server or not using below command :
 
sudo ntpdate ntp-server-name
 
For Example : sudo ntpdate 172.20.1.200

 
To Install NTP-Server on Ubuntu Server.
 
1.Download NTP-server from internet
 
ntp_4.2.8p4+dfsg-3ubuntu5.3_amd64.deb
 
2.Upload the downloaded file on ubuntu server.
 
3.Execute the below command to install the downloaded package
 
sudo dpkg -i ntp_4.2.8p4+dfsg-3ubuntu5.3_amd64.deb
 
4.Once the installation will complete, we can check the NTP daemon is running or not by using below command
 
ps -aef | grep ntpd
 
Hope it helps 🙂

 
Thanks ,
Sayali | Helical IT Solutions

Grails

What is grails?
According to grail documentation
Grails is a full stack framework and attempts to solve as many pieces of the web development puzzle through the core technology.

What are the built-in features of Grails?

Data management:
It is easy with built-in Object Relational Mapping (ORM) using Hibernate

Cool view technology:
Similar to JSP groovy has Groovy Server Pages (GSP)

MVC
Integrated Spring MVC

Commandline
An interactive command line environment and build system based on Gradle

Servlet container: Has embedded Tomcat container with hot loading feature.

Global language
Support for internationalization (i18n) built on Spring

Transaction
A transactional service layer built on Spring’s transaction abstraction
All of these are made easy to use through the power of the Groovy language and the extensive use of Domain Specific Languages (DSLs)

HttpClient

What is HttpClient?
Any program or software that can make call to any url or webapplication using http proptocol is HttpClient.

What are the dependencies of HttpClient?
1. commons-codec jar 1.2
2. commons-logging jar 1.0.4
3. junit
What are the steps for using HttpClient?
1. Intantiate HttpClient.
2. Use any methods (GetMethod/Post etc) .The URL can be passed in the constructor
3. Invoke HttpClient to execute the method.
4. Read the response using stream and store it as string or any other object.
5. Release the connection.
6. Use the response.
Method that can be used in HttpClient
Options: It helps to retrive the options available.
Get: The get method which has the url and query parameters.
Head: The HEAD method similar GET except here message-body is not present in the response.
Post: For a POST request, with large request in the body
Multipart: Generally it is used for file uploads etc.
Put, Delete and Trace are also usable

How to use Google maps in Jaspersoft Studio

How to Use Google Maps in Jaspersoft Studio

The report in this example is developed on Jaspersoft Studio version: 6.3.1 final and deployed on Jaspersoft server version: 6.3.0 Enterprise edition.

Step 1: Create a report in jasper studio. In this example report is called GoogleMapsTest.

Screenshot (193)

1- Delete all the bands except Title and summary band. Use the query “Select 1 as one” in main data-set query.

Step 2: Adding the jasper map component to report.

Screenshot (173)

1- Add the text box to Title band and insert any relevant text into it.

2- Drag the Map component from Palette on to the Summary band of the report.

3- Add a data-set named “Markers” to the report, and use this query in it.

Query:

select ‘NorthCharleston’ as CityName,’-80.02842′ as Longitude,’32.92896′ as Latitude
union
select ‘GREENVILLE’ as CityName ,’-82.33626515′ as Longitude ,’34.79360263′ as Latitude
union
select ‘ZIBO’ as CityName,’117.8351787′ as Longitude,’37.17124911′ as Latitude
union
select ‘Mumbai’ as CityName, ‘72.9227767’ as Longitude, ‘19.10662726’ as Latitude

Step 3: Add markers to the map.

Screenshot (175)      Screenshot (176)

1- Click on the Map and select the Markers tab from the Properties section.

2- Add the created data-set to the marker by selecting the Data-set tab.

Step 4:Add markers properties.

Screenshot (177)

1- For adding marker properties, click the add button on right side, and insert the following details.

i.) Longitude & Latitude : For adding Longitude & Latitude, click the the properties icon on the right hand corner of the input text field. Check the “Use Expression” box and provide the latitude and longitude fields in their appropriate input boxes.

Screenshot (179)

ii.) Adding Mouse-over(Information appearing when hovered on marker) : For mouse-over information, insert data into the “Title” box in the window.

Screenshot (181)

iii.) Adding Pop-up window content(Content shown after clicking the marker) : For adding pop up window content, use the “Info Window”  text boxes.

Screenshot (183)

Step 5: Deploying the report in jaspersoft server

Publish the report on jasper server.

Screenshot (192)

………. and Voila..!!!!!!!!!!!

Thank You

Ravi Bhatta

 

 

Create A Custom Table Report using Helical Insight (Dynamically Picking the Columns Names and Data)

Create A Custom Table Report using HI (Dynamically Picking the Columns Names and Data)

If you have already had a Hands-On experience on the Helical Insight Tool [HI tool] then this blog would be helpful

For a creating a report there are 4 files required
1. EFW
2. HTML
3. EFWD
4. EFWVF

the Report Layout lies on the HTML page, the SQL queries lies within the EFWD file and the visualization lies in the EFWVF File.
Hence once the Query is fired it comes to the visualization file to create a table as that’s our goal. With the help of the following code below it can be created with ease.

The below code here is a template of our EFWVF looks like
<Charts>
<Chart id=”1″>
<prop>
<name>Table</name>
<type>Custom</type>
<DataSource>1</DataSource>
<script>

//Your Visualization Code Goes Here

</script>
</prop>
</Chart>
</Charts>

Chart ID here is 1 which is unique
Type is of custom
DataSource 1 is the Unique ID defined in the EFWVF File. ie (<DataMap id=”1″ )
Now within the <script> </script>
we Paste the following:

<![CDATA[
//The If Block Does Return A Message NO Data when there is No Data
if(data.length == 0)
{
$(‘#chart_1′).html(“<div ><h4 style=’text-align:CENTER;color:black; padding-top:60px;’>No Data Available For Current Selection</h4></div>”);
return;
}
//The Else Block Returns The Table if there is a Table
else
{
//Here the funtion Tabluate Returns the Data in Tabular Form
function tabulate(elem, data, columns)
//Function Start
{
var table = d3.select(elem).append(“table”)
.attr(“class”,” table display compact width:100%;cellspacing:1 “)
.attr(“id”,”table”)
thead = table.append(“thead”),
tbody = table.append(“tbody”);

//Append the header row
thead.append(“tr”)
.selectAll(“th”)
.data(columns)
.enter()
.append(“th”)
.text(function(column) { return column; })
.attr(‘class’, function(d, i){ return “colH_” + i; })
.style(‘background-color’,’#ededed’)
.style(‘color’,’black’)
.style(‘@media print’,’display:none’)
.style(‘padding-left’,’25px’);

// create a row for each object in the data
var rows = tbody.selectAll(“tr”)
.data(data)
.enter()
.append(“tr”);

// create a cell in each row for each column
var cells = rows.selectAll(“td”)
.data(function(row) {
return columns.map(function(column) {
return {column: column, value: row[column]};
});
})
.enter()
.append(“td”)
.text(function(d) { return d.value; })
.attr(‘class’, function(d, i){ return “col_” + i; })
.attr(‘align’, ‘left’)
return table;
//Function END
}

//Render the table
//Object.keys(data[0]) is the Data to fetch the Column Header
//data has the data of the Table
console.log(Object.keys(data[0]));
var subjectTable = tabulate( ‘#chart_1’, data, Object.keys(data[0]));
}
]]>
Save it in the same Directory with rest of the file

Run your report on HI and you get a table.

Keep in mind you can change the query and still get the new columns from the new query.

Thanks
Sohail Izebhijie

The Print When Expression

The Print When Expression

The print when expression in Jasper Studio or Jasper iReport is very useful in so many requirements and it is defined as the name itself implies
i.e Print /Show a String, Column and so on Based on a condition passed.
Now here are some requirements on how/when we can implement The Print when expression.

Requiremnt: Show a particular column when a parameter is selected
Solution:
Here we made a parameter Called ShowLocation with values either ‘Y’ or ‘N’/blank space
now when ‘Y’ is selected then a Location column should show and when ‘N/blank space then it doesn’t show.

2 ways i did approach this:

1. I made a 2 Table Components one with the Location Column added and the other without the Location Column added.
one on top the other.
so when my parameter calls a ‘Y’ value it picks the Table with the Location Column and vice versa.

This approach is good but can be time and performance consuming

the 2nd approach was:

2. Only one Table Component was used, Opened the Table Component and selected the Column i want to restrict/show based on the parameter passed, in my scenario or requirement it is the Location column as
shown in the image below:

TableLayout

now i would select the cell button

Cellin the properties Tab and then Select the
Print When Expression.

Now here in the print when expression box i would write :
PrintExperssion
$P{showlocation}.equals(“Y”) as shown above

 

and you are good to go for a preview.
Now Let’s have to put a N or blank space in the parameter showlocation we get the following output
before
and now let’s pass ‘Y’ in the showlocation parameter

before-Y
So Now with this condition it will show the column when the parameter is checked in with a ‘Y’ Value.
and voila the output:

after

Thanks
Sohail Izebhijie.

Let’s Have Fun with Triggers in SQL

Let’s Have Fun with Triggers in SQL

What are Triggers?

Triggers are a Block of Structure that executes when a DML Statement is performed or Executed. Triggers are fun to use but from a business point of view not advisable but that a different story entirely ie the Pro’s and Con’s. but for gaining an idea about Triggers here is what the Video Covers

  1. Basic Creation of a Trigger
  2. and a simple real-time scenario on how i used triggers

The Video here illustrates or covers the above points.

Beginner’s Guide to OLTP and OLAP

Beginner’s Guide to OLTP and OLAP

In this blog we will be looking into the introduction of OLTP and OLAP. How it is defined and how we can relate it to real-time

Firstly, OLTP stands for On-line Transaction Processing and OLAP stands for On-line Analytical Processing

But to define OLTP and OLAP in the Simpliest of language is
OLTP is defined as “What is Happening” ie it deals with various transactions as it happens
and OLAP is defined as “What has happened” ie it deals with Stored Data.

OLTP is more of INSERT, DELETE, UPDATE, MODIFY i.e perform Day to Day Operations
But OLAP on the other hand is more of Fetching Data i.e Perform Analysis.

Take for Example a Supermarket Counter : the Counter receives payment for a particular product thus the event happening is a transaction which should be inputted into the Database
Hence it’s a OLTP.

But the OLAP is once the Data has been stored and the only Operation performed is Fetching the Data then it is said to be OLAP which can be used for
Business decision making and planning.

OLTP is a Relational Database and it’s 2-Dimensional in nature i.e Mathematically it is in the form of x and y axis
but OLAP on the other hand is a Multi-dimensional Database i.e Mathematically it is in the form of x, y, z, nth axis.

Now a question arises how is OLAP and OLTP represented? OLTP is represented in a Normalized Form and OLAP in a Star Schema or even at times in a Snow Flake Schema

OLAP Data- Structure is Designed in a Tree Format making it easy to Traverse.

Thanks,

Sohail

Helical Insight for ETL Tester (QA)

Helical Insight for ETL Tester (QA)

Let’s take an example that your ETL developer has just wrote an ETL script, which loaded the data into the Data Warehouse, few millions of records got inserted into the Data Warehouse.

For QA, it is super difficult to test all (millions) of the data and also a time consuming process. Moreover, there are many other problems like

  • some might not be well versed with writing complex SQL queries
  • There might be multiple data sources involved in generating the Data Warehouse.
  • Writing multiple queries on multiple data sources and then combining it in one single result set would be a sweat breaking task.

Now think of Helical Insight as your friend which helps you to do most of your tasks, allowing you to focus on more critical thinking rather than just doing stare and test whole day. Wondering how Helical Insight can help you achieve the same?

Here is your answer,

  • Helical Insight is capable of communicating with any of your data source(s) like CSV, Excel, MySql, Oracle, SQL Server, Postgres etc.
  • Helical Insight has a drag and drop feature to make your test cases ready, without even having a prior knowledge of SQL queries. Helical Insight generates the SQL queries including the joins between multiple entities. Helical Insight understands your data structure, hence it generates the query even if you are working with different set of data source(s), DW or normalized database, Helical Insight understands all.
  • Using this (drag and drop) facility you can generate all your test cases in a form of report, which you can call it as your test cases. For example, your report does a count of records for a dimension. You can save this report and check the same with source system by creating a separate report. If its a migration project the count should definitely match otherwise we know something is wrong with the ETL Script.
  • Once you have saved these reports (test cases) you can re-executes them again an again to verify your data. In the above situation you have just identified that Helical Insight is really a very useful tool with some really cool features like generating the SQL query for any database or data source(s), ability to execute the queries on any databases or data source(s), ability to save the report / test cases / SQL queries for future usage.