Healthcare I.T in the big data space: Article 2 – Getting setup, touching some code

This is the 2nd article in this series, you can start with the 1st article if you need some context.

For the infrastructure, I began small.

1. I launched an Ubuntu ec2 instance
2. I installed a vnc  server on it so i could access the desktop from a vnc client on my Mac  – I plan to use this ec2 instance as my development machine
3. I installed mirth – I’ll need this to generate valid hl7 messages that I will feed into spring xd’s big data pipeline
4. I installed eclipse luna- to develop the spring integration modules that will be uploaded into spring xd.

My 1st use case is to create a hl7 feed and ingest it using spring xd, storing the messages in hdfs.

For the 1st development effort, I plan to create a mirth channel that reads hl7 from a database, sends it over tcp to a spring xd custom source module.
The mirth configuration is shown below

Source tab of the channel

Destination tab of the channel

For the spring xd to ingest the hl7, I have to create a custom source module as none of the out of the box modules will handle hl7 mllp protocol. This module will use a  camel hl7 endpoint to accept messages from mirth and then place these messages on a spring integration channel.

The plan is to eventually upload this spring integration module into spring xd. 1st things 1st. In eclipse I created a spring boot project. The configuration class / application class looks as follows:

This being the 1st iteration , Ian writing to file just to make sure the data is coming in, ad sure enough after I run the code in eclipse and start my mirth channel, I have hl7 messages dropping into my folder . Neat!!

Spring boot/integration /camel source code iteration 1 of the inbound hl7 to a file destination:

package com.wchr.xd;

import java.io.IOException;

import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import static org.apache.camel.component.hl7.HL7.ack;
import org.apache.camel.component.hl7.HL7MLLPNettyDecoderFactory;
import org.apache.camel.component.hl7.HL7MLLPNettyEncoderFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@SpringBootApplication
@Configuration
public class Application extends RouteBuilder {

	@Autowired
	CamelContext camelContext; 

	//private static CamelContext cmContext;

    public static void main(String[] args) throws IOException, InterruptedException {
    	 SpringApplication.run(Application.class, args);
    	// cmContext = context.getBean(CamelContext.class);
    	}

    @Bean
    public HL7MLLPNettyDecoderFactory hl7decoder(){
    	return new HL7MLLPNettyDecoderFactory();
    }

    @Bean
    public HL7MLLPNettyEncoderFactory hl7encoder(){
    	return new HL7MLLPNettyEncoderFactory();
    }

	@Override
	public void configure() throws Exception {
		// TODO Auto-generated method stub
		from("netty4:tcp://localhost:9000?sync=true&encoder=#hl7encoder&decoder=#hl7decoder")
		.to("file://inbound?autoCreate=true&fileName=${date:now:yyyyMMddHHmmss}.hl7")
		.transform(ack());
	}

}

snapshot of the inbound messages from my eclipse project

 

Informatica Java Transform handling of Dates

I love working with the java transform in informatica because it gives me all the freedom in the world, and it makes my mappings compact, rather than having tens of drag and dropped components. I can usually achieve a lot with only one java transform.

One thing confounded me recently as I could not from in side a java transform assign a date value to an output port I had defined as a date/time port.
So the code below could not work.
date_port_out = New Date();

I tried a bunch of alternatives , example using simpleDateFormat to create a format that is the default in informatica, but nothing was working.
Then I decided to look at the generated code in the java transform

I couldn’t believe it ; When I create a date/time port, the java transform creates a long out of it. How does it do this, I could guess , but I sure know how to convert a java Date into the millisecond values since epoch (Jan 1st 1970).

long millis_since_epoch = new Date().getTime();

Then I assigned the long value to the output port – and it worked !!

date_port_out = new Date().getTime; 

Also important not to forget to import the Date package in the imports tab;

import java.util.Date;

 

Informatica Java Transform For Data Validation

I have been working in informatica for a while now, and every time I need to do something thats out of the ordinary filter, sort , run a query, aggregate, I usually resort to the swiss army knife that is the java transform.

One of the  main abilities I get from the java transform is the creation new rows of output from a single row of input. Why would I want to do this? Well one common use case I have seen examples of out there is to unpivot your data transpose columns into rows.

So If you have data like

Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff	Mary
10	20

And you need it to look more like

Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff	10
Mary	20

All nice and dandy. But I had another use case that I think is more interesting.

I have a set of data I need to validate for errors, and for each row, there are several errors that could occur. So one row could have 10 errors spread out in different columns.

I needed a process that goes through my set of data and creates another set of data holding the collection of validation errors.

So for example, I have the following row, and 2 errors I need to create from this: The names should be less than 20 characters , and the number (lets assume its the age) should be greater than 18

Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff

10

And I need the outputs

The name “Hubert_Blaine_Wolfeschlegelsteinhausenbergerdorff” is longer than 20 characters

The age 10 is less than 18

They way to achieve this was to use an active java transform (emphasize ‘active’) in which I have logic like below:

if(NAME.length > 20) {
error_port_out = "The name "+ name_port_in+ " is longer than 20 characters";
generateRow(); //does the magic of adding additional output rows from one input row
}

if(age > 18) {
error_port_out = "The age"+ age_port_in+ " is less than 18";
generateRow(); //again
}