Logback – Dynamic creation of log files

Some project have some log constraints that are difficult to implements with Log4j.

Logback is an alternative to log4j. It gives you more possibilities and it can easily implements your log policies.

    log1
Origami by Brian Chan

In this post, I present a simple of Logback configuration for a maven project. This configuration implements a dynamic log files generation.

Sample project description

This post example is inspired from a batch log policy I dealt with. The batch processes N files and generates two logs for each processed file. The first one is a summary of logs with no stack trace for functional users and the second one is a detailed log for programmers.

The picture bellow presents this case of study.

logback
Below the summary of log policies to implement:

  1. Logging errors without stacktrace.
  2. Dynamic file logging (switch to new log file).
  3. Read logging directory from application properties file.
  4. Create log files name with timestamp.

1. Add Maven dependency

First we need to add Slf4j and logback dependency to the project.

<dependency>
	<groupId>org.slf4j</groupId>
	<artifactId>slf4j-api</artifactId>
	<version>1.7.5</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-classic</artifactId>
	<version>1.0.13</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-core</artifactId>
	<version>1.0.13</version>
</dependency>

<dependency>
	<groupId>ch.qos.logback</groupId>
	<artifactId>logback-access</artifactId>
	<version>1.0.13</version>
</dependency>

<dependency>
	<groupId>org.slf4j</groupId>
	<artifactId>jcl-over-slf4j</artifactId>
	<version>1.7.5</version>
</dependency>

2. Create a costom layout Class

To customise logging layout, we create a class which extends LayoutBase and override the doLayout method.

public class CustomLogLayout extends LayoutBase<ILoggingEvent>{

	@Override
	public String doLayout(ILoggingEvent event) {
		StringBuffer sbuf = new StringBuffer(128);
	    sbuf.append(new SimpleDateFormat("yyyyMMdd_HHmmss").format(new Date(event.getTimeStamp())));
	    sbuf.append(" ");
	    sbuf.append(String.format("%-6s", event.getLevel()));
	    sbuf.append(" ");
	    sbuf.append(event.getFormattedMessage());
	    if((event.getLevel() == Level.ERROR || event.getLevel() == Level.WARN) && event.getThrowableProxy() !=null) {
	    	sbuf.append(" - ");
	    	sbuf.append(StringUtils.substringBefore(event.getThrowableProxy().getMessage(), IOUtils.LINE_SEPARATOR));
	    }
	    sbuf.append(CoreConstants.LINE_SEPARATOR);
	    return sbuf.toString();
	}

}

3. Logback configutration

In this step we create Logback.xml config file that will use the costomLayout we created in the previous step and will implements the logs policies.

  • Line 3 indicates the path to properties file where it will look for some values. In our case the path is a system property.
  • Line 5 specifies a key for the timestamp value.
  • STIF appender creates a custom log files with the CustomLogLayout class.
  • loggerFileName is a MDC (Mapped Diagnostic Contexts) property. Changing this value will generate a new file.
  • SIFT-TRACE appender creates a verbose log file and it uses logback.classic.PatternLayout.

<configuration>

	<property file="${appli.config.path}" />

	<timestamp key="bySecond" datePattern="yyyyMMddHHmmss" />

	<appender name="SIFT" class="ch.qos.logback.classic.sift.SiftingAppender">

		<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
			<level>INFO</level>
		</filter>
		<discriminator>
			<Key>loggerFileName</Key>
			<DefaultValue>unknown</DefaultValue>
		</discriminator>
		<sift>
			<appender name="FILE-${loggerFileName}" class="ch.qos.logback.core.FileAppender">
				<File>${log.dir}/${loggerFileName}_${bySecond}.log</File>
				<Append>false</Append>
				<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
					<layout class="com.mycompany.log.CustomLogLayout" />
				</encoder>
			</appender>
		</sift>
	</appender>

	<appender name="SIFT-TRACE" class="ch.qos.logback.classic.sift.SiftingAppender">
		<discriminator>
			<Key>loggerFileName</Key>
			<DefaultValue>unknown</DefaultValue>
		</discriminator>
		<sift>
			<appender name="TRACE-${loggerFileName}" class="ch.qos.logback.core.FileAppender">
				<File>${log.dir}/TRACE_${loggerFileName}_${bySecond}.log</File>
				<Append>false</Append>
				<layout class="ch.qos.logback.classic.PatternLayout">
					<Pattern>%d %-5level %logger{5} - %msg%n</Pattern>
				</layout>
			</appender>
		</sift>
	</appender>

	<logger name="org.springframework" level="ERROR" />
	
	<root level="DEBUG">
		<appender-ref ref="SIFT" />
		<appender-ref ref="SIFT-TRACE" />
	</root>

</configuration>

4. Test config

In this step we test the logback configuration.

setup() method set system property for the properties file path where logback will look for loging directory path.

The test simulates processing N files and at the end it verifies that we have 2xN log files.

public class LogTest {

	private static final int TEST_FILE_NUM = 5;

	private static final File TEST_DIR = new File(System.getProperty("java.io.tmpdir"), "logbak-test-dir");

	@Before
	public void setUp() throws IOException {
		System.setProperty("appli.config.path", ClassLoader.getSystemClassLoader().getResource("logback.properties").getPath());
		TEST_DIR.mkdirs();
		FileUtils.cleanDirectory(TEST_DIR);
	}


	@Test
	public void testLogbackFileConfig() throws InterruptedException {

		Logger logger = LoggerFactory.getLogger(LogTest.class);

		// assume SLF4J is bound to logback in the current environment
		LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
		JoranConfigurator configurator = new JoranConfigurator();
		configurator.setContext(context);
		context.reset();

		try {
			configurator.doConfigure(ClassLoader.getSystemClassLoader().getResource("logback.xml").getFile());
		} catch (JoranException je) {
			StatusPrinter.print(context);
			Assert.fail("Erreur de chargement des paramètre logback");
		}

		for (int i = 1; i <= TEST_FILE_NUM; i++) {
			MDC.put("loggerFileName", "file" + i);
			logger.info("log in File{}", i);
			logger.debug("log in File{}", i);
			logger.error("log in File" + i, new Exception("My test exception", new Exception("my cause of exception")));
		}

		Assert.assertEquals(TEST_FILE_NUM * 2, TEST_DIR.listFiles().length);
	}
	
}

Posted in Java | Tagged , , , , , , , , | Leave a comment

JUnit test with Mockito and Spring

It is important to be able to perform some integration testing without requiring deployment to your application server or connecting to other enterprise infrastructure.

This could be done in your Spring project by using Spring-test library and Mockito framework.

   
Photo credit: Honey

In this post, I present a simple way to create a mock and to inject it into a tested bean.

Preparation of testing with Mockito

To make Mockito available, we need to add following dependency in the pom.xml file.

		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-core</artifactId>
			<version>1.9.5</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>spring-test</artifactId>
			<version>3.1.2.RELEASE</version>
			<scope>test</scope>
		</dependency>

Deveop Mockito, Spring and JUnit tests

The code source bellow presents a JUnit test using Spring-test and Mockito framework.

  • @RunWith(SpringJUnit4ClassRunner.class) runs the test with Spring custom runner that will load Spring application context.
  • @Autowire associate invoiceService reference to the InvoiceService instance.
  • @Before annotation executes the setup method before the test.
  • MockitoAnnotations.initMocks(this) scans the class annotation, instanciates invoiceDaoMock and inject it into the InvoiceService bean.
  • when(.(any())).thenReturn() associates a return value with method execution parameters.

//Load Spring contexte
@ContextConfiguration(locations = {"classpath:/application-context.xml"})
@RunWith(SpringJUnit4ClassRunner.class)
public class InvoiceServiceTest {
	
	// Create Mock
	@Mock
	private IInvoiceDao invoiceDaoMock;
	
	@InjectMocks
	@Autowired
	private IInvoiceService invoiceService;
	
	
	@Before
	public void setUp() {
		MockitoAnnotations.initMocks(this);
	}

	
	@Test
	public void testProcessInvoice() throws SQLException {
		
		// specify mock behave when method called
		when(invoiceDaoMock.save(any(Invoice.class))).thenReturn(Long.valueOf(1));
		
		Assert.assertNotNull(invoiceService);
		Map<Product, Integer> products = new HashMap<Product, Integer>();
		products.put(new Product("labtop", BigDecimal.valueOf(1255.50)), 2);
		Invoice invoice = invoiceService.processInvoice(products);
		
		Assert.assertEquals(1255.50 * 2, invoice.getTotal().doubleValue(), 0);
		
	}
}

Posted in Java, test framework, Uncategorized | Tagged , , , , , , | 5 Comments

Web Services with Apache CXF

CXF framework is widely used to create web services. It supports a variety of protocols such as SOAP, XML/HTTP, RESTful HTTP, or CORBA and work over a variety of transports such as HTTP, JMS or JBI.

CXF has some tools like wsdl2java witch generate web service code from WSDL. This tool is very useful but it integrates some code that usually we don’t control all its details.

Photo credit: Brian A Jackson

I wrote this article to presents a concise way to create a web service with CXF and Spring. This solution has the advantage to avoid generating extra code to integrate in your project.

1 – CXF dependency

If you are using Maven you can add dependency by adding the XML code below.

<dependency>
	<groupId>org.apache.cxf</groupId>
	<artifactId>cxf-rt-frontend-jaxws</artifactId>
	<version>2.1.3</version>
</dependency>
<dependency>
	<groupId>org.apache.cxf</groupId>
	<artifactId>cxf-rt-transports-http</artifactId>
	<version>2.1.3</version>
</dependency>

2 – Web service implementation

First, add a web service interface. The code source bellow is a simple of web service that allows adding a customer to an existent JSF web application.

import javax.jws.WebService;
import com.mycompany.entity.Customer;

@WebService
public interface CustomerService {

	String SaveCustomer(Customer customer);

}


Next, write your web service implementation.

import javax.jws.WebService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import com.mycompany.dao.ICustomerDao;
import com.mycompany.entity.Customer;

@Component("customerService")
@WebService(endpointInterface = "com.mycompany.service.CustomerService")
public class CustomerServiceImpl implements CustomerService {
	
@Autowired
private ICustomerDao customerDao;

	public String SaveCustomer(Customer customer) {
		customer.setCustomerId(null);
		customer.setCustomerOrders(null);
		customerDao.save(customer);
		return "saved";
	}	
}


Finally, add spring configuration to configure the web service end point and client.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jaxws="http://cxf.apache.org/jaxws"
	xsi:schemaLocation="
	http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
	http://cxf.apache.org/jaxws http://cxf.apache.org/schemas/jaxws.xsd">

	<import resource="classpath:META-INF/cxf/cxf.xml" />
	<import resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
	<import resource="classpath:META-INF/cxf/cxf-servlet.xml" />

	<jaxws:endpoint id="customerServiceEndPoint" implementor="#customerService"
		address="/CustomerService" />

	<jaxws:client id="customerServiceClient" serviceClass="com.mycompany.service.CustomerService"
		address="http://localhost:8080/myApp_JSF_CXF/ws/CustomerService" />
</beans>

3 – Web service test

Test with SoapUi
You can use SoapUi to test your web service. In this case you don’t need to implemente a service client.

SoapUi needs a wsdl file. You can provide this URL http://localhost:8080/myApp_JSF_CXF/ws/CustomerService?wsdl.

Test with JUnit

Testing a web service with junit it’s more likely to be an integration test than a unit test because it depends on other resources. You will need to start your application server and deploy your web service before running your test.

The code source bellow uses the web service client to test the web service developed in this article.

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations={"/application-context-test.xml"})
public class CustomerServiceTest {
	
	@Autowired
	@Qualifier("customerServiceClient")
	private CustomerService customerServiceClient;

	@Test
	public void SaveCustomerClientTest() {
		Customer customer = new Customer(25, "nameTest","adresse", "city", "state", "123", "0606060606", null);
		assertEquals("saved", customerServiceClient.SaveCustomer(customer));	
	}
}

Posted in Java, Software, Uncategorized, WEB | Tagged , , , , , | Leave a comment

Spring Integration

Today’s business applications rarely live in isolation. That’s why we have often in applications development to implement a communication module to make the new application exchanging data with the existent information system. To implement those data flow we can use many technologies (JMS, SOA, Apache commons IO …). Managing all those components in a project can be tricky and not easily maintainable.

   
Photo credit: Michelangelo

Spring Integration provides an extension of the Spring programming model to support the well-known Enterprise Integration Patterns. It enables lightweight messaging within Spring-based applications and supports integration with external systems via declarative adapters.

The first part of this article presents some basic background to start using Spring Integration. The second part deals with a project sample.

1- Spring Integration Basics

Spring Integration adds essentially three components to the core Spring Framework: messages, message channels, and endpoints.

1.2-Spring Integration components

In this part we will present some basic components used in this article sample.

Message: is a data structure that will have payload and the header. The payload can be a file, String, JMS message … The header is a key-value map that contains some properties that will be used by the framework or user values.

Channel: it represents the “pipe” of a pipes-and-filters architecture. Producers send Messages to a channel, and consumers receive Messages from a channel.

Inbound Channel: it does some mapping between the Message and whatever object or resource is received-from the other system (File, HTTP Request, JMS Message, etc).

Outbound Channel: it does the same thing like the inbound Channel but in the other side. It map message to send to other system.

Service Activator: it’s a generic endpoint for connecting a service instance to the messaging system. The input Message Channel must be configured, and if the service method to be invoked is capable of returning a value, an output Message Channel may also be provided.

Transformer: is used to create a Message-transforming endpoint. It requires a “ref”. The “ref” may either point to an Object that contains the @Transformer annotation on a single method or it may be combined with an explicit method name value provided via the “method” attribute.[2]

Splitter: is a component whose role is to partition a message in several parts, and send the resulting messages to be processed independently.

To get more details about those components and to discover others components you can read the Spring integration reference manual in this link : http://static.springsource.org/spring-integration/reference/htmlsingle/

1.2- Spring integration component specification

To specify components, you can use the basic xml language or annotations.

This source code presents the transformer declaration by sing the XML language.

<integration:transformer input-channel="channel1" output-channel="channel2">
	<bean class="com.mycompany.integration.FileTransformer" />
</integration:transformer>


The same thing could be done by using Spring integration annotation.
First add to your spring application context the declaration of the package where your annotated classes are defined.

	<integration:annotation-config/>
	<context:component-scan base-package="com.mycompany.integration"/>

Next you add the annotation to your transformer class.

@Component
public class FileTransformer {	
	
	@Transformer(inputChannel="channel1", outputChannel="channel2")
	public NewMessageFormat transform(OldMessageFormat omf) throws IOException{
		//Your transformation implementation here
		return newMessageFormat;
	}
}

I recommend the xml specification because it could be used by STS to display integration graph. This graphic representation is helpful for reading and verifying the integration flow.

1.3- Create Spring integration project

The simplest way to create a Spring Integration project is to create a basic Spring project and to add Spring Integration dependencies to the pom file.

<dependency>
	<groupId>org.springframework.integration</groupId>
	<artifactId>spring-integration-core</artifactId>
	<version>${spring.integration.version}</version>
</dependency>

<dependency>
	<groupId>org.springframework.integration</groupId>
	<artifactId>spring-integration-file</artifactId>
	<version>${spring.integration.version}</version>
</dependency>

<dependency>
	<groupId>org.springframework.integration</groupId>
	<artifactId>spring-integration-jdbc</artifactId>
	<version>${spring.integration.version}</version>
</dependency>

1.4- Spring Tools Suite (STS)

You can develop your spring integration application with your favorite IDE without any extra feature. But I think it is too much easier if it will be done with STS. You can download STS from this link http://www.springsource.com/developer/sts or to install the STS plugin for eclipse form this link http://marketplace.eclipse.org/content/springsource-tool-suite.

2- Spring integration sample

In this sample, we suggest to implement data integration between two systems. The first one produces data into flat file and the second system receives a data into a database.

You can import the project sample from google code SVN link: http://project4example4.googlecode.com/svn/trunk/

The application needs to detect the new file in a folder, log the name of the detected file, read text file and integrate its content into the database.

Bellow we present the source formatted flat text file content and the destination persistent classe.

id tax name adresse city state zip phone
5 2 name5 adresse5 Paris Paris 75001 0606060606
6 2 name6 adresse6 Paris Paris 75001 0606060606
7 2 name7 adresse7 Paris Paris 75001 0606060606
8 2 name8 adresse8 Paris Paris 75001 0606060606
9 2 name9 adresse9 Paris Paris 75001 0606060606
Source file format Persistent class

The picture bellow presents Spring integration graph used to implement the solution.

The file inbound channel inspect every 5 second the input folder. If a new file is present a message with a payload of type file will be created and sent to the service activator.
The service activator in this sample will just log the file name and deliver the message to the channel1.
The Transformer gets the Message from channel1, parse it, create a list of Customers and send the new message content to channel2.
The splitter gets the message, read the payload content and split the message to massages with a payload of type Customer.
The JDBC outbound channel gets the payload and uses payload attribute to execute the query.

The code source bellow presents the xml code of the spring integration graph. You can edit your Spring integration flow with both modes graphic and XML code mode. If you change XML code, the graph will be updated and if you use the graphic mode the changes done to the graph will affect the source code.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns:integration="http://www.springframework.org/schema/integration"
  xmlns:file="http://www.springframework.org/schema/integration/file"
  xmlns:jdbc="http://www.springframework.org/schema/integration/jdbc"
  xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans.xsd


http://www.springframework.org/schema/integration


http://www.springframework.org/schema/integration/spring-integration-2.0.xsd


http://www.springframework.org/schema/integration/file


http://www.springframework.org/schema/integration/file/spring-integration-file.xsd


http://www.springframework.org/schema/integration/jdbc


http://www.springframework.org/schema/integration/jdbc/spring-integration-jdbc.xsd">

<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" />

	<file:inbound-channel-adapter id="filesIn"
		directory="${customer.file.input.path}">
		<integration:poller id="poller" fixed-delay="5000"/>
	</file:inbound-channel-adapter>

	<integration:service-activator	input-channel="filesIn" ref="handler" output-channel="channel1" />

	<bean id="handler" class="com.mycompany.integration.Handler" />
	
	<integration:channel id="channel1" />
	
	<integration:transformer input-channel="channel1" output-channel="channel2">
		<bean class="com.mycompany.integration.FileTransformer" />
	</integration:transformer>

	
	<integration:channel id="channel2" />

	<integration:splitter expression="payload" input-channel="channel2"  output-channel="dbChannel"/>	
	
	<integration:channel id="dbChannel" />

	<jdbc:outbound-channel-adapter
		query="insert into customer (customer_id, tax_id, name, adresse, city, state, zip, phone) values
		(:payload.customerId, :payload.taxId, :payload.name, :payload.adresse, :payload.city, :payload.state, :payload.city, :payload.phone)"
		channel="dbChannel" data-source="dataSource" />

</beans>

To test the application, you can run the jUnit test “testIntegrateCustomer”.

@Test
	public void testIntegrateCustomer() throws Exception {
		
		Collection<Customer> listCustomers = iCustomerDao.getAll();
		Assert.assertEquals(3, listCustomers.size());
		logger.info("customers number : {}", listCustomers.size());
		
		logger.info("wait .... ");
		Thread.sleep(2000);
		
		URL customerFileUrl = this.getClass().getResource("/input/customer.txt");
		File customerFile = new File(customerFileUrl.toURI());
		Assert.assertTrue(customerFile.exists());
		
		
		FileUtils.copyFileToDirectory(customerFile, new File(inputDirectoryPath));
		logger.info("test file {} copied to {}", customerFile.getName(), inputDirectoryPath);
		
		Thread.sleep(4000);
		
		
		listCustomers = iCustomerDao.getAll();
		logger.info("new customers number : {}", listCustomers.size());
		Assert.assertEquals(8, listCustomers.size());
		
	}


Spring Integration is a smart solution for application data integration. It provides support for the standard patterns typically used in enterprise integration. Spring Integration adds lightweight messaging and support for integrating with external systems and services using an adapter framework. Adapters enable a higher-level abstraction for communication with external systems and for Spring’s support for data and remote access as well as scheduling.

Reference

[1] http://www.eaipatterns.com/
[2] http://static.springsource.org/spring-integration/reference/htmlsingle/
[3] Dr. Mark Lui, Mario Gray, Andy H. Chan, and Josh Long, Pro. Spring Integration. Apress.

Posted in Architecture, Java, Technology | Tagged , , , , , | Leave a comment