TradeOffz

To Be Continued

2009-08-12T14:21:00.000-07:00

Web application configuration on Tomcat using Spring

2008-11-18T06:15:00.000-08:00

In order to get two sets of independent configuration (property) files, one for development and one for production, one can do the following:

In the web-inf folder place a property file, e.g. act.properties
In the applicationContext.xml define:

WEB-INF/act.properties
file:///${CATALINA_HOME}/act.properties

Birt versus Jasper Reports

2008-11-06T00:30:00.000-08:00

My experiences as a new user with Birt and Jasper.

Birt

A report grouped by week done within a one - two hours
Built-in support for group-by week (special function)
Adding JDBC drivers is easy

Jasper

I get hopelessly lost in the NetBeans based IDE. I do not understand how to use the query builder and the report builder.
Trying to build a report with group-by week is hard. No built-in support for weeks, just the standard java.util.timestamp functions. Trying to build an expression using day() results in weird validation errors (cannot cast from int to Integer()). With 1.5 you should not get this error anyway. Adding cast does not help. Bottom-line I do not understand the problem and the tool is not helping me to solve it.
NetBeans based IDE looks a lot sharper as the 3.0 classic IDE.
Selecting a SQL Server JDBC driver from dropdown result in classnotfound exception, you can select the driver, but it is not delivered with the product. You need to add your own drivers. Very confusing
A report needs to get compiled. Makes it harder to work with. Probably results in better performance but should be optional.

VB6 & REST

2008-09-11T12:05:00.000-07:00

I want to call a REST service from my VB6 Macro. This page describes how to do this using IE 6 COM objects, instead of needing to install the MS SOAP Toolkit (if it is still available)

It will probably also work for IE 7 since the difference between the two is small. Anyway something to verify.

In case the original page gets lost:

' Create objects to DOMDocument and XMLHTTP
Set objDom = CreateObject("MSXML2.DOMDocument")
Set objXmlHttp = CreateObject("MSXML2.XMLHTTP")

' Load XML
objDom.async = False
objDom.loadXML XmlBody

objXmlHttp.open "POST", AsmxUrl, False

objXmlHttp.setRequestHeader "Content-Type", "text/xml; charset=utf-8"
objXmlHttp.setRequestHeader "SOAPAction", SoapActionUrl

objXmlHttp.send objDom.xml

strRet = objXmlHttp.responseText

Set objXmlHttp = Nothing

' strRet contains result

Evaluation criteria Java Library

2008-08-13T06:58:00.000-07:00

Evaluation criteria I suggest:
- maturity of the product; years in service,
- size of the community, popularity among developers, documentation
- feature set and what is missing
- how easy to extend
- rate of new releases
- stability, performance, scalability
- integration with other 3rd party libs (e.g. Spring, Hibernate, ActiveMQ, GWT, etc)
- (commercial) support
- license
- Readability / ease of use of the configuration files

Partitioning

2008-07-03T13:19:00.000-07:00

Partitioning is a very import aspect of software development; you need to partition the application into several parts; modules, classes, units, whatsoever. Dividing the work among people is also partitioning. Choosing the right partitioning is about trade-offs.

It stroke me during my vacation that a landmass like Europe is also partitioned; it is divided into countries. It took a couple of thousand of years and a lot of fights, but finally we have a stable partitioning of land. In a way it is a self organizing system.

RBAC versus ACL

2008-04-03T08:06:00.000-07:00

RBAC stands for role-based access control
ACL stands for access control list

RBAC adds an indirection level - role.

Perfection

2008-03-27T09:22:00.001-07:00

Perfection kills innovation

Hibernate and BI

2008-03-16T05:18:00.001-07:00

Business Intelligence (BI) requires a star schema for efficient analytics. (the difference between analytics and reporting is that reporting requires you to know up front what you want to report and analytics allows you to do ad-hoc reporting). BI operates in most cases from a data warehouse based on star schemas to allow fast and time-based (historical) reporting. Reporting from the operational data sources would jeopardize the transactional/operational systems and in most cases does not allow for historical reports.

An ordinary application based on a persistence engine like Hibernate uses obviously an operational schema - making analytics hard. It would be nice to let Hibernate also update a data warehouse - similar to updating the indexes for text based retrieval.

So when a transaction is comitted, a message (asynchronously) is send to a component that updates the data warehouse. Each operation has a time stamp - allowing time based reporting. You could even use the data warehouse as a kind of trash bucket to restore data that has been deleted inadvertently.

To summarize; the application has three data sources; operational, warehouse and text-indexes.
An application in this way allows for text based search and analytics reports.

Tomcat as Windows Service

2008-03-05T06:44:00.000-08:00

To install Tomcat5 as a service follow the "Windows service HOW-TO" instructions on the Apache Tomcat side.

Calling Tomcat5 //IS//MyServiceName blabla will create an entry in the Windows registry located at:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MyServiceName

The new installed service when started does a call back to tomcat5.exe specifying the service name and the start parameter.

With tomcat5w //ES//MyServiceName you can monitor the service. Bit similar to the Windows Services application, but it has more functionality like specifying the log level and log location.

In my case the service did not start and I had an error in the log:

[2008-03-05 15:55:24] [443 javajni.c] [error] FindClass org/apache/catalina/startup/Bootstrap failed
[2008-03-05 15:55:24] [997 prunsrv.c] [error] Failed loading main org/apache/catalina/startup/Bootstrap class
[2008-03-05 15:55:24] [1260 prunsrv.c] [error] ServiceStart returned 3

So a classpath problem. I re-read the instructions and I noticed that the remark " If using tomcat5.exe, " put me on the wrong foot. I thought service.bat applied to versions prior 5. Appearently this is not the case because running service.bat worked fine. Looking into the content of service.bat I noticed that there is some classpath manipulation and after installing the service, the service is also updated with JVM options so this way is superior as to the excerpt installing Tomcat5. (In hindsight I completely misinterpreted this)

Deleting the service using //DS removes the entry from the registry.

GWT & object serialization

2008-02-29T00:29:00.000-08:00

When writing GWT business applications you have the problem that the object graph resided at the server has to be partially send to the browser/client, (perhaps) updated and send back.

A though problem to crack is how to break the graph in pieces. After all, we do not want to send the entire database from the server to the client. This is similar to the problem that Hibernate solves when communicating with the database. Hibernate will load on-demand needed objects from the database. Hibernate will also optimize the updates to the database.

I think it is key to have something similar to Hibernate for the GWT arena. Instead of manually loading additional objects a framework should handle this.

A key difference is that (in most cases) the Hibernate instance is a singleton, thus allowing caching techniques. Caching at the browser is tricky since other clients can have updated the data.

My hope was that a framework like XSTM would solve this problem. The sad things it that it uses code generation. I'm not in favor of code generation and especially not when it involves the domain objects. Reason is that it makes the usage of the library very intrusive through your entire application. Advantage of code generation in this case instead of byte code manipulation is of course that it runs in GWT.

Anyway when I look at the sample the objects replicated are stored in a share, which is a generic container and thus requiring a lot of casting. My idea that you could retrieve objects with a query kind of language is not the case, you have to manually put in the share and get it out. So this solution will work well for games I suspect - but not for business applications with huge databases. In this case you have to write code at client side to notify the server which objects to put in the share. Again plumming that I hoped the framework would fix. Looks like that XSTM requires an additional layer of abstraction on top of it that will make it hide the clouds/shares and just work with regular objects.

All problems in computer science can be solved by another level of indirection

2008-02-18T03:50:00.000-08:00

Finally found the guy(s) whom the quote is from.

Butler Lampson and/or David Wheeler

Cross-region software development

2008-02-16T13:44:00.000-08:00

At my company we have been struggling finding out the optimal partitioning of the source tree of a product that is developed in three different countries (US, India and NL).

Some people prefer to have a single source tree. Others (among myself) prefer to have a stricter separation between the teams in different locations and time zones. One of the major reasons is that if the build is broken or disfunctional caused by team A, it impacts the productivity of team B and C. The developers of team B or C enter in the morning and face a build in which they cannot check. First the failure has to be solved before continuing regular development. Admitted, this can be done by a single person, but still the broken build impacts all the developers of a team since they can't check-in and might have updated from source control before realizing something was wrong. Since a team performs optimal when it is running at constant speed - similar to a car's engine - you need to avoid peaks and breaks.

Another reason for keeping sources separate is that you want to avoid mixing integration and regular development cross teams. Otherwise you cannot isolate external factors easily. It is much more efficient when you have accomplished your work of team development, all tests are passing and then integrate new releases of other teams or 3'rd party artifacts. If something fails, the cause is much easier to spot then when you also have changed your code.

Lesson: separate daily development from integration. We effectuated this (or tried to) by giving each team and the application its own Maven artifact. Admitted, you need Maven mavens to get the whole thing working. (BTW we never got to the point where we could fully demonstrate this Maven approach to be more efficient, in the past we used branches)

An often heard counter argument for having separate source trees is the price you have to pay for refactorings, especially renaming classes / methods. This can be solved in the following way: at the end of an iteration, the teams have released their artifacts and hopefully the product is working with all the parts integrated. So the start of the next iteration is an excellent time to get all the source code into a single view / project and refactor some elements. Yes, indeed you do have to wait. That is a small price to pay. However note that for really big products, getting all the source code into a single project probably makes the IDE too slow. This is something you do not want to work with the whole day. Besides renaming there is also a refactoring like changing the signature. The same can be accomplished with deprecating and introducing new methods.

Another mistake often made is the focus on compilation - which can easily be accomplished with a single source tree. IF the code compiles together then we are OK. But compiling is only just a small part of creating the product, much more important is to have a stable product and you know what is in it. That the code compiles together, does not mean a thing. It gets much better if the artifacts have passed their unit tests, are released and composed into a new product - that should pass some automated acceptance test.

There is much more to this discussion, I will have to add that another time.

Panta rhei

2008-02-16T03:50:00.000-08:00

Everything flows - Heraklitus.

Well known fact; when you write down the specs, the world is changing. By the time you've finished the implementation the world is no longer the same. This why we all went agile.

There is another side to this; with an agile approach you take shortcuts in the first iteration. Most simple thing you can do, blabla bla. However a couple of iterations down the road, you forgot you took some shortcuts. People start cursing the whacky implementation, forgetting that this was a deliberate choice and now the time has come to crank up the implementations.

Iterations also pose a challenge for QA. In my experience they are still used to the waterfall process. It is hard for them; each iteration new functionality is delivered but still some things are not (yet) fully functional. The deliverable of each iteration has to be carefully defined to enable them to properly test the deliverable. However, how much time you are going to invest writing a detailed document that is obsolete after the next iteration? Worse, how are you going to get the full spec after all iteration are finished? How efficient are UI designers if they know that each iteration the UI will change? Do they start with a wizard or tab-based approach because they know the next iteration requires more widgets?

There is some kind of impedance mismatch between the development process (iterative) and the product as itself. As usual a trade-off.

Scala

2008-02-12T02:04:00.000-08:00

I've been reading the “Scala for Java Refugees” series. I'm amazed by the power of this language. Tuples, closure, etc, everything that annoys me about Java is 'fixed'. The language itself is a bit overwhelming, reminds me of university with its magical and impressive calculus notations.

Reading on the implicit type conversion in serie 6, I got similar feeling when reading on AOP for the first time. It might become complex to understand the code, especially in the case of Scala where a smart compiler can find paths the developer has not thought off.

Probably the answer is the same as with AOP, use it scarce, wise and effective. In other words not all over the place, but where it pays off, eg security, transactions, all those clear cross cutting concerns.

I'm impressed by the expressive power of the language. On ScalaTest the expect construct is close to natural language (ok, a bit exaggerated). The freedom it gives to clearly state the intent, is something you never get with Java.

JODReports and MS-Word

2008-02-07T11:49:00.000-08:00

I finally implemented my document generation application using JODReports. The basic idea is as follows; user creates a bunch of templates using the JODReports variable mechanism and stores them in a certain folder. My application picks the templates up, converts to OO format, merges it with the data from the database, converts it back to Word and writes the files to a certain folder.

Works pretty good so far. A couple of issues I stumbled into:

Editing the Word document by pasting a variable name from html (in my case the instructions page in the browser) gives weird Freemarker parse exceptions. Pasting as plain text workarounds the problem.
In the supplied merge data the zip code field is separated by two spaces from the city name. However the final merged document contains one space. This is also the case if the data contains three spaces...
Default error handling from Freemarker prints out to the console. There is documentation how to fix this, but requires more research.
JODReports code is not well maintained. The code of the converter and the report are out of sync (watch the package names). Luckily this does not pose a problem when using it.

It is a bit of a pity that JODReports does not support callbacks. Now you have to pass in all data up front. It would be nicer if JODReports would do a callback asking for the value of a field. Reason is that I need to supply a couple of calculated fields. Now I have to supply them all up front, whilst perhaps none is used.

Java Bug Date equals?

2008-01-29T11:45:00.000-08:00

Who would have thought that the following test would fail:

private Date getDate14April() {
Calendar calendar = Calendar.getInstance();
calendar.set(2007, 3, 14, 15, 30); // 14 april 15:30
return calendar.getTime();
}

public void testCompareDate() {
Date date = getDate14April();
for (int i=0; i < 100; i++) {
assertEquals(getDate14April, date);
}

Stacktrace (edited brackets):

junit.framework.AssertionFailedError: 60 expected: [Sat Apr 14 15:30:40 CEST 2007] but was:[Sat Apr 14 15:30:40 CEST 2007]
at org.westen.act.document.TestTemplateManager.testCompareDate(TestTemplateManager.java:212)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

JavaDoc of Date.equals:
Compares two dates for equality.
The result is true if and only if the argument is not null> and is a Date object that represents the same point in time, to the millisecond, as this object.

Hmm, I would have though that the milliseconds would have been initialized to 0 when I create a date object.

Hibernate many-to-many & inverse

2008-01-29T07:47:00.000-08:00

I had the problem that two entities A and B had a many-to-many relation but deleting an object a resulted in a constraint violation because the object was used by an object b.

To my surprise fiddling with the inverse="true", inverse="false" and cascade="none" fixed the problem.

My understanding is that deleting an object always deletes the row from the join table. (cascade="none"). Setting cascade to "delete" will also force the deletion of the object(s) on the other side.

However why inverse="true" and "false" make a difference I do not understand. You would assume that specifying one would be enough and imply the other. Apparently there is more to it than I understand. Needs a follow up...

So I'm not the first one to run into this. Found this blog "Hibernates bizarre interpretation of inverse". He basically says that inverse tells Hibernate which side of the relation to ignore. Although this does make complete sense to me, it fits my problem.

Hmmmm

GWT-EXT

2008-01-19T12:36:00.000-08:00

The last few weeks i've been struggling with GWT-Ext 1. Conclusion: the lay-out capabilities of GWT-Ext 1 are poor.

For example a table-layout is not possible. When using a form you can only place a limited number of widgets on it. Separate label widget is not present.

Ext v2 has much richer lay-our support. So we have to wait for GWT-Ext 2....

Writing a test to verify ODT content

2007-12-19T07:55:00.000-08:00

Sample to test the content of an Open Office document containing three lines ('1234', empty line, 'description'):

import javax.xml.parsers.DocumentBuilderFactory;
...
import com.artofsolving.jodconverter.DefaultDocumentFormatRegistry;
import com.artofsolving.jodconverter.DocumentConverter;
import org.w3c.dom.Document;
import org.w3c.dom.Element;

ByteArrayOutputStream output = templateManager.applyModelToTemplate(inputStream, map);
String content = getZipEntry(new ByteArrayInputStream(output.toByteArray()), "content.xml");
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
Document doc = dbf.newDocumentBuilder().parse(new ByteArrayInputStream(content.getBytes()));

Element docElement = doc.getDocumentElement();
assertEquals("office:document-content", docElement.getTagName());

Element bodyElement = (Element) docElement.getElementsByTagName("text:p").item(0);
assertEquals("1234", bodyElement.getTextContent());
// skip empty line
bodyElement = (Element) docElement.getElementsByTagName("text:p").item(2);
assertEquals("description", bodyElement.getTextContent());

Getting the entry from a ZipInputStream

2007-12-13T09:01:00.000-08:00

Finally I found the code to get an entry from an InputStream that contains zipped data. For a ZipFileInputStream this is easy, but not for a regular input stream because the size of the entry can be unknown (-1) and there is no alternative then just go through the stream.

sorry for the messy mark-up...

disclaimer: code is not optimized!

// based on http://java.sun.com/developer/technicalArticles/Programming/compression/

public static String unzipEntry(InputStream zippedInputStream, String entryName) throws Exception {
String result = null;
final int BUFFER = 2048;
BufferedOutputStream dest = null;
ZipInputStream zis = new ZipInputStream(new BufferedInputStream(zippedInputStream));
ZipEntry entry;
while((entry = zis.getNextEntry()) != null) {
int count;
byte data[] = new byte[BUFFER];
StringOutputStream fos = new StringOutputStream();
dest = new BufferedOutputStream(fos, BUFFER);
while ((count = zis.read(data, 0, BUFFER)) != -1) {
dest.write(data, 0, count);
}
dest.flush();
dest.close();
if (entryName.equals(entry.getName())) {
result = fos.toString();
}
}
zis.close();
return result;
}

public class StringOutputStream extends OutputStream {

// This buffer will contain the stream
protected StringBuffer buf = new StringBuffer();

public StringOutputStream() {}

public void close() {}

public void flush() {}

public void write(byte[] b) {
String str = new String(b);
this.buf.append(str);
}

public void write(byte[] b, int off, int len) {
String str = new String(b, off, len);
this.buf.append(str);
}

public void write(int b) {
String str = Integer.toString(b);
this.buf.append(str);
}

public String toString() {
return buf.toString();
}

public int contains(String string) {
return StringUtils.countOccurrencesOf(buf.toString(), string);
}

}

Pairs

2007-12-13T00:32:00.000-08:00

While driving to work I realized something funny:

If I have to work on something, for example produce an article, I will write it, put it away and pick it up a couple of days later to look at it with a fresh mind.

Within our team we also (sometimes) practice pair programming. The idea is that two know and see more than one (the power of interdependence ;-).

Effectively these two practices are one and the same; in the first case you use yourself as the second person. Time will make you a different person than you were. The two practices differ in their usage of the dimensions time and resource. Tradeoffzz..

Red/green/refactor & spikes

2007-12-10T02:50:00.001-08:00

I've developed a new way of coding new functionality. In the past I distinguished between spikes and production code development. The spike was meant for prototype code, throwaway code.

Today I do it like this:
I start with a new test that has some basic code that will do the basics of the job i'm after. I can very easily run this test from my IDE (IntelliJ in my case). After a while the test(s) will succeed. This proves that I understand the basics of the code needed. Next step is to move some code to the main folder - to production level. I will use Extract methods refactoring to do this.

This way I slowly, but steadily move from prototype to production without redoing work.

The Inner-Platform Effect

2007-12-07T01:00:00.000-08:00

The Inner-Platform Effect anti-pattern is a nice description of what I try to avoid with document generation. The task of making a document template becomes so complicated that only an expert can use it.

In a way it reminds me of XP (extreme programming); keep it as simple as possible. Wait till the latest moment adding flexibility till is really needed.

Another analogy is the J2EE / EJB programming model; programmers should focus on writing business logic. However tying the whole thing together on an application server like WAS was a nightmare and required more expertise than solving the original business problem.

RTFTemplate

2007-12-06T08:15:00.000-08:00

RTFTemplate is the Java library that does what i was looking for; it allows replacing MS-Word mail merge fields by actual data on the server side. So users can still define their document in the traditional sense using mail merge, save it as RTF and the server use it as a template to generate new documents.

The issue however is how scalable this solution is in the long term; for example how easy is to add pieces of text to documents? Converting to PDF, which is necessary in my case, is also still not solved. Using Open Office as a templating engine would solve this. However running OO as a (headless) service complicates things; it is an external process, i'm not sure about memory leaks and/or how robust it is.

Another con of using OO is that - since Word will still remain the document editor for the user - the fields have to be typed in as regular test with some markers, for example ${lastname}. This makes it a little less robust, typos are easier made, deleted by mistake, control characters within the field.

OpenOffice API sample

2007-11-27T10:52:00.000-08:00

A sample on how to use the OpenOffice is described on this blog. It gives the impression that it is not headless (perhaps my mistake). I know OpenOffice can run headless.

PDF Export in Java

2007-11-26T02:48:00.000-08:00

This site and Open Directory site mention a list of available Java libraries for PDF export (commercial and open source).

Alfresco uses PdfBox, POI and OpenOffice for its document mgt. (I suspect OpenOffice for the PDF conversion)

WebDav

2007-11-25T09:52:00.001-08:00

I did not realize that you could mount a webdav folder as a drive on your computer. For example WebDrive does this. According to this article even XP SP3 can do it out-of-the-box (SP2 has some issues, see here). This makes remote documents easy to understand for users. (In my pet applications the users are used to shared drive for all their documents. Versioning? They use backup & restore ;-)

I need a simple WebDav implementation that redirects to a folder on the server. This way I can easily link to files from my web application. My hope is that users can also save the files they open this way (should be the case). Perhaps I will use Tomcat's webdav sample application, although I need to direct the content to/from another folder as the web application's root - which is not that easy. According to this blog it is possible using a JNDI factory.

Instead of Tomcat I could use a library like JackRabbit or an application like Alfresco, but they are too heavy. (btw in my experience you should be very careful integration another application into your own). Missing using Tomcat's implementation would be a search engine - which is supported by the two above, although i'm not sure whether JackRabbit would index Word documents out of the box. (btw I do know that JspWiki can search through Word Documents using Lucene)

Executive UML

2007-11-23T13:06:00.000-08:00

I do not believe in UML that can be executed one day. Perhaps for a very small, well defined domain it might work - and then only for the first 20%, the rest, the tweaking, small UI enhancements will require manual coding. Hopefully in a language that is powerful enough and does not require tons of code. (Java is an example of how it not should be done, with the writers, outputstreams, etc. You have always to pay the price for flexibility, certainly not a good trade-off)

Alternative:

From the requirements you distill the concepts. For example a requirement could be: "as end-user I should be able to add a contact person". Contact person would be a concept.
The requirements would be translated in automated acceptance tests in terms of these concepts. The automated acceptance test would talk to an API. (call addContactPerson and getAllContactPersons). Next step would be to translate these tests to UI level - probably requiring human interaction.

Obvious benefit would be that you could change the implementation and still verify whether the requirements are met or not.

Modern Times

2007-11-23T07:03:00.001-08:00

Ain't it weird: we fear privacy intrusion, but everybody is writing blogs. We dislike traditions like female genital mutilation, but women these days do surgery on their body everywhere.

Word 2003 docx format - mail merge

2007-11-23T06:01:00.000-08:00

Hopefully i'm doing something wrong, but look at the screenshot to the left that displays the document.xml of a docx when a mail merge field has been defined. Notice the use of begin and end instead of making use of the hierarchical nature of xml.

But it gets even weirder; if you define a document in Word (i'm using 2003 with the docx plugin) with the mail merge fields and you do a preview before saving the document then the document.xml looks like this:

The xml hierarchy of the last one makes sense, but the first one definitely not. Why the two are completely different, i've no clue...

After doing some research I found out that others have run into a similar issue. See the hyperlink discussion on this blog.

Partially copying XML using XSL

2007-11-20T04:57:00.000-08:00

Subject: Sample xls that copies entire XML source tree, but skips an element as specified in the xls.

Note: the indentation is messed up due to the xml characters used.

The source xml file includes a link to the xsl for display purposes (xml and xls as stored in the same folder):

<?xml version="1.0" encoding="ISO-8859-1"?>
<?xml-stylesheet type="text/xsl" href="display.xsl"?>

<data>
<employee id="100">
<lastname fooAttr="123">Basta</lastname>
<insertion>van</insertion>
<firstname>Marco</firstname>
</employee>

<employee id="102">
<lastname fooAttr="123">Kruif</lastname>
<insertion></insertion>
<firstname>Jan</firstname>
</employee>

</data>

The xsl file (called display.xsl), the employee entry with firstname 'Marco' is skipped:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="*">
<xsl:copy> <xsl:apply-templates/> </xsl:copy>
</xsl:template>

<xsl:template match="/data/employee[firstname='Marco']"/>

</xsl:stylesheet>

Convert Word document to HTML

2007-11-19T03:39:00.000-08:00

I've found two ways to convert a word document in docx format to HTML, both using XSLT.

Sharepoint's DocX2Html.xsl, no idea if it is open source
xslt on OpenXML

This would make it possible to do some kind of mail merge preview, replacing the mail merge fields and showing the output both using XSLT.

Similar topic, but different implementation (not open source) from Softinterface. Not sure how they get this working without installing any additional software. (perhaps they use the OLE object's preview for images, but how about the entire Word document?).

Architectural Quality Attributes

2007-10-31T01:23:00.000-07:00

Two quality attributes to which an application architecture should adhere:

I) It is important that an application architecture can accommodate new requirements without big changes in the implementation. Changes should be as local as possible, keeping the impact small. (not everything can be tested in an automated manner).

This means that the delta in requirements should be linear towards the delta in implementation changes. I call this an architectural quality attribute. (see also book Software Architecture in Practice)

II) An architecture should have one way of doing things and allow for exceptions.

For example an architecture that has two different ways of sharing information, for example through a remote service call or storing information in a database, does not adhere to this quality attribute.

Using EasyMock2

2007-10-17T11:19:00.000-07:00

The recording and playback phase of EasyMock has always drove me away from it, but I got sick of implementing mocks myself, so I gave it a try.

The simplicity is appealing. The record and playback is not as annoying as I thought it would be. However what is a bit frustrating is that i'm not interested so much in the collaboration between the class-to-test and the mock, but just in the-test-to-class. If i'm not mistaken; I want to use the mock just as a stub.

Example: i've a component that uses a dao. I want to mock the dao. For example return a fixed collection of items for a particular method. This is easily feasible with EasyMock.

Other example: i've a component that uses a HttpServletResponse. My component sets the header, content-type and some other stuff. I'm only interested what is written to the response. So I do not want to implement the setContentType and other methods. However afaik you can't tell EasyMock just to ignore calls for all methods - except for the ones you have specified during recording.

Objects have identity, components don't

2007-09-16T01:38:00.000-07:00

A couple of rules:

Objects have identity, components don't.
Components have an emphasis on behavior, objects on state.
Object can expose methods to expose state and validate state.
Components with state are considered harmful - but sometimes inevitable
Events in components should always make it explicit if they are before or after the fact occurred. To make it explicit in which state the component is and not in some kind of transition
Services are components designed for remote access. (coarse granular)

To me, a service is instantiated and accessible at a certain physical location, however this does not match with the situation where a service is deployed in an embedded way. Mmmm.

Data Duplication - Source of all Evil

2007-09-16T01:22:00.001-07:00

Several disciplines in the software world have their means to deal with data duplication; database guys use normalization, programmers use refactorings like extract-method. The goal is to have the definition/specification in one place to reduce the maintenance burden.

Other disciplines like data warehousing don't care about the data duplication because they do not have to update the data, only to add.

The problem in programming is that it is hard to detect duplication. (especially when we span the dimensions; large & multiple teams + time). There are known algorithms to detect duplication (eg IntelliJ provides something like this). However I question; do we want to remove all duplication? If there is a single implementation in a large system; won't it become impossible to change the implementation just because doing the impact analysis will take a very long time?

In this case of components it will be hard to prevent duplication. Take for example a simple method that replaces all double quotes with single quotes (one Java 1.5 call). This method occurs in two different components. The cost of extracting this (and thus introducing a new shared component) does not weigh up against living with the duplication.

Another killer is of course 'semantics'; although the implementation is identical; are the semantics identical? Determining this won't be always easy - especially when another guy wrote the other component.

Fact; we have to live with code duplication. It is inevitable.

Programming languages with build-in support for unit testing

2007-09-14T01:51:00.000-07:00

How many times have you made a private method package local just for unit testing? Each time you have to add some comment 'package local for testing'. Annoyance!

Java's successor must have some kind of access descriptor dedicated for unit testing.

'Class names in Plural' smell

2007-09-12T07:06:00.000-07:00

Just ran into a piece of code where somebody was using a class that was in plural, like for example Computers. Well, that smells. If you have a collection/list/bag/set/whatever of computers, then use the appropriate collection class. Since this was not the case probably more is going on. This is a collection of items with a certain characteristic that they share. Probably a bunch of computers forming a domain with additional properties that rise above the collection level. I'm arguing that you should use a class name that reflects the item(s) in common - most likely in singular.

svn ignore for multiple folder / directories

2007-09-06T03:24:00.000-07:00

I was struggling with the svn:ignore property to specify multiple folders. This is how I solved it:

set the environment variable EDITOR. I'm using ultra edit:
>set EDITOR="C:\Program Files\UltraEdit\uedit32.exe" (note the quotes)

Type in (note the period at the end, indicating the current folder is the target)
>svn propedit svn:ignore .

This will open the editor, UltraEdit in my case.
Enter the folder names separated on each line
Save and exit the editor.

That's it. Use svn status to see if indeed the folders are ignored.

Import / Export Database from / to XML

2007-09-05T06:35:00.000-07:00

I'm using Hibernate to manage my database persistency and I wanted to add export/import to/from XML capabilities. Hibernate does support XML but the documentation is very limited. Next, it is not capable of generating a schema (xsd). XStream has the same problem. So that left only JAXB. Issue with JAXB is that it is very document (web service) oriented. The HyperJAXB - project uses the web services xml as a starting point. I want it the other way; my domain classes are the source of truth; they form the basis from which I want to persist and export/import.

Since i'm neither a JAXB nor a Hibernate expert it has proven to be quite an exercise. Especially Hibernate's merge operation in combination with Spring's IdTransferringMergeEventListener caused a lot of headache. I did not fix this properly, but replaced the merge() with saveOrUpdate().

Anyway the basic steps to export/import a database (aka repository) are as follows:

Create a container class that contains collections of the top-level domain classes. My sample is based on two domain classes Person and FiledCase. The class Repository contains two collections of these classes. The collections are extensions of a special HibernateCollectionAdapter class. This class takes care of loading from and persisting to the database for a domain class. The container class is passed to JAXB for import or export
Annotate the database id field in each domain class with @XmlTransient, we do not use this field since it is not guaranteed to be unique across all classes. (if it would be you could use this field as xml id)
Add an xml id field to the domain classes that can be referred to. Annotate the get method with @XmlID. The get method should return the fully quantified classname appended with the database id. (to make the ID unique across all classes)
Annotate the classes that needs to be exported with @XmlRootElement
Mark the get methods that refer to other objects that are not aggregated, but have an association type of relationship with @XmlIDREF
Make sure that the classes are exported in the order of least referenced, use the annotation @XmlType with field proporder to specify the order of the properties. This is very important during import. JAXB will patch references of objects afterwards when they cannot be resolved immediately (because the object is not yet imported). However when the object is already persisted it will not reflect the updated references.
The HibernateCollectionAdapter class implements the Collection interface. In the add() method it will store() an object and in the contains() it delegates to a find() method. The store and find methods are overridden in the descended classes. There are a couple of other methods that should also be overriden.

Sample is uploaded, see comment below

Controlling Change Requests

2007-08-25T11:16:00.000-07:00

One of the challenges facing the software delopment process is controlling the number of change requests (CR). You got tons of exotic enhancement requests that will never make it into the product. Next, you have enormous amounts of small bugs that should be fixed. Among those thousands CRs a bunch is already fixed and it is too expensive to find out which ones are fixed.

I plead that enhancement requests older than 2 years (2 major versions) are removed. Apparently they are not important enough. If they are important, they will come back.

Another recommendation: when doing triage (deciding if and when the CR will be fixed), the corresponding code should be annotated with the CR number. This will make it easier for the programmer to see which CRs are applicable to this part. He/she can decide to fix the CR as well, update the CR text, or perhaps even close the CR as already fixed.

Today's bugtracking systems integrate with version control systems enabling you to see which CRs is fixed in which piece. This should be improved so you can see where future CRs reside. This will enable you to see which parts are the weakest.

Your own framework

2007-08-19T10:31:00.000-07:00

Your own framework that is gonna solve all the problems. It is going to be faster, easier to understand, more robust and lightweight.

There is one problem; it takes time. Probably about 3 major releases to get it right. And guess what happened after 3 years of hard work? A new technology has arisen making the existing technologies obsolete.

Take Lucene - i've been told a very elegant framework. There is new major v5 on the horizon to get everything right. Problem however is that frameworks like GWT and J2S solve a core problem; easy way of writing client side code.

Lesson: as a company don't write your own frameworks unless it is core business. Even then take into account that it is going to be out-dated in 5 years.

unit testing

2007-08-14T11:14:00.001-07:00

Unit testing aka white box testing means you write a test that is aware of the implementation. It means that you can write fine grained tests specific to a set of requirements. Integration tests (or whatever you wanna call them) test multiple artifacts cooperating together. Theoretically a unit test tests the implementation of one method and nothing else. However this distinction is hard to make when the tests evolve.

Suppose you start with two different test folders: unit-test and integration-test. First couple of days everything is working nicely. One day you decide to refactor a piece of code. Luckily you are disciplined enough to move the tests along with the code to the new packages. However one of the methods that was white box tested had a piece that was extracted to a new method (let's say within the same class). Now suddenly this test you had written is no longer testing solely one method. Hmmm.

Thinking of this; if I write a unit test for a method and this method calls a couple of JVM routines, is it a unit test or an integration test from the beginning?

Point i'm trying to make is that if you want to be really strict about separating unit and integration tests it takes an enormous effort to keep it sound. With large teams - people come and go - in time it will be an impossible task.

2007-08-09T02:27:00.000-07:00

In the last couple of years software code and architecture moved from being porcelain to rubber, but it still not fluid. We need fluidity; we need some kind of dynamic flow that will re-establish balance within the code. For example some packages or classes have become too large, obsolete code that needs to be removed. This is now all manual labor. Organism have something called homestase , a mechanism to restore the internal equilibrium. The premise is already fulfilled; we have the metrics, we have refactorings. Only some kind of automation is needed to put it together. Might be a little be odd to come into the office in the morning and to find out you're whole application is turned upside down.

Unit testing

2007-08-06T11:14:00.000-07:00

After a couple of years of unit testing I still haven't found an elegant way to make sure that a method calls another method. For example I want to test method x that calls method y. Method y ensures a business rule, for example it checks whether certain reserved characters are not present in a string that is passed as parameter. I can do some litmus tests against x, indirectly testing whether it calls y. However this gives not a 100% guarantee. In fact, it introduces an extra dependency. If the business rule change, y changes and i've to update some tests of x...
Another approach is of course to use mocks. However this requires interfaces, factories/ioc, etc. A lot of overhead for a simple test.

A futuristic alternative is to define an aspect that checks that at least one method called from x is y. Admittedly, I've no idea how to implement this, but it won't be easy / flexible to do in a TDD manner.

Another desperate attempt; if y would throw an exception in case of failure, you could check the stacktrace. Obivously, still brittle.

So the question remains how to solve this. It gives me an unpleasant feeling that such a basic problem is not yet solved. Perhaps someone else has a bright idea.