Waxing Cerebral

Microsoft. Outlook. It sends attachments in its own weird an wonderful format as a single file called winmail.dat, for those of us fortunate enough not to have Outlook as our email client to unpack them. There's an opensource project, TNEF which provides a tool that unpacks these well

1. Download the tnef tar
2. Untar it (tar -xf .., or just double click)
3. Follow the README steps (note the make install needs to be run with sudo)

JAXB extension plugins (1) give the user more power than simple binding alterations, plus can encourage clean schemas by keeping the code extensions out of the xsd's. But finding up-to-date documentation on this can be a pain, most predates maven.

In hindsight, the process is actually quite simple and there are some good references, but it still takes collation of quite a few sites to make it work. Here I hope to capture them all for a cohesive approach.

Note that 'plugin' is a ubiquitous term and here can mean many things. For disambiguation I've added a postscript number to each use of the term. See footnotes at the bottom for meanings.

JAXB runtime

This post assumes you're using maven. And if so, the JAXB plugin (2) to use is maven-jaxb2-plugin. Add the latest version to your pom http://mvnrepository.com/artifact/org.jvnet.jaxb2.maven2/maven-jaxb2-plugin. Then get your schema generation working OK as documented plenty of places elsewhere on the web. If you're looking at this page then you're probably well past that step anyway.

Plugin project skeleton

Create a new maven project to hold your plugin (1). For this example we'll call it my-jaxb-plugin.

In the pom add a dependency on jaxb xjc, eg:

Create a new class, call it MyPlugin1 in package com.eg, extending from com.sun.tools.xjc.Plugin. For now just implement the required methods returning "XmyJaxb" from getOptionName(), and implement run() as System.out.println("***working!"); (or similar :).

Now create new folders META-INF/services on the classpath (eg src/main/resources). Add a text file in there called com.sun.tools.xjc.Plugin.In this file is a list of all the plugins that this project provides, one per line. So for now add your new plugin class:

Run a maven install on the project to make it available (not needed later if you use workspace resolution with eclipse maven plugin (3))

Employ the plugin (1)

Maven build your JAXB project and your should see in the console output your ***working! comment. Congrats
The -debug arg can be removed later as needed. You may also find it useful to use -X after your mvn install command to get verbose output from the maven build.

More

For taking your plugin (1) further, read the key post from 2005, resources incl javadoc for jaxb-xjc, and the sun codemodel javadocs which you'll be using to modify the code generation.

Footnotes

There's the obvious choice of client drivers in your language of choice for application-based mongodb connections, but for server-side admin and scripting, js is best. But from the shell and cron, often they get wrapped in shell scripts. This table-esque post outlines some of the decision points to use when approaching this, plus some guidance as to how to write them.

Interactive shell

--eval

• see printjson

js file

eval + js file

All within one sh script

• Cannot be run inside an --eval
• From mapreduce docs { inline : 1} - With this option, no collection will be created, and the whole map-reduce operation will happen in RAM. Also, the results of the map-reduce will be returned within the result object. Note that this option is possible only when the result set fits within the 16MB limit of a single document. In v2.0, this is your only available option on a replica set secondary.
  
  
• Queries (such as count(), sort()) cannot be run on inline results (as of mongodb 2.1+ new aggregation features may allow this). For versions <2.1, seeing as cannot run these queries on inline results, and can't have temp collections on secondary servers, have to run these on primary server so can save to a temp collection and run queries against that.

TurnKey is yet another Linux distro, but it's come a long way since I last checked it out. Initially providing images for LAMP and 2x CMS stacks only, it now boasts nearly 50 different appliances, with an impending launch of up to 150. There are now commercial options for managed cloud deployments as well. Nice offerring

Migrate repositories using the eclipse subclipse plugin.
Note this method will not bring revision history into the new repository, so assumes you will still have ongoing access to the old respository. This technique is useful if the history isn't that rich, and you want a quick switch over, or if you don't directly manage the repositories, so svnadmin commands aren't available (like our case, we use a hosted JIRA solution).

Steps:

1. Create the new project top-level folder in the new repository. Eg: /NEWREPO/projectA/trunk. Using 'new remote folder' in SVN repositories view is the easiest.
2. Open the existing project in eclipse
3. Choose team> switch to another branch/tag/revision.
4. Choose intended top-level project folder (eg /NEWREPO/projectA/trunk). Note this folder name doesn't have to become the eclipse project name
5. team> revert the project to remove local changes
6. team> merge (Install into eclipse the fantastic CollabNet Merge client, if haven't already)
7. Untick best-practices check and choose 'Merge range of revisions', then Next
8. Select, then under Root, browse to the exisiting project folder (original repo) (eg /OLDREPO/projectA/trunk)
9. Commit the project

Using top, ps aux, etc etc gives all kinds of memory usage with lots of blog posts and forum questions explaining how difficult they are to understand, and how typically aren't actually that useful.

As of kernel 2.6.14, smaps has been introduced which means each process reports its own memory usage breakdown.

smaps detail for a process can be seen by:

where PID is the process ID of the process you're interested in. Note you won't see any content if not the process owner or root.

So, use top, find the big memory hoggers first. Then look at the smaps file for the swap usage. You can use, for example:
where 30888 is an eg PID.

For more advanced smaps analysis, try the smaps.pl tool. Not sure at this stage, though, whether that tool gives separate swap values.

In Debian at time of this post, the pmap tool has been patched to use smaps, which may then offer swap values in its output.

FitNesse suites can grow enormous, so managing them becomes tough. Here's a simple script to find all pages in a given suite that are not marked as either a Test or as a Suite:

Running Python scripts via launchd doesn't seem to flush stdout print statements (ie. using the basic 'print' command). This can be confirmed by importing sys and calling sys.stdout.flush() after the print statements to see them actually output.

The solution? Add the -u flag to tell python not to buffer its stdio.

In an OSX launchd plist file, your solution may include:

I have just had a new Mac Mini arrive that I want to use for mostly automated and headless tasks. But I also want to use this as a home computer for my wife, but it has no screen and no mic, plus not much room on the bench where it lives to fit a keyboard and mouse. My wife has a decrepit (no battery; no harddrive) laptop she's been using, so hey, why not use that as a terminal?

Laptop is running puppy linux (4.3.1). Great little lightweight distro. It boots off CD and uses a USB key for storage.
- Installed tightvnc-client
Mac is running Snow Leopard

So by setting up the script below as a menu option (add to ~/.jwmrc), all she has to do is click it, and the connection begins, including sending all mic input sound (via dd with /dev/dsp) through an audio cable from laptop's headphone jack to Mac's input jack, registering as input on the Mac (Skype-ho!).

Beauty

[caveat: I cannot be held responsible for damage to hardware caused by plugging hardware into each other. Headphone jacks are pretty low voltage though, as I understand it, and it's worked for me]

Currently I'm on a project that uses an Atlassian-hosted JIRA Studio. We put scripts and config files as wiki page (confluence) attachments, then needed to access them from remote machines via the command line, even as part of a script. JIRA has it's own security methods, not simple http auth, which would be a piece of cake. So, another solution was needed.

A page in the JIRA Community Space describes how to do this using wget with cookies. But it didn't work for me. Whether this is because we are using the hosted variant, or due to changes since that page was written, I found we had to access a different url to log in.

So here's a current working solution that firstly interactively prompts for username and password to log in and creates a cookie:

Replace OURDOMAIN to make the url relevant for your site. This needs to be run only once per machine, at least until the cookie expires. The cookie file will be saved in your home dir.

Next, to retrieve any files, use:

Again replacing OURDOMAIN. Run this for each file, pasting the full url to the file to retrieve when prompted.

From these you should be able to see how to strip out the interactivity to make them scriptable, too.

It happens sometimes, committing something in svn then realising the comment was wrongly for something completely different, or the main thing was missed out. Here's how to change it.

If you don't have the repo configured to allow post-commit changes (as per; it is off by default), then you can easily do it on the server as admin with no config changes:

This command is documented in svn versions 1.4 to 1.6.


Trac
If Trac is being used on the project, then prompt it to re-read the repo and pick up the change by:

where ## is the revision number to resync. If it is left out then all revisions will be resynchronised.

Netbeans' Matisse Swing editor is a great tool. Keeping it running is the challenge. That is, carefully ensuring that .form files are kept in sync with the .java files, and no external editing gets too trigger-happy. The other challenge, it turns out, can be when migrating to a different dev box / dev environment.
I was getting an error when trying to open forms that had opened fine in my previous environment, and I knew they were committed to my version control.

The error:

And then if you carry on to view the form, in navigator you get [invalid component] instead of classnames next to those components.

Infuriating when you know it should work. But I've found it to be only caused by one of 2 reasons:

1. Compile the class! Ensure that every one of the beans that the form is trying to load is currently compiled (do 'clean and build' on the whole project if in doubt).
2. Netbeans' JRE incompatible For example, my current project is in Java 6, and the projects were configured fine in Netbeans to use Java 6. However, Netbeans itself was launching in Java 5, which is fine, except that you then get the above error. It would be so easy for the Netbeans team to add this compatbility check one would think, but oh well. So, if Netbeans' JRE (tools> Java Platforms) is lower than your project JRE, change Netbeans' default runtime by either (as of NB 6.5):
   • running the IDE with the --jdkhome switch on the command line
   • or by entering the path to the JDK in the netbeans_jdkhome property of your INSTALLATION_DIRECTORY/etc/netbeans.conf file.

If these don't help, check out [user directory]/var/log/messages.log (eg. for OSX, user dir is '~/.netbeans/[version]/'), and your answer should lie in an exception thrown in there.

The Swing GUI editor in NetBeans is great. It has a number of samples to build from as well, one of which is the Master/Detail template. That's an example of one that generates a JPanel, and I wanted to convert to a JFrame. Here are the steps, assuming a form named NewForm (with NetBeans 6.5):

1. In NewForm.java:
   1. Change extends JPanel to extends JFrame.
   2. Change the static void main.. method content to be:
      java.awt.EventQueue.invokeLater(new Runnable() {
      public void run() {
      new NewForm().setVisible(true);
      }
      });


2. In NewForm.form (may need to edit this outside of NetBeans):
   1. Change the type attribute of Form (near the top) from type="org.netbeans.modules.form.forminfo.JPanelFormInfo" to type="org.netbeans.modules.form.forminfo.JFrameFormInfo".
   
   


3. In the graphical editor, select the Frame and choose the Code tab. Under Form Size Policy, change to Generate pack().

That should be it, good to go. May need to recompile, and/or open close the form to get NetBeans to recognise the changes.

It's great that CheckPoint has a SecureClient for the Mac. It works really well. What doesn't work, is making it stop.

There is no way to prevent it booting at startup. Oh, there are ways referred to by CheckPoint's docs, but as they say in the fine print, that just stops the gui from starting - any security policies (read: complete firewall lockdown) will still be in place. And if you do have the gui up and choose 'Stop VPN-1 SecureClient', again it's only the gui that goes, and you go mad like me having lost your svn, http, etc server, not knowing why. There are ways to control it via the command line which you could script, but your admins have to have allowed it to allow that via some centralised settings at their end.

So, if you do need SC, always launch the gui. If you need access to services on that box, then always choose 'Tools>Disable Security Policy' in SC. And if you don't need it anymore, then say goodbye to it, like I'm about to do.

I'm sure there's a better design methodology to do this, but I have an issue in Java where the PostgreSQL JDBC driver can't execute createBlob() (in JDBC3 spec), so I want to overwrite its Connection. I can't subclass it, as it's returned from the call DriverManager.getConnection(...). So what I need to do is use the Wrapper design pattern, aka Decorator, aka Delegator. Oh wouldn't it be great to use via? But alas, until my cry is heard, or someone corrects me, the solution is to:

1. Create a new class that also implements Connection.
2. Use this class to wrap the obtained PostgreSQLConnection
3. Painstakingly implement each and every method in Connection to pass through to the wrapped connection, except for the methods I want to meddle with.
4. Pass out of my Connection-obtainer class, not the raw Connection, but my new wrapper.
5. Accept resignedly that when the Connection interface changes, my Connection wrapper will now not fulfil the new interface and will break. Really good reason for via.

So enough whinging. In Eclipse, it is just a matter of creating a new class, and implementing Connection. If the methods didn't appear then the class name will have an error; Ctrl+1 on this gives the option 'Add unimplemented methods'. And there they will be, all 50 of them (rough count), ready to type this.connection.blah() in each. But hey, Eclipse can do multiline regex finds, and we can use regular expressions in the find and replace, so....

To fix all wrapped method calls that return a value:
In the new class, Ctrl-F to bring up the find/replace dialog. Make sure 'Wrap search' and 'Regular expression' are ticked, and 'Scope' is 'All', then copy and paste into Find:

This searches for any lines having the word public, not followed by class, and having a return statement.
Hit Find a few times to validate it's matching correctly.

Now, in Replace:

Takes the first match (\1) and appends the method call (\2) to the field name.

Click 'Replace All' to see it happen.

To fix the remaining method calls with no return value (ie void)
In Find:

Now, in Replace:

And 'Replace All' again.

As yet the above regex statements don't remove parameter types from the calling statements, so you will have to go through and remove them, but that should be relatively little work.

For a long time I have been looking for an open source database management tool that would be able to access any JDBC compliant database. I've used the great Oracle product, SQLDeveloper, for a long time and found it great for SQL Server management (usually performing quicker than using Microsoft's own Management Studio!). But now I've started using Derby (aka JavaDB aka Cloudscape) and SQLDeveloper is limited to Oracle, SQL Server, and MySQL.

Step in SQuirreL SQL. It has a pretty smooth interface, with great features like SQL syntax highlighting and even code completion. Released under the LGPL license, it is open source and contributions are welcomed, particularly for plugins. The plugins are important as JDBC is obviously a restricted subset of a database's functionality - a number of the existing plugins enable extra features for specific database types (eg. Derby's triggers).

Another very notable plugin is called graph which enables production of simple entity relationship diagrams (ERD) from a database connection. While it doesn't display multiplicity indicators, and some of the image exporting is a bit clunky, it is great to have this functionality in an open source product. (I now use NetBeans' UML Modeling which is great for the common UML diagrams, but lacking ERDs).

My notes to help remember the processes to get a secure svn server using Apache 2 on OS X Tiger. I know these are documented in lots of places, but it seemed that parts of various HowTos were required in order to do it with current Apache (2.2) and with Tiger. For anyone with Leopard onwards, you can skip the installation of Apache2 as this is now standard from Leopard on (Tiger and earlier used Apache 1.3 and earlier) and here is probably your best bet.

Edit: I drafted this a year ago and never finished (or published) it. Now the server has reverted back to apache1.3 after an update and my ssl cert had expired - time to revive the post! Unfortunately I didn't capture all the steps, especially the setting up of mods-enabled etc, but I'll just use my install as reference. Ask if you need those bits. Meantime, this link may be helpful. Brad.
Edit 2 (16/4/2009): It did it again. If svn access starts to fail, log in to a directory-displaying page or any page that gives the apache http server version - if it says 1.3 then it's reverted back from 2.0 (alternatively, if ls -l /usr/sbin/apachectl shows a single file rather than a symlink to /sw/sbin directory, that also is an indicator it's reverted). Easy fix is (after stopping 'Personal Web Sharing' in System Prefs): sudo ln -sf /sw/sbin/apache2ctl /usr/sbin/apachectl as below. Then start web sharing again; should now be 2.0 running.

Installing Apache httpd 2
1. Apache2 Installation

The bulk of this guide comes from Tim Fanelli's great howto (just the Fink, Apache2, Subversion, and WebDav steps for now). If you don't have XCode Tools installed prior to this, you can get them here (requires free registration). Extra notes to the sections in Tim's guide follow:

• Apache2: At the end of this section you can start up Apache2. Be sure to shut down the 1.3 WebServer first via the Sharing panel of SysPrefs, if it is running. You should then be able to view the Apache2 welcome screen by browsing to http://localhost. You probably won't be able to access this from another machine yet though (even within LAN) as the Mac's firewall won't be allowing it. We'll get to that later.
• Subversion: If you will be carrying this on to use svn, you might as well do this and the WebDAV steps now. You'll need svn-ssl, but may not require svn-client-ssl. I didn't install it.

That's all we need from that blog for setting up the Apache2 server

2. Apache2 Configuration
Next we'll configure Apache2 to be handled by the Sharing panel of SysPrefs, instead of old 1.3.

• cd /usr/sbin
• mv apachectl apachectl1.3 {–> This renames default apache1.3/apachectl command}
• ln -s /sw/sbin/apache2ctl apachectl {–> This creates symlink for Apache2/apachectl command}

Edit /sw/etc/apache2/apache2.conf:

1. Change the pidfile location to:
   
   
   /private/var/run/httpd.pid


2. In order to easily view logs from Console, and to get free log rotation, change the ErrorLog parameter to:
   
   
   /var/log/httpd/error_log


3. And add a new entry:
   
   
   CustomLog /var/log/httpd/access_log common

3. SSL Certificate
Go back to Tim's post and the SSL step. You'll want to follow the link and generate the certificate Important: Ensure you create a CommonName attribute in the certificate. If it is missing then many svn clients will fail to access the site. After doing that though, a couple of the commands given back in the main howto are wrong. Change sudo cp ~/server.key /sw/etc/apache2/ssl.key/ to sudo cp ~/sslcert/server.key /sw/etc/apache2/ssl.key/ and the same for .crt.



For better security, don't do the decryption suggested. However, for
pragmatism and to be able to control the server via the SysPrefs, just
do it :)

Launch mysqld (server) on Linux (if not already running):
sudo /etc/init.d/mysql start

Connect to server [specific database]:
mysql [-h host] -u user -p [dbname to use]

Use the SHOW statement to find out what databases currently exist on the server:
mysql> SHOW DATABASES;

Pick one:
mysql> USE dbname

Create new one:
mysql> CREATE DATABASE dbname;

List tables:
mysql> SHOW TABLES;

Create table:
mysql> CREATE TABLE pet (name VARCHAR(20), owner VARCHAR(20),
-> species VARCHAR(20), sex CHAR(1), birth DATE, death DATE);

List tables columns:
mysql> DESCRIBE tablename;

Generate schema:
mysql> SHOW CREATE TABLE tablename;

On the surface, external configuration files in a JEE environment provide a simple mechanism to store environment-specific data, but they can be a pain to access. How to access it when file system access isn't allowed? Since it's outside the classpath, where is the file in each environment? If I'm using Spring, how do I access this movable file?

The best solution I've found is to use jndi resources. The following is a solution using Websphere (6) and Spring (2.01).

Step 1: Configure the jndi reference for Websphere
This step was based on information from IBM's page "Using URL resources to manage J2EE property files in IBM WebSphere Application Server V5", steps A and B. Websphere Studio isn't required, however, so briefly:
a) Navigate to Resources > URL > URLs and create a new URL. Make up a JNDI name starting with "url/". In 'specification', enter the path to the properties file as a URI, eg. "file:///E:/project.properties". So now Websphere has a URL Resource pointing to the properties file for this environment.
b) In the code, edit web.xml. Configure a new resource-ref as shown in IBM's figure 6. 'res-ref-name' is the jndi name we set up in a). Then in ibm-web-bnd.xmi, add a new resRefBindings as shown in IBM's figure 7.
That completes the configuration of Websphere and the jndi configuration in the code.

Step 2: Next Spring needs to be able to load the properties file by looking up the jndi location.
c) I'll assume that you want the properties file to be set as a property on a class 'pkg.MyClass'. To do this, we use a PropertyFactoryBean to convert from properties file to Properties class. The PropertyFactoryBean takes a Resource as location property, so we create a UrlResource bean for this, with the java.net.Url as constructor argument. This java.net.Url is the result of using a JndiObjectFactoryBean to look up the jndi name and return the Url object. The following bean config shows these conversions:

And that's it. In summary, Spring looks up a jndi URL reference to a properties file, configured in the JEE server. Spring beans are created that convert the URL to a URLResource to a Properties object, available for injection into your custom class.

There are a number of posts around the place on getting gprs or 3g cellphone connections working on linux. I wasn't interested in bluetooth as I have limited batteries and a usb cable. And you don't always know what your locla settings should be.

Here's all I did on Xubuntu Hardy:

1. Plug the phone into the computer using the usb cable.
2. In a terminal, type "sudo wvdialconf". (This sets up available baud rates, etc)
   
3. In a terminal, type "sudo vi /etc/wvdial.conf".
4. Edit this file with:
5. phone = [for my nokia on Vodafone New Zealand, the number is "*99#". I phoned them to find out.]
6. name = [the login name, if your mobile company requires it. Vfone NZ doesn't, but wvdial needed something here. I used "name"].
7. password = [as with name. I used "password"].
8. Save and close the file.
9. Connect to the internet via your mobile by typing "wvdial" in a terminal window.
10. When done, press ctrl-c in that window.

So easy and so great!