captain holly java blog

It seems like the worlds of statistics and Java don’t talk to one another enough.

Small cell Suppression is a statistical term for not allowing users to be able to infer what should be private information from public sets of data. For example, consider a survey on athletes with staph infections that was queryable by age, county, sport and race. If there were statistically small number of hispanic wrestlers in Otter Tail County, you could probably guess who had a staph infection. So, if a population is identified as statistically vulnerable to this inference, then that data is suppressed.
The Washington State Dept. of Health page has a pretty good explanation:

Why are small numbers a concern in public health assessment?

Public health policy decisions are fuelled by information. Often, this information is in the form of statistical data. Questions concerning health outcomes and related health behaviors and environmental factors often are studied within small subgroups of a population. Continuing improvements in the performance and availability of computing resources, including geographic information systems, and the need to better understand the relationships between environment, behavior, and consequent health effects have led to increased demand for data on small populations. These demands are often at odds with the need to preserve privacy and data confidentiality. Small numbers also raise statistical issues concerning the accuracy, and thus usefulness, of the data.

In general, problems with confidentiality arise when there are small denominators (population size represented in a specific cell in a table); and, problems with data reliability arise when there are small numerators (cases in a specific cell in a table).

Definitions
The broader term for these controls is “Statistical Disclosure Control”. The challenge is to use optimal levels since too little control leaks public data and too much control makes published survey data useless.
“Imputation” is the practice of substituting values for missing data items. If we are leaving out data to protect confidentiality, then substitute data must be imputed so as to not skew the overall results.
“Inference”: The practice of finding secret data in published survey results. By measuring inference, we can find out if disclosure control is an issue.
Spearman’s Rank Correlation: a statistical tool for inference. It can find out how closely two variables are tied. This web page will perform this correlation for you (if you are ready to hand type your data into a web form).

I could only find one tool related to this in the java world. I’m surprised it isn’t more of a booming field since it touches on survey data, health and financial data, and security and privacy. Is that too small of a niche? I doubt it. Inattention to the dangers of leaking information in this way could potentially cause a lot of harm and cost a lot of money.

The stats package SAS has small cell suppression features. This document (Word Doc) discusses how to deal with the holes in the data that result from suppression.

So, how to have this feature in my java app?
R = the open source statistical package
CRAN = a list of packages for use with the R language
sdcTable: statistical disclosure control for tabular data
lPSolve: an R package that sdcTable depends on
rJava = an R package that allows R to create java objects and, through the JRI package that is now part of rJava, allows java run R in a single thread and make calls to it.
JGR = java GUI tool that makes use of rJava for a java GUI interface to R. R binaries must be installed and the JGR jar then allows java to call it. The source of JGR has good, production quality examples of how to call R from java.
Using all that, one should be able to create an ad-hoc query front end for survey data, run submitted queries through small cell suppression rules in R, and
return safe data.
There, I solved your small cell suppression problems. I’ll leave the details to the reader. What could be easier than integrating a stack of open source C and Java projects into your web app? or, rather, tune in for part II: implementing this stack O’ fun.

I recently switched over from working primarily with Eclipse/ MyEclipse and these are some large and small obstacles and how to overcome them.

1. I want to ignore persistence framework errors. Go to Project Structure –> JPA facet Delete Data Sources Mappings (but not JPA Configuration Descriptor!)
2. Web application doesn’t reflect changes to html, xhtml, jsp, etc. Go to Project Structure –> Java EE build settings. Make sure Exploded Directory Project compile output path is the same one the server is using (ie where your project lives on disk) Also make sure compile output path is the same as where your project lives and not some crazy intelliJ invented directory..
3. I want editor to be linked with menu, like in Eclipse. This is autoscroll from source, a button in the top row of the project pane.
4. I used Ctrl-shift R (for resource) all the time in Eclipse. In IntelliJ IDEA, the same function is CTRL-Shift-N (for name)
5. Auto complete does not work! In my case, this was due to the La Clojure plugin (0.2.172) When I disabled this plugin and restarted, autocomplete (and several other features) came back. A web search on this turned up nothing. Maybe now it will.
6. How to integrate CVS
• If CVS is not connected, go to Version Control –> –> CVS –> Configure CVS Roots –> Test Connection. This appeared to reset the connection for me.
• To Setup CVS repo Version Control –> CVS –> configure CVS Roots –> click “plus” button to make a new root. Enter your cvs info
• Import existing project into your IntellJ IDEA File –> open project –> browse to find .pom file
7. How to get vim keyboard mappings in intelliJ. go to settings –> plugins –> available –> right click on IDEAVIM to install. The step I skipped screwed me up big time: You must copy the keymap file according to these directions.
8. I hacked the authentication mechanism on an app so I wouldn’t have to log in every time during testing, and I was afraid I might accidentally commit it to CVS. So I had to ensure this file never got mixed in with the rest of our code. This is a CVS question rather than an IntelliJ IDEA question, but the answer is to create a new branch. Right-click on file –>CVS–>create branch (name it “DEAD_BRANCH” or something) and check the “Switch to this branch” box. The next time you go to commit that file or the directory it is in, that file will show up as [switched to tag DEAD_BRANCH] and if committed, will only be committed to that branch, so that your co-workers, when they update, will not get your screwed up file.
9. Keystroke goodness. The following keystrokes are indispensable. For a complete keystroke chart, go to help –> keystroke reference
   • move lines or blocks of code. This comes in handy on almost a daily basis and for some reason isn’t in the keystroke chart. Ctrl-Shift up arrow moves a line or selected block up. Ctrl-shift down arrow moves a line down. If it does not work, try hitting escape.
   • IntelliJ has a history of clipboard (buffer) contents. To paste from it, use Ctrl-Shift-V
   • Rename: Shift -F6
   • Generate Getters and Setters: Alt-Insert
   • Find usages: alt-F7
   • Duplicate Line or selection: Ctrl-D.

I’m trying to summarize several discussions about alternate languages on the JVM that I absorbed at the No Fluff Just Stuff conference. Can I become a language evangalist based on a weekend at a conference? I suppose not, but there were a lot of compelling arguments for why we should be looking at some of these new functional languages on the JVM. It was put forward that most of the reasons we like Java have to do with the JVM and not with the Java language:

1. Cross platform
2. stability
3. Performance
4. security
5. huge world of libraries

These will hold true with any language that compiles to the JVM.

Why are they even considering new languages? Multiple reasons bubble up from conference as a whole.

Extensibility.
Discussed the example of hadoop. It is an open source framework that handles huge amounts of data in a distributed way. It is inspired by Google’s MapReduce papers. They evidently found some of the core java classes insufficient for their needs. If you look at the docs for org.apache.hadoop.io.text, it says, “It provides methods to serialize, deserialize, and compare texts at byte level…. In addition, it provides methods for string traversal without converting the byte array to a string.” Does this point to an extensibility problem in Java? If not, why couldn’t they reuse any code from String? Someone at the conference asked why can’t I make Object define toXmlString() so that every one of my classes that descends from Object automatically has a toXMLString() ? This is extensibility and Java doesn’t do it as completely as some other languages might.

A language shouldn’t limit what you can do. Certain language constructs not available in java (closures, switch statements, folding) enable developers to be far more efficient.

OO might be failing us. We try to think of Objects as changing in place. Rich Hickey, the creator of Clojure, rejects this: ” The future is a function of the past, it doesn’t change it. ” If we stop thinking of data as persisting and changing over time and instead recognize that a thing is immutable and when it changes it becomes a different immutable thing. Like a date, or an account balance. The state of an account a point in time is immutable. Adding money to it does not change it, it creates a new state. This 55 minute video of Rich Hickey explaining some of these ideas was recommended at the conference and is amazing. As he explains, all of our concurrency problems come from the notion of objects changing in place.

We all understand that when a checkbox is not checked on a form, it is not present in the request object. This is the basis for many headaches in web application programming, especially when using multiple form pages. When using multiple form pages, as in a wizard, the struts way around is to have a reset() method that contains some logic for setting the value to false if it doesn’t exist in the request. Again, this applies to situations with a session scoped form.

The documentation for the html:checkbox tag says:

WARNING: In order to correctly recognize unchecked checkboxes, the ActionForm bean associated with this form must include a statement setting the corresponding boolean property to false in the reset() method.
In practice, the only properties that need to be reset are those which represent checkboxes on a session-scoped form. Otherwise, properties can be given initial values where the field is declared.

public void reset(){
    this.citizen = false;
}

There are several confusing posts out there in forums about how to populate checkboxes when viewing forms with existing data. One says to have a hidden form field with the same name as the checkbox. Another has us jumping out of struts and using regular JSP tags with logic. Both of these are unnecessary and have potentially bad repercussions later.
The real solution is to use a html:checkbox with a name equal to that of a bean and the property equal to the name of the boolean variable in that bean that the checkbox captures. The following will check or uncheck the checkbox depending on the value of “citizen” in the applicantBean:

<html:checkbox name="applicantBean" property="citizen" value="true">

to work this, your code must invent an empty applicant bean before loading the blank form, or struts will whine that there is no such thing as “applicantBean” in any scope.

The Denied and Restricted Parties List (DRPL) is kind of a No-Fly list for export restrictions and since Sun has some encryption related technology, it is a national security concern that someone might take the SCJP exam.

After initially being informed that my request to take the exam was denied, today I got an email from SUN saying that I’m not, after all, someone who might do bad stuff.
here is some background email that was attached to my email:

The following individual, as a result of screening, has been identified as being as a potentially non-compliant export customer:

Search Key: US2121923
First and Last Name: Tim McGuire

City: St. Paul

Country: US

Result of DRPL check: Detected

Date and time of denial: Mon Jul 27 10:10:15 MDT 2009

Course Order Numbers: No Numbers Generated.

Reason for denial: Not Available from service.

From here and here I see that a bunch of people have been inconvenienced and aggravated to varying degrees because some mouth breather hasn’t figured out that no-fly lists are a fake idea. Imagine if Macy’s did a surprise sniff test on every 100th customer in their underwear department? This is just like that.

I installed jadclipse to see if it was a way to view source of all the libraries I use but don’t have the source code for.

Installation is simple, with the small extra step of downloading jad and telling eclipse where to find the jad.exe. (Window –> Preferences –> Java –> Decompilers –> Jad)

All the jad command line options are represented in the jad dialog.

inner classes and static blocks were decompiled just fine. I did notice that jad left out any braces in if-then blocks that weren’t needed. Nitpicking a bit here, but I can’t read code without braces. The braces were put in when I checked the “show redundant braces” options, but not until I restarted eclipse.

Finally, some classes seem to have a stuck setting causing them to open in class viewer. For instance, Struts Action gets decompiled just fine, but Struts DispatchAction just shows itself in that awful class viewer. Restarting Eclipse made this problem go away too.

Big thumbs up for this very useful eclipse plugin.

thumbs up