Category Archives: programming

Shotwell Plugins, Part I – Setup

Here’s a quick overview on how to start writing a custom publishing plugin. This is being done on Ubuntu 14.04, so no promises it will function on any other version.

  1. Install valac-0.22, libgphoto2-dev, gnome-doc-utils,libgstreamer-plugins-base1.0-dev, libgee-0.8-dev libsqlite3-dev libraw-dev librest-dev libwebkitgtk-3.0-dev libgexiv2-dev libgudev-1.0-dev libgtk-3-dev libjson-glib-dev
  2. Download the shotwell 0.20.2 sources and not the current version from github. The current version in get uses some new gtk features which are not available in ubuntu 14.04.
  3. Copy the shotwell/samples/simple-plugin from the shotwell git repo to a new directory
  4. Build/install shotwell 0.20
    $ ./configure --install-headers
    $ sudo make -j6 install

     

  5. In your new plugin, run ‘make; make install’ to ensure the basic build works.
  6. Rename simple-plugin.vala to your publishing plugin name (ie, OnedrivePublishing.vala)
  7. Modify the Makefile and set the PROGRAM to your plugin name (ie, OnedrivePublishing)
  8. Running make should compile your new empty plugin.

Now that that’s done we can start creating out publishing plugin.

The plugin sample implements the Spit.Pluggable interface, in order to create a publishing plugin, we’ll need to use that to return our publishing module and create a new class to implement the Spit.Pluggable and Spit.Publishing.Service interface as well. Rename that class and include all the necessary interfaces. We’ll use the ShotwellPublishingCoreServices as a template for how to bootstrap out publishing service.

The basic do-nothing module which compiles w/ one warning (the return null) now contains the following:

extern const string _VERSION;
private class OnedriveModule : Object, Spit.Module {
    private Spit.Pluggable[] pluggables = new Spit.Pluggable[0];

    public OnedriveModule() {
        pluggables += new OnedriveService();
    }
    
    public unowned string get_module_name() {
        return _("OneDrive Publishing Services");
    }
    
    public unowned string get_version() {
        return _VERSION;
    }
    
    public unowned string get_id() {
        return "org.yorba.shotwell.publishing.onedrive";
    }
    
    public unowned Spit.Pluggable[]? get_pluggables() {
        return pluggables;
    }
}
// This is our new publishing class
private class OnedriveService : Object, Spit.Pluggable, Spit.Publishing.Service {
        

    public OnedriveService() {
    }

    public unowned string get_id() {
        return "org.yorba.shotwell.publishing.onedrive";
    }
    
    public Spit.Publishing.Publisher.MediaType get_supported_media() {
        return (Spit.Publishing.Publisher.MediaType.PHOTO |
            Spit.Publishing.Publisher.MediaType.VIDEO);
    }
    public Spit.Publishing.Publisher create_publisher(Spit.Publishing.PluginHost host) {
        //TODO
        return null;
    }

    public void get_info(ref Spit.PluggableInfo info) {
        info.authors = "Mike Smorul";
        info.version = _VERSION;
        info.is_license_wordwrapped = false;
        
    }    
    public unowned string get_pluggable_name() {
        return "OneDrive";
    }

    public int get_pluggable_interface(int min_host_interface, int max_host_interface) {
        return Spit.negotiate_interfaces(min_host_interface, max_host_interface,
            Spit.Publishing.CURRENT_INTERFACE);
    }
    
    public void activation(bool enabled) {
    }
}
// This entry point is required for all SPIT modules.
public Spit.Module? spit_entry_point(Spit.EntryPointParams *params) {
    params->module_spit_interface = Spit.negotiate_interfaces(params->host_min_spit_interface,
        params->host_max_spit_interface, Spit.CURRENT_INTERFACE);

    return (params->module_spit_interface != Spit.UNSUPPORTED_INTERFACE)
        ? new OnedriveModule() : null;
}

private void dummy_main() {
}

You can now compile this by:

$ make clean; make ; make install

This will install your new module into your local modules directory. To make sure it works, open up shotwell, go to Edit -> Preferences -> Plugins and you should see your new plugin listed under the Publishing section with a generic graphic next to it. If you enable the module you’ll notice the following error that will be fixed when we start implementing functionality

 GSettingsEngine.vala:457: GSettingsConfigurationEngine: error: schema 'org.yorba.shotwell.plugins.enable-state' does not define key 'publishing-onedrive'

Useful Links

PBS, FD_CLOEXEC and Java

The PBS/Torque scheduler that ships w/ Ubuntu 12.04 uses an interesting method to verify that user requests from a submission node cannot impersonate anyone else. In a nutshell, any Torque command (qsub, qstat, etc) calls a suid program (pbs_iff) which connects to the pbs server from a privileged port and notifies the server the client port and what user will be sending commands from that port. pbs_iff receives this information by looking at the source port on the file handle passed to if during its clone. The whole handshake looks like this:

  1. Unprivileged client opens a socket to the pbs server
  2. Client calls clone and passes the file handle number to a suid pbs_iff as an argument
  3. pbs_iff reads the source port off of the file handle
  4. pbs_iff opens a socket from a priviliged port to the pbs server and sends invoking user and source port .
  5. The pbs server now trusts that commands from the initial socket belong to the user passed by pbs_iff
  6. pbs_iff terminates and the original client sends whatever commands it desires.

This works nice in C where the default is to pass all file handles to the child process on a fork. However, many languages frown on this file handle leaking for a number of reasons and have decided this default is a bad idea. Java is one of these, so it nicely sets FD_CLOEXEC on all file handles it opens. This means when you use the ProcessBuilder or call Runtime.exec, you can’t see any file handles you previously had open thereby breaking Torque’s security mechanism.

Fun with DC’s gis data, part 1

It looks like DC has kindly released quite a bit of gis data for public consumption. One of the more interesting sets is the regularly updated Owner Polygon dataset available from data.dc.gov. This is a shapefile containing current property records for everything in the District. Unfortunately, it’s not available kml for easy display in google’s tools. However the 70MB esri shapefile is available. Using Open Layers, PostGIS, and and GeoServer, we can get start displaying everything, but what if we want to use google maps and do things the hard way?

To solve that, there’s a few simple steps to allow polygon querying, selection, and display on google maps.

  1. Import data into PostGIS
  2. Create GIS servlet
  3. Draw the data on google maps
  4. Query PostGIS for google’s lat/long
  5. Select Properties from the map

We’re going to work on step one today, import your data into PostGIS.

Prepare PostGIS

I’m running Ubuntu 11.04, PostgreSQL 8.4 with postGIS 1.5.1 installed from the default software repo.

  1. PostGIS 1.5 manual
  2. nad 83, maryland projection
psql (8.4.8)
Type "help" for help.

postgres=# create database propertymap;
CREATE DATABASE
postgres=# \q
~$ createlang plpgsql propertymap;
~$ cd /usr/share/postgresql/8.4/contrib/postgis-1.5/
postgis-1.5$ psql -f postgis.sql propertymap
postgis-1.5$ psql -f ../postgis_comments.sql propertymap;
postgis-1.5$ psql -f spatial_ref_sys.sql propertymap;

Convert Shapefile

Create a ton of insert statements using shp2pgsql:

poly$ shp2pgsql -s 926985 OwnerPly.shp ownertable > inserts.sql
Shapefile type: Polygon
Postgis type: MULTIPOLYGON[2]

If we look at the .prj file included, we see that the projection for the data is NAD_1983_StatePlane_Maryland_FIPS_1900. We need to add the projection from spatialreference.org in to our database

propertymap=# INSERT into spatial_ref_sys (srid, auth_name, .......66666666],UNIT["Meter",1.0]]');
INSERT 0 1
propertymap=# \i inserts.sql

Run your first query

propertymap=# select ownername,square,lot,premiseadd from ownertable where premiseadd like '%1600 PENNSYLVANIA%';
        ownername         | square | lot  |       premiseadd        
--------------------------+--------+------+-------------------------
 UNITED STATES OF AMERICA | 0187   | 0800 | 1600 PENNSYLVANIA AV NW
 UNITED STATES OF AMERICA | 0187   | 0802 | 1600 PENNSYLVANIA AV NW
 UNITED STATES OF AMERICA | 0187   | 0801 | 1600 PENNSYLVANIA AV NW

First, a little background on what we asked for. DC property records are based on square, suffix, and lot. Square generally refers to a city block and goes all the way back to the original city planning in the old part of the city. Lot is a lot within a square/suffix. For the most part, you can ignore suffix as it’s rarely used.

Next time, create a simple servlet to expose all of this.

log4j and Pivot

Here’s a simple way to consume log4j messages in pivot for use in a log console or similar.

First create a custom appender which sents a log message to the pivot message bus.

public class MessageBusAppender extends AppenderSkeleton {
    
    @Override
    protected void append(LoggingEvent event) {
        MessageBus.sendMessage(new LogMessage(layout, event));
    }

    @Override
    public boolean requiresLayout() {
        return true;
    }

    @Override
    public void close() {
       //nop
    }

}

public class LogMessage {

    private Layout logLayout;
    private LoggingEvent event;

    public LoggingEvent getEvent() {
        return event;
    }

    public Layout getLogLayout() {
        return logLayout;
    }

    public LogMessage(Layout logLayout, LoggingEvent event) {
        this.logLayout = logLayout;
        this.event = event;
    }
    
}

In any component needs to display log messages, just listen for the messages and update as appropriately. Here’s an example updating a textpane:

public class LogPane extends Border {

    @BXML
    private TextPane logTxt;
    @BXML
    private PushButton clearBtn;
...
...
        logTxt.setDocument(new Document());

        MessageBus.subscribe(LogMessage.class, new MessageBusListener() {

            public void messageSent(final LogMessage message) {
                ApplicationContext.queueCallback(new Runnable() {

                    @Override
                    public void run() {
                        String text = message.getLogLayout().format(message.getEvent());
                        logTxt.getDocument().add(new Paragraph(text));
                        if (message.getEvent().getThrowableInformation() != null)
                        {
                            StringBuilder sb = new StringBuilder();
                            for (String s : message.getEvent().getThrowableInformation().getThrowableStrRep())
                            {
                                sb.append("  ");
                                sb.append(s);
                                sb.append("\n");
                            }
                            logTxt.getDocument().add(new Paragraph(sb.toString()));
                        }
                    }
                });
            }
        });

        clearBtn.getButtonPressListeners().add(new ButtonPressListener() {

            public void buttonPressed(Button button) {
                logTxt.setDocument(new Document());

            }
        });
 ...
 ...
}

Now tie it together in your log4j config:

log4j.rootLogger=ERROR, Pivot

log4j.appender.Pivot=sample.MessageBusAppender
log4j.appender.Pivot.layout=org.apache.log4j.PatternLayout
log4j.appender.Pivot.layout.ConversionPattern=%-6r [%15.15t] %-5p %30.30c %x - %m%n

Counting bits

Here’s a simple input stream which will tell you the current position. You should insert this after any buffered input streams if you want an accurate position after a read.

public class CountingInputStream extends PushbackInputStream {

    private long bytesRead;

    public CountingInputStream(InputStream is) {
        super(is);
    }
    public CountingInputStream(InputStream is,int len) {
        super(is,len);
    }

    public long getPosition() {
        return bytesRead;
    }

    @Override
    public void unread(byte[] b, int off, int len) throws IOException {
        bytesRead -= len;
        super.unread(b, off, len);
    }

    @Override
    public void unread(int b) throws IOException {
        bytesRead--;
        super.unread(b);
    }

    @Override
    public void unread(byte[] b) throws IOException {
        bytesRead -= b.length;
        super.unread(b);
    }

    @Override
    public boolean markSupported() {
        return false;
    }

    @Override
    public synchronized void reset() throws IOException {
        throw new IOException("Mark not supported");
    }

    @Override
    public int read() throws IOException {
        int rd = super.read();
        if (rd != -1) {
            bytesRead++;
        }
        return rd;
    }

    @Override
    public int read(byte[] b) throws IOException {
        int read = super.read(b);
        if (read != -1) {
            bytesRead += read;
        }
        return read;
    }

    @Override
    public int read(byte[] b, int off, int len) throws IOException {
        int read = super.read(b, off, len);
        if (read != -1) {
            bytesRead += read;
        }
        return read;
    }

    @Override
    public long skip(long n) throws IOException {
        long skipped = super.skip(n);
        bytesRead += skipped;
        return skipped;
    }
}

GZip pushback input stream

When dealing w/ Java’s GZIPInputStream, it has an annoying habit of eating bits past the end of the gzip trailer. Normally when reading a file, this isn’t a problem, but when reading concatenated GZIP’d files or gzip data embedded in other inputstreams, you would really like to have those overread bits back. Using this inputstream, you can.

public class GzipPushbackStream extends GZIPInputStream {

    private boolean pushed = false;
    private static final int TRAILER_LEN = 8; // 8 byte tailer

    public GzipPushbackStream(PushbackInputStream pis) throws IOException {
        super(pis);
    }

    @Override
    public int read(byte[] buf, int off, int len) throws IOException {
        int read = super.read(buf, off, len);
        if (eos && !pushed) {
            int n = inf.getRemaining();
            if (n > TRAILER_LEN) {
                int offset = super.len - (n -TRAILER_LEN);
                ((PushbackInputStream) in).unread(super.buf, offset, n - TRAILER_LEN);
            }
            pushed = true;
        }
        return read;
    }
}

Java digesting performance

For the ACE audit manager, reading and generating digests on files is slowest part of auditing. In previous (1.6 and lower) versions of ACE digesting was done in a simple, read/update digest loop using Java’s DigestInputStream. This appeared to work good enough, however I wanted to see what effect large blocks have on this model. When reading from remote resources, we up the block size from a standard 4-32k to 1MG.

Using that model, here’s how performance looked for a variety of block sized, both aligned and unaligned. The following table compares both only reading and reading then updating. The test data ~16G of large (27MG) files on a SATA disk, running on a machine w/ 8G ram and an Intel Q9550 @2.83Ghz. The digest algorithm was SHA-256, using Sun’s provider.

Block size read digest only read+digest
4096 96.5 MB/s 69.6 MB/s 53.1 MB/s
8192 95.9 MB/s 69.6 MB/s 54.8 MB/s
10000 94.6 MB/s 69.2 MB/s 54.0 MB/s
1048576 97.8 MB/s 72.1 MB/s 34.3 MB/s

The interesting take-away is how much worse large block preforms when you follow the read then process model.

The next step is to change the model and split the digesting and reading into separate threads. The new model uses two threads and two queues. The two threads exchange a pre-allocated set of byte arrays using an empty-array queue and a filled-array queue. These queues were Java LinkedBlockingQueues.

Read Thread:

  1. Pop byte buffer from empty-array queue
  2. Read into the byte array.
  3. Push buffer into filled-array queue.

Digest Thread:

  1. Pop byte buffer from filled-array queue
  2. Update digest using byte array.
  3. Push buffer into empty-array queue.

On the following test runs, I used 5 byte buffers of the above sizes to see how performance would vary.

Block size read digest only read+digest
4096 92.2 MB/s 65.63 MB/s 54.7 MB/s
8192 92.4 MB/s 67.5 MB/s 57.0 MB/s
10000 87.8 MS/s 68.0 MB/s 57.2 MB/s
1048576 97.4 MB/s 72.2 MB/s 64.7 MB/s

Small block performance is pretty much unchanged, however on large block, large file there is a substantial speedup running at almost 90% of the possible digest speed vs 48% possible speed. The next version of ACE will switch to this method of data reading.

ACE 1.6 released

ACE 1.6 has been released. Among it’s notable features are:

  • Support for exporting token stores.
  • New maven-based build system, for easier 3rd party collaboration
  • checkm manifest exporting
  • mproved database storage of tokens.
  • Open subversion repository, and nightly builds.

Download here: ACE Download

A complete list of all improvements and bug fixes is available at:
Changelog

When upgrading to 1.6, ACE will attempt to migrate all existing tokens from
the older table layout to the newer. This means, the first time you run 1.6,
it may take several minutes to perform this migration. The web interface will
be unavailable during this migration. You can follow the status of the migration
by looking at the ace log file, located in /tmp/aceam.log.