FOSS4G 2010 conference

Last week, FOSS4G conference came to visit us in Barcelona, and of course we couldn't miss this event. There were tons of interesting conferences and topics, such as 3D and raster support in PostGIS, and using CSS to style WMS services.

We gave a presentation about BeETLe project on Thursday afternoon, with a nice amount of attendees and several interested people. We are proud to have brought BeETLe to FOSS4G, as there was a big competition this year (more than 200 abstracts were discarded during the FOSS4G selection process). [Side note: If you don't have OpenOffice, you can grab the PDF version of the talk].

We also attended Thierry Badard's talk about GeoKettle (right after ours), who announced several new features and a new version of GeoKettle for October. Our projects should definitely have a closer collaboration, since they are very complementary and I think we could make bigger improvements working together.

Thierry also gave a talk about SOLAPLayers, which is a visualization and exploration web application for Spatial-OLAP cubes stored in GeoMondrian. ETC-LUSI is also running a similar project with CamptoCamp, which was also presented as a poster in FOSS4G. Our focus is a simple and intuitive interface to build the queries, and several interactive tools to explore the data. The application interface provides maps, tables and charts to show the queried data in a visual way.

We were also talking with Thierry about collaboration on this field and maybe an eventual product merge. Hopefully, we are not replicating work here, as we have taken complementary approaches. For example, we are sending raster (WMS) data to the client, which is supposed to load faster, while they are serving vector (WFS) data, which is useful for some interaction tools. For the moment, they use a raw MDX query to launch the queries, while we already have a nice query builder in place. We will provide some links to the project website as soon as we release it.

BeETLe project presentations

On March 10th-12th, the city of Girona (Spain) hosted the annual Free GIS Conference [Spanish], which is one of the most important GIS events in Spain.

We had the opportunity to present BeETLe project to this eager public, and shared some lovely days with the leaders of the Spanish Free-GIS scene.

I hope the organization publish the videos soon, but in the meantime you can have a look to our .odt presentation (in Spanish). You can also find a brief version of the same presentation in English (.odf) at the BeETLE project page at OSOR (alternative format: PDF) .

BeETLE project will also be presented at FOSS4G 2010 conference, which will take place in Barcelona on September. This is a great opportunity to know about the project and meet BeETLe project team, so we hope to see you there!!

Working with big data

One of the main goals of the project is the ability to process large datasets (mainly raster and vector layers, but also database tables). In the way of this goal, we have found several problems in the base libraries (mainly GeoTools, Java Image I/O Ext and Sextante), which had problems when accessing big raster data. Therefore, a part of our work has focused in reinforcing these libraries.

Some of these problems have already been solved:
- The ASCII GRID driver of Java Image I/O Ext is not able to read large files
- Sextante-GeoTools bindings load the whole raster layer in memory before processing it
- Sextante-GeoTools bindings load the whole dbf table in memory before processing it
- Sextante-GeoTools bindings creates the whole dbf table in memory before writing it to disk

However, there are still some pending tasks, all of them are due to GeoTools and Image I/O limitations:
- Raster layers must completely be created in memory before being written to disk.
- No BigTiff support (TIFF files are limited to 4GB)
- No binary grid support

Currently, these remaining tasks are considered to be low priority tasks, but we hope to be able to solve them in future phases of the project. Collaborations are welcome!

EPSG:3035

Most of the data that ETC-LUSI has to process or create in its daily work, are TIFF files using EPSG:3035 projection (Lambert Azimutal Equal Area projection using the ETRS89 datum).

However, we've discovered that ArcGIS (at least on 9.3 version) is not able to properly encode EPSG:3035 using standard GeoTiff tags, so it uses an external auxiliary file (using a proprietary format) to store the spatial reference information.

GeoTools is not able to read these proprietary files, and therefore it refuses to read these (false Geo-)TIFF files. Fortunately, the library can still be convinced to read them by using the DEFAULT_COORDINATE_REFERENCE_SYSTEM Hint during GeoTiff reader creation.

For the moment, we can live with this workaround, but we would be really pleased to see ArcGIS generating correct standard GeoTIFF files for EPSG:3035 projection, as is the reference projection at the European Environment Agency.

Project Milestones

The project has quite ambitious goals, but we prefer to reach them step by step. For the moment we have defined 2 milestones:

MILESTONE 1: Sextante as GeoKettle Job Entries.
This milestone will be available for the end of 2009, and its main goals include:

• The full set of Sextante algorithms available in GeoKettle as Job Entries.
• New Sextante algorithms: Tabulate Area and Zonal Statistics.
• Geospatial Input and Output Job Entries based on GeoTools.
• Improved support for huge raster datasets in GeoTools ArcGrid driver.
  
• Improved support for huge raster and table data in Sextante-GeoTools bindings.

MILESTONE 2: Sextante as GeoKettle Transformation Steps.
This milestone will be developed during 2010. Main goals include:

• A selected set of Sextante algorithms available as Kettle Transformation steps.
• Raster support for GeoKettle: Raster input and output steps based on GeoTools.
• Full power of GeoKettle transformation applied to Sextante: distributed execution using computer grids.

The result of each milestone will be a fully functional product, matching the planned goals.
You can find more detailed, technical documentation at the documentation section of the BeETLe project at OSOR.

BeETLe Project Introduction

The BeETLe project aims to extend GeoKettle with advanced spatial analysis tools. The main idea involves integrating Sextante into GeoKettle, making GeoKettle a more powerful and versatile application.






This project has been pushed by ETC-LUSI, in coordination with other institutions: Extremadura University (Sextante developers), Laval University (GeoKettle developers), and Junta de Andalucía (which shares vision and needs with ETC-LUSI).