Metadata Generation Options

I've been spending some time examining some of the options we have regarding metadata. Right now I'm working with three tools; all of which have their advantages and disadvantages. I think we need to make some choice soon, since I want to get data sets ready for distribution and I don't want to document the data several times over. My experience is going to be repeated by the user community at large. I'd like to have a tool that is likely to be used.

Michael Lefsky has done some excellent legwork in uncovering just what tools are available in the realm of spatial data set documentation. Essentially they fall into FGDC-compliant and non-compliant tools. Since we already have a fairly robust non-compliant tool at our disposal (fsldoc), I've chosen to use that one in my analysis. The other two tools that I'm investigating are the DOCUMENT tool that is part of arc/info and a METADATA extension to ArcView developed by NASA. These latter two tools are, for the most part, FGDC-compliant.

1. DOCUMENT appears to be fully FGDC-compliant. A great deal of the metadata standard is generated automatically by the software. The documentation is maintained in info files so that the metadata travels with the dataset when it is copied or renamed or, most importantly, exported. DOCUMENT produces an ascii file of the documentation and an HTML file.

   DOCUMENT confronts the users with a barrage of screens that are not always easy to decipher. No doubt practice and some hardcopy examples will help, but the program seems to ramble all over the place - a solid overview of the process is critical. Near the end of the documentation process DOCUMENT presents the user with a text editor screen with a variety of entries that require narrative. Some of the requested information seems to be a repeat of earlier menu entries. (However the entries are used slightly differently.) Most of our user community is not familiar with the text editors available in DOCUMENT. This will be very frustrating and may serve as an excuse not to document. Perhaps the greatest shortcoming is that only arc/info coverages and grids are documented using this tool. Finally, ESRI has announced that there will be no additional development of this tool. The company will be developing a tool to replace it, presumably in response to the rumored development of 'core' FGDC standards.

2. The METADATA extension (see Metadata Collection in ArcView) is accessed via ArcView. In my initial experiments with the tool, while the output appeared to FGDC-compliant, it seemed less complete that the output from DOCUMENT. The tool works with any theme in an ArcView project - arc/info coverages and grids, arcview shapefiles, and images. Output files are in ascii and/or HTML and arc/info coverages can have an info file created that will travel with the coverage. As more of our users do their work in ArcView and are less familiar with arc/info, I believe having a metadata tool in ArcView will be increasingly important.

   The METADATA extension also delivers a procession of screens to be filled out by the user. Again an overview of the process and some examples in hand should reduce the confusion level. The screens seem to be limiting. There seems to be no way to paste in already developed information without actually editing the output file. My fear is that actually including a complete set of FGDC-compliant standards will require significant file editing - which misses the point of having a tool in the first place. There may be modifications that can be made to the program to improve the tool, but many of us will have to learn a lot more about the Avenue programming language. Since this is a tool developed outside of ESRI, we should not expect support or improvements (other than our own). I suspect that any documentation tool in arcview will follow a path similar to METADATA, but if we commit to it we may have to redo a lot of metadata in the future.

3. FSLDOC is an arc macro language program that has been under development and revision for over six years. It was developed primarily to provide internal documentation for our user community - it was at least something! The output from FSLDOC is an ascii file. It is neither FGDC-compliant nor does it travel with the coverage. It is extremely robust and easy to use and captures what we've deemed to be most important for users who share data - attribute codes. Neither of the above mentioned tools provide an easy way to capture these codes. The documentation is specific to arc/info coverages, grids, and info files - shapefiles and images are not handled. While Barbara Marks, Hazel Hammond, and I have invested a good deal of time and energy in the development and updating of this aml, when we leave, interest in supporting this tool may leave as well. I'm reluctant to abandon FSLDOC, since it does relatively painlessly capture a lot of information that is valuable in supplementing the other metadata tools.

   I've taken a stab at rewriting gopherdoc.aml to add some features and to improve performance. The aml resides in /data/gisinfo/aml/fsldoc.aml. This program creates ascii abstract files or full documentation files. It can document arc/info coverages or grids using feature types; points, arcs, polygons, regions, routes, and grids. In addition info data files, relates, amls, and arcview project files can be documented. The files created are not FGDC compliant, however they do give some structure to the part of the documentation process that the arc/info DOCUMENT utility doesn't address well - attribute codes. Documentation created with FSLDOC will be quite useful in preparing documentation with DOCUMENT. The goal remains to have abstracts available in a dbms catalog for browsing and to somehow integrate information entered in FSLDOC into the arc/info DOCUMENT utility.

   Please give FSLDOC a try. It runs with pop-up menus and blanks to fill in. To run it, start arc/info, move to the directory that includes the coverage or file to be documented and type &run /data/gisinfo/aml/fsldoc. Before you begin you should be certain that the coverage has a projection defined. You'll also find that it's helpful to review the log for the coverage to determine when you deemed the layer complete, and to note the dates of revisions. The output files are written to the directory where you are working, so any mistakes you make can be edited later with vi, or textedit, or emacs. I'd be interested in hearing your ideas for improvements or additions or any bugs you discover.

My conclusion: The arcview tool seems to be the future. It needs improvement, but it certainly could serve our interim needs.