Dwight,

Thanks for your letter. I agree with you that the information should go into the DAT files. I am working on a new data format that will allow all sorts of new information to be included in the files. I had hoped to have the new format done and implemented a couple years ago, but things have been very busy for me over the last year and a half. In the last couple of months, I have gotten a lot of feed back about the new file format, so maybe the delay has been useful.

If you are interested in the changes that I am planning, I have attached a copy of the newsletter I sent out a couple years ago. Some of the features mentioned have already been implemented. I have also spent the past few months converting CaveX so it would work under Vista (not an easy task.) I have also converted all the help files to a Vista compatible format. Since there are nearly 1000 pages of documentation with Compass, it has been quite a tedious project.

My schedule will be pretty full until after the first of year. My current work project is software for the Big Bear Solar Telescope in California

http://www.bbso.njit.edu/newtelescope/index.html

I’m hoping my schedule will finally slow down after the first of the year and I will be able to finish up all the changes that are half finished.

Larry

=====================================================================

COMPASS NEWSLETTER

Judging from my emails, there seems to be quite a bit of interest in the changes I am planning for Compass. As a result, I thought it might be useful to list in detail the changes I am planning on making over the next year or so.

1. NEW FILE FORMAT. I originally developed the Compass file format around 1979. At the time I was working on a mainframe computer at the School Of Mines here in Colorado and only had a few thousand feet of cave data from Groaning Cave to work on. I never anticipated that it would become so widely used.

Since then, the format has been modified a few times to handle new data items, but Compass can still read and display files I have archived from the early 1980's. However, as time goes on, I am having more and problems adding new features to the format. For this reason, I am thinking about shifting to a new XML-based format that will make it much easier to add new data items and make changes to the format.

I am planning on using some ideas developed by Devin Kouts:

http://www.psc-cavers.org/xml/#designRequirements

For those who don't know what XML is, it is data format similar to HTML that allows you to create your own specialized document formats. It uses tags, just like HTML, except that you get to define your own tags. For example, the tags for a cave survey file might look something like this:

<CaveSurvey>

 <SurveyName>Groaning Cave</SurveyName> </CaveSurvey>

There are several advantages to XML. First, it is "extensible".

That means it is very easy to add new items without breaking old programs that don't know about the new data. Second, the general XML format is already defined so it is easy for people to write programs that can read any XML file. In fact, there are general purpose utility programs and libraries that can read an XML file without knowing anything about the actual data that resides in the file.

The biggest disadvantage is that it is verbose. This means that the files can be much bigger than with simpler formats.

I have already written some code to write the new format and have discovered that the new format would be about 8 times bigger than the old files. The worst part is that data takes about 10 times longer to read or write. For example, writing one of the sections of Lechuguilla takes 0.4 seconds using the old format and 5 seconds under the new format even on my

2.8 Ghz Pentium-IV. This will be less of a problem as computers get faster, but it is probably too slow for some older computer. For example, someone with a 280-Mhz Pentium would take 50 second to read one Lechuguilla file and several minutes to read the whole cave. I think I can speed things up by simplifying the data format for the shot data and not breaking everything into independent tags.

Even with the new file format, I will continue to support the old format for a long time to come. I will make sure all the Compass programs can read the old format and  I will have programs that can convert the new data back to the old format.

I have written a test program that reads the old Compass files and can write either the old or new XML format. If you would like to look at a sample of the new data or play with the program, you can down load a copy here:

http://www.fountainware.com/download/xml.zip

WARNING: Be careful how you use this program particularly when you save the files to the old format. It has not been tested extensive and it could damage your cave data. Make sure you only test it on backup copies of your data.

2. USING MEMORY. Up until recently, Compass' strategy for processing files was to parse the data on the fly, reading one character at a time and only retaining the data it needed to address the problem at hand. This reduced the memory requirements of Compass and allowed it to process large caves even with the limited memory available to the early Apples and PC's. The main problem is that you only have a certain amount of information in memory and this limits what things you can display. For example, the Viewer only has the shots and LRUDs data in memory. As a result, it is impossible to do other things like highlight individual loops in the Viewer.

Today, memory is no longer a limitation and on the typical PC, it is very easy to allocate multi-megabyte arrays. As a result, I can now keep a lot more information in memory including loops, shots, comments, etc. I figure I need about 150 bytes per shot to do everything I want to do. For a cave like Lechuguilla, that would equate to 7 or 8 megabytes. For a cave like Mammoth, the requirement would be about 20 megabytes. These number are well within the capability of modern PCs.

3. UNDERLYING DATA STRUCTURES. Another aspect that will change is the structure of the data stored by Compass. In the past Compass used a fairly simple data structure, with just enough information to keep track of station locations and to find loops.

The new data structure will be designed to support "graph theory" operations. Graph theory is the branch of mathematics that deals with the kind of "networks" of interconnected lines that cave surveys generate. In keeping with the Graph Theory model, the new data structure will have "edges,"

"vertices" and an "adjacency" information.

Graph theory will allow Compass to add a bunch of new features such as finding the shortest distance between station, and even finding the easiest path between stations.

Most of the new data structure has been built and code has been written to work with it. The only remaining task, (probably the hardest part), is to integrate it with the rest of Compass.

4. NEW LOOP CLOSURE TOOLS. With the new data structure, I will be able to do a better job of finding loops. Currently, under certain circumstances, Compass does not find the smallest set of loops in the cave.

I have already written routines that do a much better job of finding loops. In fact, I have written an article about the techniques I am using for the NSS publication "Compass and Tape." You can get a preview of the article here:

http://www.fountainware.com/compass/FindingLoops.doc

You can also download a test program that generates large quantities of overlapped and randomly connected loops. The link is in the article.

4. NEW ROOM MODELING. I am also working on a system for handling large rooms. It will be based on using "splay"

shots. Basically, you choose a station near the center of the room and take shots to the wall in various directions.

Since the user might enter any where from one to a dozen splay shots, the plan is to have Compass interpolate and convert the splay shots into 8 or 10 evenly spaced pie-shaped wedges that would describe the room.

Ideally, you would have at least four shots at evenly spaced angles to the wall, but the system would handle any number with any spacing. Of course, the more shots, the more precisely it would define the room.

The splay shots would be entered just like a regular shots except that the "To" station would be a special symbol.

Currently, the plan is to use a right parenthesis ")" for the "To" station. (It turns out that parenthesis is one of the few character that exists on most of the non-English key boards. It also vaguely resembles a passage wall.)

The Up and Down dimension would apply to the "To" station of the shot. In other words, the Up and Down dimension would describe the wall height where the shot makes contact with the wall. The height of the center of the room would be controlled by the LRUDs of the "From" station. This should produce a reasonably realistic room shape which could be either convex or concaved. However, it would not model a side branch of the a room wanders around a corner.

I have written test code and it seems to work well and I may put this feature in Compass fairly soon.

5. RUNNING PROFILES. If you have been reading the Compass Newsgroup you know that there have been some extensive discussion about what is know as "Running Profiles" or "Developed Profiles." These are profiles where the turns and bends in the cave passage are "unrolled" and the passage is displayed as though all the shots were in a straight line.

In an ordinary profile, it may not be possible to see all the ups and downs of the passages because some shots maybe pointing directly toward or away from the viewer. Rotating the cave does not help, because as you rotate one shot into an optimal position, other shots will rotate into a bad position.

The feature is very popular in Europe where there are lots of deep pits and not many maze caves. The feature works best in relatively simple caves that have few loops and interconnects. In the US where there are lots of maze caves, the feature is only useful on small segments of a cave.

Again, I have already written some test code to experiment with the feature. I think the feature will be easier to implement when I have the new data structures in place. However, there may be a way to implement the feature sooner.

6. SVG/BITMAP WARPING. For several years I have been looking at ideas to help cavers create finished maps. One problem with finished cave maps is that they become obsolete whenever a new survey is added or loop errors are corrected. Ideally, cartographers would like to draw the map once, and then have a computer program warp the map to adapt it to changes.

There are several ways Compass could address the problem:

A. BITMAP MORPHING. First, it could import scanned bitmap images of survey sketches and then register various stations to locations on the bitmap. If a passage moved as a result of resurveying a passage, Compass would warp the bitmap using the kind of "Afine" transform used for "morphing" images.

This is the approach used by Garry Petrie's Karst program and it is relatively easy to do. It works fairly well on small caves, but it does not really produce finished maps. It is also tedious to piece together multiple sketches to produce a full-scale cave map.

I am just finishing up some changes to Compass that will take the first steps toward adding this feature. The next release will have the ability to register a bitmap image to the cave so that no matter how much you zoom, pan and rotate, the image will stay locked to the cave.

B. ROUND TRIPPING. Another approach is to export and import cave survey data to a drawing program. For example, Walls exports and imports cave data in the SVG format; a format created and promoted by Adobe. The cave data is then imported into Adobe Illustrator and then passage wall details, symbols etc. are added. If the cave changes, the SVG data is loaded back in the Survey Program and the map is warped to match the changed survey data.

This model has several advantages, the main one being that with Adobe Illustrator, you can produce very beautiful maps maps. One disadvantage is that Adobe Illustrator is fairly expensive and maybe not the best tool for cavers. I also have questions about how well this works with larger or complex caves. Finally, there are questions about how universal SVG will become and whether it will be adopted by other programs.

Another option would be to do something similar using a less expensive drawing program such as Corel Draw, or Canvas. DXF would be the ideal format because it is a nearly universal file format.

7. GLOBAL STATIONS/PREFIXES. One of the problems with cave survey data occurs when you have duplicate stations. This usually happens when try to combine several caves into regional map. As you probably know, Compass uses a "Linking"

system to deal with this issue. Although, it works well, people are often confused about how to apply it, particularly when the linking stations have to be "carried" between files.

A different option that might be easier to understand would be to allow the user to declare certain stations in each file to be "Public" or "Global". These stations would be the only stations that would be visible to other files. In practice, it works similar to the "Links" but it might be easier to understand and use.

Another option would be to add prefixes to each survey file or even each survey. The prefixes would then be added to each station name to help prevent name conflicts. This is what Survex does to deal with British cave data which often has duplicate station name even in a relatively small cave.

8. LOOP CLOSURE. As you may know, there has been some controversy about the efficacy of various loop closure techniques. I am convinced that the routine described by John Halleck are superior to other methods:

http://www.cc.utah.edu/~nahaj/cave/survey/

My plan is implement John's ideas after have I changed the file format and data structures.

9. INCREMENTAL UPDATES. While I am working on all these changes, I will continue to make updates to the current version of Compass. I get lots of requests for new features and many of them are relatively easy to do (or at least fit into the current data structure.) For example, I have made about 40 changes to Compass in just the last six months.

Currently, I am working on adding a standard-deviation field for Fixed Station. This is needed because GPS readings are often inaccurate and can distort the cave if more than one entrance is tied to a GPS fix. I am also finishing up the feature that locks a bitmap image to the cave.

I expect to have these two features finished in a couple weeks.

Although many of these changes have in place for months, I will probably make a public announcement in the next few weeks so that everyone will be aware of the new features.

The big changes will come later.

10. TIME CONSTRAINTS. I work on Compass whenever I have free time. This depends on how much other work I have. Typically, I have several months each year when I am between contracts to work on Compass. As a result, the time frame for completing all the ideas listed above will depend on how much other work I have. I'm hoping to have the majority of the changes done within six months or a year.

In case you are curious about what kind of work I do when I am not working on Compass, the last five years I have been writing control software for large astronomical telescopes manufactured by DFM engineering:

http://www.dfmengineering.com/stan_cass_tele.html

In addition to the Telescope Control Software, we have been designing Radio Telescope Control Software for colleges:

http://www.pari.edu/OnlinePARIPresentation

We have also been retrofitting NOAA's Satellite Tracking antennas in Virginia and Alaska with modern control systems and satellite tracking software:

http://www.fcdas.noaa.gov/facilities.html

http://ivs.nict.go.jp/mirror/publications/ar1999/nsgcgo/

http://www.photolib.noaa.gov/space/spind5.htm

http://www.photolib.noaa.gov/space/spac0185.htm

http://www.photolib.noaa.gov/space/spac0195.htm

Finally, I am doing a lot of work power for a company that make a very nice power and inexpensive power monitoring

tools:

https://www.doubleed.com/secure.html

Larry