requirements for a free settlement library network

> Paul,
>
> I'd very much like to thank you for your posts,

Thanks.

> this is a topic
> which I find fascinating as part of my day job includes creating
> systems for storing large amounts of data gathered from satellites
> and then being able to index into the data and perform analysis.

For fun, although it is probably not robust enough or well tested enough
yet for me to recommend it to others for production use, if you use
Python you might try playing with the Pointrel system I wrote.
http://sourceforge.net/projects/pointrel/
(I also have other versions with somewhat different architectures in a
variety of languages (C, Delphi, Scheme, Java) but I haven't released
them (except a Smalltalk version) for various reasons -- mostly related
to not wanting to support too many versions as the ideas evolve.) Most
of this Pointrel work is motivated in the end by a desire to create a
knowledge repository useful for designing space habitats. Over the last
twenty years or so I've been working on the Pointrel System on and off
(mostly time spent removing things), many other good ideas (some
predating my efforts by a long time) have gained popularity and free
implementations (RDBMS, RDF, ER, OO, etc. databases) and in practice any
of those may be better choices right now for production work (RDF is the
most similar to the Pointrel System in many ways). For Pointrel, I've
reached the point where it is settled down enough that the big issue is
writing more applications on top of it (and defining those standards for
how data is used within the system is a whole different issue).

> You've also provided a few very good links and for that I thank
> you. Now onto your post:
>
> I agree to the problems of editting documents with propietary
> information in it, however (this is coming from a non-lawyer) adding
> commentary to it should not violate the copywrite so long as you are
> not redistributing or modifying the original. This is a touchy
> point I know but by smartly linking into a text, you may be able to
> add comments without violating the copywrite.

Good point, and as you say, a complex issue of what constitutes a
derived work. My point is sort of that what we need to be more about
here in the space settlement community is the incremental collaborative
design of specific artifacts needed to make space settlements (like
airlock doors, service pods, coffee mugs, solar collectors, hydroponic
tanks, mining tools, etc.). So while discussion of ideas is important,
and as you suggest linking to web available content and commenting on it
separately is an approach that is the most likely to be legal without
prior authorization for purposes of discussion, I think that approach
will be inadequate for purposes of refining the design of an artifact.
And by artifact I sort of really mean the CAD/CAM files that somehow
tell a CNC machine or Star Trek nanotech replicator :-) in detail what
to make and how to make it, or alternatively, that detail the object
enough so it could be simulated in a virtual world for purposes of
further refining the design through simulated testing either alone or
as part of a complex system.

> But I do agree that
> for a project of this scope to work, it would be far easier if
> everyone published under a gnu-like liscense.

In the end, while one license like the GPL will be nice, I think the
system is going to have to handle material under a variety of free
licenses. Which means some form of not exactly permissions management,
but rather assertions of freedom management. See for example:
http://groups.google.com/groups?selm;0B2DD1.40BDD739%40kurtz-fernhout.com

> Have you considered a system based off a peer to peer network
> schema. This would allow for a stronger system in general
> especially if various nodes in the network mirrored each other.

Yes, that's a great approach. BitTorrent seems hot these days, as is
Freenet, and such things could potentially form a back end. And with the
recent court rulings in the US, apparently peer-to-peer systems are
gaining more legitimacy, with the winning side arguing the users of
p-to-p, like users of VCRs, are the ones liable for copyright
infringement, not the makers, something I had been worried more about
until this recent US ruling.

I personally would not want to run an unfiltered node of these types as
I do not want to generally help distribute materials that violate laws
(including copyright ones or censorship one). So, I'd want to run
peer-to-peer software which had detailed information about content that
one could filter on, perhaps using some notion of trusted peers or
separating out sharing common mostly innocuous global design information
from sharing materials only shared within trusted workgroups. At the
very least, for legal reasons, it would be prudent to use a p2p system
that uses something like the principle of XORing data with large random
blocks to create multiple overlapping unintelligeable segments, and
which requires having a short key (which could be filterable?) defining
which segments and which decoding strategy constitute a document. That
way at least there is plausible deniability by the user when handling
data that is transmitted (as the same overlapping segment might be used
to construct multiple documents). And, even in that case, I would still
want to filter out illegal or what I saw as immoral keys from being
received by or redistributed by my own workstation (even if in that case
my workstation (presumably) could presumably be legally redistributing
those large independently unintelligeable binary objects). See for
example "Mnemosyne: Peer-to-Peer Steganographic Storage"
http://www.cs.rice.edu/Conferences/IPTPS02/107.pdf
one of several such proposals.

Of course usenet newsgroups are currently functioning as a distributed
system for content distribution that has not yet been shut down for
legal reasons (claiming substantial legitimate use), so another approach
is to piggyback a more fine grained distributed design information
system on one or more usenet newsgroups as the transmission medium. I
just recently adapted an existing python newsgroup server program to use
the Pointrel System as a back end just to start to play with this idea.

> Though this could end up being a nightmare if you try to incorporate
> document change requests.

Yes, this requires thinking through the way documents are referenced,
stored, and communicated. Still, distributed versioning and/or
communication systems have been with us for a long time, and IBM/OTI's
ENVY system is just one example, and IBM/Lotus Notes in another, as in
Groove. None of these is free obviously, but there are many server based
related free projects and free standards (WebDAV, CVS, Subversion, etc.)
although none so far seem to be quite what I want in a potentially
distributed sense. I had looked into this more a few years back and
admit there may be newer projects I have not kept up with or
improvements on old ones (like WebDAV) I should really track better. I'm
sort of looking for something with more of an AI flavor, which is what
is lurking behind the Pointrel System's fine grained and consistent
internal references, even though the Pointrel System is not really an AI
system. So, I'm not exactly looking for a system to manage files or web
pages so much as a system to manage arbitrary networks of nodes (if that
makes sense).

> btw, another great analogy that more people could relate to for a
> project of this scope if Isaac Asimov's Foundation. This book is
> basically what you are talking about.

An interesting idea -- although First or Second Foundation? :-)

> glad to hear I'm not the only user of debian out here!

Woohoo! :-) I finally switched over completely about a year ago. I think
I have booted my old Microsoft Windows 2000 HD about twice since then
(mainly to do my taxes). I really seemed odd for the first month or two,
not being in Microsoft's clutches anymore but still running Mozilla and
Open Office, and thinking, wow I can now write to a free API and forget
about lots of Windows issues (although I had previously been moving to
wxWindows development anyway). It was especially amazing learning how to
do "apt-get update; apt-get dist-upgrade" and having it all work.
Personally, parts of it have also been a pain-in-the-ass (earlier
printer/scanner support for an HP OfficeJet), but overall, I'd say it
was worth it and I am not looking back. It helped also getting a Sharp
Zaurus shortly afterwards and being able to think about the commonality
of GNU/Linux on multiple platforms with different CPUs and hardware
issues. Mainly I saw it also as part of a career move aspect, since
GNU/Linux on embedded hardware is heating up (compared to VxWorks etc.
work). And of course, what else are we going to run on the computers in
free space habitats? ;-)

> Though I do
> not necessarily think debians packagin system is the best model but
> it does have some features which should be added to any system.

Agreed. It's more just an example of the sorts of issues and that people
are dealing with them.

> Most importantly is the ability to link an object with its
> dependencies. For research papers, this is a biggie as papers are
> always citing other papers. It would be nice if you could use a
> tool that can easily link to papers being cited. (which is what
> wikipedia does do)

Good point. Although as I wrote before, perhaps we are moving to a model
where the notion of "a research paper" with a well defined author may be
sort of disappearing in favor of conversations (email archives) about
artifacts which are improved (like Wiki pages). But still linking to
previous versions and supporting content remains essential. It's sort of
like how movies on DVDs aren't like ones on VCR tapes -- they often
include deleted scenes, background notes, directors and actors
commentaries, and so on.

> the seti at home people I believe are open sourcing their software
> for distributed computing. I highly recommend using something like
> this that allows users to opt for allowing their computers to be
> used for distributed processing.

Thanks for the suggestion.

> i am against in general client side processing. When we first
> implemented the system I mentioned previously, we used a fat client
> scenario similar to what you are stating by using java applets
> (necessary as our clients are on various hardware platforms). We
> quickly found out though that this didn't scale so good as race
> conditions emerged and various other problems with how jvms are
> implemented on different systems as well as distributing tools used
> for analysis

Coming from a Smalltalk background, I think poorly of Java (even in its
later versions). Also, based on my experiences with Python and Squeak
Smalltalk (and also unfortunately Java), I think a language with a main
unified codebase ported across all platforms (like Python or Squeak) is
going to be in general be far more stable across multiple platforms than
one which tries to certify independently developed VMs coded to an
interface standard (as Sun does with Java does). Also, Java has other
problems (to begin with no first class functions). I would have to say,
in my opinion, the computer world would have been better off if Java/Oak
had never got out of the lab. While I don't currently use it for
licensing reasons (despite having spent around $10K on related licenses
years back for an earlier version for a project), VisualWorks Smalltalk
is really the system that technically should have been selected wherever
Java went (but the VisualWorks product suffered from mismanagement).
Most of Java's potentially good features (hotspot optimization, Swing,
etc.) were taken from Smalltalk. As Alan Kay has said, any formal
written standard with more than three lines of text is ambiguous, and
Java's history really shows that.

So in any case, I'd be careful about generalizing from any Java project
(especially ones related to applets instead of stand alone
applications). Clearly Java's only remaining major niche is now server
based solutions, because Java server solutions entail only running code
on well defined software platforms which can be tested once and highly
controlled (although embedded JVM cellphone games also show some life).
While any client based software will suffer from issues related to
potential incompatabilities of underlying OSes and versions of
supporting libraries, in general I'd say systems built on the likes of
Python or Squeak (or Perl or GCC for that matter) do not suffer from as
many of the sorts of problems distributed Java applications suffer from.
Years ago I used to work alongside people developing IBM's HotMedia for
Java browser applets and wow did they have a lot of JVM quirks they had
to work around to get the stuff to work on most browsers. And none of
that sort of bug discovery was very much fun in many ways.

Hybrids are also possible combining server and client processing,
consider something like ActiveWorlds
http://www.activeworlds.com/
with a world maintained by a server but 3D graphics done on the client
side. And I think such a shared 3D world or worlds could eventually be
part of this effort, sort of as prototype space settlements (but with
more real logistics than an ActiveWorlds world).

> The important thing here is considering the capabilities of clients
> and of the servers. Its a hard line to draw and a point which is
> way beyond this list.

Good point, and really moving into something that might end up as a
separate discussion list.

>>== moving forward with specific recommendations ==
>
> In this section all I can say is do not limit yourself to one type
> of development application. instead embrace a way of interface like
> CORBA. This provides much greater flexibility to developers. Just
> cause you're comfortable with Python, C/C++ doesn't mean others
> are. instead adopting an interface model, you are allowing
> developers to decide.

I think you make excellent points here, and I should revisit what CORBA
can do these days. I haven't really done much with CORBA (in part
because CORBA seemed to me kind of a large and ponderous thing).
http://www.omg.org/gettingstarted/corbafaq.htm
Also, I really do believe in the Alan Kay idea above of how standards
without shared reference implementations often spell lots of trouble
(like with the failure of Java's promise of write once, run everywhere,
which things like VisualWorks had delivered on a decade earlier). But it
is true that much of the success of the Linux kernel has been attributed
to Linus Torvalds' designing of good APIs, with the idea that once an
interface function convention is defined, the interface is sort of set
in stone as everybody starts to use it, but the underlying
implementation can always be easily changed and improved. So that argues
for your point. Still, I can see there being a lot of new code showing
up in the core of the project, and I think it makes sense to pick some
good approaches. Also, sometimes, arguments for shared interfaces rather
than shared code come about because someone wants to make a proprietary
solution to work to the interface. So, while I'm all for good free
shared interfaces, I guess I am also for good shared free code.

You have a good point in handling various languages, although as for
Python/C, actually I'm more comfortable in Smalltalk in many ways. Using
Python (despite some of its flaws) is a compromise I make. Part of it is
that for free software projects that desire others to work on them, the
bulk of software developers are generally more familiar with the syntax
of C++ and Java and the process of using their favorite text editor, and
so they can adapt to Python more readily than Smalltalk and its better
integrated development environment. Most C or Java programmers seem to
pick up the core of Python in about an hour or two
http://www.python.org/doc/current/tut/tut.html
(although naturally it takes much longer to really get to know the
language's standard libraries, but often that can be learn as you go).
Also, there are free licensing issues in Python's favor, i.e. the best
Smalltalk, VisualWorks, is proprietary, and the others all have various
technical and/or licensing issues. Also, Python is easily maintainable
in large projects with multiple developers of varying skills (unlike,
say, Perl or Forth or even C++). I shudder to think of the pain of
anyone trying to coordinate volunteers programming in Perl or C++ -- you
would need to enforce draconian coding standards, etc. Python also has a
growing user base.
http://www.python.org/

But, again, this issue is probably best discussed in depth if needed
elsewhere, and I acknowledge contributions might come in a variety of
languages, especially when related to preexisting useful packages. The
important thing is that unlike at the time of the Apollo design effort,
when the average engineer at best could use punched cards in an
institutionally owned mainframe (if they were not using a slide rule and
paper) and had to do multiple mechanical tests of their ideas, we now
have free solutions involving networked personal computers that can
amplify the engineering efforts doable in people's spare time and allow
much testing in desktop simulation.

> while on this subject of tool, I would also suggest those who add
> papers to such a project look at a common format. My favorite for
> documents is xml as you can include a fair amount of metadata within
> it and still imbed objects. I cannot stress how important coming up
> with some sort of common document interface such a project needs. I
> am a big fan of data that describes itself (hence almost my leaning
> towards CORBA as an interface). Again, I'll skirt away from in
> depth discussion on this as I believe the details would only
> interest software engineers.

Yes, I think the adoption of free and open formats for all project
communications is absolutely essential.

> [snip]

> As you've mentioned already, ITARs are definitely a thorny issue and
> much space design work falls under this category. Even NASA has in
> some of its tools provisions that only US citizens cane use it
> (http://itos.gsfc.nasa.gov/)

That web page is scary just looking at it! :-)

Looking up ITAR:
http://www.epic.org/crypto/export_controls/itar.html
One big problem as I think more about the specifics of "International
Traffic in Arms Regulations" is this: "authorizes the President to
control the export and import of defense articles and defense services."
Is a rock defensive? Denying a potential future enemy food could be seen
as defensive these days under a policy of "preventive warfare" (not even
simply "preemptive" anymore). So I guess ITAR compliance for projects
that aim to have everyone on the planet and in space live together in
peace and prosperity is going to revolve around a very bureaucratic
process of deciding what is currently labeled defensive or not and to
what extent, especially if these definitions change over time. Again, a
good argument for a distributed framework so people can implement these
policies on their own workstations.

>>== the future of engineering as a profession as open source ==
>>
> while this is true in the software world, I am not so confident this
> applies to any other branch of engineering. I've never seen open
> source version of an ion-thruster design or lithium battery.

That's exactly my point. And that is what needs to change if we are to
have free space habitats, as opposed to proprietary, say, a GE space
habitats or NASA space habitats which in reality translates into *no*
space habitats (since it's easier for GE to make money on the ground or
to use automation and AIs in space given a thirty year timeframe).

>>== 1970s technology is all we need for a first version ==
>>
> nice in thought, but I'd hate to have to develop on a 8086
> motherboard! There has been many advancements made since then in
> space related hardware which greatly reduce the costs. I'd hate to
> ignore it all, but then again, you have correctly pointed out the
> NDAs can be bothersome and most of those only apply to more current
> technologies.

Yes, the more current stuff we can include the better. And certainly as
a first cut, one can think of as trying to make 99% by mass of the
system (or any other arbitrary percent) as based on free technology.
Then one can address the other 1% of "proprietary vitamins" in due time
(hopefully).

One of the big problems beyond NDAs is simply employment agreements
limiting outside engineering work (even as a volunteer) which many
engineers in the US sign as a condition of employment. Some of Europe,
and I believe also the state of California, have laws against such
requirements though. Still, contributions by such legally chained
engineers could pose legal problems for any such free projects they
contribute to unless the employer signs off on the volunteer work.

> Again, thanx for an interesting topic!

You are welcome, thanks for the detailed reply.

> Ryan Z

--Paul Fernhout
http://www.pointrel.org

[This document is hereby placed into the public domain (except as noted,
which is presumed fair use in the orginal publication context, your use
may vary); but remember that moral and legal laws other than copyright
say you can not modify it in a defamatory way such as to indicate I
wrote the modified version or to claim that you wrote it if
substantially unaltered.]