This document provides an analysis of the requirements which the system is intended to meet. See also the system's master QA document.
Melati is being developed to provide the infrastructure for the all future Paneris sites, and will hopefully be used by others when it is open-sourced.
The Melati project came to life out of discussions on the JAL messageboard.
JAL (Java Application Layer) is a class library layer which mediates between a relational database---currently Postgres---and webmacro controllers and templates, encapsulating database table rows as objects which handle tasks such as resolving linked records and generating HTML input fields semi-transparently. It also offers a comprehensive administration system for any database whose schema adheres to certain idioms, without further customisation. The present implementation of JAL is held to suffer from
The users who are going to interact with the system fall into the following classes:
Paneris programmers. Paneris programmers will use Melati as the core infrastructure for constructing most if not all of their customers' web sites.
The open source community. It is hoped that Melati will become popular among open-source developers outside Paneris. This will bring Paneris the familiar benefits of `free' debugging and further development, and, to Paneris's customers, the reassurance of (free) dual-sourcing.
Customers' end users. End users will interact directly with Melati, as they do with JAL, through its login screen (part of the access control facility).
Customers' site administrators. Melati will preserve JAL's very powerful generic database administration system, which allows all but the most domain-specific administration tasks to be carried out remotely through a web interface.
Customers' installers. These are the people who install Melati-based software on a fresh machine.
The services the system will offer its users are:
A facility for defining the database schema, including linked fields (`one to one' and `one to many' relationships) and field display preferences, in a convenient notation perhaps similar to that of Java class definitions. It is important that this should be treated as a minimum definition: administrators should be able to add other tables and fields, so that they can be used in templates and the generic administration system, directly over the web interface.
In the long run, it might be nice to allow inheritance relationships between tables to be supported automatically (but not by exploiting the Postgres inheritance facility, for portability reasons).
Database objects returned to the programmer via the Melati API should appear to her as Java objects with type-safe get/set method pairs corresponding to the fields given in the data structure definition. All fields, including extra ones defined dynamically by administrators etc., should also be accessible by their (string) names, to facilitate the writing of generic code; in particular, it must be possible to iterate over a persistent object's fields in a webmacro template (e.g. that for a generic data entry screen).
The programmer should be able to add domain-specific `convenience methods' to the bare persistent objects.
Explicit retrieval of linked records should not be necessary: the programmer should be able to treat the records as normally linked Java objects.
The API should provide for any kind of search to be performed, if necessary by means of arbitrary WHERE clauses. The special case of retrieval by a canonical ID field should be made trivial (but it must not be a necessary that this field is actually called id).
In general, the basic unit of retrieval should be the record/object. However, it must be possible to execute arbitrary SQL SELECTs returning a selective join of fields---on the understanding that this will bypass the cache.
It may also be worth thinking about providing an API for retrieving only some of only some of an object's fields (the remainder being fetched on demand), and, conversely, for pre-fetching linked objects using a single call to the backend.
At least a simple kind of `transactional' capability must be provided: e.g. commit/rollback of all changes made to the database during the processing of a particular HTTP request.
The programmer should be able to control the manner in which Melati caches data from the RDBMS: provisionally, the maximum size of the cache for each table, and perhaps whether the results of each individual operation which might trigger a SELECT should be cached.
The cacheing mechanism must interact consistently with SQL transactions. If a cached row is changed through a non-autocommitting connection, the change must be visible through that connection only, and must revert correctly if the transaction is rolled back.
Melati should provide support for convenient internationalisation: language-specific table columns should be localised transparently or semi-transparently.
It must be possible to enforce restrictions which ensure that only groups of named users (as determined by knowledge of a password to go with their username) are able to access certain services, defined on a per-record basis, on a Melati-based site. For the programmer's convenience and to avoid security bugs creeping in due to programmer oversight, the checks implementing access control should be made as close to the data as possible, so that in principle all she has to worry about is passing in the right `user' object or authentication token (once only). There should also be a way for her to perform specific checks of her own if she wishes to catch authentication-failure situations before wasting any of the user's time.
For similar reasons of convenience and robustness, Melati should allow the programmer to define access control policies---for instance, one to the effect that an invoice should be visible both to its issuing party and to its receiving party. (But arbitrary, per-row permissions setting must still be possible for the administrator.)
The system should support the definition of user groups (and of group-groups, etc.) for use in authentication rules.
Basic access levels to table records should include `none', `read only' and `read-write'.
There should be a simple and unintrusive way of placing markup implementing views of and data entry fields for objects in a webmacro template, and of reading fields back into objects when they return from the user.
This facility should be markup-language neutral, so that we can as a minimum write templates in any of HTML, XML and WML.
Javascript validation checks should be added automatically or semi-automatically to input fields where appropriate.
The services provided for Paneris programmers must be sufficiently accessible---easy to grasp, convenient to use, well documented---for people from the wider open source community to want to use them too.
End users must be able to log in to Melati-based sites for authentication purposes, either on demand (when they attempt to do something for which authentication is required) or explicitly. They should also be able to logout or re-login under a different username at any time. It would be nice if they had the option of storing their logged-in-ness across browser sessions in (e.g.) a cookie (but cookies must not be forced on them).
Melati must provide a facility for reminding users of their password, by some reasonably secure means, if they have forgotten it, and allow them to change it at will.
Melati must support the functionality of JAL's current generic administration system: briefly,
For an example of JAL's admin system in action, see: http://paneris.org/webmacro/org.paneris.jal.controller.Admin?db=paneris&action=tables
The admin system should support localisation of an internationalised database in some convenient form.
At the request of the installer, a Melati-based site should be able to (re-)create and initialise a fresh database so that this doesn't have to be done manually.
When a Melati-based site starts up, it should be able to verify that the database schema in the RDBMS is safely consistent with the program's idea of what should be there, and fail with an error message if it isn't.
Melati is targetted to run on a Linux machine (2.2 and up) as a Java process (1.1.7 and up) using Postgres as a backend RDBMS (6.5 and up) and webmacro for the templates. We should feel confident that we can replace Postgres with any other full-featured SQL RDBMS for all Melati-based sites inside a week at most. For the benefit of the wider open-source community, it's important that Melati should ultimately be portable to NT4/W2K plus SQL Server (for developers as well as for running live sites).
It is acceptable that no other processes should be allowed write access to the database in the backend RDBMS while Melati is using it (apart from those run by hacker-administrators who know what they are doing; without this restriction, cacheing makes no sense, and Melati will have continually to check that the database schema is not changing under its feet).
Melati-based sites should be able to handle HTTP requests for simple templates including linked records at a rate of twenty hits per second. This should suffice for all applications except heavy-duty ones for which the tradeoff point between development effort/maintainability and implementation simplicity/speed falls unusually far towards the latter.
Melati-based sites should offer reasonable security: we should feel happy about allowing it to control access to user preferences, messageboards etc., but facilities for spending money and deeply private information will have to be protected by an external mechanism.
Given the consistency of Melati's cacheing mechanism, and low-level data integrity checks, the site's robustness should be limited only by that of the RDBMS, and of course the site-specific code.
We may want to look into running a watchdog process which restarts httpd and everything else every minute or two if the system fails a test transaction, and sends a warning email or whatever.
It should be very surprising and worrying for a Melati/Postgres/Linux site to go down.
It's unlikely that the requirements proposed in the present document, once reviewed, will turn out to be seriously wrong, because JAL and the sites developed using it provide an excellent case study of what is needed and what can in practice be supported.
The main doubt relates to the data structure definition: it may be that a simple `language' turns out to be too inflexible to map onto the kind of CREATE TABLE we need in complicated cases.
William Chesters <williamc@paneris.org>
Most recent CVS $Author$ @paneris.org
The current quality level of this document is: Beta.
The important points in the life of this document are listed below (for everyday update information, consult its CVS log).
The CVS log for this document is: $Log$ Revision 1.1 2005/11/21 22:01:49 timp Moved from site/doc Revision 1.12 2000/02/29 09:53:02 williamc Finish recovering from disaster; point out in the docs that you can 'add methods to table rows' Revision 1.2 2000/02/04 18:28:34 williamc Add QA stub; explain DSD-admin interactino a little better