The ORM Foundation

Jakob, BrianC (and others),

My ActiveFacts project has produced:

• A metamodel in ORM2 which is capable of fully(*) representing ORM2 (and more - like units support, multiple overlapping models, multiple natural languages for the same model, reference populations as well as an example one, join constraints beyond what's possible in ORM2, etc)
  
• A textual language (CQL, the Constellation Query Language) that is concise and may be used for the expression and interchange of models,
• A converter for NORMA models to CQL (not the reverse, though only diagram layout would be difficult).
  
• A comprehensive implementation of my Rmap algorithm is included (it improves over NORMA's implementation),
• An initial SQL generator for MS SQL Server (additional generators are only an hour or so of work),
• An object-model generator, which is instantiated as a generator for the Ruby programming language (however UML class diagrams would also be a simple artefact - again, except for automatic layout).
• An information management API for in-memory representation and manipulation of information pertinent to any model (this supports the generated object-oriented code).
  

All this is available as open source for any platform, using the very liberal zlib license. For more information, visit http://dataconstellation.com/ActiveFacts.

(*) ORM2 Model Notes are not currently imported, and the CQL syntax for annotating subtype mapping modes isn't yet available.

CQL further includes a query language (as you might guess from the name!) however the implementation is incomplete at present - work is ongoing. The language is designed to be easy to learn and use, suitable for generation from a point&click interface, and have greater expressive power than SQL select.

The relational mapper is constructed so as to be able to apply schema transforms during the process. For example, with a small plugin extension, a database using PostgreSQL could be generated using the inheritance features of PG. Ruby on Rails strongly prefers every table to have an AutoCounter ID field as a primary key - this too will be a small plugin option during mapping.

The generator for the Ruby programming language is supported by the Constellation API, on which work is continuing on an entirely new way of programming database applications - a fully fact-oriented semantic approach, with no exposure of the underlying relational machinery.

I also plan commercial implementations for C# and Java, and have been working on a web-hosted (Flash-based) graphical ORM2 designer and query builder, so as to allow folk to try semantic modeling without even having to subscribe, let alone to install any software.

Yes, I'm tired now. I've been working on this full-time for most of the last two years, and I have nothing to sell yet, nor have I yet made a substantial business from consulting in the area. Obviously I hope to... but then I hoped that this community would catch the vision a bit better than they have as well! I've found some folk in the community to be a bit tendentious and closed in regard to sharing (though with notable exceptions, Terry and the NORMA team!), and I find that very wearing. That is what I believe is the reason fact-oriented modeling hasn't made more progress - it's underlying the lack of the kind of demonstrated project and community successes that lead to marketing impacts. The longer it fails to flourish, the harder it will be.

The other factor that has kept ORM from the mainstream is that there have only ever been design tools for it. The tools have only solved problems for which existing solutions (though perhaps inferior) are widely available and well-understood, so the development community has stayed with the mainstream. We must produce tools that make adoption mandatory in order to stay competitive; the development industry has too much inertia. This is the reason for me developing a semantic query language and a semantic application runtime, and for making them both freely available.

I still live in hope that I can help unify this community around a technology base that can truly deliver the promise of this amazing foundation.

Ken Evans:

the ORM graphical model because it is precise and less verbose than any of its transforms

 

I have found this claim to be quite false.

A model converted from NORMA to CQL takes almost exactly the same amount of paper to print (to an equivalent level of readability) and in fact contains  the data that is hidden in property sheets in a NORMA model. Yet it still encodes the same meaning, has the same formal semantics, and moreover is directly readable by an untrained user.

The software that supports CQL is command-line oriented, so it may be easily used in batch mode software build processes, unlike NORMA.

In addition, it can be emailed to any user who can peruse it regardless of what software they have available. That includes the business experts who alone can add the distinctive value of semantic modeling over traditional modeling approaches. These people are the ones who matter, and they will almost never have Microsoft Visual Studio available to them.

 A common textual modeling language is the only path to the success of the ORM2 technology - all the other tools are worthwhile but peripheral.

*** Heath:

A model converted from NORMA to CQL takes almost exactly the same amount of paper to print

Hi ***, thanks for your observation about paper volumes. However, it seems to me that the "volume of paper" argument is a different issue.

I don't doubt the expressive power of your CQL but my point was about precision and verbosity. 
In my mind this is to do with human perception and ease of understanding - which is not at all the same thing as the surface area covered by the "document".
I still maintain that an information model expressed in ORM graphical language has many advantages over a textual representation. Maybe I should add that an object-role model expressed in logic notation is probably the most accurate transform but most people find it hard to read the symbols used by logicians.

*** Heath:

In addition, it can be emailed to any user who can peruse it regardless of what software they have available. That includes the business experts who alone can add the distinctive value of semantic modeling over traditional modeling approaches. These people are the ones who matter, and they will almost never have Microsoft Visual Studio available to them.

You are probably right that most "business experts" will not have access to Visual Studio. Nor would they take the time to learn how to use it.
However, in my mind, an ORM tool (regardless of platform) is the tool of an "ORM expert" not of a "Business expert". 
As I see it, the role of the "business expert" is to agree or disagree with the verbalizations generated from an ORM tool.

*** Heath:

 A common textual modeling language is the only path to the success of the ORM2 technology

If you mean that "the business expert should be able to understand the verbalizations generated by an ORM tool" then I agree with you.
But it seems to me that this feature already exists in NORMA (and it also exists in the ORM 1 tools VEA and VisioModeler).

So maybe you had better clarify what you see as the difference between "verbalization" (as in NORMA) and  "a common textual modeling language".
Common with what?

Ken

Dear Clifford and all of you,

The discussion touched several interesting points and helped to get a deeper understanding of ORM. Especially some more philosophical points that Ken mentioned are very interesting. To start, I will focus on more technical issues. My current understanding is that the .orm file format as exported by NORMA, which is an XML Schema, is the de-facto exchange format for ORM2 models. Although the state of specification and implementation could be improved, there are several open source tools that help to process .orm files and ORM2 models (reading, writing, transformation to other languages etc.), namely GanttPV (Python), DogmaModeler (Java), ActiveFacts (Ruby), and NORMA (C# and XSLT, requires Visual Studio). The XSLT files of NORMA look useful to build on; GanttPV (GPL2) and ActiveFacts (zlib license) are even Free Software with compatible licenses. I have not found out what the difference between ORM-LITE and GanttPV is and whether DogmaModeler (available on request) is GPLed or not. Apart from the NORMA layout engine, which probably hard to reuse, this projects should be enough to build on for ORM processing :-)

Thanks and Greetings,
Jakob 

Hi Jakob,

Firstly, thanks for starting this very useful thread.

Secondly, what's next?

After reviewing the whole thread (both parts) I see a need for a related "context discussion" that covers "problems", "objectives", "solutions" and "tools".
1: Problems & Objectives 
Many years ago in an IBM far far away I learned the valuable lesson that a "problem" is "something that prevents you from achieving an objective"
Thus, if you don't have at least one properly defined objective then you can't possibly have any problems.
In this context, a "properly defined objective" is an "outcome" and the "outcome definition" must include metrics to allow you to measure when the objective has been achieved and a date by when the objective must be reached. Thus, to conform to requirements, a statement of objectives MUST include metrics and a target date. The absence of metrics and/or date means you don't have an objective.

2: Solutions and tools 
A solution can be implemented in a tool so in this context, these words are synonyms.

Now my many years of ORM experience (InfoModeler, VisoModeler, VEA and NORMA) has given me a deep insight into the kinds of problems that ORM tools are able to solve.

However, since I am not familiar with the other tools that you mention, I must confess to a lack of understanding of the problems that they are designed to solve. So now to my questions:

In addition to the problems that can be solved by VEA and NORMA, exactly what kind of problems are the other tools you mention intended to solve?
I have put my own perceptions in here - maybe you or others can add to them? (The "objectives" can be inferred from the problem statement)

ORM-Lite (GantPV) -  solves the problem called "ORM does not run on platforms other than Windows" 
ActiveFacts - solves the problem called "Meets Clifford's perceived need for a textual ORM language"
DogmaModeler - I have no experience with this tool so maybe you can add something here.

Ken

Ken Evans:

Clifford Heath:

A model converted from NORMA to CQL takes almost exactly the same amount of paper to print

Hi Clifford, thanks for your observation about paper volumes. However, it seems to me that the "volume of paper" argument is a different issue.

 

 Perhaps you can come up with a better way of defining the "verbosity" of an ORM diagram? After all, it's largely non-verbal, in the traditional sense. It's not even a form of hieroglyphics.

The ability of the eye to resolve detail and the amount of detail that can be extracted from a given surface area actually is the only sensible way to think about it, in my opinion.

Note that the discussion isn't about comprehensibility - you used the terms "precise" and "verbosity", not about comprehensibility. I willingly admit that many facets of comprehensibility are easier in a graphical representation - however other facets favour a textual form. And of course, comprehensibility is zero if I lack the ability to even view the model - text wins every time on accessibility.

If you only expose the business experts to the generated verbalisations, you cheat them of their opportunity to contribute to the source documents. 

Ken Evans:

However, in my mind, an ORM tool (regardless of platform) is the tool of an "ORM expert" not of a "Business expert".

That point of view is the main reason why ORM continues to fail. Every candidate "ORM expert" already has a way of doing what they need (constructing valid schemas), and they want to protect and control their territory, so they won't adopt a tool that opens them up to control by the business.

Yet control is the very thing that the business wants, needs, and must have. They commonly hate dealing with the IT department because it is seen as vigorously defending its control - yet the business cannot move forward without IT. The nexus of control is in the wrong place, and a tool that can restore it, by being directly accessable to the business analysts and experts, can succeed for that reason alone.

jakob.voss:

My current understanding is that the .orm file format as exported by NORMA, which is an XML Schema, is the de-facto exchange format for ORM2 models.

That is incorrect. ORM files are the only format available for export from NORMA. Nothing else generates them, nor should, in my view. 

 

jakob.voss:

The XSLT files of NORMA look useful to build on

That's unfortunately also false. The XSLT is an artefact generated from an object model built using the VS DSL Tools. It is hugely over-referenced (every tag has an ID, even ones that cannot be referenced) and contains many items that are related to the internal operations of NORMA. This additional detail is subject to change, and is not part of any sensible semantic model of ORM2, so third-party tools should not attempt to generate it. The dog-food principle says that we who care about modeling techniques should use them, but that doesn't seem to have happened in this case.

So far as I've seen, every attempt at collaboration to construct a meta-model has failed to even get started. I got so frustrated that I built my own, more than a year ago, and despite some early and very helpful feedback, have had no indications that anyone, anywhere has actually studied it. It's by no means perfect, but it clearly does work.

All my tools are built on the Ruby code generated from this model (yes Virginia, it's possible to use ORM for more than just database schemas), and they clearly work. I'm not asking for it to be adopted - but can we at least have a sensible and detailed discussion about it sometime? Any request for a common file format is going to fail until we agree on what information it should represent.

 

jakob.voss:

I have not found out what the difference between ORM-LITE and GanttPV is

GanttPV is a tool for managing project timelines. ORM-LITE is an application that has been added within its framework - quite a strange choice if you ask me, yet it seems to work. 

 

Ken Evans:

ActiveFacts - solves the problem called "Meets Clifford's perceived need for a textual ORM language"

 

Ken, I find that excessively patronising.

I represent three decades of direct experience with software development. Not only project management, actual development. I have also experienced hundreds of organisations' processes and software development challenges second-hand, through my career as a professional maker of software tools. I've also experienced the way the market for software tools operate from the vendor's side, not just as a user of software tools. I learnt the hard way about the psychology of organisations' decisions to procure and invest time into software tools.

*** Heath:

So far as I've seen, every attempt at collaboration to construct a meta-model has failed to even get started. I got so frustrated that I built my own, more than a year ago, and despite some early and very helpful feedback, have had no indications that anyone, anywhere has actually studied it. It's by no means perfect, but it clearly does work.

Yes, there might be benefit in opening up the discussion on the ORM metamodel.
My interpretation of Terry's earlier comments is that he is working with PNA to develop a single metamodel that meets both the requirements of "being an ORM 2 metamodel" and of "being a metamodel that meets PNA's needs"

I agree with you that it would also be helpful to add "and that meets the needs of ***'s CQL metamodel" (correct me if I did not express that correctly). 

Anyway, I would be happy to help you to get some folks to "study it".
So may I suggest the following procedure in order to move this forward:

1: You post your CQL metamodel to the Library
2: You prepare a list of questions that you feel should be answered by people who are willing to "study it"
3: I will encode your questions into my survey subsystem and provide a link that can be posted or emailed as you wish.
4: After a pre-defined time, I will run the reports in the survey subsystem and post the results here for discussion.

No offence if you don't want to do this.
Just trying to help.

*** Heath:

Any request for a common file format is going to fail until we agree on what information it should represent.

Yes Virginia, you are spot on with that observation. Well said! 

Ken

Ken Evans:

...Terry's earlier comments is that he is working with PNA

 

PNA also offered to adapt my work to their needs for a language, including for process modeling (a long-term goal of mine also), but I couldn't agree to my open source being closed off, and they couldn't agree to it being open. I couldn't agree to a solution being Windows-only, or installable-product only, whereas they want a product they can sell. That's the sort of disagreement which if continued, will retard our success as a community, and the success of ORM as a result. You can only effectively sell a tool for a technique after the technique has become established - something I spent a decade learning.

Ken Evans:

and that meets the needs of Clifford's CQL metamodel

CQL does have one or two needs in addition to ORM2 because of the need to parse the verbal form. Things like like distinguishing hyphen-bindings as adjectives, and rules about how they should be used - similarly with role names (an area I've discussed with Terry without full agreement yet). Also CQL uses the role order of the first (preferred) reading as definitive, whereas NORMA has a hidden order not present in ORM.

My metamodel also has various extensions (multiple vocabularies, units) and clarifications (value types vs data types) that are either not present or not well-expressed in current ORM tools. It also represents explicit join-paths, and I'm not aware of other ORM tools that do that (though it's in the ORM literature). It will also grow to reflect my extended query language.

But on the whole, I've tried to keep as close to ORM as possible.

Ken Evans:

may I suggest the following procedure

Thanks for the suggestion, but I already maintain a published version, in NORMA, CQL and as images. Because it evolves from time to time, I'd rather not maintain multiple definitive versions of those.

Further, the value of any questions I might already have is less than the value of the questions I haven't asked, or haven't formulated. A vote on questions isn't a design procedure, as it will only serve to highlight different people's preconceptions about the correct forward direction, not help make whatever compromises might be needed. Discussion of each question is needed before a vote on it. I'm aware of my existing compromises, and where possible the current model reflects my judgment (things like re-using the RoleReference object type, for example). It's up to others to indicate which parts (if any) they find unpalatable, and raise the issues for discussion.

The most important goals are, in order, to:

• To find any areas which are incorrect (possibly refining the definition of "correct")
• To find anything lacking which is required by current or upcoming tools
  
• To find better ways to represent things that are otherwise correct
  
• To more fully constrain the model against disallowed situations.
  

Clifford Heath:

Ken, I find that excessively patronising.

 

Hmm, curious - I don't see what's patronising about it. An unintended consequence of the way I chose to express myself.
As I think I have indicated elsewhere, I have a high regard for your experience level and had no intention of calling into question either your experience or your dedication to the cause.  From my side of this exchange it seems to me like a case of the "Long Toes" problem about which I learned when I was living in Belgium many years ago. (The etymology being that some people have specially lengthened their toes in order to increase the probability that someone will step on them)  - Again - no offence intended - just one possible explanation - seen from my side of course.

What I was trying to express was the following:

1: I don't understand the need for an "ORM textual language"
2: I don't understand which problems that such a language aims to solve.    
3: The amount of effort you have put into CQL suggests that you DO understand the NEED and the problems.

So the closest I could get to expressing my perception was to reference your perception and imply that this is something that you clearly have a good grasp of but that I don't.
Thus my encoding of this message as "Clifford's perceived need".

Hope this helps.

Ken

 

Ok, sorry for being a little touchy. I'll offer some of my perceptions of the social dynamic of the use of software tools.

Organisations will not invest in software development using tools from companies that may fail, or where the tool is seen as risky or dead-end. A technology has to be established in order to provide an escape route.

Designers of software are motivated mainly by kudos - if their success might be put down to a tool, they aren't as motivated to use it. They thing that motivates them is to produce the nice software (that's already in their head) in the shortest possible time. They like tools when they lessen the work without reducing the quality. Tool output is often seen as a compromise and suboptimal, so there really has to be a big time saving to impress these people. Existing ORM design tools haven't produced the right artefacts to shorten schedules - they've been targeted at doing things better and getting them right more often. The developer's hubris doesn't allow them to see this as an advantage, since they think if left alone, they could produce perfect software without such tools, and that's what they want to be admired for.

People who want to be "top dog" in a development team, wielding control in excess of their work output, like to use tools - because their knowledge of the tool gives them a special place, they pull the strings. But such people produce so little of the final artefacts of a project they often contribute little to the success of projects anyhow. Instead they create turmoil by insisting that things be done their way, and redone even if a solution is already working.

These last two paragraphs explain why the CASE tool movement of the 1980's failed so miserably, not because the tools didn't work.

The IT function as a whole, is often viewed by the business as having far too much control. In part this is a natural reality, as the business can't move forward without the IT changes, and they don't have enough understanding of the challenges of succeeding in software development to trust IT. But on the other part, IT uses its power to gain some control over the business direction, sometimes with legitimacy but not always. So the IT function is further distrusted. In addition, IT often fails to deliver adequate functionality in a timely way, and so are seen as less competent than other areas of the business.

On the other side, the business isn't often much good at writing specifications. The language used is too vague, and doesn't reflect an understanding of how the IT systems will support business changes; because the business is concerned with what and why - as it should be. So they employ business analysts, who are meant to bridge the gap, but often fall too much on one side or the other. When IT try to explain that a feature cannot be implemented, or is incompletely specified, they have great trouble explaining why there's a problem. In part, that's because they think in terms of how, since that's the natural tendency of the engineering mind. The failure of the business to understand the problem is seen as legitimizing the degree of control that IT asserts.

All this is down to communication and language. A semantic modelling language must make it easier for the business and IT to work together, not as opponents, engaging in paper warfare, but really collaborating. The best way to do that is to create a single language that both groups can read and write, that can provide both with what they need - precision and consistency for the IT folk, and verifiability against the business rules and process for the business folk.

In the process, the language can also be used to generate the artefacts that both groups need (schemas, code, documentation of business rules) - but it's main attraction to the business is the way it changes the communication process. The generated artefacts are about reducing the project schedules while ensuring continuous compliance with the specification - but they must be of high quality, and preferably, the generators must be tweakable (open source).

That's the language I hope CQL will become.

Hi Clifford,

I'm a brand new member in this forum, though not new to this subject. I'm a bit in a hurry right now (to get into bed at last, before my wife kills me, what kind of a Valentine is this anyway ), but I couldn't agree more that the development industry has too much inertia. I'm working on automation of software engineering for the last 10 years at least. One domain in which this sector is not willing to believe automation is possible, without resulting in ugly or not workable products, is (G)UI. Most analysts feel that a GUI is unimportant or secundary, but most developers seem to be incapable of producing user friendly and maintainable GUIs. My point of view is that the domain model logically defines the possible choices for the best GUI. Most work seems to have been done on automating database generation (mapping objects on the (mostly) relational databases), but none on GUI. However, even if it were only to produce prototypes, it would help a lot communicate (parts of) complex models to the users community. For them the GUI is the only visual proof of a domain model existance and most of them see the GUI as THE system. So that will allow them to visualize/understand the internals, on top of viewing a textual (natural language) interpretation of the facts and rules. But having a scenario-based walk-through allows for very productive meetings with (end) users [I produced and used with success such a tool around 2000-2001]. Feedback of (even "stupid") users is more important than many people believe.

So I will study your project with much of interest, and although I can't promise anything yet (having 5 kids I'd have to go underground to work full-time on anything, but I compensate at night ;-)), I hope I may contribute from my side in some way. Thanks for your contribution anyway. Sleep tight !

Marc De Valckenaere
sales@ickware.be
Belgium

I found this simplified metamodel for ORM, which goes in the direction of an exchange format. Unfortunately, I can't read the PDF file. Could someone fix this?  Is the work still in progress ?

Pierre C.

 

PDF File problems
Sorry about that. I have replaced the corrupted pdf file with a pps file.
Let me know if this is OK for you.

Is the work still in progress?
Yes. But its going rather slowly.

Ken