Mike Vizdos and I would like to invite you spend a few moments taking the 2009
Agile Practice Adoption survey at
http://www.surveymonkey.com/s.aspx?sm=3lZZ3fYRxvV3y6XsnXPDEw_3d_3d .  The goal
of this survey is to find out what practices agile practitioners find useful,
which they're finding difficult to adopt, and which they're abandoning.  There
are 17 questions in this survey, although you may not be asked all of them, and
it should take you at most 5 minutes to complete.

To entice you to fill out the survey we'll be drawing for 10 copies of "Stand
Back and Deliver: Accelerating Business Agility" by Pollyanna Pixton, Niel
Nickolaisen, Todd Little, and Kent McDonald published in June 2009 by Addison
Wesley.  I've been reading through this book the past few days and I'm finding a
lot of great insights in it.  More information can be found at
http://www.informit.com/store/product.aspx?isbn=0321572882

The results of this survey will be summarized in my "Agile by the Numbers"
presentation at Agile 2009 in Chicago (Aug 24-28). Furthermore, this is an open
survey, so the source data (without identifying information to protect your
privacy), a summary slide deck, and the original source questions will be posted
at www.ambysoft.com/surveys/ so that others may analyze the data for their own
purposes. Data from previous surveys have been used by university students and
professors for their research papers, and hopefully the same will be true of the
data from this survey. The results from several other surveys are already posted
there, so please feel free to take advantage of this resource.

We apologize if you've received several copies of this request.

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale: http://www.ibm.com/developerworks/blogs/page/ambler


       __________________________________________________________________
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
http://ca.toolbar.yahoo.com.

I just wanted to thank those of you who participated in the survey.  As I
indicated earlier, the results of this survey will be included in my "Agile by
the Numbers" presentation at Agile 2009, see http://www.agile2009.org/node/263 .
After the presentation the details from this survey will be posted at
http://www.ambysoft.com/surveys/practices2009.html as I usually do.

To give you a feel for the results of the survey, some of the findings (which
I'll expand upon in August) include:
1. The three practices which are seen as most effective were Continuous
Integration (CI), Daily Stand Up Meetings, and Developer Test-Driven Development
(TDD).
2. The three practices which are seen as easiest to learn are Daily Stand Up
Meeting, Continuous Integration (CI), and Iteration Planning.
3. The three practices which are seen as hardest to learn are Developer TDD,
Acceptance/Story TDD, and Pair Programming.
4. The three practices which were most likely to be tried and then abandoned
were Pair Programming, Burndown Tracking, and Potentially Shippable Software
Each Iteration.
5. The three practices which people want to adopt but have not yet done so were
Acceptance TDD, Potentially Shippable Software Each Iteration, and Developer
TDD.

If you're thinking about running a survey, I've posted some thoughts at
http://www.ambysoft.com/surveys/agileSurveys.html which might prove helpful. 
Feedback is always appreciated.

Also, I've sent emails to the ten winners of "Stand Back and Deliver", so if you
haven't recieved an email you haven't won.  Better luck next time!

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale: http://www.ibm.com/developerworks/blogs/page/ambler
--- In agileDatabases@yahoogroups.com, Scott Ambler <scottwambler@...> wrote:
>
>
> Mike Vizdos and I would like to invite you spend a few moments taking the 2009
Agile Practice Adoption survey at
http://www.surveymonkey.com/s.aspx?sm=3lZZ3fYRxvV3y6XsnXPDEw_3d_3d .  The goal
of this survey is to find out what practices agile practitioners find useful,
which they're finding difficult to adopt, and which they're abandoning.  There
are 17 questions in this survey, although you may not be asked all of them, and
it should take you at most 5 minutes to complete.
>
> To entice you to fill out the survey we'll be drawing for 10 copies of "Stand
Back and Deliver: Accelerating Business Agility" by Pollyanna Pixton, Niel
Nickolaisen, Todd Little, and Kent McDonald published in June 2009 by Addison
Wesley.  I've been reading through this book the past few days and I'm finding a
lot of great insights in it.  More information can be found at
http://www.informit.com/store/product.aspx?isbn=0321572882
>
> The results of this survey will be summarized in my "Agile by the Numbers"
presentation at Agile 2009 in Chicago (Aug 24-28). Furthermore, this is an open
survey, so the source data (without identifying information to protect your
privacy), a summary slide deck, and the original source questions will be posted
at www.ambysoft.com/surveys/ so that others may analyze the data for their own
purposes. Data from previous surveys have been used by university students and
professors for their research papers, and hopefully the same will be true of the
data from this survey. The results from several other surveys are already posted
there, so please feel free to take advantage of this resource.
>
> We apologize if you've received several copies of this request.
>
> - Scott
> Scott W. Ambler
> Chief Methodologist/Agile, IBM Rational
> Agile at Scale: http://www.ibm.com/developerworks/blogs/page/ambler
>
>
>       __________________________________________________________________
> Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
> http://ca.toolbar.yahoo.com.
>

A few questions:

1) Are there competing UML Data Modeling profiles (for relational databases) or
is only one being considered? One of Scott's articles mentions that an official
data modeling profile RFP appeared at OMG in December, 2005. It is now ~3.5
years later. Where does that stand:

http://www.agiledata.org/essays/umlDataModelingProfile.html#RFP

2) If there's only one being considered, how far away are we from it becoming an
accepted standard?

3) How are UML modeling tool vendors handling this absence of a standard,
accepted data modeling profile? Will vendors create transforms to the standard
from whatever UML data modeling profile(s) they currently support?

4) How are agile database designers handling this absence of an approved
standard, and is it causing problems with database designers accustomed to older
ways of doing things (e.g. committed to ER diagrams, etc). I am looking for
resources that explore the controversy of using UML for database design vs.
older methods.

From what I can tell, there is a deep divide between traditional "non
programmer" database designers and the new breed of agile focused, app-centric
database designers. I'm trying to sort all of this out as I help recommend a
methodology for database design and diagramming to our organization for the next
decade.

Thanks very much.

Dana

I’m often asked fundamental questions by people new to agile regarding how
people go about getting an agile project started.  What strategies do people use
for estimating?  What approaches to project funding do people typically use? 
What did you have to do to justify the project?  How much modeling occurs and to
what extent?  How long does this typically take?  Without decent answers to many
of these questions some people are still reticent to experiment with agile
approaches to software development.

So, to get answers to these sorts of questions Mike Vizdos and I have put
together a quick survey around agile project initiation.  The survey can be
accessed at http://www.surveymonkey.com/s.aspx?sm=qZI7E9V8TuF55s_2bCQrgMEw_3d_3d
There are 17 questions in this survey, although you may not be asked all of
them, and it should take you at most 5 minutes to complete. Thank you very much
for investing your valuable time to fill it out.

To entice you to fill out the survey we'll be drawing for 10 copies of "Agile
Project Management: Creating Innovative Products, 2nd Ed." by Jim Highsmith
published in July 2009 by Addison Wesley. For more information about the book,
visit http://www.informit.com/store/product.aspx?isbn=0321658396

The results of this survey will be summarized in Scott Ambler’s "Agile by the
Numbers" presentation at Agile 2009 in Chicago on August 27th. Furthermore, this
is an open survey, so the source data (without identifying information to
protect your privacy), a summary slide deck, and the original source questions
will be posted at http://www.ambysoft.com/surveys/ so that others may analyze
the data for their own purposes. Data from previous surveys have been used by
university students and professors for their research papers, and hopefully the
same will be true of the data from this survey. The results from several other
surveys are already posted there, so please feel free to take advantage of this
resource.

For anyone who is reticent about filling out a survey, or who doesn’t see the
value in them, you might want to take a look at
http://www.ambysoft.com/surveys/agileSurveys.html#Valuable for some arguments as
to why you should consider investing your valuable time to help us out.  Thanks
in advance.

Also, we apologize if you’ve received several copies of this email.

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale: http://www.ibm.com/developerworks/blogs/page/ambler


       __________________________________________________________________
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
http://ca.toolbar.yahoo.com.

Answering myself here for posterity. Still interested to know list members'
opinions to my original post. With 2,696 of you I hope someone feels comfortable
responding.

> 4) How are agile database designers handling this absence of an
> approved standard, and is it causing problems with database
> designers accustomed to older ways of doing things (e.g. committed
> to ER diagrams, etc). I am looking for resources that explore the
> controversy of using UML for database design vs. older methods.

There are a few posts I created on a vendor's forum that may, or may not, turn
out to answer how at least one vendor is, or will be, handling the absence of an
approved. The rules say steer clear of vendor mentions. Hope it's ok to post
these. If not, moderator, please delete them:

http://www.sparxsystems.com/cgi-bin/yabb/YaBB.cgi?num=1248611946

http://www.sparxsystems.com/cgi-bin/yabb/YaBB.cgi?num=1213360383/0

> From what I can tell, there is a deep divide between
> traditional "non programmer" database designers and the new breed
> of agile focused, app-centric database designers. I'm trying to
> sort all of this out as I help recommend a methodology for
> database design and diagramming to our organization for the next
> decade.

Two articles I found very useful on the cultural and technical "impedance
mismatch" between agile app developers and traditional data professionals. Now I
have a name for the controversy:

http://www.agiledata.org/essays/culturalImpedanceMismatch.html

http://en.wikipedia.org/wiki/Object-relational_impedance_mismatch

Private replies are welcome, although public replies will help more people.

Dana

hello, i'm new to testing. i'm using junit and dbunit for testing our web app.
this is my first job being a unit tester and i'm reaaly lost in testing the
schema of database as well as testing the database. i have read many articles
regarding dbunit, but can't seem to find the right answer in testing the schema.
please help. thanks so much

Posted to comp.software-eng and did get one useful response about the time-table
for an approved UML data modeling profile. I'm posting the text here for
reference. Hope this is useful to others. It was to me. I hope more people start
talking about this stuff:

http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c2f3a4ede\
47c5ac#

Responding to dana...

> 5) What precisely *is* an RFP? Is the OMG simply in a mode where it is
> soliciting entire candidate UML data modeling profiles, or has it put
> out a draft and is it asking for refinements? If it's the former, how
> many years might one expect a standard to selected, refined, and
> ratified?

Request For Proposal. IOW, OMG is looking for ideas on what a standard
should be.

Currently UML only supports very basic data modeling (i.e., RDB schemas)
in a Class Diagram, using the Class Diagram as a traditional Entity
Relationship Diagram. (Lots of tools use other diagtrams for data
management contexts, but that usage is not standardized.) As the
citation you provided indicates, data management has come a long way
since ERDs were sufficient due to warehousing, middleware, etc., etc..
So there is a need for a more comprehensive standard that goes beyond
RDB schemas.

OMG is soliciting proposals for such a standard, probably because UML
already provides a lot of the needed constructs for OOA/D. That is, one
only needs to re-interpret the OOA/D semantics to the data management
domain. OMG's MDA initiative already provides semantic infrastructure
for doing that through the profile concept. I haven't read it but my
guess is that the RFP is primarily about defining an MDA profile for
using UML as a notation for general data management (and adding any
syntactic elements, if needed). That is, it provides a unique semantic
interpretation at the meta-model level for the notation.

Progress at standards groups tends to be glacial, so I wouldn't expect
much for at least a couple of years. Typically OMG gets an initial
standard ratified in about four years. But that will usually be a sort
of trial balloon good for "vanilla" modeling and will likely be
significantly modified over another four years to get the details right.

> 6) What can the community do to accelerate the process of getting a
> UML Data Modeling profile ratified? Does one have to be a prof or in
> the upper echelons of one's field, or can "ordinary" professionals
> contribute?

Other than joining OMG, the only thing the community can do is contact
the working group members and lobby them. Typically working groups are
made up of representatives of software vendors who volunteered because
that have vested interests in getting their proprietary view of the
world anointed as the standard, so the political infighting can be
intense. Providing support like white papers on particular issues can be
useful as a lobbying technique by supporting particular working group
members in a concrete way. But being a member of the working group is
far and away the best way to influence the standard.

Anybody can participate. However, to participate in a working group
initially defining the standard you have to be a member of OMG (either
representing a corporation or as an individual). [If OMG gets too many
volunteers for an efficient working group, they will usually break up
the working group into multiple working groups with more narrowly
defined scope. So if you are an OMG member and want to participate, you
very likely can somehow.]

Once a draft proposal is created OMG will make it publicly available for
review and anyone can critique it without OMG membership. But by that
time there is a lot of inertia and the public review is going to have to
identify some really glaring problem to cause anything but cosmetic changes.

--
===========

Dana here again.

I posted on comp.databases.theory and got replies that were by turns accusatory,
dismissive, and sarcastic. Not precisely the type of professional discourse I'd
hoped for. One person denied the existence of an O-R impedance mismatch:

http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c2f3a4ede\
47c5ac#

Guess I chose the wrong "theory" (e.g. not pure RM?) and the wrong newsgroup--as
the grail keeper in Indiana Jones & The Last Crusade said, "Choose wisely." It
could be said of me--me being he--"He chose... unwisely."

Dana

Dear Dan,
On Tue, Jul 28, 2009 at 12:40 PM, dananrg<dananrg@...> wrote:
> Answering myself here for posterity. Still interested to know list members'
> opinions to my original post. With 2,696 of you I hope someone feels
> comfortable responding.
I posted a profile related question to this list some time ago and did
not receive any reply either:
http://tech.groups.yahoo.com/group/agileDatabases/message/2296

Fortunately, there was some follow-up discussion on the PUML list:
http://www.cs.york.ac.uk/puml/puml-list-archive/1644.html  You could
try there as well.

Sincerely,
Pieter

Anna,
Any detail what you want to test?

I test constraints defined on tables (especially those concerned to
multiple-record data sets). Since the model is generated from ERwin (including
these constraints, defaults, CRUD code, etc) it's easy to lose some of them, so
I created set of tests checking all that.

ak

--- In agileDatabases@yahoogroups.com, "anna_sniper18" <anna_sniper18@...>
wrote:
>
> hello, i'm new to testing. i'm using junit and dbunit for testing our web app.
this is my first job being a unit tester and i'm reaaly lost in testing the
schema of database as well as testing the database. i have read many articles
regarding dbunit, but can't seem to find the right answer in testing the schema.
please help. thanks so much
>

--- In agileDatabases@yahoogroups.com, "anna_sniper18" <anna_sniper18@...>
wrote:
>
> hello, i'm new to testing. i'm using junit and dbunit for testing our web app.
this is my first job being a unit tester and i'm reaaly lost in testing the
schema of database as well as testing the database. i have read many articles
regarding dbunit, but can't seem to find the right answer in testing the schema.
please help. thanks so much
>

We use DbFit, a variant of the FitNesse testing platform to test our databases.
It offers benefits for automating the testing, being able to organize tests into
discrete areas of concern, and can test constraints, data, stored procedures,
etc. Just about anything you'd want or need to test on the database level.

Tue, Aug 11 - Tue, Sep 22 '09 (6 live sessions)
Agility has allowed development teams to simultaneously reduce cost, increase
the rate at which value is delivered, and improve responsiveness to changing
market forces. As great as this is, things can still get better. Databases have
always been a bottleneck when it comes to change. They are clunky,
mission-critical and (worst of all) they have inertia, the bane of Agility.
True database agility comes from the recognition of how data and programs are
fundamentally different. This course breaks the practices of agile software
development down into a set of principles and then uses those principles to
build up a set of Agile practices in the context of database development.
This online training is true training, including lectures, readings, exercises
and question & answer periods.

http://www.netobjectives.com/course-schedule/database-agility-online-aug-2009

Hi Dana, thanks for keeping this thread alive.  I agree it is an
important issue that deserves more input from the community.

I just found http://www.omg.org/docs/ad/05-06-09.pdf while searching
with http://www.google.be/search?q="data+modeling"+profile+site%3Aomg.org
Perhaps the authors are already involved as an OMG member and they can
learn us more already?

I was wondering: the commercial profiles you have used, do they
support two layers of abstraction (physical data modeling, conceptual
data modeling)?  In my own case studies, I have found Scott's profile
elements to be incomplete when it comes to key representation.

Kind regards,
Pieter Van Gorp, Assistant Professor (Universitair Docent)
     Information Systems Group, School of Industrial Engineering
     Eindhoven University of Technology (TU/e), The Netherlands
     Phone: +31 40 247 2062, Skype: pvgorp, Fax: +31 40 243 2612
     http://is.tm.tue.nl/staff/pvgorp/research/

On Thu, Jul 30, 2009 at 10:51 AM, dananrg<dananrg@...> wrote:
>
>
> Posted to comp.software-eng and did get one useful response about the
> time-table for an approved UML data modeling profile. I'm posting the text
> here for reference. Hope this is useful to others. It was to me. I hope more
> people start talking about this stuff:
>
>
http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c2f3a4ede\
47c5ac#
>
> Responding to dana...
>
>> 5) What precisely *is* an RFP? Is the OMG simply in a mode where it is
>> soliciting entire candidate UML data modeling profiles, or has it put
>> out a draft and is it asking for refinements? If it's the former, how
>> many years might one expect a standard to selected, refined, and
>> ratified?
>
> Request For Proposal. IOW, OMG is looking for ideas on what a standard
> should be.
>
> Currently UML only supports very basic data modeling (i.e., RDB schemas)
> in a Class Diagram, using the Class Diagram as a traditional Entity
> Relationship Diagram. (Lots of tools use other diagtrams for data
> management contexts, but that usage is not standardized.) As the
> citation you provided indicates, data management has come a long way
> since ERDs were sufficient due to warehousing, middleware, etc., etc..
> So there is a need for a more comprehensive standard that goes beyond
> RDB schemas.
>
> OMG is soliciting proposals for such a standard, probably because UML
> already provides a lot of the needed constructs for OOA/D. That is, one
> only needs to re-interpret the OOA/D semantics to the data management
> domain. OMG's MDA initiative already provides semantic infrastructure
> for doing that through the profile concept. I haven't read it but my
> guess is that the RFP is primarily about defining an MDA profile for
> using UML as a notation for general data management (and adding any
> syntactic elements, if needed). That is, it provides a unique semantic
> interpretation at the meta-model level for the notation.
>
> Progress at standards groups tends to be glacial, so I wouldn't expect
> much for at least a couple of years. Typically OMG gets an initial
> standard ratified in about four years. But that will usually be a sort
> of trial balloon good for "vanilla" modeling and will likely be
> significantly modified over another four years to get the details right.
>
>> 6) What can the community do to accelerate the process of getting a
>> UML Data Modeling profile ratified? Does one have to be a prof or in
>> the upper echelons of one's field, or can "ordinary" professionals
>> contribute?
>
> Other than joining OMG, the only thing the community can do is contact
> the working group members and lobby them. Typically working groups are
> made up of representatives of software vendors who volunteered because
> that have vested interests in getting their proprietary view of the
> world anointed as the standard, so the political infighting can be
> intense. Providing support like white papers on particular issues can be
> useful as a lobbying technique by supporting particular working group
> members in a concrete way. But being a member of the working group is
> far and away the best way to influence the standard.
>
> Anybody can participate. However, to participate in a working group
> initially defining the standard you have to be a member of OMG (either
> representing a corporation or as an individual). [If OMG gets too many
> volunteers for an efficient working group, they will usually break up
> the working group into multiple working groups with more narrowly
> defined scope. So if you are an OMG member and want to participate, you
> very likely can somehow.]
>
> Once a draft proposal is created OMG will make it publicly available for
> review and anyone can critique it without OMG membership. But by that
> time there is a lot of inertia and the public review is going to have to
> identify some really glaring problem to cause anything but cosmetic changes.
>
> --
> ===========
>
> Dana here again.
>
> I posted on comp.databases.theory and got replies that were by turns
> accusatory, dismissive, and sarcastic. Not precisely the type of
> professional discourse I'd hoped for. One person denied the existence of an
> O-R impedance mismatch:
>
>
http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c2f3a4ede\
47c5ac#
>
> Guess I chose the wrong "theory" (e.g. not pure RM?) and the wrong
> newsgroup--as the grail keeper in Indiana Jones & The Last Crusade said,
> "Choose wisely." It could be said of me--me being he--"He chose...
> unwisely."
>
> Dana
>
>

--- On Sun, 8/2/09, Pieter Van Gorp <pieter@...> wrote:

>
> I was wondering: the commercial profiles you have used, do
> they
> support two layers of abstraction (physical data modeling,
> conceptual
> data modeling)?  In my own case studies, I have found
> Scott's profile
> elements to be incomplete when it comes to key
> representation.

What do you think is missing?

- Scott

Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
Follow me on Twitter: http://twitter.com/scottwambler



       __________________________________________________________________
Looking for the perfect gift? Give the gift of Flickr!

http://www.flickr.com/gift/

Hi Scott, thanks for your reply.
>> In my own case studies, I have found
>> Scott's profile
>> elements to be incomplete when it comes to key
>> representation.
> What do you think is missing?
It is impossible to include to-one association ends in key
definitions.  When you can only include attributes in key definitions,
you are forced to using an attribute even when you wish to use a
to-one association.  In conceptual data modeling, I find this often
undesirable.  See
http://www.cs.york.ac.uk/puml/puml-list-archive/1644.html and the
surrounding messages for clarification (Daniel Jackson) and feedback
(Stefano Merenda).

Kind regards,
Pieter

On Tue, Aug 4, 2009 at 1:23 AM, Scott Ambler<scottwambler@...> wrote:
>
>
> --- On Sun, 8/2/09, Pieter Van Gorp <pieter@...> wrote:
>
>>
>> I was wondering: the commercial profiles you have used, do
>> they
>> support two layers of abstraction (physical data modeling,
>> conceptual
>> data modeling)?  In my own case studies, I have found
>> Scott's profile
>> elements to be incomplete when it comes to key
>> representation.
>
> What do you think is missing?
>
> - Scott
>

I agree that this is a critical issue to the software development
community. UML is the de facto standard for representing the design (and
sometimes the requirements) of software. E/R models are the de facto
standard for representing the requirements and designs for relational
databases. We badly need the two to live together in harmony if our
software is to run correctly against the databases and the databases are
to appropriately persist data processed within the software.

May I direct you to a series of articles written by David Hay and
Michael Lynott intended to reconcile UML class diagrams with E/R
diagrams. The purpose of the articles is:

"This series of articles has two audiences: The data modelers who have
been convinced that UML has nothing to do with them; and UML experts who
don't realize that data modeling really is different from object
modeling (and the differences are important). The authors' objective is
to finally bring the two groups together in peace.
The series is in three Parts. Part I, here, sets the stage, describing
the basic differences between the notations and how they can be
reconciled. Part II, which will be published in the next issue of The
Data Administration Newsletter, goes into more detail, addressing
sub-types and constraints, along with both what in UML should not be
used for a data model, and what "stereotype" has to be added. And, since
the whole point of preparing a data model is to present it to management
for validation, Part III discusses the aesthetics of preparing and
presenting data models - no matter what notation is used. "

The articles are here, in 4 parts:

http://www.tdan.com/view-articles/8457

http://www.tdan.com/view-articles/8589

http://www.tdan.com/view-articles/8914

http://www.tdan.com/view-articles/9219

________________________________

From: agileDatabases@yahoogroups.com
[mailto:agileDatabases@yahoogroups.com] On Behalf Of Pieter Van Gorp
Sent: Sunday, August 02, 2009 2:42 AM
To: agileDatabases@yahoogroups.com
Subject: Re: [agileDatabases] Re: UML Data Modeling Profile--how close
are we to adoption of a single standard?




Hi Dana, thanks for keeping this thread alive. I agree it is an
important issue that deserves more input from the community.

I just found http://www.omg.org/docs/ad/05-06-09.pdf
<http://www.omg.org/docs/ad/05-06-09.pdf>  while searching
with http://www.google.be/search?q= <http://www.google.be/search?q=>
"data+modeling"+profile+site%3Aomg.org
Perhaps the authors are already involved as an OMG member and they can
learn us more already?

I was wondering: the commercial profiles you have used, do they
support two layers of abstraction (physical data modeling, conceptual
data modeling)? In my own case studies, I have found Scott's profile
elements to be incomplete when it comes to key representation.

Kind regards,
Pieter Van Gorp, Assistant Professor (Universitair Docent)
Information Systems Group, School of Industrial Engineering
Eindhoven University of Technology (TU/e), The Netherlands
Phone: +31 40 247 2062, Skype: pvgorp, Fax: +31 40 243 2612
http://is.tm.tue.nl/staff/pvgorp/research/
<http://is.tm.tue.nl/staff/pvgorp/research/>

On Thu, Jul 30, 2009 at 10:51 AM, dananrg<dananrg@...
<mailto:dananrg%40yahoo.com> > wrote:
>
>
> Posted to comp.software-eng and did get one useful response about the
> time-table for an approved UML data modeling profile. I'm posting the
text
> here for reference. Hope this is useful to others. It was to me. I
hope more
> people start talking about this stuff:
>
>
http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c
2f3a4ede47c5ac#
<http://groups.google.com/group/comp.software-eng/browse_thread/thread/4
c2f3a4ede47c5ac#>
>
> Responding to dana...
>
>> 5) What precisely *is* an RFP? Is the OMG simply in a mode where it
is
>> soliciting entire candidate UML data modeling profiles, or has it put
>> out a draft and is it asking for refinements? If it's the former, how
>> many years might one expect a standard to selected, refined, and
>> ratified?
>
> Request For Proposal. IOW, OMG is looking for ideas on what a standard
> should be.
>
> Currently UML only supports very basic data modeling (i.e., RDB
schemas)
> in a Class Diagram, using the Class Diagram as a traditional Entity
> Relationship Diagram. (Lots of tools use other diagtrams for data
> management contexts, but that usage is not standardized.) As the
> citation you provided indicates, data management has come a long way
> since ERDs were sufficient due to warehousing, middleware, etc., etc..
> So there is a need for a more comprehensive standard that goes beyond
> RDB schemas.
>
> OMG is soliciting proposals for such a standard, probably because UML
> already provides a lot of the needed constructs for OOA/D. That is,
one
> only needs to re-interpret the OOA/D semantics to the data management
> domain. OMG's MDA initiative already provides semantic infrastructure
> for doing that through the profile concept. I haven't read it but my
> guess is that the RFP is primarily about defining an MDA profile for
> using UML as a notation for general data management (and adding any
> syntactic elements, if needed). That is, it provides a unique semantic
> interpretation at the meta-model level for the notation.
>
> Progress at standards groups tends to be glacial, so I wouldn't expect
> much for at least a couple of years. Typically OMG gets an initial
> standard ratified in about four years. But that will usually be a sort
> of trial balloon good for "vanilla" modeling and will likely be
> significantly modified over another four years to get the details
right.
>
>> 6) What can the community do to accelerate the process of getting a
>> UML Data Modeling profile ratified? Does one have to be a prof or in
>> the upper echelons of one's field, or can "ordinary" professionals
>> contribute?
>
> Other than joining OMG, the only thing the community can do is contact
> the working group members and lobby them. Typically working groups are
> made up of representatives of software vendors who volunteered because
> that have vested interests in getting their proprietary view of the
> world anointed as the standard, so the political infighting can be
> intense. Providing support like white papers on particular issues can
be
> useful as a lobbying technique by supporting particular working group
> members in a concrete way. But being a member of the working group is
> far and away the best way to influence the standard.
>
> Anybody can participate. However, to participate in a working group
> initially defining the standard you have to be a member of OMG (either
> representing a corporation or as an individual). [If OMG gets too many
> volunteers for an efficient working group, they will usually break up
> the working group into multiple working groups with more narrowly
> defined scope. So if you are an OMG member and want to participate,
you
> very likely can somehow.]
>
> Once a draft proposal is created OMG will make it publicly available
for
> review and anyone can critique it without OMG membership. But by that
> time there is a lot of inertia and the public review is going to have
to
> identify some really glaring problem to cause anything but cosmetic
changes.
>
> --
> ===========
>
> Dana here again.
>
> I posted on comp.databases.theory and got replies that were by turns
> accusatory, dismissive, and sarcastic. Not precisely the type of
> professional discourse I'd hoped for. One person denied the existence
of an
> O-R impedance mismatch:
>
>
http://groups.google.com/group/comp.software-eng/browse_thread/thread/4c
2f3a4ede47c5ac#
<http://groups.google.com/group/comp.software-eng/browse_thread/thread/4
c2f3a4ede47c5ac#>
>
> Guess I chose the wrong "theory" (e.g. not pure RM?) and the wrong
> newsgroup--as the grail keeper in Indiana Jones & The Last Crusade
said,
> "Choose wisely." It could be said of me--me being he--"He chose...
> unwisely."
>
> Dana
>
>





[Non-text portions of this message have been removed]

Several years ago I wrote in Agile Database Techniques about the cultural
impedance mismatch between the object community and the data community, which
refers specifically to the difficulties that object-oriented developers and data
professionals experience when working together and generally to the
dysfunctional politics between the two communities that occur within IT
organizations and even the IT industry itself. Worse yet, this impedance
mismatch has become even more pronounced between the agile and data communities.
This has been to the detriment of both, but the data community has particularly
suffered as a result. I believe that it doesn’t have to be this way, and
provide some strategies for addressing the problem.   I've reprised my thoughts
in "The Cultural Impedance Mismatch" published in the current issue of TDAN. 
The URL is http://www.tdan.com/view-articles/11066 .  Hope you find it
interesting.

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
Follow me on Twitter: http://twitter.com/scottwambler


       __________________________________________________________________
Connect with friends from any web browser - no download required. Try the new
Yahoo! Canada Messenger for the Web BETA at
http://ca.messenger.yahoo.com/webmessengerpromo.php

--- On Tue, 8/4/09, Pieter Van Gorp <pieter@...> wrote:

> > What do you think is missing?
> It is impossible to include to-one association ends in key
> definitions.  When you can only include attributes in
> key definitions,
> you are forced to using an attribute even when you wish to
> use a
> to-one association. 

Wouldn't that be indicated by a multiplicity of 1 on the association line?  Or,
if that's not sufficient, wouldn't an OCL constraint get the job done?

If not, surely there must be a fairly straightforward strategy for doing so.

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
Follow me on Twitter: http://twitter.com/scottwambler


       __________________________________________________________________
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your
favourite sites. Download it now
http://ca.toolbar.yahoo.com.

2009/8/4 Scott Ambler <scottwambler@...>:
>
>
> --- On Sun, 8/2/09, Pieter Van Gorp <pieter@...> wrote:
>
>>
>> I was wondering: the commercial profiles you have used, do
>> they
>> support two layers of abstraction (physical data modeling,
>> conceptual
>> data modeling)?  In my own case studies, I have found
>> Scott's profile
>> elements to be incomplete when it comes to key
>> representation.
>
> What do you think is missing?
>
> - Scott
>

Isn't there a danger that by trying to make one profile suit every
variety of data model (relational, SQL, network, hierarchical) it
becomes a harder task for modelling tools and humans to produce a
valid model? For example: attribute ordering within keys isn't valid
in a relational model but some SQL systems require it; associations
need not be foreign keys; candidate keys may be mandatory but primary
keys optional.

Another minor point. I'd like a notation for a key with an empty
attribute list: {}. I've often had reason to create single-row
"parameter" tables but modelling tools frequently have no way to
represent this. Having a notation for that case is better than leaving
out the key altogether.

David

On 06/08/2009, at 9:24 AM, Scott Ambler wrote:
> I've reprised my thoughts in "The Cultural Impedance Mismatch"
> published in the current issue of TDAN.

Scott,

  > 19% felt the data group offered too little value.

The data group doesn't exist to offer value to the developers,
but to the business as a whole. That means defending the
stability of database schemas for the benefit of consumers
outside the development groups. I suspect that this defensive
response is a large cause of the development group's complaint,
but it isn't easily avoided. Agile techniques are all very well, but
must be applied across all consumers of the data, and that's a
significant political problem which the data community can only
influence, not control. It would help if they started trying, of course!

Many of the problems that are avoided by good data modeling
only occur during scaling. The agile development teams can
easily be painted as heroes because of the speed with which
they get an effective system into production, but are not blamed
for subsequent DBMS scaling issues; that's the data group's
problem, right? Wrong! The point is that good agile practise
must involve performance testing *at scale*. With computers
having many GB of RAM, that requires large data sets; at least
10x the available RAM is a good starting point. In my work, I
make that easier by testing in a VM where the DBMS has only
very limited RAM available for its buffer space; from 64-256MB.
It's then much easier to demonstrate DBMS scaling issues.
But this doesn't easily mesh with a short TDD cycle.

Your conclusion that the data community is to blame for not
providing the training that developers require is a false one.
If developers need the skills, they should seek education, not
demand it be thrust at them. Saying "we're here to stay, so nyah
nyah nyah" doesn't help. If "we" are producing bad designs, then
"we" need to take responsibility to educate ourselves.

Ultimately, though objects may be the most effective known way
to design software, they've been proven less effective than relations
in designing databases. Objectbases didn't fail because of technical
issues, they failed because they didn't have an adequate focus on
correct transactional behaviour, or even a mathematical framework
which could *define* such correctness. For all that relational DBMS
don't seem as flexible as developers would like, what they do is at
least correct to an agreed standard of correctness. A large part of
the residual problems lie in the lack of adequate support for automatic
mapping from conceptual to physical models. SQL isn't exclusively
physical, but almost always gets used that way.

The truth is that objects and relations differ mainly because they
hang attributes on different frameworks. That's a problem which
*simply doesn't exist* in attribute-free modeling approaches like
Object Role Modeling. ORM2 can be used to create models that
contain *pure* relations (fact types), and from those, both good
object models *and* good RDBMS models can be mechanistically
derived (that is, in reference to the static aspects of the objects).
Starting with either attribute-based approach prevents this mapping
being effectively automated. In other words, ORM2 unifies object
modeling and relational modeling while achieving the goals of those
who espouse "truly" relational databases. It avoids the O/RM Vietnam
by removing the need to create an O/R mapping; both are artefacts
of a unifying higher-level model and always provably congruent.

Both the object and the data communities would be well-served by
an understanding of this revolutionary (but old) approach. Both can
find common ground and both improve on their best achievements
by adopting it.

Clifford Heath, Data Constellation, http://dataconstellation.com
Agile Information Management and Design.

--- On Wed, 8/5/09, Clifford Heath <clifford.heath@...> wrote:

>
>  > 19% felt the data group offered too little value.
>
> The data group doesn't exist to offer value to the
> developers,
> but to the business as a whole.

Yes, the developers are part of that whole business and are a major customer of
the data group.  If the data group can't interact effectively with the
development teams then there is very little hope of there ever being quality
data sources.  And, as we know, most organizations suffer from data quality
challenges.


> That means defending the
> stability of database schemas for the benefit of consumers
> outside the development groups. I suspect that this
> defensive
> response is a large cause of the development group's
> complaint,
> but it isn't easily avoided.

Actually, it could be easily avoided if data groups were to adopt more
collaborative and evolutionary techniques.


> Agile techniques are all very
> well, but
> must be applied across all consumers of the data, and
> that's a
> significant political problem which the data community can
> only
> influence, not control. It would help if they started
> trying, of course!

Exactly.

However, although it would be nice if all consumers of the data were to apply
agile techniques it's not realistic. As a result the data groups must be
flexible enough to apply both agile and traditional strategies, and everything
in between, when they're working with various groups.  Different groups will
work in different ways, and the data folks need to be flexible enough to
interact with them accordingly.  As the lean community points out, we need to
optimize the whole, not the parts.  The implication is that one data strategy,
an optimization of the "data part", will not fit all.


>
> Many of the problems that are avoided by good data
> modeling
> only occur during scaling. The agile development teams can
> easily be painted as heroes because of the speed with
> which
> they get an effective system into production, but are not
> blamed
> for subsequent DBMS scaling issues; that's the data
> group's
> problem, right? Wrong!

If the data groups are responsible for the data sources, then it is their
problem.  If they know that development teams are doing whatever it is that
they're doing, then they need to interact with those teams appropriately to work
with them to ensure data quality.


> The point is that good agile
> practise
> must involve performance testing *at scale*. With
> computers
> having many GB of RAM, that requires large data sets; at
> least
> 10x the available RAM is a good starting point. In my work,
> I
> make that easier by testing in a VM where the DBMS has
> only
> very limited RAM available for its buffer space; from
> 64-256MB.
> It's then much easier to demonstrate DBMS scaling issues.
> But this doesn't easily mesh with a short TDD cycle.

TDD isn't the only approach to testing which agile teams employ, and some TDD
cycles can be very long.


>
> Your conclusion that the data community is to blame for
> not
> providing the training that developers require is a false
> one.

I disagree.  I often run into data professionals who complain about data quality
issues, who complain that development teams are doing data design, and that
they're doing a poor job of it.  I see these things too all the time.  But when
I ask them what they're doing about the problem, their solution is for them to
have more control over the data issues, even though they're already being
ignored successfully by the development teams.  Their strategy is a political
solution that does little more than try to shore up their eroding positions, a
strategy that sadly hasn't been working very well for them historically. 
Instead, they'd be better off finding ways to collaborate effectively and help
transfer their data skills to developers, either through close collaboration,
mentoring, or training.  Similarly, they should be trying to pick up skills from
others to help modernize their own skillsets.  This is a more difficult approach
for most data professionals,
  one that is undoubtably threatening to many.

> If developers need the skills, they should seek education,
> not
> demand it be thrust at them.

Very few developers will do this because they perceive data design to be
relatively trivial compared with other design issues that they're dealing with.

> Saying "we're here to stay, so
> nyah
> nyah nyah" doesn't help. If "we" are producing bad designs,
> then
> "we" need to take responsibility to educate ourselves.

But they don't perceive them as bad designs.  Because the interaction between
data people and developers is problematic in many organizations, the developers
don't have the background to identify what the real source of the problem is. 
They can't ask for help if they don't know that they need to.  They've also got
a lot more than just data on their plates, and those issues are often viewed as
more interesting to them.

If data professionals perceive the problem, which they often do, then they need
to act.  And they need to do so in a manner which reflects the realities of the
situation.
<snip>

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
Follow me on Twitter: http://twitter.com/scottwambler




       __________________________________________________________________
Make your browsing faster, safer, and easier with the new Internet Explorer® 8.
Optimized for Yahoo! Get it Now for Free! at
http://downloads.yahoo.com/ca/internetexplorer/

Scott, it seems to me that you're requiring a tremendous amount from the
data group.

(1) You wrote: " ... the data groups must be flexible enough to apply
both agile and traditional strategies, and everything in between, when
they're working with different groups." Wow, that seems like an awful
lot to expect. These are data professionals and not methodologists. Many
people have the capability to learn and apply a single process. To
expect them to apply everything from A to Z and to make every variation
work I think is way too high a bar.

For example, I became a DBA in 2003. In the absence of a data group work
process, I developed one over the course of the next few years that
works very well in my shop. Were I to have to develop a different
process, it would take me months or years to get it right and there
would be some stumbling and reworking of the process along the way that
I'm sure my developers would not appreciate. So I stick to my process
(which seems to support pretty well their unstructured, iterative
approach to application development).

(2) You wrote: "Instead, they'd be better off finding ways to
collaborate effectively and help transfer their data skills to
developers ..." Scott, I think you fall naturally into the role of
mentor and trainer. Not all of us have that skill set. I have no
training in how to train others although I happen to have a natural love
of mentoring others one-on-one. However not everyone can effectively
train another person. I can think of a certain brilliant DBA I worked
with that had no capability at all of transferring his knowledge to me.

I agree that data group members should do more training and mentoring of
application developers. But we also shouldn't expect the developers to
master the more difficult aspects of our jobs--after all, that is why
the organization hired DBAs with specialized expertise. Clifford's
example of scaling is a good one, and there are many more.

________________________________

From: agileDatabases@yahoogroups.com
[mailto:agileDatabases@yahoogroups.com] On Behalf Of Scott Ambler
Sent: Thursday, August 06, 2009 4:49 AM
To: agileDatabases@yahoogroups.com
Subject: Re: [agileDatabases] The Cultural Impedance Mismatch (between
developers and data professionals)






--- On Wed, 8/5/09, Clifford Heath <clifford.heath@...
<mailto:clifford.heath%40gmail.com> > wrote:

>
> > 19% felt the data group offered too little value.
>
> The data group doesn't exist to offer value to the
> developers,
> but to the business as a whole.

Yes, the developers are part of that whole business and are a major
customer of the data group. If the data group can't interact effectively
with the development teams then there is very little hope of there ever
being quality data sources. And, as we know, most organizations suffer
from data quality challenges.

> That means defending the
> stability of database schemas for the benefit of consumers
> outside the development groups. I suspect that this
> defensive
> response is a large cause of the development group's
> complaint,
> but it isn't easily avoided.

Actually, it could be easily avoided if data groups were to adopt more
collaborative and evolutionary techniques.

> Agile techniques are all very
> well, but
> must be applied across all consumers of the data, and
> that's a
> significant political problem which the data community can
> only
> influence, not control. It would help if they started
> trying, of course!

Exactly.

However, although it would be nice if all consumers of the data were to
apply agile techniques it's not realistic. As a result the data groups
must be flexible enough to apply both agile and traditional strategies,
and everything in between, when they're working with various groups.
Different groups will work in different ways, and the data folks need to
be flexible enough to interact with them accordingly. As the lean
community points out, we need to optimize the whole, not the parts. The
implication is that one data strategy, an optimization of the "data
part", will not fit all.

>
> Many of the problems that are avoided by good data
> modeling
> only occur during scaling. The agile development teams can
> easily be painted as heroes because of the speed with
> which
> they get an effective system into production, but are not
> blamed
> for subsequent DBMS scaling issues; that's the data
> group's
> problem, right? Wrong!

If the data groups are responsible for the data sources, then it is
their problem. If they know that development teams are doing whatever it
is that they're doing, then they need to interact with those teams
appropriately to work with them to ensure data quality.

> The point is that good agile
> practise
> must involve performance testing *at scale*. With
> computers
> having many GB of RAM, that requires large data sets; at
> least
> 10x the available RAM is a good starting point. In my work,
> I
> make that easier by testing in a VM where the DBMS has
> only
> very limited RAM available for its buffer space; from
> 64-256MB.
> It's then much easier to demonstrate DBMS scaling issues.
> But this doesn't easily mesh with a short TDD cycle.

TDD isn't the only approach to testing which agile teams employ, and
some TDD cycles can be very long.

>
> Your conclusion that the data community is to blame for
> not
> providing the training that developers require is a false
> one.

I disagree. I often run into data professionals who complain about data
quality issues, who complain that development teams are doing data
design, and that they're doing a poor job of it. I see these things too
all the time. But when I ask them what they're doing about the problem,
their solution is for them to have more control over the data issues,
even though they're already being ignored successfully by the
development teams. Their strategy is a political solution that does
little more than try to shore up their eroding positions, a strategy
that sadly hasn't been working very well for them historically. Instead,
they'd be better off finding ways to collaborate effectively and help
transfer their data skills to developers, either through close
collaboration, mentoring, or training. Similarly, they should be trying
to pick up skills from others to help modernize their own skillsets.
This is a more difficult approach for most data professionals,
one that is undoubtably threatening to many.

> If developers need the skills, they should seek education,
> not
> demand it be thrust at them.

Very few developers will do this because they perceive data design to be
relatively trivial compared with other design issues that they're
dealing with.

> Saying "we're here to stay, so
> nyah
> nyah nyah" doesn't help. If "we" are producing bad designs,
> then
> "we" need to take responsibility to educate ourselves.

But they don't perceive them as bad designs. Because the interaction
between data people and developers is problematic in many organizations,
the developers don't have the background to identify what the real
source of the problem is. They can't ask for help if they don't know
that they need to. They've also got a lot more than just data on their
plates, and those issues are often viewed as more interesting to them.

If data professionals perceive the problem, which they often do, then
they need to act. And they need to do so in a manner which reflects the
realities of the situation.
<snip>

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
<http://www.ibm.com/developerworks/blogs/page/ambler>
Follow me on Twitter: http://twitter.com/scottwambler
<http://twitter.com/scottwambler>

__________________________________________________________
Make your browsing faster, safer, and easier with the new Internet
Explorer(r) 8. Optimized for Yahoo! Get it Now for Free! at
http://downloads.yahoo.com/ca/internetexplorer/
<http://downloads.yahoo.com/ca/internetexplorer/>





[Non-text portions of this message have been removed]

>> If developers need the skills, they should seek education,
>> not demand it be thrust at them.
>
> Very few developers will do this because they perceive data design
> to be relatively trivial compared with other design issues that
> they're dealing with.

A typical head-in-the-sand mentality, which doesn't reflect the truth.

>> Saying "we're here to stay, so
>> nyah nyah nyah" doesn't help. If "we" are producing bad designs,
>> then "we" need to take responsibility to educate ourselves.
>
> But they don't perceive them as bad designs.

Because their head is in the sand. It doesn't mean they're
not bad designs. If the consequence of poor data models
is poor scaling, how can that be "somebody else's problem"?
It can only be the responsibility of the modeller, whatever
their perceptions of the matter.

> Because the interaction between data people and developers is
> problematic in many organizations, the developers don't have the
> background to identify what the real source of the problem is.

And yet they know when to call in a UI specialist, or any
other kind of specialist... but they assume they can do
modeling because they can get their software to work
correctly when it has five records in the DBMS.

> <snip>

I note that you snipped the real meat of what I was saying, and
used your response as yet another opportunity to perpetuate
your biases instead.

There is a solution to this problem, but it can only be implemented
after not just data people, but developers also admit that just as
relations are "a hammer looking for a nail", so are their objects.
They both suffer a common weakness in that they're both attribute
oriented, and hence bias any conceptual model towards the
physical *before* it can be properly analysed. Objects that have
relationships instead of attributes are facts, and in this respect only
fact-oriented modeling can be truly conceptual. It's such an easy
thing to see, when you realise that it's exactly how our natural
communication works... unless you're too invested in another way
of thinking, and don't *want* to see. Just like the folk you're bashing!

Clifford Heath.

On 06/08/2009, at 9:24 AM, Scott Ambler wrote:
> I've reprised my thoughts in "The Cultural Impedance Mismatch"
> published in the current issue of TDAN.
>

I do agree with Clifford's fine summary. Scott's view that data
management professionals aren't trying hard enough to educate
application developers is not my experience. Instead I've sometimes
found it hard to motivate developers to learn about enterprise data
management issues, and that includes agile teams as well. Perhaps
especially agile teams in fact. Increasingly, those developers rely
too much on mapping their object models to relational models and
assume they don't need to bother with relational database modelling at
all.

On explaining to one developer why join dependency was an important
consideration in database design I was told that it was "too
theoretical" and not relevant to real development. Of course it may be
that I'm not a very good teacher. Perhaps I'm not. But most of the
learning material that developers are exposed to totally fails to deal
with fundamental data management issues, which means there is a
mountain to climb when it comes to teaching them how to solve
practical problems.

Scott, if you really believe in educating developers about data
management issues then I can't understand why you seem to discourage
them from learning database fundamentals
(http://www.agiledata.org/essays/relationalTheory.html). The urgent
need for educators in my opinion is not to teach "applying SQL in
practice"(!!) instead more education is needed to give IT
professionals a grounding in the basics so that they will then stand a
chance of understanding more advanced issues and techniques.

David

Hi David,

On Wed, Aug 5, 2009 at 9:15 PM, David Portas<dportas@...> wrote:
> Isn't there a danger that by trying to make one profile suit every
> variety of data model (relational, SQL, network, hierarchical) it
> becomes a harder task for modelling tools and humans to produce a
> valid model?
So far, I think the proposed profiles are not too complex... but I'm
interested in your specific examples...

> For example: attribute ordering within keys isn't valid
> in a relational model but some SQL systems require it;
Which SQL systems require that?  I haven't experienced that problem so
far.  The good news is that the UML profiles for data modeling are
relying on lists (=> ordered) in the tag definitions for keys so the
order that you specify can be taken into account (when desirable).  It
should be noted though, that it is rather accidental that the order
between the attribute references is preserved in your UML model... So
I think it should be taken into account when using a new version of
the UML (2010+ ?), since at some point, it could become possible to
define tags based on sets or bags rather than lists only.

> associations
> need not be foreign keys
I think that you should always map association ends to foreign keys
when translating a conceptual data model in a physical data model.
Even when your database does not support foreigh keys (e.g., mysql) it
is useful to know the implicit integrity rules.  Not?

> candidate keys may be mandatory but primary
> keys optional.
Do you mean that one should not include the concept of a primary key
in the OMG approved version of the profile for data modeling?  If so,
why not?

> Another minor point. I'd like a notation for a key with an empty
> attribute list: {}. I've often had reason to create single-row
> "parameter" tables but modelling tools frequently have no way to
> represent this. Having a notation for that case is better than leaving
> out the key altogether.
Interesting case.  If I am not mistaking, you are expecting the
following logical integrity rule to be preserved:

all tup1,tup2: YourTable | false implies tup1=tup2

If you would have a non-empty attribute list as a key (e.g., {a1,a2})
the 'false' would correspond to (tup1.a1=tup2.a1 and
tup1.a2=tup2.a2)...

Obviously, the rule "all tup1,tup2: YourTable | false implies
tup1=tup2" is vey inefficient and I only wrote it to better understand
how this special case relates to the general case.  A more efficient
and also more simple way to enforce the empty-attribute-list-key case
would be to check whether there is at most one tuple in the table.
Please correct me if I am mistaking.

Regards,
Pieter

The fundamental problem with databases and agile development is that they
contain data.

If we could throw away the data then agile database development would be a
no-brainer.

Unfortunately we cannot throw away the data
1.  It probably drives a wide range of systems
2.  It has a very real value, in some cases that value is extreme!
3.  It remains useful over a very long period of time, in fact its use may
increase in value over time.

I think the key thing is to separate out the tasks necessary for development
from the tasks necessary to deploy to a production environment until the
deploy-to-production tasks are needed.

If this means the last story in every itteration is to work out how to deploy to
production without incurring unacceptable down-time then so be it.  In fact I
would say this should be a compulsory step.

I would suggest a mid-itteration task involving DBAs to ask "is it feasible to
deploy this to production"?  No point in having a task to amend the structure of
a 1 billion row table if you need 99.999% up-time!

Hi Scott,

On Thu, Aug 6, 2009 at 1:29 AM, Scott Ambler<scottwambler@...> wrote:
>> > What do you think is missing?
>> It is impossible to include to-one association ends in key
>> definitions.  When you can only include attributes in
>> key definitions,
>> you are forced to using an attribute even when you wish to
>> use a
>> to-one association.
> Wouldn't that be indicated by a multiplicity of 1 on the association line?
I think you are referring to an approach such as
http://is.tm.tue.nl/staff/pvgorp/fora/key-issue1.png (which I
discussed on http://www.cs.york.ac.uk/puml/puml-list-archive/1644.html)?
  In that example, each "Internet Access" instance needs unique values
for (user,host).  Indeed, the example relies on association ends with
multiplicity one.  But... (see below)...

> Or, if that's not sufficient, wouldn't an OCL constraint get the job done?
Exactly, http://is.tm.tue.nl/staff/pvgorp/fora/key-issue1.png needs
two OCL expressions to express the key/unique property.  As discussed
on http://www.cs.york.ac.uk/puml/puml-list-archive, this has several
disadvantages.  Among other issues, it is quite hard to build a
UML2SQL translator that can interpret such OCL expressions (and it is
overkill to build a general purpose OCL to SQL translator).

> If not, surely there must be a fairly straightforward strategy for doing so.
I think http://is.tm.tue.nl/staff/pvgorp/fora/key-issue2a.png would be
a quite simple and adequate solution.  Again,
http://www.cs.york.ac.uk/puml/puml-list-archive/1644.html mentions
some further tool issues but they can be resolved.  Any feedback?

Best regards,
Pieter

Pieter,

Throughout I'm thinking that it's desirable to have models that can be
validated by software. Different validation rules apply to different
types of model, which is why I question the desirability of having a
single profile for all. It might be sufficient to have a stereotype
that identifies the model type but then users and CASE tool vendors
will still want to know which grammar is the correct one for each
type.

2009/8/6 Pieter Van Gorp <pieter@...>:
>
>> For example: attribute ordering within keys isn't valid
>> in a relational model but some SQL systems require it;
> Which SQL systems require that? I haven't experienced that problem so
> far. The good news is that the UML profiles for data modeling are
> relying on lists (=> ordered) in the tag definitions for keys so the
> order that you specify can be taken into account (when desirable). It
> should be noted though, that it is rather accidental that the order
> between the attribute references is preserved in your UML model... So
> I think it should be taken into account when using a new version of
> the UML (2010+ ?), since at some point, it could become possible to
> define tags based on sets or bags rather than lists only.
>

Microsoft SQL Server matches a compound foreign key to a
UNIQUE/PRIMARY key constraint based on the ordered list of attributes
specified for those keys. As far as I can tell the SQL standard is not
specific about whether foreign keys should match to UNIQUE/PRIMARY
constraints by name or by position. (ISO/IEC 9075-2:2003 secion 11.8).

>> associations
>> need not be foreign keys
> I think that you should always map association ends to foreign keys
> when translating a conceptual data model in a physical data model.
> Even when your database does not support foreigh keys (e.g., mysql) it
> is useful to know the implicit integrity rules. Not?
>

I agree it's probably a good idea most of the time but a foreign key
is really just a special case of an inclusion dependency, ie the
general case that A is a subset of B. I think we ought to be able to
represent inclusion dependency in models but always labelling it a
"foreign key" is wrong and confusing and might cause problems
validating the model.

>> candidate keys may be mandatory but primary
>> keys optional.
> Do you mean that one should not include the concept of a primary key
> in the OMG approved version of the profile for data modeling? If so,
> why not?
>

I think it should be included but if we need to model SQL systems then
it must be optional because in SQL all keys are optional.

>> Another minor point. I'd like a notation for a key with an empty
>> attribute list: {}. I've often had reason to create single-row
>> "parameter" tables but modelling tools frequently have no way to
>> represent this. Having a notation for that case is better than leaving
>> out the key altogether.
> Interesting case. If I am not mistaking, you are expecting the
> following logical integrity rule to be preserved:
>
> all tup1,tup2: YourTable | false implies tup1=tup2
>
> If you would have a non-empty attribute list as a key (e.g., {a1,a2})
> the 'false' would correspond to (tup1.a1=tup2.a1 and
> tup1.a2=tup2.a2)...
>
> Obviously, the rule "all tup1,tup2: YourTable | false implies
> tup1=tup2" is vey inefficient and I only wrote it to better understand
> how this special case relates to the general case. A more efficient
> and also more simple way to enforce the empty-attribute-list-key case
> would be to check whether there is at most one tuple in the table.
> Please correct me if I am mistaking.

That's correct. It should be clearly identifiable as a key and not any
other type of constraint because, again, in a (not SQL) relational
model keys ought to be mandatory for every table.

David

Hi again,

On Thu, Aug 6, 2009 at 10:27 PM, David Portas<dportas@...> wrote:
> Throughout I'm thinking that it's desirable to have models that can be
> validated by software. Different validation rules apply to different
> types of model, which is why I question the desirability of having a
> single profile for all. It might be sufficient to have a stereotype
> that identifies the model type but then users and CASE tool vendors
> will still want to know which grammar is the correct one for each
> type.
This can be realized using the standards UML profile mechanism.  You
don't need an old-school grammar for that.  More specifically, you can
add so-called OCL well-formedness rules (WFRs) to your profile(s).
You can use this to state for example that if the stereotype
<<MS-SQL>> is attached to a package, that other stereotypes and tagged
values should be applied in a specific way in that package.  I suggest
that the OMG profile only has WFRs based on ISO SQL whereas different
vendors add WFRs for their products.  That architecture will only work
if the vendors check that their WFRs are not in conflict with the OMG
ones (which requires probably database knowledge beyond the OCL
semantics that can be checked automatically).  Still, this
architecture would solve already quite a number of
incompatibilty/conversion issues, right?

Best regards,
Pieter

PS: for an example OCL WFR from a profile from another domain, see the
appendix from http://is.ieis.tue.nl/staff/pvgorp/research/#gtvmt2006

I've been reading Agile Database Techniques (great book), but I am perplexed
about Scott's definitions of Normal Forms. I'm not sure I understand them myself
after all these years, as I see conflicting definitions from different authors.
I feel I understand them well enough to design great tables, but perhaps not as
well as I should understand relational theory.

Long post here and I mean absolutely no disrespect to anyone. No mean-spirited
intentions here, so please do not take them that way. Just trying to get to
objective facts of the matter and seeing where there is genuine disagreement.
Here goes:

Is there "genuine" disagreement on what 1NF, 2NF and 3NF mean? Or is the variety
of contradictory in-print definitions of  Normal Forms due to misunderstanding
from authors or from authors trying to "protect" readers? Also, where can I find
an accurate, concise (less than 3 sentences for each NF), and
easy-for-any-IT-person-to-understand, definition of Normal Forms 1 through 3?

My understanding of Normal Forms 1 through 3 is as follows. I look forward to
being corrected. I'm leaving out any mention of relations and relvars here,
because I don't expect the typical IT person to know what those are (although it
can be argued, convincingly I think, that they should):

1NF:

1) No multivalued attributes (e.g. every attribute in every tuple should be
atomic, containing only one "fact" that can't further be decomposed; although
there may be legit exceptions, e.g. zip code)

2) No repeating groups of attributes are allowed (e.g. SKILL_1, SKILL_2, SKILL_n
are out). Repeating groups of attributes, of course, lead to data anomalies and
growth problems.

Most common definitions I've read in books on database design miss point #1
completely. Is that because #1 is disputably not part of 1NF, because the
authors are "protecting" the reader, or because of authors' incomplete
understandings of what it is for something to be in 1NF?

2NF:

1) Must already be in 1NF
2) Every non-key attribute must rely exclusively on the key, e.g. no Functional
Dependencies.

Here's something I really need clarification on. Is it the case, or is it *not*
the case, that 2NF "only applies" in cases where there is a composite key? This
is my understanding, but I may be dead wrong. If so, I need to know sooner or
later. I have seen this point glossed over or missed by various authors. And I
believe Scott uses the words "fully dependent", not Functionally dependent. He
may have his reasons (e.g. protecting the reader by not scaring the reader
away).

3NF:

1) Must already be in 2NF
2) No transitive dependencies. All non-key attributes must rely exclusively on
the key.

I would like some concise and clear definitions of functional and transitive
dependencies, because 2NF and 3NF sound similar to me. So I am not surprised
that, perhaps, some have missed nuances--myself included.

Pet peeve:

I dislike the mantra, "the key, the whole key, and nothing but the key" to
characterize Normal Forms 1 through 3. What is evidently intended as a helpful
mantra seems to give a false sense of understanding. I'm all about some
mnemonics, but understanding must preceed the mnemonic, right? And a mnemonic
should not promote incorrect understanding?

Finally, what are some good books out there on database design for neophytes and
experienced folks alike that gets the theory correct while not getting mired in
what relational databases *could* be. I read a really great book by Fabian
Pascal, the title of which escapes me, which I need to go and buy--believe it
had the word Practitioner in it. I want to be faithful to relational theory as
much as possible while getting work done with products currently on the market.
This seems to be Scott's intention in Agile Database Techniques.

I believe Fabian Pascal stated that theory was immensely important for practice;
that it was not about a bunch of curmudgeonly academics bickering about how many
angels can stand on the head of a pin (which it can seem to be at times). Is
Fabian Pascal correct? If so, to what degree? While it's essential that
methodologists and theorists are out there visioning what databases can or
should be, I agree that practitioners still have to get projects completed with
existing resources (e.g. the SQL language, leading RDBMSes, etc).

Thanks very much.

Dana

--- On Thu, 8/6/09, David Portas <dportas@...> wrote:

> >
>
> I do agree with Clifford's fine summary. Scott's view that
> data
> management professionals aren't trying hard enough to
> educate
> application developers is not my experience. Instead I've
> sometimes
> found it hard to motivate developers to learn about
> enterprise data
> management issues, and that includes agile teams as well.

The issue isn't how hard you're trying, it's how well you're succeeding.  If
you're not finding ways to motivate people to learn this stuff, then we've still
got a problem.

> Perhaps
> especially agile teams in fact. Increasingly, those
> developers rely
> too much on mapping their object models to relational
> models and
> assume they don't need to bother with relational database
> modelling at
> all.

Exactly.  And unless you find ways to motivate people to go beyond that, the
situation will never get better.  The OR framework folks found a way to motivate
people to get interesting in OR mapping -- They solved a perceived need and
provided (relatively) consumable tooling to help address that.

The development community must perceive the need to improve data quality, or at
least not make the problem worse, and I suspect for the most part that they do. 
They might not understand the solution, and they likely don't have decent tools
(yet).

>
> On explaining to one developer why join dependency was an
> important
> consideration in database design I was told that it was
> "too
> theoretical" and not relevant to real development. Of
> course it may be
> that I'm not a very good teacher. Perhaps I'm not. But most
> of the
> learning material that developers are exposed to totally
> fails to deal
> with fundamental data management issues, which means there
> is a
> mountain to climb when it comes to teaching them how to
> solve
> practical problems.

As to my previous point, I'm sure you tried hard but it didn't work out.  I see
this all the time too.  Some data professionals are trying to transfer skills,
I'm sure some are succeeding, some are trying and not succeeding as often as
they'd like, but many aren't even trying.

Perhaps if we discussed issues like this in terms that developers are
(hopefully) interested in, such as quality and performance, we'd likely have
better results.

I'd be surprised to discover that more than 1 in 3 orgs choose to give
developers even something as simple as a 1 or 2 day course in database design,
let alone significant training and mentoring.

>
> Scott, if you really believe in educating developers about
> data
> management issues then I can't understand why you seem to
> discourage
> them from learning database fundamentals
> (http://www.agiledata.org/essays/relationalTheory.html).

Because there's very little of practical value in it for developers.  Most
developers are tired of wading through all the theory to get to the few nuggets
of practical information that they can use to do a good job.  Until the data
community comes up with a viable way of communicating the nuggets, and doing so
in a non-theoretical-sounding manner, the development community will very likely
continue to ignore them.  The story that you told above is an example, or
perhaps a symptom, of this very thing.

> The urgent
> need for educators in my opinion is not to teach "applying
> SQL in
> practice"(!!) instead more education is needed to give IT
> professionals a grounding in the basics so that they will
> then stand a
> chance of understanding more advanced issues and
> techniques.

And how many decades has the data community been thrashing on this?  Do you
honestly see a light at the end of this tunnel?

- Scott
Scott W. Ambler
Chief Methodologist/Agile, IBM Rational
Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
Follow me on Twitter: http://twitter.com/scottwambler



       __________________________________________________________________
Connect with friends from any web browser - no download required. Try the new
Yahoo! Canada Messenger for the Web BETA at
http://ca.messenger.yahoo.com/webmessengerpromo.php