Hello Andre,

Yes, should read " to double ".  Fixed in subversion.  You are reading carefully
to find that!

Note that exceptions generally include useful information, such as the filed and
bad value in this case.

Please post issues to the mailing list.

Anthony

At 11:41 PM 13/09/2009, you wrote:
>I was looking the SimpleORM's code and found something weird:
>File: SimpleORM_Root_Folder/dataset/SRecordGeneric.java, line 113.
>
>public float getFloat(String field) {
>        Object val = this.get(field);
>        try {
>            return convertToFloat(val);
>        }
>        catch (NumberFormatException e) {
>            throw new SException.Data("Could not convert " + field + " to long
" + val, e); <--- line 113
>            //.setFieldMeta(field).setRecordInstance(this).setParams(val);
>        }
>    }
>
>the exception on getFloat method should be "to float" not "to long".

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Hello to you all.

Just an enquiry. About two months ago I had checked out a version of SimpleORM
which had a class called SQueryAggregate. Subsequent checkouts have not had this
class and unfortunately I was overwriting the folder that I have been using for
checkout. Is there any other class that is going to replace this one since I saw
that it had easy access to sql methods such as count etc? Kindly let me know.

Kind regards.

Abel

Hi,

SQueryAggregate has been renamed into SQueryTransient in latest
"official" release. So the API is slightly broken, but the functionality
remains the same. When refactoring, one gotcha is that SRecordGeneric
has been renamed to SRecordTransient, but another SRecordGeneric has
been created...

Another addition in this release is that records now implement Map. It
might be usefull if you want to use tools that need beans to work.

Franck


Le mardi 15 septembre 2009 à 15:18 +0300, Abel Salama Birya a écrit :
> unfortunately I was overwriting the folder that I have been using for
> checkout. Is there any other class that is going to replace this one
> since I saw that it had easy access to sql

Hello Abel,

New class is SQueryTransient, and there is a section in the white paper on it.

http://simpleorm.org/sorm/whitepaper.html#mozTocId436695

(That is the trouble with checking out from Subversion, we do change our minds
occasionally.)

It is part of the current shipped version.

Anthony

At 10:18 PM 15/09/2009, you wrote:
>
>
>Hello to you all.
>
>Just an enquiry. About two months ago I had checked out a version of SimpleORM
which had a class called SQueryAggregate. Subsequent checkouts have not had this
class and unfortunately I was overwriting the folder that I have been using for
checkout. Is there any other class that is going to replace this one since I saw
that it had easy access to sql methods such as count etc? Kindly let me know.
>
>Kind regards.
>
>Abel
>


Spreadsheet Detective,
Southern Cross Software Queensland Pty Limited
54 Gerler Street
Bardon, Queensland 4065, Australia.

Email: berglas@...
www.SpreadsheetDetective.com
Ph: +61 427 830248 (Australian Eastern Standard Time)

"If the model seems correct only because the numbers look right,
then why build the model in the first place?"

Hello All,

I have just been doing some .Net work using LINQ.  Linq is Microsoft's new very
clever ORM.  It does some amazing things, and most of the time what you want. 
The other times you tear your hair out.

The magic is that this C# looking code actually gets converted to (quite
different) SQL.  But of course, SQL Server is different in many subtle ways to
the .Net engine.

It confirms my idea that simple queries can be automated simply (like
SimpleORM), and that complex queries are much better written in the native SQL
that you are dealing with.

Here is an example of what LINQ can do.  Adding the simple term a.PublishedDate
> date mysteriously killed my performance.



         var articlesWithEntity = (from p in dc.Entities
                         where p.EntityName == EntityName
                         select p.Article).Distinct();

         var uniqueArticleIds = from p in articlesWithEntity
                             group p by p.Title // SQL cannot compare TEXT type
in p.Description:new { p.Title, p.Description };
                                 into g
                                 select g.Max(e => e.ID);

         var uniqueArticles = dc.Articles.Where(a =>
uniqueArticleIds.Contains(a.ID));

         var theArticles2 = from p in uniqueArticles orderby p.Title
                           select new {
                               p.ID, p.Title, p.PublishedDate,
                               FeedTitle = p.Feed.Title, p.Description,
                               Url = p.Url.Replace("/", "/ ").Replace("&", " &")
                           };

         var theArticles3 = theArticles2.Where(a => a.PublishedDate > date);

Results in

SELECT [t2].[ID], [t2].[Title], [t2].[PublishedDate], [t2].[Title2] AS
[FeedTitle], [t2].[Description], [t2].[value] AS [Url]
FROM (
     SELECT [t0].[ID], [t0].[Title], [t0].[PublishedDate], [t1].[Title] AS
[Title2], [t0].[Description], REPLACE(REPLACE([t0].[Url], @p0, @p1), @p2, @p3)
AS [value]
     FROM [dbo].[Articles] AS [t0]
     INNER JOIN [dbo].[Feeds] AS [t1] ON [t1].[ID] = [t0].[Source]
     ) AS [t2]
WHERE ([t2].[PublishedDate] > @p4) AND (EXISTS(
     SELECT NULL AS [EMPTY]
     FROM (
         SELECT MAX([t5].[ID]) AS [value]
         FROM (
             SELECT DISTINCT [t4].[ID], [t4].[Title], [t4].[Url],
CONVERT(NVarChar(MAX),[t4].[Description]) AS [Description], [t4].[Source],
[t4].[PublishedDate], [t4].[HasCalais]
             FROM [dbo].[Entities] AS [t3]
             INNER JOIN [dbo].[Articles] AS [t4] ON [t4].[ID] = [t3].[ArticleID]
             WHERE [t3].[EntityName] = @p5
             ) AS [t5]
         GROUP BY [t5].[Title]
         ) AS [t6]
     WHERE [t6].[value] = [t2].[ID]
     ))
ORDER BY [t2].[Title]

Wow!

(Certainly SQL could be improved -- I should be able to write
table.fkey.fkey.column without setting up subselects of joins.  And the Let
clause that lets complex queries be split up.  Those are the main advantages of
LINQ.)

Otherwise, LINQ avoids Hibernate's byte code generation by using C# Properties
to call methods directly, together with a code generator.  That part feels much
better, but is much more complex than SimpleORM for little benefit.

Anthony

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Hello Ryan,

Could do.  But for now just create subclasses for the different variants.  I
suggest just use a static class for these trivial bits.  There may even be some
real differences later.

(The bigger problem is that SDrivers represent both the instance and the
factory.  Should be a small static factory class for the factory.  Just has
never got to the top of the list to fix.)

It would be very good to support Deryb/JavaDB.  (I'm not clear about the
difference?).  This is long overdue.  And it should build and run Derby out of
the JDK 1.6 directly, with their jars.  Maybe even make Derby the default test
DB instead of HSQL?

The normal way to handle database inconsistencies is to delegate them to some
method in the SDriver hierarchy.  Null works for MSSQL and HSQL.

In SSessionJdbcHelper.flush you would have to change
     if (value == null && fieldMeta.isPrimary() && theGenerator==null)
     throw new SException.Error("Null primary key not set (after
createWithNullKey) " + instance);
     fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));

into something like

     if (value == null && fieldMeta.isPrimary())
        if (theGenerator==null)
          throw new SException.Error("Null primary key not set (after
createWithNullKey) " + instance);
       else
         session.getDriver().setEmptyGeneratedKey() // Or maybe just do nothing
here?
     else
     fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));

Make sure that whatever you do continues to work for at least HSQL, the default
test case.

Send me any changed files and I'll diff and check them in.  You are welcome to
become a committer if you make serious contributions.

There is no magic in the code.  You are not dependent on us to change things,
unlike more complex systems.

Regards,

Anthony


>I'm trying to get identity generation on insert working with Derby. I have the
driver set up so it creates the table with GENERATED BY DEFAULT AS IDENTITY for
my identity field and it works fine. The problem occurs when I call
session.ses.createWithGeneratedKey and then try to commit the new record.
SimpleORM tries to call insert passing null for the identity field. Derby
accepts an insert statement that either 1) doesn't specify a value for the
identity column or 2) passes the value "default" for the identity column. Derby
throws an error when SimpleORM tries to pass null for the generated column.
>
>How should I handle this? If there's no way then I'll have to change SimpleORM
code to pass "default" instead of "null" but I'd prefer not to do that.


At 01:00 PM 27/09/2009, you wrote:
>
>
>Hi.
>
>Have you considered allowing 1 SimpleORM driver to handle multiple JDBC
drivers? Instead of using driverName and comparing the returned string to choose
a SimpleORM driver, you could pass the actual driver name into a boolean method
of an SDriver subclass and let the driver itself decide whether it's a match.
For example, I'm working with an embedded Derby database and the driver name is
"Apache Derby Embedded JDBC Driver" but the driver name for the network client
would be "Apache Derby Network Client JDBC Driver". The same SimpleORM driver
would probably work for both.
>
>Just curious,
>--
>Ryan Boder
>1 614 598 6339
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

PS.  You will also probably need to change
   SDriver.retrieveInsertedKey
Which is called by
    theGenerator.postUpdateWithGeneratedKey(session, instance);

See HSQL.

Anthony

At 05:11 PM 28/09/2009, you wrote:
>Hello Ryan,
>
>Could do.  But for now just create subclasses for the different variants.  I
suggest just use a static class for these trivial bits.  There may even be some
real differences later.
>
>(The bigger problem is that SDrivers represent both the instance and the
factory.  Should be a small static factory class for the factory.  Just has
never got to the top of the list to fix.)
>
>It would be very good to support Deryb/JavaDB.  (I'm not clear about the
difference?).  This is long overdue.  And it should build and run Derby out of
the JDK 1.6 directly, with their jars.  Maybe even make Derby the default test
DB instead of HSQL?
>
>The normal way to handle database inconsistencies is to delegate them to some
method in the SDriver hierarchy.  Null works for MSSQL and HSQL.
>
>In SSessionJdbcHelper.flush you would have to change
>    if (value == null && fieldMeta.isPrimary() && theGenerator==null)
>    throw new SException.Error("Null primary key not set (after
createWithNullKey) " + instance);
>    fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));
>
>into something like
>
>    if (value == null && fieldMeta.isPrimary())
>       if (theGenerator==null)
>         throw new SException.Error("Null primary key not set (after
createWithNullKey) " + instance);
>      else
>        session.getDriver().setEmptyGeneratedKey() // Or maybe just do nothing
here?
>    else
>    fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));
>
>Make sure that whatever you do continues to work for at least HSQL, the default
test case.
>
>Send me any changed files and I'll diff and check them in.  You are welcome to
become a committer if you make serious contributions.
>
>There is no magic in the code.  You are not dependent on us to change things,
unlike more complex systems.
>
>Regards,
>
>Anthony
>
>
>>I'm trying to get identity generation on insert working with Derby. I have the
driver set up so it creates the table with GENERATED BY DEFAULT AS IDENTITY for
my identity field and it works fine. The problem occurs when I call
session.ses.createWithGeneratedKey and then try to commit the new record.
SimpleORM tries to call insert passing null for the identity field. Derby
accepts an insert statement that either 1) doesn't specify a value for the
identity column or 2) passes the value "default" for the identity column. Derby
throws an error when SimpleORM tries to pass null for the generated column.
>>
>>How should I handle this? If there's no way then I'll have to change SimpleORM
code to pass "default" instead of "null" but I'd prefer not to do that.
>
>
>At 01:00 PM 27/09/2009, you wrote:
>>
>>
>>Hi.
>>
>>Have you considered allowing 1 SimpleORM driver to handle multiple JDBC
drivers? Instead of using driverName and comparing the returned string to choose
a SimpleORM driver, you could pass the actual driver name into a boolean method
of an SDriver subclass and let the driver itself decide whether it's a match.
For example, I'm working with an embedded Derby database and the driver name is
"Apache Derby Embedded JDBC Driver" but the driver name for the network client
would be "Apache Derby Network Client JDBC Driver". The same SimpleORM driver
would probably work for both.
>>
>>Just curious,
>>--
>>Ryan Boder
>>1 614 598 6339
>>
>
>Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
>Just because it is possible to push twigs along the ground with ones nose
>does not necessarily mean that is the best way to collect firewood.

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Yes, you are right, more to do.

Not important, but I think that I like to see the keyword DEFAULT generated --
makes it explicit.

You could do what I suggested, but then also change SDriver.insertSql to call a
new insertValue(SFieldScalar) which by default just creates ?.  But then in
Derby driver you override this to check for generated and output DEFAULT
instead.

It is a little bit fiddly.  But either approach would be OK.

You get the general drift though.  Put hooks in the mainline code, and then
override in your driver.  Minimize per database complications in the mainline
code.

(A pain that the SQL does not standardize the most basic funtionality of key
generation!  It has lots of other escoterics, and SQL 3 for objects is bad. 
Also a pain that Sun chose Derby rather than Dafodil or maybe H2 as the Java
database.)

Regards,

Anthony

At 06:55 PM 29/09/2009, you wrote:
>
>
>Hi Anthony,
>
>I read the code and I think this will require the flush method to work a
differently for generated columns. It currently passes the NULL values for
generated columns as parameters in a prepared statement. This doesn't work with
Derby because Derby expects the keyword DEFAULT to be passed instead. The
keyword DEFAULT can not be passed as a parameter to a prepared statement. Derby
can handle the follow three syntax in a prepared statement...
>
>(*** got this from
<http://objectmix.com/apache/646246-can-derby-have-sequence-column.html>http://o\
bjectmix.com/apache/646246-can-derby-have-sequence-column.html ***)
>
>"""
>The DEFAULT <http://objectmix.com/#>keyword is part of
<http://objectmix.com/#>SQL, not ij. These SQL statements should
>all work in <http://objectmix.com/#>JDBC
>[]
><http://objectmix.com/#>
>as a PreparedStatement
>
>INSERT INTO GROUPS(ID, NAME, DESCRIPTION) VALUES (DEFAULT, ?, ?)
>
>INSERT INTO GROUPS(NAME, DESCRIPTION) VALUES (?, ?)
>
>INSERT INTO GROUPS VALUES (DEFAULT, ?, ?)
>"""
>
>So I think I'm going to have to change code a little earlier in the flush
method, from line 186 to line 236, so it doesn't handle generated columns as
parameters in the prepared statement, but rather as literals in the query
itself. For Derby the literal would be DEFAULT and for MSSQL and HSQL it would
be NULL - that decision being delegated to the driver.
>
>Another option I considered was simply not specifying a value for generated
columns as you can see in the second syntax example above, but if I'm thinking
that does not work in MSSQL. Not 100% sure though.
>
>Before I go forward, what are your thoughts? Is this the correct approach
(changing defaults to literals instead of prepared statement parameters) in your
design or do you have a better suggestion?
>
>Thanks,
>Ryan
>
>
>On Mon, Sep 28, 2009 at 3:11 AM,
<<mailto:anthony@...>anthony@...> wrote:
>Â
>
>Hello Ryan,
>
>Could do. But for now just create subclasses for the different variants. I
suggest just use a static class for these trivial bits. There may even be some
real differences later.
>
>(The bigger problem is that SDrivers represent both the instance and the
factory. Should be a small static factory class for the factory. Just has never
got to the top of the list to fix.)
>
>It would be very good to support Deryb/JavaDB. (I'm not clear about the
difference?). This is long overdue. And it should build and run Derby out of the
JDK 1.6 directly, with their jars. Maybe even make Derby the default test DB
instead of HSQL?
>
>The normal way to handle database inconsistencies is to delegate them to some
method in the SDriver hierarchy. Null works for MSSQL and HSQL.
>
>In SSessionJdbcHelper.flush you would have to change
>if (value == null && fieldMeta.isPrimary() && theGenerator==null)
>throw new SException.Error("Null primary key not set (after createWithNullKey)
" + instance);
>fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));
>
>into something like
>
>if (value == null && fieldMeta.isPrimary())
>if (theGenerator==null)
>throw new SException.Error("Null primary key not set (after createWithNullKey)
" + instance);
>else
>session.getDriver().setEmptyGeneratedKey() // Or maybe just do nothing here?
>else
>fieldMeta.writeFieldValue(ps, jx, instance.getRawArrayValue(fieldMeta));
>
>Make sure that whatever you do continues to work for at least HSQL, the default
test case.
>
>Send me any changed files and I'll diff and check them in. You are welcome to
become a committer if you make serious contributions.
>
>There is no magic in the code. You are not dependent on us to change things,
unlike more complex systems.
>
>Regards,
>
>Anthony
>
>>I'm trying to get identity generation on insert working with Derby. I have the
driver set up so it creates the table with GENERATED BY DEFAULT AS IDENTITY for
my identity field and it works fine. The problem occurs when I call
session.ses.createWithGeneratedKey and then try to commit the new record.
SimpleORM tries to call insert passing null for the identity field. Derby
accepts an insert statement that either 1) doesn't specify a value for the
identity column or 2) passes the value "default" for the identity column. Derby
throws an error when SimpleORM tries to pass null for the generated column.
>>
>>How should I handle this? If there's no way then I'll have to change SimpleORM
code to pass "default" instead of "null" but I'd prefer not to do that.
>
>
>At 01:00 PM 27/09/2009, you wrote:
>>
>>
>>Hi.
>>
>>Have you considered allowing 1 SimpleORM driver to handle multiple JDBC
drivers? Instead of using driverName and comparing the returned string to choose
a SimpleORM driver, you could pass the actual driver name into a boolean method
of an SDriver subclass and let the driver itself decide whether it's a match.
For example, I'm working with an embedded Derby database and the driver name is
"Apache Derby Embedded JDBC Driver" but the driver name for the network client
would be "Apache Derby Network Client JDBC Driver". The same SimpleORM driver
would probably work for both.
>>
>>Just curious,
>>--
>>Ryan Boder
>>1 614 598 6339
>>
>
>Dr Anthony Berglas, <mailto:anthony%40berglas.org>anthony@... Mobile:
+61 4 4838 8874
>Just because it is possible to push twigs along the ground with ones nose
>does not necessarily mean that is the best way to collect firewood.
>
>
>
>
>--
>Ryan Boder
>1 614 598 6339
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Hi,

I am making heavy use of a special UserProperty on my fields to store
the field comments.

On the one side, field comments act as a default field label in the UI
(and possibly as a key for I18N), and on the other side, they can be
used to generate a COMMENT DDL to make the database more readable.

I was wondering if the community would agree to standardize on the
following:

1) create an enum for standard field properties, with only one value for
now, being SFIELD_COMMENT. (the enum would be defined in SFieldMeta
class)

2) add two methods to SFieldMeta, setComment(String comment) and
getComment(), that would call putUserProperty(SFIELD_COMMENT, comment)
and getUserProperty(SFIELD_COMMENT);

This would make the code more readable and easier to type than the
lengthy putUserProperty form.

3) maybe change the DDL generation part to create the COMMENT ON
COLUMN ... statements (optional ?)

What do you think of it ?

Franck

Glad to see the meta data being used.  That was an original point of SimpleORM.

There are actually several "Comments".

1. PROMPT, SHORT_PROMPT: Default field name in UI, possibly long and short.

2. DESCRIPTION, or TOOL_TIP: Tool tip like comment, a line or so.

3. COMMENT?: Technical comment.

Probably the first two are the mains ones.  They get concatenated to produce the
commend DLL as
PROMPT:- DESCRIPION
(say).  Include a delimiter such as :- so that they could be reverse engineered
later.

An enum sort of defeats the purpose of user extensibility.  (Enums cannot be
extended in Java, which greatly limits their usefulnes.)  If they are well
known, then maybe they should just be properties of the object, with get/set
methods?

SHORT_PROMPT should default to PROMPT, so there is some logic.  But how does one
default DisplayLength from max length?  And know whether you have a default
value or a set value?

Long ago I had a complex properties system that Franck (fortunately) deleted.

I'm inclined to leave UserProperty for miscellaneous, and methods for known
properties.  Certainly use String as parameter to UserProperty.

To unify them, could have UserProperty $PROMPT (say) just call getPrompt(). 
Could include max size, default display length etc.  Not sure if that is useful.
Do you ever really want to enumerate all the properties?

It probably does not matter one way or the other.

Regards,

Anthony


At 07:51 PM 1/10/2009, you wrote:
>
>
>Hi,
>
>I am making heavy use of a special UserProperty on my fields to store
>the field comments.
>
>On the one side, field comments act as a default field label in the UI
>(and possibly as a key for I18N), and on the other side, they can be
>used to generate a COMMENT DDL to make the database more readable.
>
>I was wondering if the community would agree to standardize on the
>following:
>
>1) create an enum for standard field properties, with only one value for
>now, being SFIELD_COMMENT. (the enum would be defined in SFieldMeta
>class)
>
>2) add two methods to SFieldMeta, setComment(String comment) and
>getComment(), that would call putUserProperty(SFIELD_COMMENT, comment)
>and getUserProperty(SFIELD_COMMENT);
>
>This would make the code more readable and easier to type than the
>lengthy putUserProperty form.
>
>3) maybe change the DDL generation part to create the COMMENT ON
>COLUMN ... statements (optional ?)
>
>What do you think of it ?
>
>Franck
>
>


Spreadsheet Detective,
Southern Cross Software Queensland Pty Limited
54 Gerler Street
Bardon, Queensland 4065, Australia.

Email: berglas@...
www.SpreadsheetDetective.com
Ph: +61 427 830248 (Australian Eastern Standard Time)

"If the model seems correct only because the numbers look right,
then why build the model in the first place?"

Le jeudi 01 octobre 2009 à 20:32 +1000, berglas@...
a écrit :
>
> Glad to see the meta data being used. That was an original point of
> SimpleORM.
>
> There are actually several "Comments".
>
> 1. PROMPT, SHORT_PROMPT: Default field name in UI, possibly long and
> short.
>
> 2. DESCRIPTION, or TOOL_TIP: Tool tip like comment, a line or so.
>
> 3. COMMENT?: Technical comment.
>
> Probably the first two are the mains ones. They get concatenated to
> produce the commend DLL as
> PROMPT:- DESCRIPION
> (say). Include a delimiter such as :- so that they could be reverse
> engineered later.
>
> An enum sort of defeats the purpose of user extensibility. (Enums
> cannot be extended in Java, which greatly limits their usefulnes.) If
> they are well known, then maybe they should just be properties of the
> object, with get/set methods?

In fact userProperties is a Map<Object, Object>, so the key can really
be anything we want.
I was suggesting using an enum instead of strings for "standard"
properties to avoid conflicting with user keys,
eg. putUserProperty(SFieldProperty.PROMPT, "my prompt") won't conflict
with putUserProperty("PROMPT", "user did put a prompt here");

Of course, the does not mean that we would enforce using an enum as a
key, key would remain Object, and users can still use Strings, or their
own enums if they want, or anything fits their needs.

>
> SHORT_PROMPT should default to PROMPT, so there is some logic. But how
> does one default DisplayLength from max length? And know whether you
> have a default value or a set value?
>
I think we should keep it simple, ie known properties with shortcut
getters/setters.
Logic should belong to the UI layer (think also about I18N, maybe
getting translations or per-user customizations from a database, etc.)

> Long ago I had a complex properties system that Franck (fortunately)
> deleted.
>
> I'm inclined to leave UserProperty for miscellaneous, and methods for
> known properties.

If we are sure that keys won't enter in conflict (we are), I think using
userProperties even for known "standard" properties is cleaner.

> Certainly use String as parameter to UserProperty.

As I explained, I would stick with Object as a key.

>
> To unify them, could have UserProperty $PROMPT (say) just call
> getPrompt(). Could include max size, default display length etc. Not
> sure if that is useful. Do you ever really want to enumerate all the
> properties?
>
Only predefined ones.


So, what I propose is :

1) keep keys in userProperties as Object
2) use a dedicated enum to give the keys of known standard properties,
and avoid any collision with custom user properties.
3) create shortcut method that put/get from the userPropertie map
4) begin with two standard properties : PROMPT and TOOLTIP and
setPrompt, setTooltip methods.

Does this sound ok ?

Franck

>> An enum sort of defeats the purpose of user extensibility. (Enums
>> cannot be extended in Java, which greatly limits their usefulnes.) If
>> they are well known, then maybe they should just be properties of the
>> object, with get/set methods?
>
>In fact userProperties is a Map<Object, Object>, so the key can really
>be anything we want.
>I was suggesting using an enum instead of strings for "standard"
>properties to avoid conflicting with user keys,
>eg. putUserProperty(SFieldProperty.PROMPT, "my prompt") won't conflict
>with putUserProperty("PROMPT", "user did put a prompt here");

Hmm, I think you just made a good argument for not using the Enum!

Maybe an Enum.toString().  Maybe the Map should be <String, Object>?  Or maybe a
Property List?

I suppose that different aps could use different enums, but the potential for
confusion is high.

>So, what I propose is :
>
>1) keep keys in userProperties as Object
>2) use a dedicated enum to give the keys of known standard properties,
>and avoid any collision with custom user properties.
>3) create shortcut method that put/get from the userPropertie map
>4) begin with two standard properties : PROMPT and TOOLTIP and
>setPrompt, setTooltip methods.
>
>Does this sound ok ?

I think use the userProperties if you prefer, although there is a bit of a clash
with the use of fields elsewhere.  But as you say, things like defaults can be
dealt with in UI layer.

I think having different enums plus strings as keys is very odd and potentially
very nasty.  Use "$PROMPT" or "SORM:PROMPT" if you are concerned about
namespaces.

I think that I would like userProperties key changed to String now that you
bring it up.

By all means introduce an enum of Common properties.  It will help if different
aps use Simpleorm.   But mark it as just a list of strings.

Having getters and setters for userProperties is a bit odd.  If they are
standard, why not make them simple fields?  But could be handy.  Maybe get(enum)
and set(enum)?

(It could later go the other way -- expose simpleOrm fields through the property
interface as special cases.  Not actually stored as properties, just overload
get() to look up the field value.)

isDescriptive() is definitely something that should be treated like PROMPT.  The
idea was that for simple list and crud forms you could default the List fields
directly.  One could then build a generic app that worked directly off the meta
data.

It would be good to make a list of common properties.  Display size could be
one.  URL for help?  Don't go crazy, just the ones you might actually use. 
Maybe there will be too many for simple fields.

We do need to get JavaDB or possibly Daffodil up and running.  I prefer the
latter, but the former has become standardized.

Anthony


>Franck
>
>


Spreadsheet Detective,
Southern Cross Software Queensland Pty Limited
54 Gerler Street
Bardon, Queensland 4065, Australia.

Email: berglas@...
www.SpreadsheetDetective.com
Ph: +61 427 830248 (Australian Eastern Standard Time)

"If the model seems correct only because the numbers look right,
then why build the model in the first place?"

Hello Ryan,

here, we use Simpleorm in production environments on several client
installations with both Postgresql and Oracle.

No especially embedded nor lightweight but works...

Also, tests are run by default on hsql in Simpleorm (but I also run the
tests against Postgresql and Oracle regularly). I never used hsql in a
production environment, so I am not aware of the problems you mention...


Franck

Le jeudi 01 octobre 2009 à 14:50 -0400, Ryan Boder a écrit :
>
> Hi Anthony,
>
> I'm working on a production level project on a deadline, not
> experimental. It occurred to me that I can't afford to deal with Derby
> not being currently supported since other problems may appear later.
>
> I picked Derby because of JavaDB and because I read that HSQLDB could
> leave you in an inconsistent state if the system crashes. Has H2
> solved that problem? What DB do you primarily develop SimpleORM with?
> If I want to use SimpleORM for robust, easy to use Java persistence
> with no modifications required, what DB should I use? (preferably a
> Java embedded DB)
>
> Thanks,
> Ryan
>
>
>
>
> On Tue, Sep 29, 2009 at 6:12 AM, <anthony@...> wrote:
>
>         Yes, you are right, more to do.
>
>         Not important, but I think that I like to see the keyword
>         DEFAULT generated -- makes it explicit.
>
>         You could do what I suggested, but then also change
>         SDriver.insertSql to call a new insertValue(SFieldScalar)
>         which by default just creates ?. But then in Derby driver you
>         override this to check for generated and output DEFAULT
>         instead.
>
>         It is a little bit fiddly. But either approach would be OK.
>
>         You get the general drift though. Put hooks in the mainline
>         code, and then override in your driver. Minimize per database
>         complications in the mainline code.
>
>         (A pain that the SQL does not standardize the most basic
>         funtionality of key generation! It has lots of other
>         escoterics, and SQL 3 for objects is bad. Also a pain that Sun
>         chose Derby rather than Dafodil or maybe H2 as the Java
>         database.)
>
>         Regards,
>
>         Anthony
>
>
>
>         At 06:55 PM 29/09/2009, you wrote:
>         >
>         >
>         >Hi Anthony,
>         >
>         >I read the code and I think this will require the flush
>         method to work a differently for generated columns. It
>         currently passes the NULL values for generated columns as
>         parameters in a prepared statement. This doesn't work with
>         Derby because Derby expects the keyword DEFAULT to be passed
>         instead. The keyword DEFAULT can not be passed as a parameter
>         to a prepared statement. Derby can handle the follow three
>         syntax in a prepared statement...
>         >
>
>         >(*** got this from
>        
<http://objectmix.com/apache/646246-can-derby-have-sequence-column.html>http://o\
bjectmix.com/apache/646246-can-derby-have-sequence-column.html ***)
>         >
>         >"""
>         >The DEFAULT <http://objectmix.com/#>keyword is part of
>         <http://objectmix.com/#>SQL, not ij. These SQL statements
>         should
>         >all work in <http://objectmix.com/#>JDBC
>         >[]
>         ><http://objectmix.com/#>
>
>         >as a PreparedStatement
>         >
>         >INSERT INTO GROUPS(ID, NAME, DESCRIPTION) VALUES
>         (DEFAULT, ?, ?)
>         >
>         >INSERT INTO GROUPS(NAME, DESCRIPTION) VALUES (?, ?)
>         >
>         >INSERT INTO GROUPS VALUES (DEFAULT, ?, ?)
>         >"""
>         >
>         >So I think I'm going to have to change code a little earlier
>         in the flush method, from line 186 to line 236, so it doesn't
>         handle generated columns as parameters in the prepared
>         statement, but rather as literals in the query itself. For
>         Derby the literal would be DEFAULT and for MSSQL and HSQL it
>         would be NULL - that decision being delegated to the driver.
>         >
>         >Another option I considered was simply not specifying a value
>         for generated columns as you can see in the second syntax
>         example above, but if I'm thinking that does not work in
>         MSSQL. Not 100% sure though.
>         >
>         >Before I go forward, what are your thoughts? Is this the
>         correct approach (changing defaults to literals instead of
>         prepared statement parameters) in your design or do you have a
>         better suggestion?
>         >
>         >Thanks,
>         >Ryan
>         >
>         >
>
>         >On Mon, Sep 28, 2009 at 3:11 AM,
>         <<mailto:anthony@...>anthony@...> wrote:
>         >Â
>
>         >
>         >Hello Ryan,
>         >
>         >Could do. But for now just create subclasses for the
>         different variants. I suggest just use a static class for
>         these trivial bits. There may even be some real differences
>         later.
>         >
>         >(The bigger problem is that SDrivers represent both the
>         instance and the factory. Should be a small static factory
>         class for the factory. Just has never got to the top of the
>         list to fix.)
>         >
>         >It would be very good to support Deryb/JavaDB. (I'm not clear
>         about the difference?). This is long overdue. And it should
>         build and run Derby out of the JDK 1.6 directly, with their
>         jars. Maybe even make Derby the default test DB instead of
>         HSQL?
>         >
>         >The normal way to handle database inconsistencies is to
>         delegate them to some method in the SDriver hierarchy. Null
>         works for MSSQL and HSQL.
>         >
>         >In SSessionJdbcHelper.flush you would have to change
>         >if (value == null && fieldMeta.isPrimary() &&
>         theGenerator==null)
>         >throw new SException.Error("Null primary key not set (after
>         createWithNullKey) " + instance);
>         >fieldMeta.writeFieldValue(ps, jx,
>         instance.getRawArrayValue(fieldMeta));
>         >
>         >into something like
>         >
>         >if (value == null && fieldMeta.isPrimary())
>         >if (theGenerator==null)
>         >throw new SException.Error("Null primary key not set (after
>         createWithNullKey) " + instance);
>         >else
>         >session.getDriver().setEmptyGeneratedKey() // Or maybe just
>         do nothing here?
>         >else
>         >fieldMeta.writeFieldValue(ps, jx,
>         instance.getRawArrayValue(fieldMeta));
>         >
>         >Make sure that whatever you do continues to work for at least
>         HSQL, the default test case.
>         >
>         >Send me any changed files and I'll diff and check them in.
>         You are welcome to become a committer if you make serious
>         contributions.
>         >
>         >There is no magic in the code. You are not dependent on us to
>         change things, unlike more complex systems.
>         >
>         >Regards,
>         >
>         >Anthony
>         >
>         >>I'm trying to get identity generation on insert working with
>         Derby. I have the driver set up so it creates the table with
>         GENERATED BY DEFAULT AS IDENTITY for my identity field and it
>         works fine. The problem occurs when I call
>         session.ses.createWithGeneratedKey and then try to commit the
>         new record. SimpleORM tries to call insert passing null for
>         the identity field. Derby accepts an insert statement that
>         either 1) doesn't specify a value for the identity column or
>         2) passes the value "default" for the identity column. Derby
>         throws an error when SimpleORM tries to pass null for the
>         generated column.
>         >>
>         >>How should I handle this? If there's no way then I'll have
>         to change SimpleORM code to pass "default" instead of "null"
>         but I'd prefer not to do that.
>         >
>         >
>         >At 01:00 PM 27/09/2009, you wrote:
>         >>
>         >>
>         >>Hi.
>         >>
>         >>Have you considered allowing 1 SimpleORM driver to handle
>         multiple JDBC drivers? Instead of using driverName and
>         comparing the returned string to choose a SimpleORM driver,
>         you could pass the actual driver name into a boolean method of
>         an SDriver subclass and let the driver itself decide whether
>         it's a match. For example, I'm working with an embedded Derby
>         database and the driver name is "Apache Derby Embedded JDBC
>         Driver" but the driver name for the network client would be
>         "Apache Derby Network Client JDBC Driver". The same SimpleORM
>         driver would probably work for both.
>         >>
>         >>Just curious,
>         >>--
>         >>Ryan Boder
>         >>1 614 598 6339
>         >>
>         >
>
>         >Dr Anthony Berglas, <mailto:anthony%
>         40berglas.org>anthony@... Mobile: +61 4 4838 8874
>
>         >Just because it is possible to push twigs along the ground
>         with ones nose
>         >does not necessarily mean that is the best way to collect
>         firewood.
>         >
>         >
>         >
>         >
>         >--
>         >Ryan Boder
>         >1 614 598 6339
>         >
>
>         Dr Anthony Berglas, anthony@... Mobile: +61 4 4838
>         8874
>         Just because it is possible to push twigs along the ground
>         with ones nose
>         does not necessarily mean that is the best way to collect
>         firewood.
>
>
>
>
>
> --
> Ryan Boder
> 1 614 598 6339
>
>
>
>

Hello All,

I have added support for JavaDB -- long overdue.

The Insert Identity code now works.

One outstanding issue is how to convert BYTES to JavaDB.  Advice by someone more
familiar with it would be good.

(I suspect that Daffodil / One$db is a better Java database.  But Derby has the
IBM/Sun stamp.  Certainly both should be good for real applications, for which
HSQL is a bit scary.  H2 is also not to be ignored.  I would love to see a good
comparison of the Java databases.  I suspect that Sun did not make one when they
chose Derby.)

Regards,

Anthony

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Hi Anthony,

>
> Hmm, I think you just made a good argument for not using the Enum!
>
Well, I don't get you here...
You seem to assume everybody expects that keys in Maps are String, and
that no one will notice that SFieldProperty.PROMPT and "PROMPT" are not
the same objects !

I think on the contrary that using an enum object as the key for the
library's own "user" properties makes it perfectly clear that they are
internal to the library and will ensure that they won't conflict with
the user namespaces (be they "clever" Strings or any other object).
Moreover, users will use the setPrompt / getPrompt shortcuts, so they
will probably never notice what the key is.

> I suppose that different aps could use different enums, but the
> potential for confusion is high.
>
On the contrary, enums are the perfect namespaces for Maps. Using
Strings with prefixes to try to mimic what enums do perfectly seems odd
to me...

What do you think I'm missing ?

Regards,
Franck

Hello Franck

I am simply concerned that when you do toString on an Enum you do not see the
type.  Just the name.  Leading to horrible bugs.  Why is this PROMPT different
to that PROMPT?  I can see PROMPT in the table, but get("PROMPT") returns null? 
etc.

I am also wondering if we really need getPrompt() instead of get(Prompt).  The
latter is very Simplormish.

How about
1.  Declare and use the enum.
2.  For in builts give a special name, eg. "SPrompt" or S_PROMPT

Maybe change getUserProperties to getProperty(Enum property).  Would make it
clearer.  But would be a pain for extendable enums.

Maybe make Descriptive a property later.

We can then argue about whether the key should be String or Object.  Won't
really matter if "SPrompt" is used, so that it is clear.  Maybe make the actual
Enum SPrompt, so that it can be statically imported.

Put a fat comment on getPropety to say that enums are being used as name spaces.
Can one just have a type Enum?

Strings are simple.

Do what you think is best.

Anthony


At 08:08 PM 5/10/2009, you wrote:
>
>
>Hi Anthony,
>
>>
>> Hmm, I think you just made a good argument for not using the Enum!
>>
>Well, I don't get you here...
>You seem to assume everybody expects that keys in Maps are String, and
>that no one will notice that SFieldProperty.PROMPT and "PROMPT" are not
>the same objects !
>
>I think on the contrary that using an enum object as the key for the
>library's own "user" properties makes it perfectly clear that they are
>internal to the library and will ensure that they won't conflict with
>the user namespaces (be they "clever" Strings or any other object).
>Moreover, users will use the setPrompt / getPrompt shortcuts, so they
>will probably never notice what the key is.
>
>> I suppose that different aps could use different enums, but the
>> potential for confusion is high.
>>
>On the contrary, enums are the perfect namespaces for Maps. Using
>Strings with prefixes to try to mimic what enums do perfectly seems odd
>to me...
>
>What do you think I'm missing ?
>
>Regards,
>Franck
>
>


Spreadsheet Detective,
Southern Cross Software Queensland Pty Limited
54 Gerler Street
Bardon, Queensland 4065, Australia.

Email: berglas@...
www.SpreadsheetDetective.com
Ph: +61 427 830248 (Australian Eastern Standard Time)

"If the model seems correct only because the numbers look right,
then why build the model in the first place?"

I tried to post the message below on Oct 30 but I sent it from the wrong
address, so it didn't get through.

Since then, we haven't had any ideas on how to replace findUsingPrototype, and
would welcome any thoughts on it.  (It's deprecated, so there must be some
thoughts to be shared...)

However, we have made some progress on understanding the threading.  It seems
the design is to have one session per thread, and one (or zero) session per
dataset, and one current dataset per session.  Is that correct?

If that is the design, then we'd like to have two threads that have their own
sessions, that create datasets containing batches of records.  The batches of
records (and their associated datasets) would be disassociated from the session,
then put in a queue for the main thread to process, and ultimately destroy.

We don't want to create sessions for each batch, just datasets for each batch. 
So we'd need to create a new dataset for the session after the current dataset
has been disassociated from the session (haven't figured out how to do this
yet.)

We'd appreciate any comments or suggestions on how to replace findUsingPrototype
(it's too slow) and on how to best use SimpleORM in a multithreaded application.

--
John Abraham
jabraham@...

*************
** Original message (that didn't make it to the list on Friday Oct 30) **

Hello there.  We have been working at speeding up our SimpleORM project which
processes 65 million records in sequence, in about 4000 batches.  We've profiled
the code (using hprof) and would like to do two things now to increase
performance:

1) we'd like to spin off two threads to pre-fetch batches of records (average
batch size 65,000,000/4000 = 16,250) and put the resulting ArrayLists of
SRecordInstances in a FIFO queue. That way we can process previous batches of
records off the head of the queue on a separate processor while the database is
engaged fetching the next batches for the tail of the queue. The Java program
spends about 30% of its time waiting for the database to return the next batch
and even more time (about 8%) for SimpleORM to create the resulting ArrayList;
we might as well ask for a few batches in advance.  Question:  is SimpleORM
thread safe for this type of operation?

2) findUsingPrototype is quite slow for repeated finds of prefetched records: 
SDataSet.findOrCreate() has to create the prototype object, then
SDataSet.finder() has to populate its fields, then there is the processing of
SRecordInstance.equals() and SRecordInstance.hashCode() to find the record in
the cache.  All of these together are, I think, about 30% of our runtime.  Is
there a way to speed this up?  I notice findUsingPrototype() is deprecated,
suggesting that someone already has a new plan for how to do this.  For some of
our tables we know the underlying data is static, so our current plan is to
build our own faster hashmaps for certain types of records (especially those
with a single integer primary key), and check our own cache first before
checking SimpleORM's cache.  But I'd rather speed up SimpleORM than work around
it.

So those are my two questions: 1) Threadsafe?  2) are there already ideas or
bits of code to replace findUsingPrototype() with something faster?

This is just a note to let you know that this multi-threaded design did work. 
All indications are that this *should* be much faster, but for some reason it
isn't.  We're blaming SQL Server, Microsoft's JDBC driver, and/or the database
configuration, because the code is now spending most of it's time in
socketRead() waiting for the query result.  Investigations are continuing.  With
PostgreSQL (instead of SQL Server) it is faster.

I had some other thoughts about alternative designs.  What would be really cool
is if the query methods in SimpleORM could immediately return an iterator to a
List object that is actually populated by another thread.  The getNext() method
in the iterator would block if the other thread hadn't pulled the data from the
database yet. But I think the whole design of one-dataset-per-session and
one-session-per-dataset wouldn't permit this?  Also we don't want to make
SimpleORM into NotSimpleORM.

We had to add a method to SSessionJdbc which attaches a new dataset to a
session, to replace the one that is detached and handed over to the processing
thread.

--
John Abraham


--- In SimpleORM@yahoogroups.com, jabraham@... wrote:
>
> I tried to post the message below on Oct 30 but I sent it from the wrong
address, so it didn't get through.
>
> Since then, we haven't had any ideas on how to replace findUsingPrototype, and
would welcome any thoughts on it.  (It's deprecated, so there must be some
thoughts to be shared...)
>
> However, we have made some progress on understanding the threading.  It seems
the design is to have one session per thread, and one (or zero) session per
dataset, and one current dataset per session.  Is that correct?
>
> If that is the design, then we'd like to have two threads that have their own
sessions, that create datasets containing batches of records.  The batches of
records (and their associated datasets) would be disassociated from the session,
then put in a queue for the main thread to process, and ultimately destroy.
>
> We don't want to create sessions for each batch, just datasets for each batch.
So we'd need to create a new dataset for the session after the current dataset
has been disassociated from the session (haven't figured out how to do this
yet.)
>
> We'd appreciate any comments or suggestions on how to replace
findUsingPrototype (it's too slow) and on how to best use SimpleORM in a
multithreaded application.
>
> --
> John Abraham
> jabraham@...
>
> *************
> ** Original message (that didn't make it to the list on Friday Oct 30) **
>
> Hello there.  We have been working at speeding up our SimpleORM project which
processes 65 million records in sequence, in about 4000 batches.  We've profiled
the code (using hprof) and would like to do two things now to increase
performance:
>
> 1) we'd like to spin off two threads to pre-fetch batches of records (average
batch size 65,000,000/4000 = 16,250) and put the resulting ArrayLists of
SRecordInstances in a FIFO queue. That way we can process previous batches of
records off the head of the queue on a separate processor while the database is
engaged fetching the next batches for the tail of the queue. The Java program
spends about 30% of its time waiting for the database to return the next batch
and even more time (about 8%) for SimpleORM to create the resulting ArrayList;
we might as well ask for a few batches in advance.  Question:  is SimpleORM
thread safe for this type of operation?
>
> 2) findUsingPrototype is quite slow for repeated finds of prefetched records: 
SDataSet.findOrCreate() has to create the prototype object, then
SDataSet.finder() has to populate its fields, then there is the processing of
SRecordInstance.equals() and SRecordInstance.hashCode() to find the record in
the cache.  All of these together are, I think, about 30% of our runtime.  Is
there a way to speed this up?  I notice findUsingPrototype() is deprecated,
suggesting that someone already has a new plan for how to do this.  For some of
our tables we know the underlying data is static, so our current plan is to
build our own faster hashmaps for certain types of records (especially those
with a single integer primary key), and check our own cache first before
checking SimpleORM's cache.  But I'd rather speed up SimpleORM than work around
it.
>
> So those are my two questions: 1) Threadsafe?  2) are there already ideas or
bits of code to replace findUsingPrototype() with something faster?
>

Hello John,

THREADING

Your threading approach is OK.

In normal operation, you would have one session per thread, one thread per
session.  This can be overridden in special cases, but normally there are sanity
checks to ensure that it is the case.  Sharing a JDBC connection between threads
is certainly asking for deep trouble.

Queing the SDataSets is a reasonable approach.   There are examples in the
LongTransactionTest as to how to detach and reattach SDataSets.

An alternative would be to simply have two threads, each of which does the
reading and the processing, and then synchronize to ensure that only one ever
does critical processing at a time.  Avoids need for a queue.

But generally, I would not introduce complex architectures for maximum gain of
only 30%.

More importantly, can the processing itself be multi threaded?  You have not
indicated the application, so hard for me to know.  But a quad core machine
might increase throughput by 400%, much better than 30% improvement.

One way to go much faster is to use a Java embeded database.  HSQL, for example
can be well over 1000% faster than Oracle.  Daffodil, H2 and JavaDB are probably
more reliable, have not tested for speed.  But again, if the database is only
consuming 30% of your time, then maybe not worth it.

And of course Stored Procedures will be much faster where applicable.

And of course, make sure you are using the "server" JVM to compile properly.

FIND USING PROTOTYPE

findUsingPrototype is deprecated because we do not want people using it
directly, but it is public because it provides a back door from SSessionJdbc to
avoid creating an object.  The main method is find(meta, keys...).

But we have not found it to be slow -- the overhead of a couple of temporary
objects is negligible.  Are you sure that it really is a significant bottle neck
for you?

We could make a special case of the common case of a single primary key.    But
I prefer to avoid special cases unless a significant performance benefit could
be demonstrated.  (The changes would mainly be in SDataSet.  Eg. in find() if
(keys.count() == 1) records.get(key);  Bit fiddly because HashTable does
key.equals(k) instead of k.equals(key), so the single key case would be quite
different indeed and need to be implemented consistently.)

(One thing that is slow is the standard Java implementation of HashMap.  It
creates an Entry object for each entry instead of using odd and even slots in
the array.  In our case the key and the value are both the same object.  If you
felt like writing a new HashMap implementation that could help a bit.  But
unfortunately it cannot implement Map, which has the Entry object implicitly
built into it (very bad design of a core element).)

Hope this helps,

Anthony

At 02:56 AM 4/11/2009, you wrote:
>
>
>I tried to post the message below on Oct 30 but I sent it from the wrong
address, so it didn't get through.
>
>Since then, we haven't had any ideas on how to replace findUsingPrototype, and
would welcome any thoughts on it. (It's deprecated, so there must be some
thoughts to be shared...)
>
>However, we have made some progress on understanding the threading. It seems
the design is to have one session per thread, and one (or zero) session per
dataset, and one current dataset per session. Is that correct?
>
>If that is the design, then we'd like to have two threads that have their own
sessions, that create datasets containing batches of records. The batches of
records (and their associated datasets) would be disassociated from the session,
then put in a queue for the main thread to process, and ultimately destroy.
>
>We don't want to create sessions for each batch, just datasets for each batch.
So we'd need to create a new dataset for the session after the current dataset
has been disassociated from the session (haven't figured out how to do this
yet.)
>
>We'd appreciate any comments or suggestions on how to replace
findUsingPrototype (it's too slow) and on how to best use SimpleORM in a
multithreaded application.
>
>--
>John Abraham
><mailto:jabraham%40ucalgary.ca>jabraham@...
>
>*************
>** Original message (that didn't make it to the list on Friday Oct 30) **
>
>Hello there. We have been working at speeding up our SimpleORM project which
processes 65 million records in sequence, in about 4000 batches. We've profiled
the code (using hprof) and would like to do two things now to increase
performance:
>
>1) we'd like to spin off two threads to pre-fetch batches of records (average
batch size 65,000,000/4000 = 16,250) and put the resulting ArrayLists of
SRecordInstances in a FIFO queue. That way we can process previous batches of
records off the head of the queue on a separate processor while the database is
engaged fetching the next batches for the tail of the queue. The Java program
spends about 30% of its time waiting for the database to return the next batch
and even more time (about 8%) for SimpleORM to create the resulting ArrayList;
we might as well ask for a few batches in advance. Question: is SimpleORM thread
safe for this type of operation?
>
>2) findUsingPrototype is quite slow for repeated finds of prefetched records:
SDataSet.findOrCreate() has to create the prototype object, then
SDataSet.finder() has to populate its fields, then there is the processing of
SRecordInstance.equals() and SRecordInstance.hashCode() to find the record in
the cache. All of these together are, I think, about 30% of our runtime. Is
there a way to speed this up? I notice findUsingPrototype() is deprecated,
suggesting that someone already has a new plan for how to do this. For some of
our tables we know the underlying data is static, so our current plan is to
build our own faster hashmaps for certain types of records (especially those
with a single integer primary key), and check our own cache first before
checking SimpleORM's cache. But I'd rather speed up SimpleORM than work around
it.
>
>So those are my two questions: 1) Threadsafe? 2) are there already ideas or
bits of code to replace findUsingPrototype() with something faster?
>
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

Hello all.
I have been using SimpleORM for a few months now and it has been a
great help especially for backend data migration scripts. I shall be
writing my first full app in about 2 months time. I have been reading
a lot on thread safety and I wanted to know whether the
getThreadLocalSession method is enough to ensure thread safety when
accessing the db or whether I need to have my own implementation to
ensure that if 30 people or so are connected actions such as inserting
data to the db are safe and data integrity is maintained. Kindly
advise. Kind regards. Abel

SimpleORM checks for common threading errors such as trying to use the same
*Connection* from two threads.

It is quite OK and normal to have multiple threads in multiple JVMs accessing
the same database, with each thread having its own connection.

But keeping accesses from different connections from interfering is the job of
the database.  In normal usage, with normal locking, you should be OK.  And
SimpleORM double checks all updates with optimistic locks.  But the details can
be very complex.  The white paper has a section.

Anthony

At 01:31 PM 7/11/2009, you wrote:
>
>
>Hello all.
>I have been using SimpleORM for a few months now and it has been a
>great help especially for backend data migration scripts. I shall be
>writing my first full app in about 2 months time. I have been reading
>a lot on thread safety and I wanted to know whether the
>getThreadLocalSession method is enough to ensure thread safety when
>accessing the db or whether I need to have my own implementation to
>ensure that if 30 people or so are connected actions such as inserting
>data to the db are safe and data integrity is maintained. Kindly
>advise. Kind regards. Abel
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

You can get it from the source forge repository.

However, as the author I do not really recommend it, as it heavily uses JSPs and
JSPs are just awful technology.  It evolved, I was pressured, ...

A better approach is SimpleServlets.  Can be used in two modes, plain procedures
or an object model.  Not very developed, but a sound design based on experience,
and very easy to work with an extend as necessary.  No XML, of course.

svn co
http://simpleorm.svn.sourceforge.net/svnroot/simpleorm/trunk/simpleservlets

NOTE that I am going to have to rename this, as irritatingly someone else has
since created a (very weak) project called simpleservlets and registered the
domain name.

Please send future requests to the mailing list.

Anthony

At 04:25 AM 8/11/2009, you wrote:
>I'm interested in SimpleWebApp.  Is it available for download?
>
>Mark
>
>Mark Anderson                           voice:   703-983-6508
>Netwk Sys&Dist Sys Eng, Sr         fax:   703-983-7978
>The MITRE Corporation               email:  
<mailto:anderson@...>anderson@...
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.

If you have non-trivial queries for MSSQL try

UPDATE STATISTICS

on the main tables.  I have found that sometimes it just looses track and starts
running slowly for no apparent reason.

Anthony

At 01:18 PM 7/11/2009, you wrote:
>
>
>Thanks for the feedback!  Right now the database is responsible for basically
100% of the run time because the main thread (which is actually querying the
dataset, not the database) is always faster than the database query thread.  So
there's no point speeding up findUsingPrototype (used in the main thread)
anymore until we get the database query thread working faster.
>
>But that could just be because something odd is happening over in SQL Server. 
For now, we're able to blame Microsoft for our problems.
>
>Maybe we'll try HSQLDB.  I wasn't aware that it did hybrid disk/memory
operations (our 65 million records will NOT fit into memory all at once), and I
wasn't aware that it could process .CSV files directly.  Those two features
together with a potential speed advantage make it look very promising.  Our
users like .CSV files for the smaller tables.
>
>--
>John
>
>
>On 6-Nov-09, at 7:19 PM, <mailto:anthony@...>anthony@... wrote:
>
>>Hello John,
>>
>>THREADING
>>
>>Your threading approach is OK.
>>
>>In normal operation, you would have one session per thread, one thread per
session. This can be overridden in special cases, but normally there are sanity
checks to ensure that it is the case. Sharing a JDBC connection between threads
is certainly asking for deep trouble.
>>
>>Queing the SDataSets is a reasonable approach. There are examples in the
LongTransactionTest as to how to detach and reattach SDataSets.
>>
>>An alternative would be to simply have two threads, each of which does the
reading and the processing, and then synchronize to ensure that only one ever
does critical processing at a time. Avoids need for a queue.
>>
>>But generally, I would not introduce complex architectures for maximum gain of
only 30%.
>>
>>More importantly, can the processing itself be multi threaded? You have not
indicated the application, so hard for me to know. But a quad core machine might
increase throughput by 400%, much better than 30% improvement.
>>
>>One way to go much faster is to use a Java embeded database. HSQL, for example
can be well over 1000% faster than Oracle. Daffodil, H2 and JavaDB are probably
more reliable, have not tested for speed. But again, if the database is only
consuming 30% of your time,
>>
>
>Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
>Just because it is possible to push twigs along the ground with ones nose
>does not necessarily mean that is the best way to collect firewood.

Not normally necessary.  And locking is about more than Synchronized.  Unlike
most software, SimpleORM takes care to try to fail fast if you have threading
errors.

Anthony

At 05:35 PM 9/11/2009, you wrote:
>
>
>Hello Anthony,
>Thanks for the reply and I have to say I do understand exactly what you mean.
What I am really concerned with is whether I may have to write synchronised
methods to restrict how resources are accessed. Is this really necessary while
using this package.
>
>On Sun, Nov 8, 2009 at 4:14 AM,
<<mailto:anthony@...>anthony@...> wrote:
>
>
>SimpleORM checks for common threading errors such as trying to use the same
*Connection* from two threads.
>
>It is quite OK and normal to have multiple threads in multiple JVMs accessing
the same database, with each thread having its own connection.
>
>But keeping accesses from different connections from interfering is the job of
the database. In normal usage, with normal locking, you should be OK. And
SimpleORM double checks all updates with optimistic locks. But the details can
be very complex. The white paper has a section.
>
>Anthony
>
>
>At 01:31 PM 7/11/2009, you wrote:
>>
>>
>>Hello all.
>>I have been using SimpleORM for a few months now and it has been a
>>great help especially for backend data migration scripts. I shall be
>>writing my first full app in about 2 months time. I have been reading
>>a lot on thread safety and I wanted to know whether the
>>getThreadLocalSession method is enough to ensure thread safety when
>>accessing the db or whether I need to have my own implementation to
>>ensure that if 30 people or so are connected actions such as inserting
>>data to the db are safe and data integrity is maintained. Kindly
>>advise. Kind regards. Abel
>>
>
>Dr Anthony Berglas, <mailto:anthony%40berglas.org>anthony@... Mobile:
+61 4 4838 8874
>Just because it is possible to push twigs along the ground with ones nose
>does not necessarily mean that is the best way to collect firewood.
>
>
>

Dr Anthony Berglas, anthony@...       Mobile: +61 4 4838 8874
Just because it is possible to push twigs along the ground with ones nose
does not necessarily mean that is the best way to collect firewood.