Big restriction: Dart's Stream does not support synchronous mode

When I tried to support getter property of another referenced entity, I came to recognize that it is NOT possible in current Dart.

This is ridiculous restriction.
And it was supported until M3 deprecated the previous synchronous Stream classes.
I think this is pretty much nutty stupid decision.
It must have been done by young limited experienced people who can believe such single bullet approach can solve every thing. Probably it may solve every thing they knew.
That is the problem of the lack of experience.

Whole Dart development somehow has resemblance to early stage of Java around 1995. Probably many of developer of Dart does not remember.

Java has been introduced as a language for Applet. And marketed as such, then it was mainly used for server side. The main reason for the adaption can be contributed several reasons. one aspect is it was free, and it provided a lot of library. Compared to C/C++, these are big difference. Also from language point of view, Java supported class, object oriented features and garbage collection.(and lack of pointer). So compared to C/C++, it was significantly high level language, it was good enough for the most of server side application.

(but it was really backward step compared to other advanced moduler languages, like Mesa/Cedar, Ada, Modula-2, etc. but they never became known to wider audiences, that may have helped Java to proliferated also..)

For Dart,  the current marketing Buz is for browser side language(to replace JavaScript (eventually..)), but as a language, it has also unique feature integrating object/functional idioms in a single language.
Although there are many of such languages(Like Scala), there are no other language which can support both browser and server side.

I think  these two aspects are quite important. Although current marketing focus is in browser side, like polymer, dart:io etc, but it will be used more on server side.
Since the combination of object/functional style is very powerful, once people realized how it is productive, they will never want to write huge amount of equivalent codes in Java. Also sharing the same class from both sides is big plus.

Bottom line is  Dart should target such server side development in much higher priority. Right now, many people seems not developing significant application, they will not complain, but eventually, more people start using Dart, this kind of biased API design will bring a lot of complaints.

ORMapping in Dart (Part2)

I started the implementation of Object Relational Mapping tool like JPA in Dart.
The approach follows the outline I posted before.
But there were several differences.

One is basically I decided to go with minimalist approach.
Always it creates more problem in the end if we rely on many outer resources.
In this case, I just went with my own Postgres scheme generation, sql generation etc.
In fact, it is not so complicated.
And for this kind of tool, we don't need much feature of SQL.

The reason I chose Postgres was the Dart lib for Postgres had BSD license.
MySQL driver in Dart was GPL. But Postgres may be better choice than MySQL technically.

Right now, it can generate DB scheme from annotated class definition, and it support CRUD*create/read(query) /update/delete operations.
persistence object has several level of state.

remaining things are:

1.  how to support reference and relation. These are actually subtle issue inherent for many ORM system.
   Since if we fetch all associated objects, it may need to copy whole db content.
   So it is important to minimize fetching data.
   
2.  how to support relationship. Even though we define relationship as List or reference, but, in particular, the List representation are sort of useless.
   Conceptually, relationship don't have to be represented as entity properties.
   In JPA, it also supports separate query, so it is better not to use List attribute.

 For reference, if we uses strict evaluation, it may end up fetching all data, so we must use lazy evaluation for fetching reference.
There is  some interesting issue for this, since most of postgres API is using Future, how can we avoid asynchronous operation for getting referenced object?
We may need to introduce some operator to change Future to synchronous call by  monitoring the asynchronous call was finished or not.

Also in order to support lazy evaluation, some fieldstate object associated to the filed may need to keep entity id in it, and if the filed is accessed,  it need to fetch from db and assign to the actual object field.
These are a bit complicated, but relatively straightforward.

But the relationship, List field is quite bad.
Essentially if there are large number of associated entities, such whole set is useless. If it cannot be used in general situation, it is better not to use it.

Actually, I created this binding scheme, so I can also change the type to represent relationship.
So instead of List<B>, I may use Rel<A, B>, Ref<A>. and Rel<A, B> class will provide query methods. and it will not maintains fetched objects. In fact, query and managing entity objects for transactional scoped are contradicting requirement.
Often fetched object may not have to be managed(read only).
So it will be useful to provide separate fetching mode so that it will reduce managed fetched objects.

For creating one/many to many relationship, it may be better to reflect such change immediately in database, and do not keep such info in memory unless corresponding object is managed in memory(such case, it need to reflect both sides). this simplify fetching data from database consistently.

class A {
  @OneToMany(mappegBy:"a")
  Rel<A, B> bs;

}
class B {
  @ManyToOne
  Ref<A> a;
}

Following is the test program.

@Table(name: "A")
class A extends Persistence {
    @Id()
    @Basic(optional: false)
    @Column(name: "id")
    String _id;

    @Basic(optional: false)
    @Column(name: "i")
    int _i;
    
    @ManyToOne()
    B _b;

    @OneToMany(mappedBy: "a")
    List<C> _cs;
    
    String get id => get(const Symbol("id"));
    void set id(String v)=>set(const Symbol("id"), v);

    int get i => get(const Symbol("i"));
    void set i(int v)=>set(const Symbol("i"), v);

    B get b => get(const Symbol("b"));
    void set b(B v)=>set(const Symbol("b"), v);

    List<C> get cs => get(const Symbol("cs"));
    void set cs(List<C> v)=>set(const Symbol("cs"), v);
    
    String toString() => "A(id: ${id}, i: ${i}, b: ${b}, cs: ${cs})";
}

@Table(name: "B")
class B extends Persistence {
  @Id()
  @Basic(optional: false)
  @Column(name: "id")
  String _id;

  @Basic(optional: false)
  @Column(name: "s")
  String _s;
  
  @ManyToOne()
  A _a;

  @OneToMany(mappedBy: "a")
  List<C> _cs;
  
  String get id => get(const Symbol("id"));
  void set id(String v)=>set(const Symbol("id"), v);

  String get s => get(const Symbol("s"));
  void set s(String v)=>set(const Symbol("s"), v);

  A get a => get(const Symbol("a"));
  void set a(A v)=>set(const Symbol("a"), v);

  List<C> get cs => get(const Symbol("cs"));
  void set cs(List<C> v)=>set(const Symbol("cs"), v);
  
  String toString() => "B(id: ${id}, s: ${s})";
}

@Table(name: "C")
class C extends Persistence {
  @Id()
  @Basic(optional: false)
  @Column(name: "id")
  String _id;

  @Basic(optional: false)
  @Column(name: "i")
  int _i;
  
  @ManyToOne()
  B _b;

  @OneToMany(mappedBy: "a")
  List<C> _cs;
  
  String get id => get(const Symbol("id"));
  void set id(String v)=>set(const Symbol("id"), v);

  int get i => get(const Symbol("i"));
  void set i(int v)=>set(const Symbol("i"), v);

  B get b => get(const Symbol("b"));
  void set b(B v)=>set(const Symbol("b"), v);

  List<C> get cs => get(const Symbol("cs"));
  void set cs(List<C> v)=>set(const Symbol("cs"), v);
  
  String toString() => "C(id: ${id}, s: ${i})";
}

Future<int> test0(_) =>
    persistenceMgr.deleteTables([A, B, C]).then((int n) {
      print("deleteTables done. ${n}");
    });

Future<int> test1(_) => persistenceMgr.createTables([B, C, A]);

Future<bool> test2(_) {
  A a1 = new A()
    ..managed = true
    ..id = "a1"
    ..i = 10;
  
  A a2 = new A()
    ..managed = true
        ..id = "a2"
        ..i = 20;
 
  return persistenceMgr.commit();
}

Future<bool> test3(_) {
  IClassMirror cmirror = ClassMirrorFactory.reflectClass(A);
  Selector<A> selector = new Selector<A>(cmirror);
  return selector.query(where: "id = 'a1'")
    .then((A a){
      print("==>> a: ${a}");
      a.i = 100;
      return persistenceMgr.commit();
    })
    .then((_)=>selector.queryAll())
    .then((List<A> as){
      print("==>> as: ${as}");
      return true;
    });
}

Future test4a(_) {
  IClassMirror cmirror = ClassMirrorFactory.reflectClass(A);
  Selector<A> selector = new Selector<A>(cmirror);
  return selector.query(where: "id = 'a2'")
    .then((A a){
      print("1a==>> a: ${a}");
      if (a == null) {
        return false;
      }
      print("1b==>> a: ${a}");
      return a.delete();
    })
    .then((ok) {
      if (!ok) {
        return false;
      }
      return selector.query(where: "id = 'a1'")
      .then((A a){
        print("2==>> a: ${a}");
      });    
    });
}

Future test4b(_) {
  IClassMirror cmirror = ClassMirrorFactory.reflectClass(A);
  Selector<A> selector = new Selector<A>(cmirror);
  return selector.queryAll()
  .then((List<A> as)=>Future.forEach(as, (A a)=>a.delete()))
  .then((ok) => selector.query(where: "id = 'a1'"))
  .then((A a){
      print("2==>> a: ${a}");
      //a.delete();
   });
}

void main() {
  
  // register reflection factory
  initClassMirrorFactory();
  
  setDBAdaptor(new PostgresDBAdaptor());
  
  test1(null).then(test2).then(test3).then(test4b).then(test0).then((_) { print(">>> completed."); });
}

And this is the output.

==>createTables: CREATE TABLE B (
id varchar(256) primary key,
s varchar(256),
a varchar(256));

CREATE TABLE C (
id varchar(256) primary key,
i int,
b varchar(256));

CREATE TABLE A (
id varchar(256) primary key,
i int,
b varchar(256));

ALTER TABLE B ADD CONSTRAINT b_a_fkey FOREIGN KEY (a) REFERENCES A(id);
ALTER TABLE C ADD CONSTRAINT c_b_fkey FOREIGN KEY (b) REFERENCES B(id);
ALTER TABLE A ADD CONSTRAINT a_b_fkey FOREIGN KEY (b) REFERENCES B(id);

>> createInsert: INSERT INTO A (id, i, b) VALUES
( E'a1' , 10, null);
>> createInsert: INSERT INTO A (id, i, b) VALUES
( E'a2' , 20, null);
>> DBTable.query: SELECT id, i, b FROM A WHERE id = 'a1'
==>> a: A(id: a1, i: 10, b: null, cs: null)
>> createUpdate: UPDATE A SET id =  E'a1' , i = 10, b = null WHERE id =  E'a1' ;
>> createUpdate: UPDATE A SET id =  E'a2' , i = 20, b = null WHERE id =  E'a2' ;
>> createUpdate: UPDATE A SET id =  E'a1' , i = 100, b = null WHERE id =  E'a1' ;
>> DBTable.queryAll: SELECT id, i, b FROM A
==>> as: [A(id: a2, i: 20, b: null, cs: null), A(id: a1, i: 100, b: null, cs: null)]
>> DBTable.queryAll: SELECT id, i, b FROM A
>> createDelete: DELETE FROM A WHERE id =  E'a2' ;
>> createDelete: DELETE FROM A WHERE id =  E'a1' ;
>> DBTable.query: SELECT id, i, b FROM A WHERE id = 'a1'
2==>> a: null
==>deleteTables: ALTER TABLE A DROP CONSTRAINT a_b_fkey;
ALTER TABLE B DROP CONSTRAINT b_a_fkey;
ALTER TABLE C DROP CONSTRAINT c_b_fkey;
DROP TABLE A, B, C;

deleteTables done. null
>>> completed.

ORMapping in Dart

A persistence support using relational DB is very basic requirement for any real world applications.
Right now, there seems no such tools available in Dart.
but there are a few DB driver for SQL server.
in this situation, what we can do in simplest way would be following.

1.  delegate generating db scheme to JPA. In order  to do this, we use the same annotation as JPA in Dart class. and from this dart class, using reflection, we generate Java source code with the same JPA annotation defined in the dart class. The rest of generating scheme in db can be automated in some script.
2.  Once we had a db scheme, we can access db. one way is to use Java itself to access db and Dart will access the object as Json through web API, but this approach may make system complicated since Dart needs Java server. So if we avoid intermediate Java and can directly access DB, that would be more simple approach.Since Dart class has already annotations, so it should be possible to provide reflection based plumbing to map row data to entity properties.

 Normally we will need to start from defining abstract(interface) class for db entity(table).
then we need to provide the implementation class for this interface.
But manual implementation of such implementation class are tedious and mechanical. It should be automated. One way is just to generate such an implementation class from the interface classes. But code generation approach often create more problem for maintenance etc. So it is better to avoid this approach. Also code generation bloat the code size, and it will not good approach for running on client side.

but if we define abstract class we need to define another class implementing the methods, so in this approach, we cannot avoid code generation.
So we need to define a concrete class which has no db mapping codes.

The simple solution for this is just using inheritance.

@Entity()
@Table(name: "A")
class A extends Persistence {
    @Id()
    @Basic(optional: false)
    @Column(name: "id")
    String _id;

    @Basic(optional: false)
    @Column(name: "i")
    int _i;
    
    @ManyToOne()
    B _b;

    @OneToMany(mappedBy: "a")
    List<C> _cs;
    
    String get id => get(const Symbol("id"));
    void set id(String v)=>set(const Symbol("id"), v);

    int get i => get(const Symbol("i"));
    void set i(int v)=>set(const Symbol("i"), v);

    B get b => get(const Symbol("b"));
    void set b(B v)=>set(const Symbol("b"), v);

    List<C> get cs => get(const Symbol("cs"));
    void set cs(List<C> v)=>set(const Symbol("cs"), v);
}

we define persistence root class called Persistence. This class will use reflection for intercepting getter/setter to access database. The only annoyance is we needs to define getter/setter calling methods get/set defined in the Persistence class. This would be a reasonable compromise. If dart has a build-in intercepting mechanism for getter/setter, we may further skip this definitions, and it may be a good proposal for Dart language..

The basic idea required to implement the Persistent class is the same as Monitor class, but it uses different annotation(JPA style), and will have SQL Driver access codes.

Another aspect of OR mapping is query. Often they introduce new query language mainly to avoid join operation. So we may create such query language or implement some standard query language in Dart, but we may just use ordinary SQL for this purpose. if it uses many-to-many relationship,  JPA creates intermediate table so writing query require such knowledge, but other relationships , the db table representation is simple. (and it is better to avoid many-to-may to simplify the db scheme).

So I think I will just go with SQL query based approach.
 But still some idea from https://github.com/polux/parsers, may be used to implement simple query language.

JsonMapper and JPA

One area I haven't investigated for Jason Mapper is the case where entity  have general network structure rather than tree structure where no sharing of objects occurs

I need to check how dart handle such cases, for parse, stringify.
When I checked the dart:json code, there was a code to check repeated occurrence  (for the recent version of Json lib.)

for stringify, if it does not check, either the same entity will be serialized repeatedly, and worst, if there are circular references, it will go into infinite loop.

for parse, essentially original json string need to have some id information so that other part of json code can refer to it.
when json see new json id, it may not be loaded yet, so substituting the actual object must be delayed until that object is created. so the fromJson will be more complicated.(but it will be still simple if we push closure to assign to the entity field  with given parameter, and when object is created with given id, it should  retrieve those closures and apply them with the newly created entity object.)

There is a Java project called jackson, I guess it will handle such cases, and if there is a some notation to represent such reference in json, Dart's json mapper should follow the convention.

This is important, since json is good format to communicate Java and Dart.
 If both are following the same format, we can pass Dart's object to Java, and vice versa.
This may happen through Dart's VM and JVM message passing protocol similar to Dart's  javascript call mechanisms (although I don't know if such lib exist now), or through HTTP protocol using REST style API such as CouchDB.
Both approaches are relying on converting an entity object to json string.

The useful application of this approach will be the support of JPA(Java Persistence API) in Dart.
It will be achieved by a Restfull service in HTTP server implemented by Servlet where the mapped Java object are to be serialized to Json string(using Jackson).

Since building JPA like service in Dart will be not simple task, this will be a quickest  way. the main problem will be some performance issues for server side Dart's access to JPA.(For client side, if it directly calls this servlet service, there should be no performance issues.) 
But such loosely coupled architecture will allow to use separate CPU cores for JVM to run Java Servlet for JPA and Dart VM to run HTTP client(server) to consume(provide) JPA services from servlet, so it may not be so inefficient.

Also if we provide such restful service, it may be called directly from client side (browser).
We may consider an additional query for Restful API in which query condition can be expressed as a dart expression  like CouchDB (CouchDB uses JavaScript for such filter function).
This may require some dynamic loading of Dart code, so it may not be so straightforward, but it may be interesting to investigate the possibility.

For such a binding, there should be a tool to map between Dart and Java persistent classes.
One direction is to start from annotated Dart classes, and generate corresponding Java JPA entity classes with the same annotation.

The other direction is to start JPA Entity Java classes, and generate Dart annotated classes.

For this end, we should define Dart annotation classes corresponding to JPA annotation classes in Java.