JsonMapper and JPA

One area I haven't investigated for Jason Mapper is the case where entity  have general network structure rather than tree structure where no sharing of objects occurs

I need to check how dart handle such cases, for parse, stringify.
When I checked the dart:json code, there was a code to check repeated occurrence  (for the recent version of Json lib.)

for stringify, if it does not check, either the same entity will be serialized repeatedly, and worst, if there are circular references, it will go into infinite loop.

for parse, essentially original json string need to have some id information so that other part of json code can refer to it.
when json see new json id, it may not be loaded yet, so substituting the actual object must be delayed until that object is created. so the fromJson will be more complicated.(but it will be still simple if we push closure to assign to the entity field  with given parameter, and when object is created with given id, it should  retrieve those closures and apply them with the newly created entity object.)

There is a Java project called jackson, I guess it will handle such cases, and if there is a some notation to represent such reference in json, Dart's json mapper should follow the convention.

This is important, since json is good format to communicate Java and Dart.
 If both are following the same format, we can pass Dart's object to Java, and vice versa.
This may happen through Dart's VM and JVM message passing protocol similar to Dart's  javascript call mechanisms (although I don't know if such lib exist now), or through HTTP protocol using REST style API such as CouchDB.
Both approaches are relying on converting an entity object to json string.

The useful application of this approach will be the support of JPA(Java Persistence API) in Dart.
It will be achieved by a Restfull service in HTTP server implemented by Servlet where the mapped Java object are to be serialized to Json string(using Jackson).

Since building JPA like service in Dart will be not simple task, this will be a quickest  way. the main problem will be some performance issues for server side Dart's access to JPA.(For client side, if it directly calls this servlet service, there should be no performance issues.) 
But such loosely coupled architecture will allow to use separate CPU cores for JVM to run Java Servlet for JPA and Dart VM to run HTTP client(server) to consume(provide) JPA services from servlet, so it may not be so inefficient.

Also if we provide such restful service, it may be called directly from client side (browser).
We may consider an additional query for Restful API in which query condition can be expressed as a dart expression  like CouchDB (CouchDB uses JavaScript for such filter function).
This may require some dynamic loading of Dart code, so it may not be so straightforward, but it may be interesting to investigate the possibility.

For such a binding, there should be a tool to map between Dart and Java persistent classes.
One direction is to start from annotated Dart classes, and generate corresponding Java JPA entity classes with the same annotation.

The other direction is to start JPA Entity Java classes, and generate Dart annotated classes.

For this end, we should define Dart annotation classes corresponding to JPA annotation classes in Java.

JsonMapper application for CouchDB HTTP data handling

There is a neat application of JsonMapper for CouchDB HTTP data conversion. Couch DB is based on json data. So it is good application area to apply jsonMapper between entity object and serialized object. (Here we see an interesting combination of generic type. Namely by defining generic classes, we can create a flow to pass the entity type to the 'doc' field type at runtime. )

In order to extract entities list from json HTTP output, and do some none standard plumbing of attributes(id must be mapped from _id, rev is from _rev), we can use a single fromJson method.

we define following generic class:

class CouchDBFetchData<T> {
  List<CouchDBFetchRowData<T>> _rows;
  
  List<CouchDBFetchRowData<T>> get rows => _rows;
  void set rows( List<CouchDBFetchRowData<T>> rows) { _rows = rows; }
}

class CouchDBFetchRowData<T> {
  T _doc;
  T get doc => _doc;
  void set doc(T doc) { _doc = doc; }
}

This class does not define all json properties, but they are ignored when json is mapped to the entity object.
then we can apply fromJson to the json output. This is the all things we need to get the list of entities. But unfortunately, right now, since there is a bug (as mentioned in previous post), this will fails. but if it is fixed, this should work.

  Future> getAll() {
    return _getData("/$dbName/_all_docs?include_docs=true")
        .then( (json) {
          if (json != "") {
            CouchDBFetchData<T> data = jsonMapper.fromJson(
              new CouchDBFetchData<T>().runtimeType, json, attr_redirect_map: {'id':'_id', 'rev':'_rev'});
            return data.rows.fold([], (list, CouchDBFetchRowData<T> row)=>list..add(row.doc));
          } else {
            return new List();
          }
        }).catchError((_)=>[]);
  }

here is the original inefficient, inelegant approach.

  Future> getAll() {
    return _getData("/$dbName/_all_docs?include_docs=true")
        .then( (json) {
          if (json != "") {
            Map data = parse(json);
            for (var rowData in data["rows"]) {
              Map map = rowData["doc"];
              map['id'] = map['_id'];
              map.remove('_id');
              map['rev'] = map['_rev'];
              map.remove('_rev');
              String json1 = stringify(map); // ugry
              T t = jsonMapper.fromJson(modelType, json1);
              ts.add(t);
          } else {
            return new List();
          }
        }).catchError((_)=>[]);
  }

Reflective JsonMapper (Part 3)

Originally I planed to use generics, but now it is better not to call it generic, but we call it reflective.
  
I completed the support of List(and Map) property in Entity class, and this works for both dynamic(with dart:mirrors lib for Dart VM) and static mode(for javascript).
But I haven't tested on javascript mode yet, but it should work. Also I haven't tested the map case. I will do it later.
This version was committed to Github.

Following is the sample code to show the support of List field in a class.

class A {
  int _i = 0;
  List<B> _bs = [];
  
  A(this._i, this._bs);
  factory A.create() => new A(111, []);
  
  int get i => _i;
  void set i(int ii) { _i == ii; }
  
  List<B> get bs => _bs;
  void set bs(List<B> bs) {_bs = bs; }
}

class B {
  String _s = 'zz';
  B(this._s);
  factory B.create() => new B('ss');
  String get s => _s;
  void set s(String ss) { _s == ss; }
}

test_simple_model() {
      A a = new A.create()..i = 10
        ..bs.add(new B.create()..s = "vvv")
        ..bs.add(new B.create()..s = "eee");
      
      String json = sampleJsonMapper.toJson(a);
      print(">>A@1 test_toJson json: ${json}");
      
      A a1 = sampleJsonMapper.fromJson(A, json);
      print(">>A@2 test_Json a1: ${a1}");
}

And the output of execution:

>>A@1 test_toJson json: {"i": 111,"bs": [{"s": "ss"},{"s": "ss"}]}
>> obj: Instance of 'A'
>>A@2 test_Json a1: Instance of 'A'
>> json: {"i": 10,"bs": [{"s": "ss"},{"s": "vv"}]}

Actually the support of generic type property was a bit difficult due to some confusing behavior of reflectedType property of ClassMirror. And some irreversible operation of types and  typeArguments behavior.

But by keeping type instead of extracting from ClassMirror, I could avoid the current restriction of Miror library. Probably Dart is still early stage, such mirror behavior should be refined.

Following are the problems I found:

1) why we cannot do followings:

var t1 = List<A>;

var cmirr = reflectClass(List<A>);

while we can do for none generic class:

car t = A;

var cmirr = reflectClass<A);

2) how can I create ClassMirror object which has reflectedType.

if we create CirrorClass through reflectClass(new List<B>().runtimeType),

it does not have reflectedType. but if we get ClassMirror from  bs of  A, it has a reflectedType. So there should be some way to create Mirror class which has reflectedType.

3) the reflectClass<List<A>) has all information to make cmirror to have reflectedType(since argumentType A is provided).

It should return ClassMirror with reflectedType.

I think this is very inconvenient behavior, and should be called a bug.

library dart_generics_test;

import 'dart:mirrors';

class A {
  int _i = 0;
  List<B> _bs = [];
  
  int get i => _i;
  void set i(int ii) { _i == ii; }
  
  List<B> get bs => _bs;
  void set bs(List<B> bs) {_bs = bs; }
}

class B {
  String _s = 'zz';
  
  String get s => _s;
  void set s(String ss) { _s == ss; }
}

test_generics() {
  {
    /*
    ClassMirror immir0 = reflectClass(List<B>); // <== this cannot be compiled!
    Type t0 = immir0.reflectedType;
    print(">> ${t0}");
    */
    
    // following code is to get a Type of List<B>, a work around to avoid above problem.  
    InstanceMirror immir = reflect(new List<B>());
    Type t = immir.type.reflectedType;
    print(">>1.1 ${t}");
    
    /*
    ClassMirror cmirr = reflectClass(t);
    Type t2 = cmirr.reflectedType; // here t2 == t, but it throws exception
    print(">>1.2 ${t2}");
    */
  /*
   * Unhandled exception:
Unsupported operation: Declarations of generics have no reflected type
#0      _LocalClassMirrorImpl.reflectedType (dart:mirrors-patch/mirrors_impl.dart:304:7)
#1      test_generics (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:76:21)
#2      main (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:81:16)
   * 
   */
  }

  {
    Type t1 = new B("nn").runtimeType;
    print(">>2.1 ${t1}");
  
    ClassMirror cmirr1 = reflectClass(t1);
    Type t2 = cmirr1.reflectedType; // this is ok
    print(">>2.1 ${t2}");
  }

  {
    Type t1 = new List<B>().runtimeType; // return List<B>
    print(">>3.1 t1: ${t1}");
  
    ClassMirror cmirr1 = reflectClass(t1);
    Type t2 = cmirr1.runtimeType; // return _LocalClassMirrorImpl
    print(">>3.2 t2: ${t2}"); 
    print(">>3.3 cmirr1.hasReflectedType: ${cmirr1.hasReflectedType}"); 
    
    Type t4 = cmirr1.reflectedType; // here it should be t4 == t0, but it throws exception
    print(">>3.3 t4: ${t4}");
  }

  {
    ClassMirror cmirr = reflectClass(A);
    cmirr.getters.forEach((k, MethodMirror md){
      Type tm = (md.returnType as ClassMirror).reflectedType; // this is OK
      print(">>4.1 k: ${k}, tm: ${tm}");
    });
  }
}

main() {
  test_generics();
}

Generic Class is not a first class citizen?

I tried to complete fromJson implementation, a method to instantiate an object of given type from its serialized json string, then I found some interesting limitation of Dart language.
I guess this is not planed feature, but something which are to be improved later.

for instance, the support of getClass in Java is not supported in dart now. (this will be obj.type getter). but I read a proposal, this is going to be supported.

there are two issues.

if we have a function taking Type value as the argument,  like f(A), this works as expected. but we cannot write f(A<B>)!.
also Type a = A<B> is not allowed.
So in a sense, generic class is not a first class citizen in Dart now.

Also similar, but more strange problem is following:

  InstanceMirror immir = reflect(new List<B>());
  Type t = immir.type.reflectedType;
  print(">> ${t}");
  ClassMirror mirr = reflectClass(t);
  print("2>> ${mirr.reflectedType}");

Since we cannot use List<B>, we need to create this type by reflect(new List<B>()).type.reflectedType. and this will be shown as List<B>.
but if we use this type to create ClassMirror, the getter call, mirr.reflectedType will cause runtime error.

>> List<B>
Unhandled exception:
Unsupported operation: Declarations of generics have no reflected type
#0      _LocalClassMirrorImpl.reflectedType (dart:mirrors-patch/mirrors_impl.dart:304:7)
#1      test_generics (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:77:21)
#2      main (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:82:16)

This is actually not good for portable_mirror librray.
Since creating Type through dart:mirrors library is not allows in javascript mode.

For example, couchdb_client_dao(in browser side) need to convert json string of list to list of entity. So ideally, this should be parsed by jsonMapper's fromJson. and it requires type value in the first argument, e.g, fromJson(List<modelType>, json).

This issue would not be limited to javascript mode. but it would also affect to the Dart VM mode.
So in order to avoid this problem in DartVM mode(as well as JavaScript mode), it will be necessary to avoid using Type. Instead, it should always use IClassMirror.

Although that may fix the jsonMapper, but this issue may influence in entire application
So one approach is not to worry about the generic type case(including list/map), and wait until this mirror issue is fixed..

Generic Json Mapper(Part 2)

In the previous post, I showed how to generate entity object from json string and type information.
Although I have already written the reverse direction(generating json from entity object),  it was written from scratch(since it was simple), and I wanted to see how I can use dart:json library in a similar way.

Then I found there is a class _JsonStringifier in json.dart in Dart sdk lib.
But as the name shows, it is private class, so I cannot extend the class and override some of the methods. If it is possible, I can do the similar things.
 That was a bit of disappointment. Actually the class  should be changed to public class.

By the way, I found Dartson was posted just 1 hour after I posted to Dartisans.
I think Json is quite universal data structure in web application, and this kind of binding are very powerful tool.

For the Dartson, I clone the project, and saw the code.
It is well written, although it is not yet supporting for javascript mode usage, it seemed planed to support it.

Essentially the main difference between our approaches is that  I took 'SAX' parser approach, and Dartson took DOM node approach. But right now, I just extended the library class, during the parsing it still creates maps, and uses the map to create an entity, then the map is immediately discarded.

If we use this approach, such an intermediate map creation can be avoided. We can just use stack to push all the key, value objects.
If we use such approach, it will be faster than Dartson.

Since Dartson is actually using json library to create intermediate json object(consisting List, Map, primitive values), the parsing quality will be as good as dart json library.

But Dartson seems not supporting irregular cases, like DateTime parsing/serialization, none Bean like classes where we need to call special factory/constructor(like ExpenseType).

I think it is better to have an ability to adjust such exceptional cases, otherwise, the tool's applicability will be unnecessarily limited.

//
// entity(including list, map) stringifier
//
class EntityJsonStringifier extends _JsonStringifier {
  final ISpecialTypeMapHandler mapHandler;
  
  EntityJsonStringifier(this.mapHandler): super(null);
  
  String toJson(final obj, {StringSink output}) {
     this..sink = (output != null)?output:new StringBuffer()
    ..seen = [];
    stringifyValue(obj);
    return sink.toString();
  }

  // @Override
  void stringifyValue(final object) {
    if (!stringifyJsonValue(object)) {
      checkCycle(object);
      try {
        // if toJson is defined, it will be used.
        var customJson = object.toJson();
        if (!stringifyJsonValue(customJson)) {
          throw new JsonUnsupportedObjectError(object);
        }
      } catch (e) {
        // if toJson is not defined..
        if (!stringifyJsonValue(object)) {
          stringifyEntity(object);
        }
      }
      seen.removeLast();
    }
  }
  
  void stringifyEntity(final object) {
    // this require dirt:mirrors
    Type t = ClassMirrorFactory.getType(object);
    StringifierFun stringfier = mapHandler.stringifier(t);
    if (stringfier != null) {
      sink.write(stringfier(object));
      return;
    }
    
    //
    IClassMirror cmirror = ClassMirrorFactory.reflectClass(t);
    IInstanceMirror iimirr = cmirror.reflect(object);
    
    sink.write('{');
    int idx = 0;
    Map fmap = cmirror.fieldTypes;
    int lastIdx = fmap.length-1;
    fmap.forEach((k, IFieldType ft){
      sink.write('"${ft.name}": ');
      stringifyValue(iimirr.getField(k).value);
      sink.write((idx == lastIdx)?"":",");
      idx++;
    });
    sink.write('}');   
  }
}

Generic Json Mapper(Part 1)

I finished the json mapper implementation based on restricted (my) mirror library.
 The idea is by providing a target class information for a given Json string, it automatically creates the instance of the given class filled with the json data.

Following is the sample codes:

 final Map<Type, ConstructorFun> entityCtors = {
    ExpenseType: (Map map)=>new ExpenseType(map['name'], map['code'])};

  final Map<Type, StringifierFun> stringifiers = {
    ExpenseType: (ExpenseType et) => '{"name": "${et.name}", "code": "${et.code}"}'};

var jsonMapper = new JsonMapper(new SpecialTypeMapHandler(entityCtors, stringifiers: stringifiers));

Expense e = new Expense.Default()
  ..id = "aaa"
  ..rev = "mmm"
  ..amount = 111
  ..date = new DateTime.now()
  ..detail = "asdjah"
  ..isClaimed = true
  ..expenseType = const ExpenseType("Hotel", "HT");

  String json = jsonMapper.toJson(obj);
  print(">> json: ${json}");

  Object obj = jsonMapper.fromJson(Expense, json);
  print(">> obj: ${obj}");
 

I have not yet finished two cases, when attribute has List or Map value.
I will do it later.
But actually there are a bit of difficulty for these cases.
In order to get List type's parameter type information, the current simple mirror API must be extended.
And a list may contains an instance of the different subclass of List's element type.
Map has a similar problem.
Also even if the field type is defined as ordinary class, the actual instance may be of subclass of that class.
So the current implementation is not covering such cases(yet).
But if we know all the available classes at the run time, we may be able to uniquely identify appropriate class. anyway if there are some ambiguity, that is inherent to the given json itself.

Also if we provide possible subclass information in the list value property as annotation, we can determine the class  efficiently.

This code was actually done by extending existing dart:json library. so it covers quite rigorously the spec of json.(but in fact, it uses old version of json code. what is the difference between the latest and older code?

This code does ot suport List, Map properties yet..

library json_mapper_v1;

import "dart:json";

import 'package:portable_mirror/mirror_api_lib.dart';

part 'src/dart_JsonStringifier.dart'; // this should be eliminated, dart:json should have pblic class JsonStringifier instead of _JsonStringifier

typedef dynamic ConstructorFun(Map map);
typedef dynamic StringifierFun(Object obj);
typedef dynamic ConvertFun(Object obj);

abstract class ISpecialTypeMapHandler {
  ConstructorFun entityCtor(Type type);
  StringifierFun stringifier(Type type);
  ConvertFun convert(Type type);
}

class SpecialTypeMapHandler implements ISpecialTypeMapHandler {
  final Map entityCtors;
  final Map _converts = {DateTime: (Object value)=>DateTime.parse(value)};
  final Map _stringifiers = {DateTime: (DateTime dt) => '"${dt.toString()}"'};
  
  SpecialTypeMapHandler(this.entityCtors, {Map converts, Map stringifiers}) {
    if (converts != null) _converts.addAll(converts);
    if (stringifiers != null) _stringifiers.addAll(stringifiers);
  }
  
  ConstructorFun entityCtor(Type type)=>(entityCtors == null)?null:entityCtors[type];
  ConvertFun convert(Type type)=>_converts[type];
  StringifierFun stringifier(Type type)=>_stringifiers[type];
}

//
//
//
abstract class IJsonMapper {
  Object fromJson(Type modelType, String json);
  
  String toJson(final object, {StringSink output});
}

class JsonMapper implements IJsonMapper {
  EntityJsonParser parser;
  EntityJsonStringifier stringifier;
  ISpecialTypeMapHandler mapHandler;
  
  JsonMapper(this.mapHandler, {_Reviver reviver}) {
    parser = new EntityJsonParser(mapHandler, reviver: reviver);
    stringifier = new EntityJsonStringifier(mapHandler);
  }

  Object fromJson(Type modelType, String json) => parser.parse(modelType, json);
  String toJson(final object, {StringSink output}) => stringifier.toJson(object, output: output);
}

//
// json parser
//
typedef _Reviver(var key, var value);

class EntityJsonParser {
  //EntityBuildJsonListener listener; // TODO.. this would be effcient way..
  ISpecialTypeMapHandler mapHandler;
  _Reviver _reviver;
  
  EntityJsonParser(this.mapHandler, {_Reviver reviver}) {
    _reviver = reviver;
  }
  
  EntityBuildJsonListener getListener(Type modelType) =>(_reviver == null)?new EntityBuildJsonListener(mapHandler, modelType)
      :new EntityReviverJsonListener(mapHandler, modelType, _reviver);

  dynamic parse(Type modelType, String json) { 
    EntityBuildJsonListener listener =  getListener(modelType);
    new JsonParser(json, listener).parse();
    return listener.result;
  }
}

class EntityBuildJsonListener extends BuildJsonListener {
  final ISpecialTypeMapHandler mapHandler;
  IClassMirror currentCmirror = null;
  List cmirrorStack = [];
  
  EntityBuildJsonListener(this.mapHandler, Type modelType) {
    currentCmirror = ClassMirrorFactory.reflectClass(modelType);
  }
  
  /** Pushes the currently active container (and key, if a [Map]). */
  void pushContainer() {
    super.pushContainer();
    cmirrorStack.add(currentCmirror);
  }

  /** Pops the top container from the [stack], including a key if applicable. */
  void popContainer() {
    super.popContainer();
    currentCmirror = cmirrorStack.removeLast();
  }
  
  void beginObject() {
    super.beginObject();
    if (key != null) {
      IFieldType ft = currentCmirror.fieldTypes[new Symbol(key)];
      if (ft != null) {
        currentCmirror = ClassMirrorFactory.reflectClass(ft.type);
      } else {
        print('>> beginObject ${key}');
        currentCmirror = null;
      }
    }
  }

  void endObject() {
    Map map = currentContainer;
    ConstructorFun spCtor = mapHandler.entityCtor(currentCmirror.type);
    if (spCtor != null) {
      currentContainer = spCtor(map);
    } else {
      // Dart Beans
      IInstanceMirror imiror = currentCmirror.newInstance();
      currentCmirror.fieldTypes.forEach((_, IFieldType ft){
        ConstructorFun vCtor = mapHandler.convert(ft.type);
        var value = map[ft.name];
        imiror.getField(ft.symbol).value = (vCtor != null)?vCtor(value):value;
      });
      currentContainer = imiror.reflectee;
    }
    super.endObject();
  }
}

class EntityReviverJsonListener extends EntityBuildJsonListener {
  final _Reviver reviver;
  EntityReviverJsonListener(ISpecialTypeMapHandler mapHandler, Type modelType, reviver(key, value))
    : super(mapHandler, modelType), this.reviver = reviver;

  void arrayElement() {
    List list = currentContainer;
    value = reviver(list.length, value);
    super.arrayElement();
  }

  void propertyValue() {
    value = reviver(key, value);
    super.propertyValue();
  }

  get result {
    return reviver("", value);
  }
}