Dart vs Java performance comparison with Recursive Maxtrix Multiplication (Part 3)

I ported the last Java implementation of recursive matrix multiplication to Dart.
It is interesting to see the actual performance difference of two languages.
Such comparison will depend on the type of application and level of expected performance.

As we have seen, Java allows considerable fast implementation of recursive matrix multiplication. It is interesting to compare the performance with C++.
But definitely its 1.6 Gaga flop(1 billion multiplication in 0.6 sec) is definitely assembler level performance.

Following is the result of the performance measurement of Dart.
From this, Dart is about 4 times slower than Java if we compare with 1 thread version of Java. but if we compare with 4 threads(4 cores) version of Java, it is about 13 times slower.
So 4 times difference is in a sense close to Java. but Dart does not allow to share memory among isolates, so even we use isolate, it will be more costly to copy matrix elements, but still it is interesting to see the performance using  isolates.
Even Dart does not support array, if the List representation internally uses array(by delegating to external code), it will not cause major performance difference. Other reason will be the representation of int. We may have similar result in Java if we replace long by BigInteger etc.

[Dart]

rec_test_loop: 12
>> dim: 2, elapsed time: 5 milli sec, dim^3; 8, ration(time/dim^3)=625000.0
>> dim: 4, elapsed time: 0 milli sec, dim^3; 64, ration(time/dim^3)=0.0
>> dim: 8, elapsed time: 1 milli sec, dim^3; 512, ration(time/dim^3)=1953.125
>> dim: 16, elapsed time: 32 milli sec, dim^3; 4096, ration(time/dim^3)=7812.5
>> dim: 32, elapsed time: 4 milli sec, dim^3; 32768, ration(time/dim^3)=122.0703125
>> dim: 64, elapsed time: 5 milli sec, dim^3; 262144, ration(time/dim^3)=19.073486328125
>> dim: 128, elapsed time: 16 milli sec, dim^3; 2097152, ration(time/dim^3)=7.62939453125
>> dim: 256, elapsed time: 119 milli sec, dim^3; 16777216, ration(time/dim^3)=7.092952728271484
>> dim: 512, elapsed time: 1239 milli sec, dim^3; 134217728, ration(time/dim^3)=9.231269359588623
>> dim: 1024, elapsed time: 8027 milli sec, dim^3; 1073741824, ration(time/dim^3)=7.475726306438446
>> dim: 2048, elapsed time: 60587 milli sec, dim^3; 8589934592, ration(time/dim^3)=7.0532551035285
>> dim: 4096, elapsed time: 484160 milli sec, dim^3; 68719476736, ration(time/dim^3)=7.045455276966095

[Java with single thread]

>> dim: 2, elapsed time: 1 milli sec
>> dim: 4, elapsed time: 0 milli sec
>> dim: 8, elapsed time: 0 milli sec
>> dim: 16, elapsed time: 0 milli sec
>> dim: 32, elapsed time: 5 milli sec
>> dim: 64, elapsed time: 15 milli sec
>> dim: 128, elapsed time: 5 milli sec
>> dim: 256, elapsed time: 30 milli sec
>> dim: 512, elapsed time: 217 milli sec
>> dim: 1024, elapsed time: 1915 milli sec
>> dim: 2048, elapsed time: 14691 milli sec
>> dim: 4096, elapsed time: 115860 milli sec

[Java with 4 threads(4 cores)]

>> dim: 2, elapsed time: 0.0 milli sec, dim^p; 8, ration(time/dim^3)=0
>> dim: 4, elapsed time: 0.0 milli sec, dim^p; 64, ration(time/dim^3)=0
>> dim: 8, elapsed time: 1.0 milli sec, dim^p; 512, ration(time/dim^3)=1,953.125
>> dim: 16, elapsed time: 1.0 milli sec, dim^p; 4,096, ration(time/dim^3)=244.141
>> dim: 32, elapsed time: 3.0 milli sec, dim^p; 32,768, ration(time/dim^3)=91.553
>> dim: 64, elapsed time: 17.0 milli sec, dim^p; 262,144, ration(time/dim^3)=64.85
>> dim: 128, elapsed time: 4.0 milli sec, dim^p; 2,097,152, ration(time/dim^3)=1.907
>> dim: 256, elapsed time: 18.0 milli sec, dim^p; 16,777,216, ration(time/dim^3)=1.073
>> dim: 512, elapsed time: 105.0 milli sec, dim^p; 134,217,728, ration(time/dim^3)=0.782
>> dim: 1024, elapsed time: 602.0 milli sec, dim^p; 1,073,741,824, ration(time/dim^3)=0.561
>> dim: 2048, elapsed time: 4771.0 milli sec, dim^p; 8,589,934,592, ration(time/dim^3)=0.555
>> dim: 4096, elapsed time: 38704.0 milli sec, dim^p; 68,719,476,736, ration(time/dim^3)=0.563

Here is again a puzzling result for the performance degradation of Simple Matrix multiplication in Dart. until dim is 512, simple matrix is faster than recursive matrix, but after that, simple matrix become slower than recursive version, and at 4096, it is 3.3 times slower.

[Simple Matrix]

>> dim: 2, elapsed time: 4 milli sec
>> dim: 4, elapsed time: 0 milli sec
>> dim: 8, elapsed time: 0 milli sec
>> dim: 16, elapsed time: 4 milli sec
>> dim: 32, elapsed time: 6 milli sec
>> dim: 64, elapsed time: 2 milli sec
>> dim: 128, elapsed time: 7 milli sec
>> dim: 256, elapsed time: 55 milli sec
>> dim: 512, elapsed time: 483 milli sec
>> dim: 1024, elapsed time: 12269 milli sec
>> dim: 2048, elapsed time: 118535 milli sec
>> dim: 4096, elapsed time: 1597558 milli sec

[Recursive Matrix]

>> dim: 2, elapsed time: 5 milli sec, dim^3; 8, ration(time/dim^3)=625000.0
>> dim: 4, elapsed time: 0 milli sec, dim^3; 64, ration(time/dim^3)=0.0
>> dim: 8, elapsed time: 1 milli sec, dim^3; 512, ration(time/dim^3)=1953.125
>> dim: 16, elapsed time: 32 milli sec, dim^3; 4096, ration(time/dim^3)=7812.5
>> dim: 32, elapsed time: 4 milli sec, dim^3; 32768, ration(time/dim^3)=122.0703125
>> dim: 64, elapsed time: 5 milli sec, dim^3; 262144, ration(time/dim^3)=19.073486328125
>> dim: 128, elapsed time: 16 milli sec, dim^3; 2097152, ration(time/dim^3)=7.62939453125
>> dim: 256, elapsed time: 119 milli sec, dim^3; 16777216, ration(time/dim^3)=7.092952728271484
>> dim: 512, elapsed time: 1239 milli sec, dim^3; 134217728, ration(time/dim^3)=9.231269359588623
>> dim: 1024, elapsed time: 8027 milli sec, dim^3; 1073741824, ration(time/dim^3)=7.475726306438446
>> dim: 2048, elapsed time: 60587 milli sec, dim^3; 8589934592, ration(time/dim^3)=7.0532551035285
>> dim: 4096, elapsed time: 484160 milli sec, dim^3; 68719476736, ration(time/dim^3)=7.045455276966095

Following is the Dart code used for this benchmark test.

library rec_matrix_v6;

import "dart:math";

Random rand = new Random(); 

List<List<int>> _randomMatrix(int dim) {
  //final List<List<int>> matrix = new int[dim][dim];
  final List<List<int>> matrix = new List(dim);
  for (int i = 0; i < dim; i++) {
    matrix[i] = new List(dim);
    for (int j = 0; j < dim; j++) {
      matrix[i][j] = rand.nextInt(50);
    }
  }
  return matrix;
}
List<List<int>> _zeroMatrix(int dim) {
  final List<List<int>> matrix = new List(dim);
  for (int i = 0; i < dim; i++) {
    matrix[i] = new List(dim);
    for (int j = 0; j < dim; j++) {
      matrix[i][j] = 0;
    }
  }
  return matrix;
}

//
// AbstractMatrix
//
abstract class AbstractMatrix {
  int _dim;
  List<List<int>> _matrix;
  
  AbstractMatrix(List<List<int>> matrix) {
    this._dim = matrix.length;
    this._matrix = matrix;
  }
  
  int dim() {
    return _dim;
  }

  SimpleMatrix add(SimpleMatrix m) {
    List<List<int>> m0 = _matrix;
    List<List<int>> m1 = m._matrix;
    List<List<int>> m2 = new List(_dim);
    for (int i = 0; i < _dim; i++) {
      m2[i] = new List(_dim);
      for (int j = 0; j < _dim; j++) {
        m2[i][j] = m0[i][j]+m1[i][j];
      }
    }
    return new SimpleMatrix(m2);
  }

  SimpleMatrix subtract(SimpleMatrix m) {
    List<List<int>> m0 = _matrix;
    List<List<int>> m1 = m._matrix;
    List<List<int>> m2 = new List(_dim);
    for (int i = 0; i < _dim; i++) {
      m2[i] = new List(_dim);
      for (int j = 0; j < _dim; j++) {
        m2[i][j] = m0[i][j]-m1[i][j];
      }
    }
    return new SimpleMatrix(m2);
  }
  
  String toString() {
    return toString1(_matrix);
  }
  
  String toString1(List<List<int>> matrix) {
    int dim = matrix.length;
    StringBuffer sb = new StringBuffer();
    sb.write("(\n");
    for (int i = 0; i < dim; i++) {
      for (int j = 0; j < dim; j++) {
        int v = matrix[i][j];
        if (j != 0) {
          sb.write(", ");
        }
        sb.write(v);
      }
      sb.write("\n");
    }
    sb.write(")\n");
    return sb.toString();
  }
  
  bool equals(Object obj) {
    if (obj is SimpleMatrix) {
      SimpleMatrix m = obj;
      if (_dim != m._dim) {
        return false;
      }
      List<List<int>>matrix1 = m._matrix;
      for (int i = 0; i < _dim; i++) {
        for (int j = 0; j < _dim; j++) {
          if (_matrix[i][j] != matrix1[i][j]) {
            return false;
          }
        }
      }
      return true;
    } else {
      return false;
    }
  }
}

//
// SimpleMatrix
//
class SimpleMatrix extends AbstractMatrix {
  SimpleMatrix(List<List<int>> matrix): super(matrix) {}
  
  SimpleMatrix mult(SimpleMatrix m) {
    List<List<int>> m0 = _matrix;
    List<List<int>> m1 = m._matrix;
    List<List<int>> m2 = new List(_dim);
    for (int i = 0; i < _dim; i++) {
      m2[i] = new List(_dim);
      for (int j = 0; j < _dim; j++) {
        int v = 0;
        for (int k = 0; k < _dim; k++) {
          v += m0[i][k]*m1[k][j];
        }
        m2[i][j] = v;
      }
    }
    return new SimpleMatrix(m2);
  }
}

//
// RecMatrix
//
class RecMatrix extends AbstractMatrix {
  RecMatrix(List<List<int>> matrix): super(matrix) {}
  
  static RecMatrix randomMatrix(int dim) {
    List<List<int>> matrix = _randomMatrix(dim);
    return new RecMatrix(matrix);
  }

  RecMatrix mult(RecMatrix m) {
    RecMatrix m2 = new RecMatrix(_zeroMatrix(_dim));
    m2.mult0(_dim, _matrix, 0, 0, m._matrix, 0, 0);
    return m2;
  }
  
  void mult0(final int dim, final List<List<int>> m1, final int row_index1, final int column_index1, final List<List<int>> m2, final int row_index2, final int column_index2) {
    if (dim == 2) {
      for (int i = 0; i < 2; i++) {
        for (int j = 0; j < 2; j++) {
          int i1 = row_index1 | i;
          int j2 = column_index2 | j;
          int v = 0;
          for (int k = 0; k < 2; k++) {
            int j1 = column_index1 | k;
            int i2 = row_index2 | k;
            v += m1[i1][j1]*m2[i2][j2];
          }
          _matrix[i1][j2] += v;
        }
      }
    } else {
      final int dim0 = dim >> 1;
      for (int i = 0; i < 2; i++) {
        final int r_idx1 = (i == 0) ? row_index1 : (row_index1 | dim0);
        for (int j = 0; j < 2; j++) {
          final int c_idx2 = (j == 0) ? column_index2: (column_index2 | dim0);
          for (int k = 0; k < 2; k++) {
            final int c_idx1 = (k == 0) ? column_index1: (column_index1 | dim0);
            final int r_idx2 = (k == 0) ? row_index2 : (row_index2 | dim0);
            mult0(dim0, m1, r_idx1, c_idx1, m2, r_idx2, c_idx2);
          }
        }
      }
    }
  }
}

//
// Tests
//

// simple_matrix_test1
SimpleMatrix simple_matrix_test1(List<List<int>> matrx1, List<List<int>> matrx2, bool show) {
  SimpleMatrix m1 = new SimpleMatrix(matrx1);
  SimpleMatrix m2 = new SimpleMatrix(matrx2);
  SimpleMatrix m3 = m1.mult(m2);
  if (show) {
    print("m1: ${m1}");
    print("m2: ${m2}");
    print("m3: ${m3}");
  }
  return m3;
}

DateTime simple_matrix_test2(int dim) {
  List<List<int>> matrx1 = _randomMatrix(dim);
  List<List<int>> matrx2 = _randomMatrix(dim);
  
  simple_matrix_test1(matrx1, matrx2, false);

  return new DateTime.now();
}

void simple_test_loop(int max) {
  int dim = 1;
  for (int i = 1; i <= max; i++) {
    dim = dim*2;
    DateTime startTime = new DateTime.now();
    DateTime endTime = simple_matrix_test2(dim);
    Duration du = endTime.difference(startTime);
    print(">> dim: ${dim}, elapsed time: ${du.inMilliseconds} milli sec");
  }
}

// rec_matrix_test1
RecMatrix rec_matrix_test1(List<List<int>> matrx1, List<List<int>> matrx2, bool show) {
  RecMatrix rm1 = new RecMatrix(matrx1);
  RecMatrix rm2 = new RecMatrix(matrx2);
  RecMatrix rm3 = rm1.mult(rm2);
  if (show) {
    print("m1: ${rm1}");
    print("m2: ${rm2}");
    print("m3: ${rm3}");
  }
  return rm3;
}

DateTime rec_matrix_test2(int dim) {
  List<List<int>> matrx1 = _randomMatrix(dim);
  List<List<int>> matrx2 = _randomMatrix(dim);
  
  rec_matrix_test1(matrx1, matrx2, false);
  return new DateTime.now();
}

void rec_test_loop(int max) {
  int dim = 1;
  for (int i = 1; i <= max; i++) {
    dim = dim*2;
    DateTime startTime = new DateTime.now();
    DateTime endTime = rec_matrix_test2(dim);
    Duration duration = endTime.difference(startTime);
    int b_duration = duration.inMilliseconds;
    int dim_p3 = dim*dim*dim;
    double b_r = (b_duration/dim_p3)*1000000;
    print(">> dim: ${dim}, elapsed time: ${b_duration} milli sec, dim^3; ${dim_p3}, ration(time/dim^3)=${b_r}");
  }
}

// verify_matrix_test1
void verify_matrix_test1(int dim, bool show) {
  List<List<int>> matrx1 = _randomMatrix(dim);
  List<List<int>> matrx2 = _randomMatrix(dim);
  
  SimpleMatrix sm3 = simple_matrix_test1(matrx1, matrx2, show);
  RecMatrix rm3 = rec_matrix_test1(matrx1, matrx2, show);
  SimpleMatrix srm3 = new SimpleMatrix(rm3._matrix);
  if (!sm3.equals(srm3)) {
    print("!!verify_matrix_test1: not equals ");
    print("m1: ${new SimpleMatrix(matrx1)}");
    print("m2: ${new SimpleMatrix(matrx2)}");
    print("sm3: ${sm3}");
    print("srm3:${srm3}");
    throw new Exception();
  } else if (show) {
    print("m1: ${new SimpleMatrix(matrx1)}");
    print("m2: ${new SimpleMatrix(matrx2)}");
    print("m3: ${sm3}");
  }
}

void verify_test_loop(int max, bool show) {
  int dim = 1;
  for (int i = 1; i <= max; i++) {
    dim = dim*2;
    verify_matrix_test1(dim, show);
  }
}

void test1(int max) {
  /*
  print("verify_test_loop: "+max);
  verify_test_loop(max, false);
  
  print("simple_test_loop: "+max);
  simple_test_loop(max);
   */
  print("rec_test_loop: ${max}");
  rec_test_loop(max);
}

void main() {
  int max = 12;
  test1(max);
  print(">> done");
  //verify_matrix_test1(2);
  //verify_matrix_test1(4);
  //verify_test_loop(max);
  
  //simple_test_loop(max);
  
  //rec_matrix_test1(4);
  //rec_test_loop(max);
}

Dart 1.0

Dart released 1.0, but it looks like some political gesture, it seems having nothing to do with the quality of the system.
There are still many basic unresolved problems, like mirror, await etc.
Essentially certain power user cannot use Dart at this level of quality.
Also editor sometimes causing suppressing internal error, and showing bad error messages.
Anyway, they would have needed to show some progress for public, that would be the only the reason for they have released 1.0.

Also the last minutes announcement of deprecation web-component and replacing it by unproven polymer is also nonsenses move.
If it is experimental nature, it is better to avoid pushing such technology. This also would have come from political decision to provide some sort of solution to support component based web development which dart supposed to address. But like JavaScript does not provide such framework, Dart actually does not have to provide such 'solution'. It doesn't have to be a part of Dart release at all.

Since Dart is new language, it may take sometime to understand the best way to utilize new language features for component based web developments.
These thing may be delegated to communities, and wait to see emerging new solutions.

Also in general, if we develop with JavaScript, we don't see any fragment of HTML. So such focus on HTML file on Polymer looks very much old fashioned approach.
It will be an obstacle for utilizing full language capabilities to design component based library.

So I feel a lot of questionable thing for Dart right now.
But if I wait Dart 2.0, it may be something I expected as 1.0.
So just I may need to wait another year to mature.

At least, if Dart does not provide proper synchronization feature for server side application, it seems a waste of time to develop something on server side right now.

bad choice for Server side Dart?

I investigated the isolate to see if it can support waiting another isolate to finish, then continue the rest of job.
But Dart's concurrency is really restrictive, there seems no way to achieve this.
Considering many poor design of asynchronity of current Dart, server side Dart seems not good option right now.

Maybe I may try 'Go' as well..
Rob pike and Thompson who are behind Go had been working on this area.

http://doc.cat-v.org/plan_9/4th_edition/papers/sleep

Of cause Java is still OK..
If we define some mapping tool from Dart annotation to JPA annotation, it is still not so bad option.But even we use java in server side, lazy evaluation on Dart client is still not possible. As long as the expected job in client side is low, this may not be a big problem though..

video:Dart Today and Beyond with Google's Gilad Bracha

Dart Today and Beyond with Google's Gilad Bracha 

This video was taken a year ago, just after m1 release. It explains some of the idea behind Dart , so it is worth watching.  

in that video, he was telling he want to reduce asynchronity. in fact, mirror dropped future from the api, but io lib added much more async api. so totally dart increased a lot of asynchronity.

Anyway, he looks kind of language proper guy, it would be better to couple with  another guy with broader application development experiences for leading language design so that it won't go to romanticist's dream world.

 

Dart's Future evaluation order

Dart's future evaluation order is not so clear from source code, so I tested using sample code.
Also this test when stream evaluation happen. listen method also started after the end of main code line and after running all futures. this means listen may create future, but not at the time listen is invoked.

import "dart:async";

main() {
    Stream stream = new Stream.periodic(const Duration(milliseconds: 1), (x) => x);
    int receivedCount = 0;
    var subscription;
    print("<1>");
    int a = 0;
    new Future.value(1).then((v) {
      print(">> 1 v: ${v}");
      new Future.value(10).then((v) {
        print(">> 10 v: ${v}");
        
      });
    });
    
    bool finished = false;
    subscription = stream.listen((data) {
      //expect(data, receivedCount);
      receivedCount++;
      print("receivedCount: ${receivedCount}");
      finished = true;
      new Future.value(30).then((v) {
        print(">> 30 v: ${v}");
        
      });
      if (receivedCount == 5) subscription.cancel();
      
    });
    new Future.value(2).then((v) {
      print(">> 2 v: ${v}");
      new Future.value(20).then((v) {
        print(">> 20 v: ${v}");
        
      });
    });
    /*
    while (!finished) {
      
    }
    */
    print("<2>");
}

<1>
<2>
>> 1 v: 1
>> 2 v: 2
>> 10 v: 10
>> 20 v: 20
receivedCount: 1
receivedCount: 2
>> 30 v: 30
>> 30 v: 30
receivedCount: 3
receivedCount: 4
>> 30 v: 30
>> 30 v: 30
receivedCount: 5
>> 30 v: 30

If we add while loop:

import "dart:async";

main() {
    Stream stream = new Stream.periodic(const Duration(milliseconds: 1), (x) => x);
    int receivedCount = 0;
    var subscription;
    print("<1>");
    int a = 0;
    new Future.value(1).then((v) {
      print(">> 1 v: ${v}");
      new Future.value(10).then((v) {
        print(">> 10 v: ${v}");
        
      });
    });
    
    bool finished = false;
    subscription = stream.listen((data) {
      //expect(data, receivedCount);
      receivedCount++;
      print("receivedCount: ${receivedCount}");
      finished = true;
      new Future.value(30).then((v) {
        print(">> 30 v: ${v}");
        
      });
      if (receivedCount == 5) subscription.cancel();
      
    });
    new Future.value(2).then((v) {
      print(">> 2 v: ${v}");
      new Future.value(20).then((v) {
        print(">> 20 v: ${v}");
        
      });
    });

    while (!finished) {
      
    }

    print("<2>");
}

The output will be just <1>.

<1>

These things means future evaluation will happen only main code line was ended.
And the order of future evaluation is the order it was created.

Big restriction: Dart's Stream does not support synchronous mode

When I tried to support getter property of another referenced entity, I came to recognize that it is NOT possible in current Dart.

This is ridiculous restriction.
And it was supported until M3 deprecated the previous synchronous Stream classes.
I think this is pretty much nutty stupid decision.
It must have been done by young limited experienced people who can believe such single bullet approach can solve every thing. Probably it may solve every thing they knew.
That is the problem of the lack of experience.

Whole Dart development somehow has resemblance to early stage of Java around 1995. Probably many of developer of Dart does not remember.

Java has been introduced as a language for Applet. And marketed as such, then it was mainly used for server side. The main reason for the adaption can be contributed several reasons. one aspect is it was free, and it provided a lot of library. Compared to C/C++, these are big difference. Also from language point of view, Java supported class, object oriented features and garbage collection.(and lack of pointer). So compared to C/C++, it was significantly high level language, it was good enough for the most of server side application.

(but it was really backward step compared to other advanced moduler languages, like Mesa/Cedar, Ada, Modula-2, etc. but they never became known to wider audiences, that may have helped Java to proliferated also..)

For Dart,  the current marketing Buz is for browser side language(to replace JavaScript (eventually..)), but as a language, it has also unique feature integrating object/functional idioms in a single language.
Although there are many of such languages(Like Scala), there are no other language which can support both browser and server side.

I think  these two aspects are quite important. Although current marketing focus is in browser side, like polymer, dart:io etc, but it will be used more on server side.
Since the combination of object/functional style is very powerful, once people realized how it is productive, they will never want to write huge amount of equivalent codes in Java. Also sharing the same class from both sides is big plus.

Bottom line is  Dart should target such server side development in much higher priority. Right now, many people seems not developing significant application, they will not complain, but eventually, more people start using Dart, this kind of biased API design will bring a lot of complaints.

Dart's reflection details

Since I needed to manipulate reflection library to develop reflective json_mapper, I investigated some fine details of library. And found strange behavior.
So I asked to dart-misc mailing list.
Fortunately these question was answered by Gilad Bracha(main designer of Dart), so it became clear for what I should expect from Dart now.

Following style will return a mirror for generic class 'declaration'.

ClassMirror cmirror = reflectClass(new List<A>().runtimeType) 
(ClassMirror cmirror = reflectClass(List<A>)) // not supported yet

the above cmirror correspond to the generic class 'declaration', which is equivalent to none applied generic type, so the type including type variables in the generic class still contain type variables when it is inspected by reflection library. This is called a generic declaration.

So the strange things are even List<A> are applied generic class(type variable are substituted by A), but the  reflectClass returns List<T> class which does not corresponding to any runtime type of an object. I don't know why this is so designed, but this is confusing, and quite useless return result for most of the applications.

But as long as there are some way to return Type of applied generic class, it will be OK.
in fact, there is such a way, but it would be difficult to know without having told the difference!

This will return a mirror for applied generic class. (but not complete yet.. since there is a bug now)

ClassMirror cmirror = reflect(new List<A>()).type;
(ClassMirror cmirror = reflectType(List<A>)) // not supported yet

This cmirror is corresponding to applied generic class, and if we do print  cmirror.reflectedType, it prints List<A>.
So it seems OK, but in fact, I already knew this was not OK.

For json_mapper, we need to get field's type in order to process sub json string.

but if we run following code, the field types are none applied generic types. namely including type variables, not substituted by applied types.

ClassMirror cmirr = reflect(new List()).type;
    print(">>0@@ cmirr.reflectedType: ${cmirr.reflectedType}");
    cmirr.getters.forEach((k, MethodMirror md){
      Type tm = null;
      if (md.returnType is ClassMirror) {

        tm = (md.returnType as ClassMirror).reflectedType;
        print(">>1@@ k: ${k}, tm: ${tm}");
      } else if (md.returnType is TypeVariableMirror) {
        TypeVariableMirror tvmirr = md.returnType;
        print(">>2@@ tvmirr: ${tvmirr.runtimeType}");
      }
    });

output:

>> type: ChouchDBFetchData
>>0@@ cmirr.reflectedType: List
>>1@@ k: Symbol("length"), tm: int
>>1@@ k: Symbol("_capacity"), tm: int
>>2@@ tvmirr: _LocalTypeVariableMirrorImpl
>>2@@ tvmirr: _LocalTypeVariableMirrorImpl
>>2@@ tvmirr: _LocalTypeVariableMirrorImpl
>>1@@ k: Symbol("isEmpty"), tm: bool
>>1@@ k: Symbol("isNotEmpty"), tm: bool
>>1@@ k: Symbol("reversed"), tm: Iterable
>>1@@ k: Symbol("iterator"), tm: Iterator

So essentially right now, there is no way to handle generic class properly.
Fortunately, Gilad agreed this is bug, and google teams is now working on fixing this problem. Actually this is a big bug.

So maybe we will get better mirror lib soon.
Other related issues seems also addressed later release.

like reflectClass(typedef List<A>)(here typedef keyword is 'required' to avoid parsing issue..)
and 'invariant'  reflectType, which probablly it returns applied generic class.

here is his answers:

I've been meaning to update the mirror tutorial on our site, but because significant changes are coming soon, I'm delaying that just a tad longer. That said, those changes have nothing to do with this issue, so I'll try and answer your question here. Bear with me, as it is slightly non-obvious and requires some background.

(A)  ClassMirrors reflect both the classes of objects and class declarations. What's the difference between the class of an object and a class declaration?  Well, in a world without generics you can leave the distinction to philosophers, but once you have generics, you want to distinguish between the one declaration of the generic (i.e, the declaration in the source code) and its many invocations (such as List<String>, List<bool> etc.), aka parameterized types.  All the invocations share the same declaration.

This matters when you ask a class mirror for its methods, and then look up a method and ask for its return type. If the mirror is on List<int>, then the return type of the method #first will be int; if it is List<Element> it will be Element etc.  If ask the same question on a declaration, the return type is E, which is a type variable (and no actual object has a type variable as its type; the type variable is always bound to some specific type).

There is never an object whose type is a generic declaration. Hence there is no Type object that corresponds to a generic declaration, and so a mirror on generic declaration has no Type object associated with it. Which is why hasReflectedType on such a mirror returns false. If you still try and get the reflectedType, it throws. 

Hence the behavior you are seeing. 

Now you might ask why.  We could have reflectClass give you a mirror on the specific Type you passed in, and then the invariant you expected would hold. However, then we'd have a problem if you wanted to get at the declaration. If you wrote reflectClass(List) you might well expect the result to be a mirror on the declaration, but you'd be disappointed. Let's say we have a class 

class G<T> ...

In Dart, the identifier G, used without type arguments, always denotes G<dynamic>.  So you would not get a mirror on the declaration of G (or of List) but a mirror on a specific parameterized type (G<dynamic> or List<dynamic> as the case might be).

We decided to keep reflectClass for the declaration. If you want a mirror on a specific parameterized type, you can get it by reflecting on an instance and asking the instance mirror for its type

reflect(new List<A>()).type

we are planning on adding a function reflectType() that would always return a mirror on the specific Type object passed to it. The the invariant

reflectType(t).reflectedType == t

Reflective JsonMapper (Part 3)

Originally I planed to use generics, but now it is better not to call it generic, but we call it reflective.
  
I completed the support of List(and Map) property in Entity class, and this works for both dynamic(with dart:mirrors lib for Dart VM) and static mode(for javascript).
But I haven't tested on javascript mode yet, but it should work. Also I haven't tested the map case. I will do it later.
This version was committed to Github.

Following is the sample code to show the support of List field in a class.

class A {
  int _i = 0;
  List<B> _bs = [];
  
  A(this._i, this._bs);
  factory A.create() => new A(111, []);
  
  int get i => _i;
  void set i(int ii) { _i == ii; }
  
  List<B> get bs => _bs;
  void set bs(List<B> bs) {_bs = bs; }
}

class B {
  String _s = 'zz';
  B(this._s);
  factory B.create() => new B('ss');
  String get s => _s;
  void set s(String ss) { _s == ss; }
}

test_simple_model() {
      A a = new A.create()..i = 10
        ..bs.add(new B.create()..s = "vvv")
        ..bs.add(new B.create()..s = "eee");
      
      String json = sampleJsonMapper.toJson(a);
      print(">>A@1 test_toJson json: ${json}");
      
      A a1 = sampleJsonMapper.fromJson(A, json);
      print(">>A@2 test_Json a1: ${a1}");
}

And the output of execution:

>>A@1 test_toJson json: {"i": 111,"bs": [{"s": "ss"},{"s": "ss"}]}
>> obj: Instance of 'A'
>>A@2 test_Json a1: Instance of 'A'
>> json: {"i": 10,"bs": [{"s": "ss"},{"s": "vv"}]}

Actually the support of generic type property was a bit difficult due to some confusing behavior of reflectedType property of ClassMirror. And some irreversible operation of types and  typeArguments behavior.

But by keeping type instead of extracting from ClassMirror, I could avoid the current restriction of Miror library. Probably Dart is still early stage, such mirror behavior should be refined.

Following are the problems I found:

1) why we cannot do followings:

var t1 = List<A>;

var cmirr = reflectClass(List<A>);

while we can do for none generic class:

car t = A;

var cmirr = reflectClass<A);

2) how can I create ClassMirror object which has reflectedType.

if we create CirrorClass through reflectClass(new List<B>().runtimeType),

it does not have reflectedType. but if we get ClassMirror from  bs of  A, it has a reflectedType. So there should be some way to create Mirror class which has reflectedType.

3) the reflectClass<List<A>) has all information to make cmirror to have reflectedType(since argumentType A is provided).

It should return ClassMirror with reflectedType.

I think this is very inconvenient behavior, and should be called a bug.

library dart_generics_test;

import 'dart:mirrors';

class A {
  int _i = 0;
  List<B> _bs = [];
  
  int get i => _i;
  void set i(int ii) { _i == ii; }
  
  List<B> get bs => _bs;
  void set bs(List<B> bs) {_bs = bs; }
}

class B {
  String _s = 'zz';
  
  String get s => _s;
  void set s(String ss) { _s == ss; }
}

test_generics() {
  {
    /*
    ClassMirror immir0 = reflectClass(List<B>); // <== this cannot be compiled!
    Type t0 = immir0.reflectedType;
    print(">> ${t0}");
    */
    
    // following code is to get a Type of List<B>, a work around to avoid above problem.  
    InstanceMirror immir = reflect(new List<B>());
    Type t = immir.type.reflectedType;
    print(">>1.1 ${t}");
    
    /*
    ClassMirror cmirr = reflectClass(t);
    Type t2 = cmirr.reflectedType; // here t2 == t, but it throws exception
    print(">>1.2 ${t2}");
    */
  /*
   * Unhandled exception:
Unsupported operation: Declarations of generics have no reflected type
#0      _LocalClassMirrorImpl.reflectedType (dart:mirrors-patch/mirrors_impl.dart:304:7)
#1      test_generics (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:76:21)
#2      main (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:81:16)
   * 
   */
  }

  {
    Type t1 = new B("nn").runtimeType;
    print(">>2.1 ${t1}");
  
    ClassMirror cmirr1 = reflectClass(t1);
    Type t2 = cmirr1.reflectedType; // this is ok
    print(">>2.1 ${t2}");
  }

  {
    Type t1 = new List<B>().runtimeType; // return List<B>
    print(">>3.1 t1: ${t1}");
  
    ClassMirror cmirr1 = reflectClass(t1);
    Type t2 = cmirr1.runtimeType; // return _LocalClassMirrorImpl
    print(">>3.2 t2: ${t2}"); 
    print(">>3.3 cmirr1.hasReflectedType: ${cmirr1.hasReflectedType}"); 
    
    Type t4 = cmirr1.reflectedType; // here it should be t4 == t0, but it throws exception
    print(">>3.3 t4: ${t4}");
  }

  {
    ClassMirror cmirr = reflectClass(A);
    cmirr.getters.forEach((k, MethodMirror md){
      Type tm = (md.returnType as ClassMirror).reflectedType; // this is OK
      print(">>4.1 k: ${k}, tm: ${tm}");
    });
  }
}

main() {
  test_generics();
}

GenericType spec in test code

It is always useful if we have a good unit tests to understand the language spec in details. It is much easier to understand from concrete examples than hight level abstract and often ambiguous description.

For dart mirror, the code API is not useful, and there are not much detailed explanation of mirror features, so I checked   /dart/tests/lib/mirrors/ folder.

following test code is useful to understand mirror for generic classes.

// Copyright (c) 2013, the Dart project authors.  Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

library test.type_arguments_test;

import 'dart:mirrors';

import 'package:expect/expect.dart';

class A<T> {}
class B extends A {}            // Same as class B extends A<dynamic>.
class C extends A<num, int> {}  // Same as class C extends A<dynamic>.
class D extends A<int> {}
class E<S> extends A<S> {}
class F<R> extends A<int> {}
class G {}
class H<A,B,C> {}

typeParameters(mirror, parameterNames) {
  Expect.listEquals(parameterNames.map((n) => new Symbol(n)).toList(),
                    mirror.typeVariables.keys.toList());
}

typeArguments(mirror, argumentMirrors) {
  Expect.listEquals(argumentMirrors,
                    mirror.typeArguments.values.toList());
  if (!mirror.isOriginalDeclaration) {
    Expect.listEquals(mirror.typeVariables.keys.toList(),
                      mirror.typeArguments.keys.toList());
  }
}

main() {
  // Declarations.
  typeParameters(reflectClass(A), ['T']);
  typeParameters(reflectClass(B), []);
  typeParameters(reflectClass(C), []);
  typeParameters(reflectClass(D), []);
  typeParameters(reflectClass(E), ['S']);
  typeParameters(reflectClass(F), ['R']);
  typeParameters(reflectClass(G), []);

  typeArguments(reflectClass(A), []);
  typeArguments(reflectClass(B), []);
  typeArguments(reflectClass(C), []);
  typeArguments(reflectClass(D), []);
  typeArguments(reflectClass(E), []);
  typeArguments(reflectClass(F), []);
  typeArguments(reflectClass(G), []);

  Expect.isTrue(reflectClass(A).isOriginalDeclaration);
  Expect.isTrue(reflectClass(B).isOriginalDeclaration);
  Expect.isTrue(reflectClass(C).isOriginalDeclaration);
  Expect.isTrue(reflectClass(D).isOriginalDeclaration);
  Expect.isTrue(reflectClass(E).isOriginalDeclaration);
  Expect.isTrue(reflectClass(F).isOriginalDeclaration);
  Expect.isTrue(reflectClass(G).isOriginalDeclaration);

  Expect.equals(reflectClass(A), reflectClass(A).originalDeclaration);
  Expect.equals(reflectClass(B), reflectClass(B).originalDeclaration);
  Expect.equals(reflectClass(C), reflectClass(C).originalDeclaration);
  Expect.equals(reflectClass(D), reflectClass(D).originalDeclaration);
  Expect.equals(reflectClass(E), reflectClass(E).originalDeclaration);
  Expect.equals(reflectClass(F), reflectClass(F).originalDeclaration);
  Expect.equals(reflectClass(G), reflectClass(G).originalDeclaration);

  // Instantiations.
  typeParameters(reflect(new A<num>()).type, ['T']);
  typeParameters(reflect(new B<num>()).type, []);
  typeParameters(reflect(new C()).type, []);
  typeParameters(reflect(new D()).type, []);
  typeParameters(reflect(new E()).type, ['S']);
  typeParameters(reflect(new F<num>()).type, ['R']);
  typeParameters(reflect(new G()).type, []);
  typeParameters(reflect(new H()).type, ['A', 'B', 'C']);

  var numMirror = reflectClass(num);
  var dynamicMirror = currentMirrorSystem().dynamicType;
  typeArguments(reflect(new A<num>()).type, [numMirror]);
  typeArguments(reflect(new A<dynamic>()).type, [dynamicMirror]);
  typeArguments(reflect(new A()).type, [dynamicMirror]);
  typeArguments(reflect(new B()).type, []);
  typeArguments(reflect(new C()).type, []);
  typeArguments(reflect(new D()).type, []);
  typeArguments(reflect(new E<num>()).type, [numMirror]);
  typeArguments(reflect(new E<dynamic>()).type, [dynamicMirror]);
  typeArguments(reflect(new E()).type, [dynamicMirror]);
  typeArguments(reflect(new F<num>()).type, [numMirror]);
  typeArguments(reflect(new F<dynamic>()).type, [dynamicMirror]);
  typeArguments(reflect(new F()).type, [dynamicMirror]);
  typeArguments(reflect(new G()).type, []);
  typeArguments(reflect(new H<dynamic, num, dynamic>()).type,
      [dynamicMirror, numMirror, dynamicMirror]);

  Expect.isFalse(reflect(new A<num>()).type.isOriginalDeclaration);
  Expect.isTrue(reflect(new B()).type.isOriginalDeclaration);
  Expect.isTrue(reflect(new C()).type.isOriginalDeclaration);
  Expect.isTrue(reflect(new D()).type.isOriginalDeclaration);
  Expect.isFalse(reflect(new E<num>()).type.isOriginalDeclaration);
  Expect.isFalse(reflect(new F<num>()).type.isOriginalDeclaration);
  Expect.isTrue(reflect(new G()).type.isOriginalDeclaration);
  Expect.isFalse(reflect(new H()).type.isOriginalDeclaration);

  Expect.equals(reflectClass(A),
                reflect(new A<num>()).type.originalDeclaration);
  Expect.equals(reflectClass(B),
                reflect(new B()).type.originalDeclaration);
  Expect.equals(reflectClass(C),
                reflect(new C()).type.originalDeclaration);
  Expect.equals(reflectClass(D),
                reflect(new D()).type.originalDeclaration);
  Expect.equals(reflectClass(E),
                reflect(new E<num>()).type.originalDeclaration);
  Expect.equals(reflectClass(F),
                reflect(new F<num>()).type.originalDeclaration);
  Expect.equals(reflectClass(G),
                reflect(new G()).type.originalDeclaration);
  Expect.equals(reflectClass(H),
                reflect(new H()).type.originalDeclaration);

  Expect.notEquals(reflect(new A<num>()).type,
                   reflect(new A<num>()).type.originalDeclaration);
  Expect.equals(reflect(new B()).type,
                reflect(new B()).type.originalDeclaration);
  Expect.equals(reflect(new C()).type,
                reflect(new C()).type.originalDeclaration);
  Expect.equals(reflect(new D()).type,
                reflect(new D()).type.originalDeclaration);
  Expect.notEquals(reflect(new E<num>()).type,
                   reflect(new E<num>()).type.originalDeclaration);
  Expect.notEquals(reflect(new F<num>()).type,
                   reflect(new F<num>()).type.originalDeclaration);
  Expect.equals(reflect(new G()).type,
                reflect(new G()).type.originalDeclaration);
  Expect.notEquals(reflect(new H()).type,
                   reflect(new H()).type.originalDeclaration);

  // Library members are all uninstantaited generics or non-generics.
  currentMirrorSystem().libraries.values.forEach((libraryMirror) {
    libraryMirror.classes.values.forEach((classMirror) {
      // TODO(12282): Deal with generic typedefs.
      if (classMirror is! TypedefMirror) {
        Expect.isTrue(classMirror.isOriginalDeclaration);
        Expect.equals(classMirror, classMirror.originalDeclaration);
      }
    });
  });
}

Generic Class is not a first class citizen?

I tried to complete fromJson implementation, a method to instantiate an object of given type from its serialized json string, then I found some interesting limitation of Dart language.
I guess this is not planed feature, but something which are to be improved later.

for instance, the support of getClass in Java is not supported in dart now. (this will be obj.type getter). but I read a proposal, this is going to be supported.

there are two issues.

if we have a function taking Type value as the argument,  like f(A), this works as expected. but we cannot write f(A<B>)!.
also Type a = A<B> is not allowed.
So in a sense, generic class is not a first class citizen in Dart now.

Also similar, but more strange problem is following:

  InstanceMirror immir = reflect(new List<B>());
  Type t = immir.type.reflectedType;
  print(">> ${t}");
  ClassMirror mirr = reflectClass(t);
  print("2>> ${mirr.reflectedType}");

Since we cannot use List<B>, we need to create this type by reflect(new List<B>()).type.reflectedType. and this will be shown as List<B>.
but if we use this type to create ClassMirror, the getter call, mirr.reflectedType will cause runtime error.

>> List<B>
Unhandled exception:
Unsupported operation: Declarations of generics have no reflected type
#0      _LocalClassMirrorImpl.reflectedType (dart:mirrors-patch/mirrors_impl.dart:304:7)
#1      test_generics (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:77:21)
#2      main (file:///opt/dart-workspace/reflective_web_dev_kit/sample_app/bin/sample_json_mapper_test.dart:82:16)

This is actually not good for portable_mirror librray.
Since creating Type through dart:mirrors library is not allows in javascript mode.

For example, couchdb_client_dao(in browser side) need to convert json string of list to list of entity. So ideally, this should be parsed by jsonMapper's fromJson. and it requires type value in the first argument, e.g, fromJson(List<modelType>, json).

This issue would not be limited to javascript mode. but it would also affect to the Dart VM mode.
So in order to avoid this problem in DartVM mode(as well as JavaScript mode), it will be necessary to avoid using Type. Instead, it should always use IClassMirror.

Although that may fix the jsonMapper, but this issue may influence in entire application
So one approach is not to worry about the generic type case(including list/map), and wait until this mirror issue is fixed..