Wednesday, September 11, 2013

Dart's 'Future'

Overall, Dart language is very powerful and the language nature is very familiar to me as long time functional language advocates.(actually mixture of functional and object oriented).

Essentially it is very powerful to abstract procedure.
Often other none functional programming languages are very weak at abstracting procedure.
Since refactoring common process can not be done easily, and it ends up writing repetitive procedure again and again.
Because of this, Dart program are tend to become very short compared to corresponding code in Java.
short, concise, abstract code is the trait of functional programming.(Thus the title of this blog, 'Zen')


Shortness is just one aspect of it, but more important aspect is to change the way to think of problem.
Also functional style will clarify the basic aspect of asynchronous programs.

the Future class of Dart is in this respect very important.
The basic  idea is to allow to write a sequential steps of asynchronous call  in sequential way.
implicitly Future is hiding some event based trigger mechanism inside the class, because of that, the code using Future looks just a sequence of functions.

Future<A> fu = f();
fu
.then((A a)=>b())
..then((B b1)=>c())
...
.then((e){...})
.then.((i)=>i);

These 'then' method (of Future class) creates another Future(of different parametric type determined by the return type of the function passed to the then method).
So essentially with 'then' future takes a function and create another future .
this is asynchronous version of function combination function from two function.

fun then('X->'Y f, 'Y-'Z g): 'X->'Z =>('X x)=>g(f(x)); // pseudo code
Although Dart does not have this kind of type scheme(see ML)) or generic type specification, but it is achieving similar things.

but in case of Future, it is not ordinarily sequence of function calls, but it determines the sequence of function calls triggered by internal events.
if all functions in the parameter are written without asynchronous call, these functions are invoked immediately in a cascade way. One after another function call, the return value is used to trigger next future.
But if there is a asynchronous call in the function body, then we can control the  triggering timing using completer.
Future<A> fu = f();
fu
.then((A a)=>{
  Completer<B> c = new Completer<B>();
  B b = new  B();
...
  stream.open(..., onDone: {
     c.complete(b); // this allows to controls when the next function(f1) is to be triggered.
  });
  return c.future; // the action must return a future which is triggered by the complete invocation above.
  }
)
..then(f1);

Prototype web app using Dart M5

I just completed porting of a sample app DartExpenseServer project associated with chap 14 of 'Dart in Action' to Dart M5.
Although I started this as porting effort to Dart M5,  I ended up rewriting whole code.
The reason was the original sample app was not fully functional and not well designed as a realistic database application.

DartExpenseServer_v1.tar.gz

This sample project will be useful to start writing realistic web app.
It demonstrates basic idioms to write client/server side web application.
Namely the basic patterns of HttpRequest in client side, and HttpServer/HttpClient in server side as well as new bidirectional asynchronous WebSocket based communication method.
Also this covers basic CRUD operations over CouchDB through HttpClient in Dart. The adaptor class was made into a generic class so that it can be used to access different entity classes.

  • supporting CRUD operation in browser client app.
    Original sample did not support delete operation. and Also the entity id handling was unrealistic hack. This was fixed by using CouchDB's id generation feature.
  • supporting websocket which allows to push an event to multiple connected clients as well as asynchronous calling(by sending message) to server.
    this feature is used to updating the number of currently connected clients to all clients from server, and synchronizing list item changes to other browsers (web clients).
    this is actually using HttpClient to update the change to server, and server use websocket to the changes to all other browsers.

    But this type of communication mechanism is not almighty. sometime we want to receive message from server after sending message. For this purpose, we can use HttpClient as explained in next item.
  • HTTPClient usage for create/update/delete/read operations.
    The original Sample code was using persistent id created in a client side and used websocket to communicate with server for saving an object in database. this may be OK  to simplify code, but it is not realistic solution.
    If we want to insert(save) a new row in  a database, that db may generates the id, and we need the id in the client side.
    In such a case, websocket based message passing is not effective way. Since the websocket listener in client side has no knowledge over which object was saved in database. it may just receive this id, and it needs to find the  saved object in client side memory. but unless client side maintains local ids for the temporarily transient objects, it will be impossible to find such object.
    In this case, we should use HttpClient instead of websocket.
  • CouchDB access using server side HttpClient, this supports CRUD operations.
    there are not much good source of information for how to use CouchAPI from http level, I needed to several trials and errors and need to find some trick to achieve CRUD operation.
  • supporting both Dartulium and JS execution. (JS was not working before)
    Since Dart has a bug for handling Element.html, I need to find workaround to have the same result on both clients.
  • Since there were many layer and several problems, debugging were not easy.In particular, logging info by print function from client and server side cleared other side of output, so essentially there are very few info remained the terminal.
    So I switched to use logging library of Dart. This is also mandatory practice in Dart app.
Todo list
There are still a few things to be done even this level of app.

  • some common library dart file should be properly packaged.
  • Although this need not be done for this sample app, but it will be interesting to use Polymer for making GUI. This will be an another project.
  • CouchDB is a bit special DB, it is sort of cloud type DB. It is interesting to replace this with PostgresSQL. there is a driver for Dart now. 
  • When we write a code using websocket, code for sending message/receiving messages are splitted in client and server side. and it will become difficult to maintain  them  consistent.
    Since Dart support same language in both server/client, the nature is better than other approach using totally separate language on each sides. For instance, these websocket message names should be defined as constants in shared files. There may be more approach for this.

Since there are so many change in Dart, this application needed a lot of change.

  • Big change of HttpRequest, HttpClient etc.
  • difficulty to understand/use CouchDB through REst API(associated with choice for put/post/get, and handling of responseText etc) 
  • bugs in Dart(setter bug, Element.html bug) and some design  pattern of modeling and websocket.)
  • no logging lib feature original code.
Some of these issues came from the obviously early state of language development.

Interesting aspects of Dart is the new way to write asynchronous program.
Future is quite powerful idiom.
Also the new Stream library in M5 support more seems less use of Future.
If you compare the original DartExpenseServer code with the new Stream based code, you will see the difference.
In new version we don't need to use Completer class(almost).

I will describe these issues in separate posts.

Dart Element.html bug

I found a bug in dart2js for handling Element.html.
when we run an app using this feature, Dartium can run properly, but js version, the  result is not correct.
It expects to show items in a table, but they are not shown in the js version.
I thought this comes from a problem handling Element.html, so I replaced it with Element.tag, and it worked on both ways.
In fact, Element.html looks too heavy way to construct page elements, so there might be a better way, (not go the other end of using tag constructors), but such difference are very annoying problem if we decide to use Dart.
So this problem should be fixed.

this is  the projects to test this problem.

I created simple test projects.
here is a snipe of code what cause the problem and how we can avoid the problem..
------------------------
// this is buggy version

  refreshUi(List<Expense> expenses) {
    _updateRootElement();

    Element tableElement = new Element.tag("table");
    Element head = new Element.html("""
          <thead>
            <td class="type">Type</td>
            <td class="date">Date</td>
            <td class="detail">Item</td>
            <td class="amount">Amount</td>
            <!--<td class="claimed>Claimed?</td>-->
            <td class="edit">&nbsp;</td>
          </thead>""");

     tableElement.nodes.add(head);

     for (Expense ex in expenses) {
       tableElement.nodes.add(_getRowElement(ex));
     }

    rootElement.nodes.add(tableElement);
  }
=====>
// this is a workaround.

  refreshUi(List<Expense> expenses) {
    _updateRootElement();

    Element tableElement = new Element.tag("table");
    rootElement.nodes.add(tableElement);
    Element thead = new Element.tag("thead");
    thead.nodes.addAll([
        newTd('type', 'Type'),
        newTd('date', 'Date'),
        newTd('detail', 'Item'),
        newTd('amount', 'Amount'),
        newTd('edit', ''),
        newTd('delete', '')
        ]);
 
    tableElement.nodes.add(thead);
    for (Expense ex in expenses) {
      tableElement.nodes.add(_getRowElement(ex));
    }
  }
Element newTd(cls, txt) {
  Element td = new Element.td();
  td.attributes['class'] = cls;
  td.text = txt;
  return td;
}


Tuesday, September 10, 2013

Dart Compiler bug

There is a strange behavior in Dart's setter declaration.
It should detect an error at compilation time(dart2js) or editing time using Eclipse dart plugin (or Dart Editor), but it is only detected at runtime.

let see following class definition.
In the class A, rev0 is defined without parameter type specification.
but only possible type allowed 'void' since the type of expression '_rev = value' must be equal to the setter's return type 'void', and that assignment expression's type is the type of value. so teh value type must be void.
This already quite ridiculous situation for setter  since we cannot declare type of a variable as void!
but aside from this issue, if we call a function in which  the return type is 'void', it works as (3).
and the null value's type seems void! anyway null is assigned to _rev.

So these things does not make sense, and practically, it introduce an error which can be detected at compilation time, but defered at the runtime.
Since this is so wird, people may not easily spot the bug in the setter definition!
As a workaround, we should always define type for the setter parameter(no dynamic also).

String _rev = null;
class A {
  String get id => _id;
  void set id(String i) { _id = i; }
  String _id = null;

  String _rev = null;
  String get rev => _rev;
  void set rev(String value) { _rev = value; }// (0) normal way..

  // void a; // compilation error

  void f(_) {}
  void set rev0(value) => _rev = value; // buggy, but no compile time error
  void set rev00(dynamic value) => _rev = value; // buggy, but no compile time error
  void set rev1(String value) => f(value); // (1) this is OK
  void set rev2(String value) => f(_rev = value); //(2)
  // void set rev3(String value) => _rev = value;  // compilation error
  String rev3(String value) => (_rev = value);

  factory A.fromJson(String json) {
    var map = parse(json);
    return new A.fromMap(map);
  }
....
}
main() {
    A e = new A.fromJson('{"id":null, "_rev":"NNNNN"}');
    log.info("1===> ${e.toJson()} ${e.rev}");
    e.id = e.rev = "ooo";
    log.info("2===>  ${e.toJson()} ${e.rev}");
    e.rev0 = e.f(1); // works!! but wird.. (3)
    log.info("3===>  ${e.toJson()} ${e.rev}");
    e.rev1 = "aa"; // works!! but wird..
    log.info("4===> ${e.toJson()} ${e.rev}");
    e.rev2 = "aa"; // works!! but wird..
    log.info("5===> ${e.toJson()} ${e.rev}");

    e.rev0 = "aa"; // runtime error!!

}
}

----------------------
The following is the execution logs.
when 'e.rev0 = "aa";' is called, it causes runtime error.
this error message "type 'String' is not a subtype of type 'void' of 'function result'." is also not easy to understand, but the reason is this rev0 is expecting void argument type!

dirt_server.log [INFO]: 1===>  {"id":null,"_rev":"NNNNN"} NNNNN
dirt_server.log [INFO]: 2===>  {"id":"ooo","_rev":"ooo"} ooo
dirt_server.log [INFO]: 3===>  {"id":"ooo"} null
dirt_server.log [INFO]: 4===>  {"id":"ooo"} null
dirt_server.log [INFO]: 5===>  {"id":"ooo", "_rev":"aa"} aa

Unhandled exception:
type 'String' is not a subtype of type 'void' of 'function result'.
#0      A.rev0= (file:///opt/A.dart:28:28)
#1      main (file:///opt/A.dart:138:34)

-----------------


Wednesday, September 4, 2013

language changes



https://www.dartlang.org/articles/m1-language-changes/


http://news.dartlang.org/2013_02_01_archive.html


check the IO changes, there are huge changes.

 We have done a full re-design of the dart:io API (file, network, directories, etc) to use the classes and concepts in the core library and in dart:async. This means that there is almost no callback registration in dart:io any more. All async operations now use streams and futures.

(These changes are available in bleeding_edge as of today, and should show up in a weekly build soon.)





We think this is a great step forward in aligning all of the “dart:” API's and for you to have fewer concepts to learn when using Dart for different tasks.

However this is a breaking change which requires rewriting code using dart:io. The good news is that most of these changes should be mechanical. The following sections describe these changes in more details and give some examples on how to migrate code to the new API. For more information take a look on the API documentation for dart:io.

Streams in dart:io
The classes InputStream and OutputStream are gone and have been replaced with classes implementing IOSink and Stream<List<int>>.

When reading from a Stream<List<int>> just use the listen method which have arguments matching the callbacks previously used. This shows how to migrate code using an InputStream to the streams-based API:

dart:io v1 code
InputStream stream = ...
stream.onData = () {
 var data = request.inputStream.read();
 /* Process data. */
};
stream.onClosed = () {
 /* All data received. */
};
stream.onError = (e) {
 /* Error on input. */
};

dart:io v2 code
Stream<List<int>> stream = ...
stream.listen(
 (data) { /* Process data. */ },
 onDone: () { /* All data received. */ },
 onError: (e) { /* Error on input. */ });

As the InputStream class is now gone so is the StringInputStream for turning a stream of bytes into strings and/or lines. Two new stream transformers StringDecoder and LineTransformer have been added to address this. The StringDecoder transforms a stream of List<int> to a stream of String and the LineTransformer transforms a stream of String to a new stream of String where each string is a line.

dart:io v1 code
InputStream stream = ...
StringInputStream stringStream = new StringInputStream(stream);
stringStream.onLine = () {
 String line = stringStream.readLine();
 /* Do something with line. */
};
stringStream.onClosed = () {
 /* No more lines */
};
stream.onError = (e) {
 /* Error on input. */
};

dart:io v2 code
Stream<<int>> stream = ...
stream
 .transform(new StringDecoder())
 .transform(new LineTransformer())
 .listen((String line) { /* Do something with line. */ },
         onDone: () { /* No more lines */ },
         onError: (e) { /* Error on input. */ });

The IOSink replaces OutputStream and this shows how to migrate code using OutputStream to use an IOSink:

dart:io v1 code
OutputStream stream = …
stream.write([72, 101, 108, 108, 111]);  // Hello
stream.writeString(", world!");
stream.close();

dart:io v2 code
IOSink sink = …
sink.add([72, 101, 108, 108, 111]);  // Hello
sink.addString(", world!");
sink.close();

The IOSink also allows you to pipe directly from a stream.

HTTP
The main changes to the HTTP server and client part of dart:io are the following:

* A new HttpServer listening for requests is created using the static method bind.
* An HttpServer is a stream of HttpRequests.
* The defaultRequestHandler setter and addRequestHandler method on HttpServer are gone.
* The HttpRequest and HttpClientResponse objects implement Stream<List<int>>.
* The HttpClientRequest and HttpResponse objects implement IOSink.

To create a new listening HTTP server use the static method bind which returns a future.

dart:io v1 code
HttpServer server = new HttpServer();
server.defaultRequestHandler = …
server.addRequestHandler(…);
server.listen(“127.0.0.1”, 8080);
// Now the HTTP server is bound and listening for requests.

dart:io v2 code
HttpServer.bind(“127.0.0.1”, 8080)
   .then((HttpServer server) {
     server.listen(
         (HttpRequest request) {
           // Handle the request.
         });
}

The request routing through defaultRequestHandler and addRequestHandler is gone and any routing to specific request handling methods should be added to the listen handling. The HttpResponse is available as the response property on the HttpRequest object.

For client side HTTP the HttpClient class still exists. The HttpClientConnection class is gone, and instead the request initiation methods get, post, open, etc. now returns a future for the HttpClientRequest object. The HttpClientRequest object has a response property which is a future for the HttpClientResponse. As a convenience the HttpClientRequest.close method also returns the future for the response. This shows how to migrate HTTP client code:

dart:io v1 code
HttpClient client = new HttpClient();
HttpClientConnection connection = client.get(...);
connection.onRequest = (HttpClientRequest request) {
 // Prepare the request.
 request.outputStream.close();
}
connection.onResponse = (HttpClientResponse response) {
 // Process the response.
}

dart:io v2 code
HttpClient client = new HttpClient();
client.get(...)
   .then((HttpClientRequest request) {
     // Prepare the request.
     return request.close();
   })
   .then((HttpClientResponse response) {
     // Process the response.
   });

Web Sockets
The web socket interface has been simplified and now uses the same class for web socket connections on both the client and the server. The WebSocket class is a stream of events.

On the server the web socket handling is implemented as a stream transformer. This transformer can transform a stream of HttpRequests into a stream of WebSockets.

dart:io v1 code
HttpServer server = new HttpServer();
server.listen(...);
WebSocketHandler handler = new WebSocketHandler();
handler.onOpen = (WebSocketConnection connection) {
 connection.onMessage = (Object message) {
   /* Handle message. */
 };
 connection.onClosed = (status, reason) {
   /* Handle closed. */
 };
};
server.defaultRequestHandler = handler.onRequest;

dart:io v2 code
HttpServer.bind(...).then((server) {
 server.transform(new WebSocketTransformer()).listen((WebSocket webSocket) {
   webSocket.listen((event) {
     if (event is MessageEvent) {
       /* Handle message. */
     } else if (event is CloseEvent) {
       /* Handle closed. */
     }
   });
 });

On the client connecting a web socket has become much simpler. Just use the WebSocket static method connect which returns a future for the web socket connection as shown here:

dart:io v1 code
HttpClient client = new HttpClient();
HttpClientConnection conn = client.openUrl(“https://127.0.0.1:8080”);
WebSocketClientConnection wsconn = new WebSocketClientConnection(conn);
wsconn.onMessage = (message) {
 /* Handle message. */
}
wsconn.onClosed = (status, reason) {
 /* Handle closed. */
};

dart:io v2 code

WebSocket.connect("ws://echo.websocket.org")
   .then((WebSocket webSocket) {
     webSocket.listen((message) {
         /* Handle message. */
       },
onDone: () {
         /* Handle closed. */
       });
   });

Process
The Process class uses Stream<List<int>> for stdout and stderr and IOSink for stdin. The exit code for the process is now available through the exitCode future.

dart:io v1 code
Process process = ...
process.stdout.onData = ...
process.stdout.onDone = ...
process.onExit = (exitCode) { /* do something with exitCode. */ }

dart:io v2 code
Process process = ...
process.stdout.listen(...);
p.exitCode.then((exitCode) { /* do something with exitCode. */ });

Likewise the types of the top level properties stdin, stdout and stderr have been changed. stdio is a Stream<List<int>> and stdout and stderr are IOSinks.

File and directory
Reading and writing a file also uses Stream<List<int>> and IOSink. To read a file change the use of openInputStream to openRead which returns a stream of type Stream<List<int>>. Likewise change the use of openOutputStream to openWrite which returns an IOSink.

The Directory.list function now returns a stream of FileSystemEntity objects. The FileSystemEntity is a superclass of both File and Directory. The previous DirectoryLister where callbacks were set is now gone.

dart:io v1 code
Directory dir = ...
DirectoryLister lister = dir.list();
lister.onDir = (Directory directory) { /* Do something with directory. */ };
lister.onFile = (File file) { /* Do something with file. */ };
lister.onDone = (bool complete) { /* Listing ended.*/ };
lister.onError = (error) { /* Listing error.*/ };


dart:io v2 code
Directory dir = ...
dir.list().listen(
   (FileSystemEntity fse) {
     if (fse is Directory) {
       Directory directory = fse;
       /* Do something with directory. */
     } else if (fse is File) {
       File file = fse;
       /* Do something with file. */
     }
   },
   onDone: () { /* Listing ended.*/ },
   onError: (error) { /* Listing error.*/ });

Sockets
The classes Socket and SecureSocket both implement Stream<List<int>> and IOSink as they support bidirectional communication. This replaces all the reading and writing previously provided through a combination of callbacks, read and write methods, and InputStream and OutputStream objects backed by sockets.

Connecting a socket now uses a static method returning a future instead of a callback.

dart:io v1 code
Socket socket = new Socket(host, port);
socket.onConnect = () { /* Socket connected. */ }

dart:io v2 code
Socket.connect(host, port).then((Socket socket) {
 /* Socket connected. */
};

The classes ServerSocket and SecureServerSocket now uses a static method returning a future when binding to an interface. A listening server socket delivers the socket connections as a stream instead of through the onConnection callback.

dart:io v1 code
ServerSocket socket = new ServerSocket(host, port);
socket.onConnection = (Socket clientSocket) {
 /* Do something with the clientSocket. */
}

dart:io v2 code
ServerSocket.bind(host, port).then((ServerSocket socket) {
 socket.listen((Socket clientSocket) {
   /* Do something with the clientSocket. */
 })
});

Raw sockets
In order to provide a low level socket API we introduced raw sockets. Raw sockets give you access to low-level socket events without giving you data in your hand (think of this as the events you get from epoll on a socket file descriptor on Linux systems). Based on the low-level events you can read out data from the socket and decide when to write more data to the socket. The high-level Socket and ServerSocket classes are built on top of the RawSocket and RawServerSocket classes. Check out the API documentation on the Raw* classes.



Mixing WebSocket and HttpRequest

http://stackoverflow.com/questions/15101858/how-do-i-register-multiple-handlers-for-a-httpserver-in-dart

// pseudo-code

var server = HttpServer;
server.register('/foo', someHandlerFunction);        // 1
server.register('/bar', someOtherHandlerFunction);   // 2
server.register('/ws', webSocketHandler);            // 3

===>
Recommended Approach

----------------
 
import 'dart:io';
import 'dart:async';

handleWebSocket(WebSocket webSocket) {
  webSocket.listen((event) {
    if (event is MessageEvent) {
      /* Handle message. */
    } else if (event is CloseEvent) {
      /* Handle closed. */
    }
  });
}

typedef bool ShouldTake(e);
typedef void RouteTo(Stream stream);
typedef void HandleEvent(e);

class TakeAndRoute<S, T> extends StreamEventTransformer<S, T> {
  ShouldTake shouldTake;
  RouteTo routeTo;
  StreamController controller = new StreamController();
  HandleEvent handler;

  TakeAndRoute(this.shouldTake, {this.routeTo, this.handler}) {
    if (routeTo != null) routeTo(controller.stream);
  }

  handleData(event, StreamSink sink) {
    print("handling");
    if (shouldTake(event)) {
      if (routeTo != null) {
        controller.add(event);
      }
      if (handler != null) {
        handler(event);
      }
    } else {
      sink.add(event);
    }
  }
}

main() {
  HttpServer.bind('127.0.0.1', 8888)
    .then((HttpServer server) {
      server
        .transform(new TakeAndRoute<HttpRequest, HttpRequest>(
          (req) => req.uri.path == '/ws',
          routeTo: (stream) => stream.transform(new WebSocketTransformer()).listen(handleWebSocket)))
        .transform(new TakeAndRoute<HttpRequest, HttpRequest>(
          (req) => req.uri.path == '/foo',
          handler: (req) {
            print('got foo');
            req.response.addString("foo");
            req.response.close();
          }))
        .listen((req) {
          print("got 404 for ${req.uri}");
          req.response.statusCode = 404;
          req.response.close();
        });
    });
}

Tuesday, September 3, 2013

ECMA workshop on Dart

There seems an activity to standardize Dart.
It is a good way to promote Dart VM on other browsers.

http://news.dartlang.org/2013/08/ecma-to-hold-workshop-on-dart.html

Considering the run-time performance of  Dart VM is now comparable to JVM, Dart may replace both language on client side JavaScript and server side Java.
That would be definitely possible future scenario, in particular Java has been used for such web related applications, and Dart is better positioned for such applications.