TypeScript Compiler : Documentation Output

Recently I have been developing my TypeScript UI project, which is hosted on CodePlex. CodePlex comes with a reasonable Documentation Wiki tab and so I have been trying to build documentation for all my classes, interfaces etc. both inline (i.e. in the code) and on the project site. However, manually converting JSDoc to Wiki Docs is slow, laborious and very hard to keep up-to-date. To add to this, my sister has agreed to translate much of the online documentation into German. This presents me with the issue of how to automate documentation generation so I can get on with coding, and how to guide her on what does and doesn’t need translating.

My solution to this problem is to modify the existing TypeScript compiler to add a “–documentation” option which will output documentation files for all the classes, interfaces and enumerables in a standard format (“.ts.wiki” files). I can then write a short C# program to parse these files and show my sister what needs translating.

This seems like it ought to be relatively simple, but this turns out to be horribly messy and tricky. This is mainly due to two big issues with the TS Compiler:

  1. There is a total lack of comments/documentation on how the compiler works or what any of the names mean
  2. It’s not designed (so far as I can tell) with a proper post-processor

I have been tackling these issues and have been making reasonable progress, so in this article I will begin to explain where I’ve got to and where it is heading.

Where I began

I began by looking at the TypeScript compiler and considering which of its existing outputs would be closest to what I wanted. The compiler in its original form has two main outputs:

  1. The JS files
  2. Declaration files

The JS files are the compiled code and not exactly useful as documentation, especially since it generally doesn’t contain the comments and isn’t the TypeScript code. I deduced then that whatever code produced the JS probably wasn’t going to help me in producing TS based documentation with JSDoc descriptions included (which are, of course, comments). Declarations files, however, contain TypeScript output, with or without comments, in a standard format and not including any of the actual code. So essentially, documentation but layout out in a different way.

To proceed I knew I would need to add a new option to the compiler and a new file format. By looking at the file names and a few bits of the code I worked out that “emitter“s are the things which use “walkers” to go down the symbol tree and emit the relevant output to a file. So I copied and pasted a version of the declarationEmitter.ts file and refactored till it was a “documentationEmitter”. Finally, by trawling the code I was able to duplicate the declarationEmitter lines of code and change them to documentationEmitter code thus adding –documentation as a compiler option and .ts.wiki as an output format.

Hacking the declarationEmitter

The next stage (and my current stage) is hacking the declarationEmitter code till it becomes a documentationEmitter. A challenge with this is that I’m not outputting actual code nor am I trying to output it to a single file nor in the order that the coee is in the script file (for instance I want to order function names alphabetically and separate functions, properties etc. into groups). This presents the post-processing issue. TS is designed to output as it parses which works fine for a compiler like this, but not for documentation. Documentation needs to be written to file out-of-order (with respect to the code) and in the full light of all relevant code around it. I have therefore, come up with a workaround to the lack of an obvious (if any) post-processor.

The declarationEmitter class contains a “close” method which is supposed to close the current emitters file output stream. I am going to use the documentationEmitter’s “close” method as my post-processor kick to emit the documentation (and emit to multiple files). The rest of the emitter code will build a documentation-block tree as an intermediate step between symbol-tree and documentation output. This means changing all the “emit” and callback methods so that instead of immediately writing to the output file, they are context aware and emit to the current documentation block (or create a new where block appropriate).

A documentation block will consist of the text for that block, what type of block it is (e.g. class block, module block, function description block, etc.), the block signature (e.g. public, private, public static, private static), a reference to the block’s parent documentation block and an array of the child documentation blocks. This will allow me to construct a tree of documentation where the text is ready it just needs piecing together in a different order (e.g. class title then what module (namespace)  it belongs to).

This re-coding should be simple, and conceptually it is, but in reality this is a laborious and tricky process. Some of the names of the emitter methods are obscure like “emitTypeNamesMember” (which so far as I can tell, emits the type information for a function, property, variable or something else e.g. number or { x: number; y:number }). It is not exactly clear for someone who doesn’t know what it does or what the exact contents of the symbols are. So at each stage I am left with the following steps:

  1. Hack it so it emits to the current documentation block but otherwise output remains the same
  2. Hack as much else of the code till it all compiles and produces some form of vague documentation-like output
  3. Deduce what incorrect documentation (either in content, format or both) came from where
  4. Work out what the hell it should have been, if it should have been there at all, and if appropriate, where in the documentation it is supposed to go.
  5. Go back and re-write the code to make it look right
  6. Repeat the above

Not the nicest way to develop since it gives me no real solid idea of how far I have gotten, how much work is left and leaves a lot of guessing (not least I have to mangle 1259 lines of code before it even compiles!)

Advice for others

If you want to do this sort of thing, good luck. It is difficult to get your head around and definitely time-consuming (unless you happen to know your way around a compiler so well that nothing is new to you!) Here’s some information that may help you:

  • Don’t get hung up in TypeScript Services or the Harness – they don’t really help you if you are trying to extend the compiler functionality.
    • Services and Harness are (so far as I can understand) wrappers for:
      • Supporting Node.js, Windows Script Host and web browser environments
      • Diagnostic services/information
  • TypeScript.ts contains the overriding control logic but doesn’t do any parsing etc. in itself – add methods here and link them up for things like calls to new emitters
  • AST – Abstract Symbol Tree – This is the breakdown of the TypeScript into symbols going down from Script (i.e. file level) through modules, classes, functions all the way to variables and their type specifiers along with comments.
    • You do not necessarily have to handle every possible type of symbol – just the ones you are interested in – you can add a general catch-all (that does nothing) for the rest
  • ASTWalker – A “walker” literally walks you down the symbol tree, symbol by symbol and you can request certain information about symbols as you go (e.g. directly related comments, symbol type, symbol name)
    • The walker has two callbacks, pre and post, pre happens just before it “walks over” the symbol, post happens just after it walks over the symbol
    • Pre and Post must return booleans:
      • returning false for pre (I think) makes the walker skip processing the symbol and its children (and you don’t get a post call)
    • Pre and Post pass you a symbol representation object which gives you everything you need about that symbol (though naming is obscure and beware not all properties are always there e.g. ASISymbol can be null)
    • Use GetASTWalkerFactory().walk(pre, post) to start walking down a tree – you can often use the same method for the pre/post callbacks with an extra parameter – see DeclarationEmitter.emitDocumentation for an example
  • Emitters – This uses a walker to walk the symbol tree and handle symbols it is interested in (e.g. declarations emitter only handles public or exported or declared symbols such as exported modules and class but not private variables or code within functions)
    • Emitters output to a particular document (file) specified when they are created (but you can create other new files within the emitter)
    • Emitters can be told to output to a single file but you may wish to ignore this
    • Emitters generally contain callbacks for each symbol type that process the symbol and then pass the essential information to emitter methods
    • Emitter methods actually write to the doc file
  • IOHost – This is global to the compiler to standardise IO to files (to make it work across Node.js, WSH, web-browser)
    • Because this is global you can use it from anywhere so you can use it to create files (there is only one instance per compiler (program) instance)
  • Process – Again this is global and has some very helpful methods for giving debug trace
    • process.stdout and process.stderr is accessible from anywhere – use .write (with “\r\n” for new lines) to emit debug info e.g.
      • “declarationEmitter.ts : Line 59 : Constructor called\r\n”
      • This si a good standard output that lets you trace back to the TS source line easily (don’t forget the \r\n or everything ends up on one line!!)
  • If you are looking for a particular bit of symbol information, think what it is compiled to, find where it gets compiled the main compiler code, copy paste what is there! It is the fasted way to work out how to get certain information. Also, look at what type of PullDecl is used – it affects what information is visible/accessible.

I hope this article helps someone with their attempts at hacking the TS compiler and I will hopefully be submitting my code to the TS CodePlex project at some stage in the future (if not, I’ll at least post the code online for others to use so check back here for updates or follow me on Twitter!).

TypeScript UI : Data Binding

I recently announced that I am developing Data Binding for TypeScript UI (to be included in version 1.0.2). This article will give some more detail about why this is a good idea and how I plan to implement Data Binding into TypeScript. Data Binding, for those who don’t know and for clarification, is a method of linking data in a source (typically a web server) to a UI control that the user sees. Data Binding involves a number of steps which can be thought of in this structure:

  1. Access
  2. Adaptation
  3. Binding
  4. Update

1. Data Access

Data Access is the process of requesting data from the data source or pushing data to the data source. In a web-based context, it has typically been a full page request or more recently AJAX requests. “HTML5” (to use the umbrella term) introduces Web Sockets – two-way streams from server to client – and these are proving particularly good for creating responsive apps. TypeScript UI will support two basic types of data accessor, two types of data and three data formats:

  1. AJAX : String data : XML or JSON
  2. Web Sockets : String/binary data : XML or JSON or Raw data

The data from data accessors gets passed to data adapters, which transform the code-readable data into human-readable data, or data which the UI control can understand.

2. Data Adaptation

Data Adaptation is the process of taking raw data from a data accessor and transforming it into human-readable data or data which a UI control can understand. Data Adapters also handle the reverse process. TypeScript UI Data Adapters will have two main functions: I2O and O2I – standing for Input to Output and Output to Input; the latter being the inverse process of the former. The main intention is for them to convert data from a data accessor to data that a UI control can understand. However, they could be used to adapt any form of data or variable arbitrarily; they will have no rigid link to the rest of the Data namespace i.e. Data Adapters will be usable as a standalone unit.

3. Data Binding

Data Binding is the magic of this whole chain. A Data Binding has two main components: the UI Control property and the Data Adapter/Accessor pair. The Data Binding links the inputs/outputs of the Data Adapter to the UI control property. When the server sends updated data, the binding will get the callback from the data accessor, process the data through the data adapter and subsequently update the UI control. Likewise, if the user changes the value of the control, the binding will send the new data to the server, via the Data Adapter/Data Accessor. That’s the simple understanding of what it is supposed to do; the implementation is rather more complex. TypeScript UI will actually implement a more flexible, powerful structure than just plain data binding. The reasoning behind the structure comes from the inclusion of Data Updaters, so I move onto that and come back to Data Binding implementation later.

4. Data Updaters

Data Update is the process of updating the UI control data from the data source or vice-versa. However, Data Update is handled mostly by the data binding, so what is a Data Updater? A Data Updater is code which organises periodic update of data (in whatever way the programmer decides). The primary aim is to allow, for example, periodic updates such as fetching new Tweets or messages from a data source. TypeScript UI will implement a simple Data Updater structure which contains an interval setting, the update method to call and three events:

  • OnUpdateInvoke – Occurs when the update method is invoked.
  • OnUpdateBegin – Occurs when the update of data actually begins.
  • OnUpdateEnd – Occurs when the update of data ends.

TypeScript UI Structure / Implementation

The structure is currently as mentioned in the paragraphs above. So to finish off the explanation from section 3 then. TypeScript UI will implement a structure which includes Binding Groups and Binding Collections (a collection is a list of groups). Each UI Control will have an instance of a Binding Collection to which all Binding Groups will be added. A Binding Group will contain the list of one or more Data Bindings in the group and an associated Data Updater that will handle updating all the bindings in that group when requested (or scheduled). The Data Updater may be omitted if no updater is wanted. The reason for this slightly more bloated structure is that it allows multiple data bindings to be updated at the same time, by the same updater without needing to store multiple references to an instance of an updater or data binding. This simplifies managing where bindings are created, held and destroyed and also allows more powerful update systems to be developed. The current working version of the full structure is shown below (image from the documentation on CodePlex):

TypeScript UI Documentation : Data Binding Diagram
TypeScript UI : Data Binding

Comments

If you have any comments specifically about TypeScript UI Data Binding, please either create a discussion on CodePlex or fill in the comment form below. All (mature) comments are welcome!

TypeScript UI Launched

On Saturday I launched TypeScript UI. This is a new web-based project which I have created. Summarised, it is the following:

TypeScript UI is a user-interface library written in TypeScript language. It is in Windows 8 Style and provides a powerful structure for creating HTML5/CSS3 web-based apps with an app/window-based design structure. It works in any HTML5 capable browser (subject to limitations of host PC) but also includes fallbacks for IE9 and Opera.

The project aims to make HTML5 app development much easier by treating the browser as a platform to build within. The library offers an app/window structure that allows you to create apps with multiple windows that all run within one page. The UI is built and handled entirely in TypeScript within the client browser which reduces server load and increases quality of animation and UI adaptability. The server is restored to its status as a data handler making servers more efficient and server side code less bloated.

TypeScript UI also aims to include some of the best features of UI libraries such as those found for Windows desktop. For example, future versions of the library will include Data Binding to allow more efficient design and data handling. The library will also provide default windows such as splash screens, login forms and other common app components. And since it is written in TypeScript, large-scale app development and maintenance is very easy.

Head over to the project on CodePlex and download the latest version! Alternitavely, check out the live Demo!

TypeScript UI Demo
TypeScript UI Demo app
TypeScript UI Demo DropDown
TypeScript UI Demo – DropDown box

1 bug in 5000 lines and it’s all thanks to…

Typescript.

TypeScript has to be one of the best new technologies that I am using this new year. After being offline for 7 days with my old Linux Laptop and just the latest Chrome browser, gEdit and the TypeScript compiler, I wrote over 5000 lines of code, testing it in just that one browser. With TypeScript picking up all my silly syntax errors and Chrome finishing off all the function/programmatic errors, the JavaScript that TypeScript helped me produce was next to flawless. I got home, tested it in all the major browsers (and some minor ones) and all I got, was one error in IE8 – even that barely counted as an error…

To clarify, the error I got in IE8 was simply the indexOf function not existing which, in a sense, is fair enough because IE8 is pre a lot of new features. (Though users who insist on not upgrading are more than a little irritating!) TypeScript and the TypeScript compiler really do make development faster, easier, much more robust and much easier to maintain.  Here’s just a few of the bets things is does: (I apologise now if this gets too close to a sales pitch for TypeScript, but it really has been that great to use!)

Static Typing

This has to be TypeScript’s most powerful tool – if it weren’t, what would be the point? I won’t discuss how to use it etc. since the TypeScript site does a good job of that but I will say that it works. Beautifully. Sure, when compared to properly typed languages such as C# it’s not quite as smooth and one or two of the jQuery definitions aren’t correct but it is smooth, easy and intuitive. There are other alternatives to TypeScript such as ones from Google, but as far as I can tell, you need to add definition comments  to everything. A messy and inconvenient solution. For every one line of code, I don’t want to have to write another that says what it was. TypeScript’s type definitions are inline using angle braces – a nice, simple solution that doesn’t impede my development at all. So a quick tip/trick then:

Got a variable that TypeScript says is of one type, but you know that it isn’t? E.g. a jQueryEventObject but you know really it is a MouseEvent? Well, there’s a nice easy hack to solve this but still keep TypeScript’s powerful type checking:

var event = <JQueryEventObject>{ };
var mouseEvent = <MouseEvent><any>event;

That simple cast to an any and then to the type you wanted solves the compiler issues and keeps TypeScript’s powerful static type checking later on in your code. Most of the time you probably won’t need this though – the provided function and type definitions are pretty accurate. I’ve only had slight issues with jQuery and had to extend the basic jQuery UI definitions file to include most of the basic of the UI widgets. My jQuery UI definitions file is downloadable for anyone to use, just click here (Note: File will be updated every so often – I will try and blog when I do so please follow me for updates).

Classes and Interfaces

JavaScript is a pain when it comes to developing large projects where you’ve got lots of UI controls and really you’d like a class for each with a common interface to handle them. To do this in JavaScript would be a nightmare to write manually, which is largely what the world has been stuck with. However, with TypeScript, life just became a whooollle lot simpler! Modules (namespaces), classes, interfaces, extension (inheritance) and implementation (implementing an interface) work just like Java as per standards but when you combine that with VS2012 intellisense and the TypeScript compiler, you get neat, efficient JavaScript that just works. And not just in IE, in any browser.

Declarations and Inferred Types

Okay so you might think I’m just listing TypeScript’s main selling points and perhaps I am but it’s only because it works – superbly. Declaration files allow you to take away that nasty native JavaScript from with your ts files and just declare what else will be included in your final HTML pages. And if you can’t be bothered to declare types for everything, the TypeScript compiler will infer the types based on first assignment – it will even infer types for native JavaScript code to help protect you! (or at least that’s what it seems to be doing 🙂 )

Overall Review

I’m sold on TypeScript. If I were in a big company now, this would be a must on the upgrades list. Microsoft have done a fantastic job with the compiler – it produces reasonably efficient, neat JavaScript which sticks to standards so well, that if it works in one browser, you can be certain it’ll work in all the major browsers (Firefox, Opera, Safari, Chrome, IE 8-10 and of course every Android or iPhone browser I have gotten my hands on). Not only that but when it comes to debugging it, the unminifed files are so perfectly laid out (retaining much of the layout of the original TypeScript), that I’ve had no issues with getting lost in the mountains of code that go into a complex UI.

A small project

A small example of just one part of a project that I am working on is shown below:

HTML5 (SVG) Gradient Picker using TypeScript
An HTML5 (SVG) gradient picker written using TypeScript.

At the top of the image you can see the gradient picker bar with the colour stops hanging below. The rest of the page is a large-scale, change-as-you-edit preview of the gradient that fills the whole page. This has compatibility with all the latest major browsers and also IE9 and mobile devices. Using SVG allows for the best compatibility while also using the least JavaScript processing time and is probably the easiest way to create linear and radial gradients that work across the web. Sadly, I can’t put the code up for the world to use but I can say that TypeScript made programming the whole thing much, much easier. I was able to extend (inherit) from an Event Manager class that I created a month ago to handle creating and firing custom onChange events etc. and easily use this jQuery ColorPicker plugin by adding to the jQuery declarations file. Even better, I am now able to use the gradient picker just like I would a UI control in C#. Just add a reference to the ts file, then create an instance of the control with its container div id! Very similar to jQuery, but using my own internal standard and written in TypeScript. If at any point I need to add to this, I can easily just inherit from my original code and then extend it as much as I like. Something that always feels a bit awkward in jQuery really…you don’t get proper extension really.

Converting to TypeScript

While converting my current JavaScript projects to TypeScript was a fairly laborious and somewhat slow process, I was able to dramatically improve the maintainability of the code. Since re-writing in TypeScript meant re-writing a lot of the code, I decided to do a full re-write from the ground up.  I should point at now that I didn’t have to re-write everything. TypeScript will allow you to put plain JavaScript in ts files and compile it – you just don’t get as many of the benefits of TypeScript.

Re-writing meant I could build in future-proofing, update from XML to JSON data format, improve stability and best of all – readability! If there is anything web developers hate JavaScript most for, it has to be that it is notoriously difficult to read. By the time you’ve finished adding in code to fix browser quirks and used all the little tricks you know to improve running speed, your JS code just looks messy. Even the TypeScript compiled JS code isn’t that nice to read. I think the best I’ve seen is jQuery but they do an extraordinary job of sticking to a neat layout – something a lot of developers just don’t do. (Not that I’m any better a lot of the time). But TypeScript has many of the same code formatting features as C#, which means your code looks good. I like the Visual Studio way of laying code out, and I love that it will now do it for TypeScript. Can it get any better? Well it certainly does. VS2012 will even allow you to collapse code blocks, comments and all the usual things you’d expect from a sane language. Unlike old versions of VS which didn’t really support JS or CSS, VS2012 does code collapse, decent syntax highlighting – the full works – for HTML, CSS, JS, TypeScript – you name it!  And with Web Essentials, CSS is even simpler too – auto add browser specific modifiers etc. – that’s a different discussion though. Ultimately, TypeScript and VS2012 are the reason I  can now open up my code and jump straight to where I want to be and read the code easily.

Conversion to TypeScript also allowed me to use proper inheritance. Duplicate copies of code is always bad. It’s even worse when you know you have duplicate copies. It becomes horrendous when you have to regularly make changes to the bit of code that you know is duplicated! I had this very same problem in my project. One bit of code, used in two places. In one place, it had it’s own “class” per se and worked fine. In the other, it was included within an extension class. Except that they were independent of each other. It wasn’t really an extension class, more of a duplication and addition. The JS was so complex, I found it impossible to get inheritance working let alone have just one copy of the code. From a maintenance point of view, this was awful. However, using TypeScript and its powerful module, class and interface capability, I was able to write just one class and then extend for my other class. One piece of code, in one place and any changes get applied in both places automatically – #wonderful.

Any projects anyone?

If you’ve got a project that you’re doing in TypeScript or you’d like a little help to get you started, please contact me! I’d love to hear about what others are doing and am always happy to help. Just visit my Contact page for my email address.

Have fun and happy coding!

(Please like/tweet if you liked this article or found it interesting/helpful!)