Nim/compiler/ic/design.rst

====================================
  Incremental Recompilations
====================================

We split the Nim compiler into a frontend and a backend.
The frontend produces a set of `.rod` files. Every `.nim` module
produces its own `.rod` file.

- The IR must be a faithful representation of the AST in memory.
- The backend can do its own caching but doesn't have to. In the
  current implementation the backend also caches its results.

Advantage of the "set of files" vs the previous global database:
- By construction, we either read from the `.rod` file or from the
  `.nim` file, there can be no inconsistency. There can also be no
  partial updates.
- No dependency to external packages (SQLite). SQLite simply is too
  slow and the old way of serialization was too slow too. We use a
  format designed for Nim and expect to base further tools on this
  file format.

References to external modules must be (moduleId, symId) pairs.
The symbol IDs are module specific. This way no global ID increment
mechanism needs to be implemented that we could get wrong. ModuleIds
are rod-file specific too.



Global state
------------

There is no global state.

Rod File Format
---------------

It's a simple binary file format. `rodfiles.nim` contains some details.


Backend
-------

Nim programmers have to come to enjoy whole-program dead code elimination,
by default. Since this is a "whole program" optimization, it does break
modularity. However, thanks to the packed AST representation we can perform
this global analysis without having to unpack anything. This is basically
a mark&sweep GC algorithm:

- Start with the top level statements. Every symbol that is referenced
  from a top level statement is not "dead" and needs to be compiled by
  the backend.
- Every symbol referenced from a referenced symbol also has to be
  compiled.

Caching logic: Only if the set of alive symbols is different from the
last run, the module has to be regenerated.