raptor/notes/incremental_parsing.txt

Incremental Parsing
--------------------

Incremental parsing is a feature where Raptor can avoid reparsing bld.infs,
mmps etc (generally known as metadata files) when it determines that they
have not changed.  This means that the makefiles that were generated in a
previous build may be reused.

The primary benefit of this feature is to reduce the test-build-run cycle
in situations where the same build is being repeated many times without
changes to metadata files, e.g. repeatedly changing a C++ source file and
recompiling the project.

Example:
	sbs -b bld.inf --ip=on

Old makefiles are only reused if they were aimed at building exactly the
same components, for exactly the same platforms and in the same environment.
The incremental parsing feature is currently sensitive to the PATH variable
only but it may be desirable to reparse metadata in other situations also.

At the moment, the decision to re-parse is "all-or-nothing" - i.e. a change to any 
single bld.inf or mmp or any of their dependencies will result in a complete 
re-parse of all of them.  A more fine-grained approach may become possible in future.

This feature is new and somewhat experimental - if you suspect that something
should have been rebuilt but hasn't while you are using the feature then 
simply remove the option from your invocation of sbs and rebuild.

Are Two Builds the Same?
------------------------

The judgement about whether a set of makefiles can be reused is made on two criteria:

  1) The environment: currently only the "EPOCROOT" and "PATH"
  environment variables are compared between two builds - if these differ
  then the builds are not equivalent. This attempts to ensure that 
  two builds are building against the same SDK with the same tools but 
  since not all tools have to be on the path there is room for some
  degree of error e.g. if you upgraded tools or the SDK itself between
  two builds.

  2) The commandline: if the commandline for two builds is different then
  the makefiles are considered different.  This *very* conservative but
  that seems right with a build system.  There are 3 exceptions.
      i) The "--ip=on" command itself is not considered to count as a 
         difference
     ii) The "clean" target is ignored.  This means that building code and 
         then cleaning it should not require the regeneration of makefiles.
    iii) The -f (logfilename) option and its argument are ignored.

  e.g. the makefiles generated for:

       sbs -b bld.inf -c armv5

  ...will be reused if this command follows:

       sbs -b bld.inf --ip=on -c armv5 clean


Build Records
--------------
A new file is created in every build with the name #MAKEFILE#.buildrecord.
This file records of the names of the makefiles involved in a build and
the corresponding commandline that was given to raptor when it created
them.

Raptor uses these records to determine if the current build can reuse
existing makefiles.  If the current build attempt's commandline and
"environment" match those of an existing record then the makefiles
referenced by the record can be reused.

Build record files are written in JSON format.

Performance and Timing Data from Build Records
-----------------------------------------------
The Build Record (as of 2.17.4) also records the start time, end time and
running time of the build in the JSON "timing" element.  The start and end
times are decimal fractions of "seconds" from Python's time.time() method
and the run time (end-start) is similarly a decimal fraction in seconds.

This provides an easy way to collect performance data over a number of builds
- particularly from users who have been experiencing problems with slow builds


Example Demonstrating incremental Parsing Use Cases
----------------------------------------------------
Step 1)
With a clean epocroot the user types:
	sbs -b bld.inf --ip=on

What happens: bld.infs and mmps are parsed, makefiles are generated, the
source code is built

Step 2)
Next the user changes a source file and issues the same build
instruction:
	sbs -b bld.inf --ip=on

What happens: bld.infs and mmps are NOT parsed, step 1 makefiles are reused,
the changed source code is rebuilt.

Step 3)
The user adds a new mmp into their bld.inf file and issues the same build
command:

What happens: bld.infs and mmps are parsed, new makefiles are created,
the new code is built.

Step 4)
The user decides to build only for arm.v5.urel.rcvt4_0

	sbs -b bld.inf --ip=on -c arm.v5.urel.rvct4_0

What happens: bld.infs and mmps are parsed because the configuration has
changed even if arm.v5.urel.rvct4_0 is only a subset of the former builds,
new makefiles are created, nothing is built since no code changed.