README.HACKING 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342
  1. Hacking on Tor Browser Build
  2. ============================
  3. This file tries to list the main things to know when making changes to
  4. the Tor Browser build.
  5. rbm documentation
  6. -----------------
  7. If you go to directory rbm/doc, you can find the rbm documentation. You
  8. can build it with 'make html' or 'make man'.
  9. Using and defining options
  10. --------------------------
  11. Options can be used in templates with the following syntax:
  12. [% c("option_name") %]
  13. More details about the templates syntax can be found in
  14. rbm/doc/rbm_templates.7 and [http://template-toolkit.org/docs/manual/index.html].
  15. Some options have a specific meaning to rbm. You can see their descriptions
  16. in rbm/doc/rbm_config.7 and rbm/doc/rbm_input_files.7.
  17. When the option does not have a specific meaning to rbm, it is a good
  18. idea to define it under var/ to avoid potential conflicts in option names
  19. with a future version of rbm.
  20. The options can be defined in different places:
  21. - rbm.conf for options available to all components
  22. - project/$project/config for options available to one component
  23. - rbm.local.conf for user defined options
  24. In each of those places, an option can be defined:
  25. - at the root of the document
  26. - under targets/$target/, in which case the definition only applies when
  27. a specific target is defined
  28. The targets are usually used to select:
  29. - the platform: torbrowser-linux-x86_64, torbrowser-linux-i686,
  30. torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
  31. torbrowser-android-armv7, torbrowser-android-x86
  32. - the channel: release, nightly, alpha
  33. The targets torbrowser-linux-x86_64, torbrowser-linux-i686,
  34. torbrowser-windows-i686, torbrowser-windows-x86_64, torbrowser-osx-x86_64,
  35. torbrowser-android-armv7, and torbrowser-android-x86 are special cases. They do
  36. not contain options directly, instead they contain a list of other targets. For
  37. instance, the torbrowser-linux-x86_64 target is pointing to the linux-x86_64 and
  38. linux targets. You should define an option under the linux target if it applies
  39. to Linux on both architectures, or under the linux-x86_64 if it only applies to
  40. the x86_64 architecture.
  41. An option that is defined at the root of rbm.conf can be overridden by
  42. an other definition under a target, or inside projects/$project/config.
  43. You can find the complete priority order in rbm/doc/rbm_config.7.
  44. Defining a project's filename
  45. -----------------------------
  46. The filename option defines the output file or directory from the build
  47. of a component. If the file or directory exists in the out/$component
  48. directory, the component will not be rebuilt when it is used as a
  49. dependency from an other component.
  50. The filename is usually something like this:
  51. filename: '[% project %]-[% c("version") %]-[% c("var/osname") %]-[% c("var/build_id") %].tar.gz'
  52. The var/build_id value is a hash that is based on the build inputs of
  53. the component:
  54. - the build script (after template evaluation)
  55. - the container definition
  56. - the input files (the filename when it is a dependency on an other
  57. project, the filename and hash of its content otherwise)
  58. This means that each time the build script, the container or one of the
  59. dependencies is modified, the output filename will change and a rebuild
  60. of the component will be required.
  61. The version and var/osname values could be removed from the filename,
  62. but they are useful to get an idea of what a file contains.
  63. Adding some Linux/Windows/OSX specific changes to a build script
  64. ----------------------------------------------------------------
  65. You can use the following template syntax in the build scripts:
  66. [% IF c("var/linux") -%]
  67. # do something for linux
  68. [% ELSIF c("var/windows") -%]
  69. # do something for windows
  70. [% ELSIF c("var/osx") -%]
  71. # do something for osx
  72. [% END -%]
  73. You can also use var/linux-x86_64 and var/linux-i686 for things that
  74. only apply to x86_64 and i686 linux builds. You can use the var/release,
  75. var/alpha and var/nightly options to do things depending on the channel.
  76. As an alternative you can define an option with a different value
  77. depending on the target:
  78. targets:
  79. linux:
  80. var:
  81. do_something: 'do something for linux'
  82. windows-i686:
  83. var:
  84. do_something: 'do something for windows'
  85. osx-x86_64:
  86. var:
  87. do_something: 'do something for osx'
  88. And in the build script, use:
  89. [% c("var/do_something") %]
  90. Evaluating a component's build script
  91. -------------------------------------
  92. If you want to look at the build script used to build a component for a
  93. specific platform/channel, you can use the following command:
  94. $ ./rbm/rbm showconf $component build --target $channel --target $platform
  95. $component should be replaced by the name of the component.
  96. $channel should be one of the following:
  97. - release
  98. - alpha
  99. - nightly
  100. $platform should be one of the following:
  101. - torbrowser-linux-x86_64
  102. - torbrowser-linux-i686
  103. - torbrowser-windows-i686
  104. - torbrowser-windows-x86_64
  105. - torbrowser-osx-x86_64
  106. - torbrowser-android-armv7
  107. - torbrowser-android-x86
  108. For example, to see tor's build script for linux x86_64 on the alpha
  109. channel, you can use:
  110. $ ./rbm/rbm showconf tor build --target alpha --target \
  111. torbrowser-linux-x86_64
  112. If the component you are looking at has many dependencies, the display
  113. can take some time as various build_id values need to be computed. If
  114. you don't care about the accuracy of input and output file names, you
  115. can add '--target no_build_id' to the command line. For instance, if you
  116. want to look at the build script for the tor-browser component (which
  117. has a lot of dependencies), you can use:
  118. $ ./rbm/rbm showconf tor build --target alpha --target \
  119. torbrowser-linux-x86_64 --target no_build_id
  120. The same type of commands can be used to look at any option values,
  121. replacing build with the name of the option you want to look at. For
  122. instance, if you want to know the output filename of tor on linux-x86_64
  123. on the alpha channel, you can use:
  124. $ ./rbm/rbm showconf tor filename --target alpha --target \
  125. torbrowser-linux-x86_64
  126. Building a single component
  127. ---------------------------
  128. When you are working on some changes to a component, before creating a
  129. full bundle with those changes, you might want to try a build of the
  130. component you modified only.
  131. This can be done with the following command:
  132. $ ./rbm/rbm build $component --target $channel --target $platform
  133. See the previous section "Evaluating a component's build script" for a
  134. list of possible values for $channel and $platform.
  135. For instance, if you want to build tor for linux-x86_64 on the alpha
  136. channel, you can run:
  137. $ ./rbm/rbm build tor --target alpha --target torbrowser-linux-x86_64
  138. To find the resulting file from the build, you can use 'ls -ltr out/tor/'
  139. to find the file with the last modification time.
  140. Patching Firefox (or an other component)
  141. ----------------------------------------
  142. If you want to test a Firefox patch, the easiest way to do it is to
  143. copy the patch file to the projects/firefox/ directory, then edit
  144. projects/firefox/config to add the new patch to the list of input_files:
  145. - filename: patch-for-XXXX.patch
  146. Then edit projects/firefox/build to add a line somewhere (probably just
  147. before running the configure script) to apply the patch:
  148. patch -p1 < $rootdir/patch-for-XXXX.patch
  149. You can now run 'make testbuild' (or an other build target) to start a
  150. build with the patch.
  151. As an alternative, if you have your patch in a git repository, you can
  152. edit projects/firefox/config to change the git_url option to point to
  153. your git repository, and change the git_hash option to point to the
  154. commit you want to build. If the new git_hash option is not pointing to
  155. a signed tag, you will also need to comment the 'tag_gpg_id: 1' line.
  156. The git_hash option can point to a tag, a commit, a branch or anything
  157. that is understood by git.
  158. If you specify a branch in the git_hash option, you don't need to prefix
  159. it with a git remote name as all branches from the git repository
  160. selected in the git_url option are created as local branches.
  161. If you want to work on a local git repository, it is not a good idea to
  162. work on the git_clones/firefox repository directly. Instead you should
  163. create a separate git repository and update the git_url option with a
  164. file:// URL pointing to your repository.
  165. After changing the git_url option or when new commits have been added
  166. to a branch, you should run "make fetch" before starting a build to get
  167. the new commits. If you want to fetch new commits automatically when
  168. starting a new build you can add a "fetch: 1" line to
  169. projects/firefox/config.
  170. Remember that the git_hash option has different definitions for alpha
  171. and nightly builds, so you should modify the right one depending on
  172. what type of build you are doing (see also the "Types of builds" section).
  173. The first alternative is not working in case patches contain binary diffs,
  174. which the patch command can't handle. However, working with submodules can
  175. make the git branch alternative tricky to use. In that case you can resort
  176. to a third option by installing git in the build container (via var/deps in
  177. the respective config file). Then you can change to the submodule directory
  178. in the build script and do a
  179. git apply patch-for-XXXX.patch
  180. after adding the patch to the project's config file (see above).
  181. The Firefox mozconfig files
  182. ---------------------------
  183. In the gitian build, we are using the mozconfig files included in the
  184. tor-browser.git repository (the .mozconfig, .mozconfig-mac and
  185. .mozconfig-mingw files).
  186. In the rbm build however, we need to make some small modifications to
  187. those files, so we are instead using mozconfig files stored in the
  188. projects/firefox/ directory, ignoring the .mozconfig files from the
  189. tor-browser.git repository.
  190. This could change in the future, when we are not using Gitian anymore.
  191. Debugging a build error
  192. -----------------------
  193. If the build fails, a debugging shell will automatically be opened in
  194. the container used for the build (unless you set debug to 0 in
  195. rbm.local.conf).
  196. Before using the debugging shell, you can first look at the build logs
  197. in an other terminal (the filename of the log file is indicated in the
  198. rbm output).
  199. In the debug shell, you can type 'bash' to get a shell with completion.
  200. You can then look at the 'build' file, and copy paste some of its commands
  201. to debug the problem. If you need to edit files, vim is not installed
  202. in the build containers, but vi can be used instead.
  203. When you press ctrl+d (twice if you opened a bash shell) the debug shell
  204. is closed and the container containing the failed build is removed.
  205. The path to the container should be printed on the screen in case you
  206. want to backup its rootfs to be able to look at it later.
  207. Testing an rbm patch
  208. --------------------
  209. When you are working on a patch to rbm, you might want to try a Tor
  210. Browser build using your patched version of rbm. You could patch the
  211. rbm in the rbm/ directory, however your patch can be reverted if you
  212. use any of the makefile rules that does a 'git submodule update'.
  213. To avoid this you can clone the rbm git repository to a separate
  214. directory, where you will apply your patch. To do a build using your
  215. patched rbm, take the command from the makefile, but replace $(rbm)
  216. with the path to your patched rbm.
  217. For example, if you want to try a Linux x86_64 alpha build, you can run:
  218. $ /path_to_rbm/rbm build release --target alpha --target \
  219. torbrowser-linux-x86_64
  220. Types of builds: nightly, alpha, release, and testbuild
  221. -------------------------------------------------------
  222. The testbuild makefile target allows you to do a build quickly in the
  223. testbuild directory, skipping the generation of all the locales and the
  224. .mar files. This is useful during development.
  225. In the case of Android builds, we are generating a multi-locale apk,
  226. contrary to the desktop builds where we have one bundle for each locale.
  227. Removing locales in a multi-locale bundle does not make a significant
  228. difference in build time, therefore we still include all the locales in
  229. the Android testbuild. There are also no .mar files generated in the
  230. Android builds, so currently, in the Android case, the only difference
  231. introduced by the testbuild target is the output directory.
  232. By default the testbuild is based on the alpha build. All the options
  233. can have a different definition for the alpha, release and nightly builds.
  234. Usually the git_hash option has a different definition for the nightly
  235. builds in order to point to the master branch.
  236. If you want your testbuild target to create builds based on nightly
  237. rather than alpha, you can edit your rbm.local.conf file and uncomment
  238. the targets/torbrowser-testbuild definition.