Buck: Trigger pack_war invocation when one of transitive deps changed Buck version was upgraded in If336ea8697. This version contains behavioral change for genrule()'s rules. Since: [1] the caching behavior was switched to input-based keys. So that the command in genrule() is executed only when one of the input keys has changed. This massive change in behavior broke Gerrit build toolchain in very subtle way: when gerrit.war was built once, it wasn't rebuilt any more unless caches were invalidated. To understand why this is the case, some background understanding needed how gerrit.war is packaged in our build toolchain. Gathering of gerrit.war is done by genrule() that executes python script that retrieves the needed libraries by issuing buck audit classpath. However, since: [1] this python script is not invoked any more, when some libraries in transitive dependency chain have changed, because only very few libraries specified as the input to this python rule and the rest of the dependencies is retrieved in dynamic way in the script itself. So when some source code was changed, and corresponding library, say gerrit-server was changed as well, the first order dependency to the python script, like gerrit-pgm wasn't changed, and thus its rule key remained stable, so that pack_war.py wasn't invoked any more, leaving gerrit.war stale. The reproducer for this problem can be found here: [2]. The upstream issue is here: [3]. The fix is straight forward: instead of passing only the first order dependencies to the war packager script (that cannot reliably be done after changed caching behavior of genrule() in: [1]) and issue buck audit classpath in python script, pass the whole classpath from Buck build file to war packager. For one this will fix the caching issue, as the rule key now reflects the whole classpath and thus would change if some deps in transitive dependency chain would change, for another we wouldn't have to retrieve the classpath twice: in Buck build itself and in war packager script. [1] https://github.com/facebook/buck/commit/143646f4ccd8737f73a2667a244c4c38dc18cb89 [2] https://github.com/davido/buck_genrule_changed_caching_behaviour_143646f [3] https://github.com/facebook/buck/issues/470 Test plan: * buck build gitiles * change some sources in gitiles-servlet project * buck build gitiles * verify that gitiles.war was rebuilt in previous step and reflect the changes made in gitiles-servlet project Reported-By: Doug Kelly <dougk.ff7@gmail.com> Change-Id: I5e22e2d46c82fa5071db34762bda1655d6d353bf
diff --git a/tools/pack_war.py b/tools/pack_war.py index cfd7963..c4ffacc 100755 --- a/tools/pack_war.py +++ b/tools/pack_war.py
@@ -18,7 +18,6 @@ from os import getcwd, chdir, makedirs, path, symlink from subprocess import check_call import sys -from util import check_output opts = OptionParser() opts.add_option('-o', help='path to write WAR to') @@ -31,12 +30,14 @@ root = war[:war.index('buck-out')] jars = set() +def prune(l): + return [j[j.find('buck-out'):] for e in l for j in e.split(':')] + def link_jars(libs, directory): makedirs(directory) while not path.isfile('.buckconfig'): chdir('..') - cp = check_output(['buck', 'audit', 'classpath'] + libs) - for j in cp.strip().splitlines(): + for j in libs: if j not in jars: jars.add(j) n = path.basename(j) @@ -45,9 +46,9 @@ symlink(path.join(root, j), path.join(directory, n)) if args.lib: - link_jars(args.lib, path.join(war, 'WEB-INF', 'lib')) + link_jars(prune(args.lib), path.join(war, 'WEB-INF', 'lib')) if args.pgmlib: - link_jars(args.pgmlib, path.join(war, 'WEB-INF', 'pgm-lib')) + link_jars(prune(args.pgmlib), path.join(war, 'WEB-INF', 'pgm-lib')) try: for s in ctx: check_call(['unzip', '-q', '-d', war, s])
diff --git a/war.bucklet b/war.bucklet index 0ac346a..adc0072 100644 --- a/war.bucklet +++ b/war.bucklet
@@ -35,20 +35,17 @@ ): cmd = ['$(exe //bucklets/tools:pack_war)', '-o', '$OUT', '--tmp', '$TMP'] for l in libs: - cmd.extend(['--lib', l]) + cmd.extend(['--lib', '$(classpath %s)' % l]) for l in pgmlibs: - cmd.extend(['--pgmlib', l]) + cmd.extend(['--pgmlib', '$(classpath %s)' % l]) - dep = [] if context: for t in context: - dep.append(t) cmd.append('$(location %s)' % t) genrule( name = name, cmd = ' '.join(cmd), - deps = libs + pgmlibs + dep + ['//bucklets/tools:pack_war'], out = name + '.war', visibility = visibility, )