JILP Special Issue for MICRO32/FDO99

 

Reviewer name: Scott McFarling

Title of paper: Comparing and Combining Profiles

 

Confidence in your evaluation (0-3): 2

 

 

Please give your written review.  = Please state specific comments and

suggestions to be communicated to the author(s).

 

%%% Comments on organization:

fine.

 

%%% Comments on clarity of explanation, including figures & = tables:

 

%%% Comments on methodology:

see below.

 

%%% Comments on describing contributions = clearly:

see below.  Some more = motivation for the results that don’t require reading Section 3 would be = helpful.

 

%%% Comments on citations:

 

%%% Other Comments:

 

 

 

This is an interesting paper.  The existing ways of combining profile information have been = around for quite a while, and it is surprising that there is a better way to do = it.

 

The main problem I have with this paper is I don’t = understand why it is better.

 

Let’s look at the three existing methods.  Depending on your assumptions, each of these seem to = be “best”.

 

  1. Polling.  If you want to have = profiles from several benchmarks and you want to predict branches the best way = on the largest number of future benchmarks, then polling seems = “best”.
  2. Unscaled.  If you believe all the = benchmarks taken together fairly represents how the program will be used, = then just adding up the counts seems to be = “best”.
  3. Scaled.  If you think each = benchmark taken individually is a good sample of how the program will be used, but = the execution counts are just a meaningless function of the input = size, then scaling the counts before adding them seems to be = “best”.

 

So the questions I have are:

  1. Am I wrong about the above methods being best?
  2. If so, can you explain why your method is better? 
  3. If not, is there some set of assumptions of how the benchmarks relate to the = expected universe of real use that I am missing?
  4. Can you explain this at a high level for readers not well versed in = information theory?

 

The regret numbers on page 21 are very interesting. 

 

Here’s another view of Table 7 (assuming I copied it = correctly):

 

 

 

ref

 

 

 

test

 

 

 

train

 

 

 

benchmark average

 

scaled

KL

delta

 

scaled

KL

delta

 

scaled

KL

delta

 

scaled

KL

delta

go

76.83%

76.69%

-0.14%

 

76.83%

76.59%

-0.24%

 

75.04%

75.04%

0.00%

 

76.23%

76.11%

-0.13%

m88ksim

92.06%

92.00%

-0.06%

 

92.04%

92.88%

0.84%

 

91.90%

91.92%

0.02%

 

92.00%

92.27%

0.27%

gcc

88.86%

88.89%

0.03%

 

88.66%

88.67%

0.01%

 

87.20%

87.20%

0.00%

 

88.24%

88.25%

0.01%

compress

73.00%

73.05%

0.05%

 

91.18%

88.73%

-2.45%

 

80.13%

82.53%

2.40%

 

81.44%

81.44%

0.00%

li

80.15%

81.36%

1.21%

 

82.82%

84.86%

2.04%

 

86.18%

87.32%

1.14%

 

83.05%

84.51%

1.46%

ijpeg

89.08%

89.08%

0.00%

 

88.94%

88.93%

-0.01%

 

89.14%

89.50%

0.36%

 

89.05%

89.17%

0.12%

perl

91.26%

90.58%

-0.68%

 

70.52%

71.07%

0.55%

 

90.84%

91.12%

0.28%

 

84.21%

84.26%

0.05%

vortex

98.51%

98.50%

-0.01%

 

98.00%

97.98%

-0.02%

 

97.78%

97.78%

0.00%

 

98.10%

98.09%

-0.01%

average

86.22%

86.27%

0.05%

 

86.12%

86.21%

0.09%

 

87.28%

87.80%

0.52%

 

86.54%

86.76%

0.22%

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

average of averages

 

 

 

 

 

 

 

 

 

 

 

 

 

86.54%<= o:p>

86.76%

0.22%

 

 

 

 

 

 

 

 

 

A couple points: