Journal Rankings in Economics:
Handle with Care
Handle with Care
Abstract
Nearly all journal rankings in economics use some weighted average of citations to
calculate a journal’s impact. These rankings are often used, formally or informally, to help
assess the publication success of individual economists or institutions. Although ranking
methods and opinions are legion, scant attention has been paid to the usefulness of any
ranking as representative of the many articles published in a journal. First, because the
distributions of citations across articles within a journal are seriously skewed, and the
skewness differs across journals, the appropriate measure of central tendency is the median
rather than the mean. Second, large shares of articles in the highest-ranked journals are
cited less frequently than typical articles in much-lower-ranked journals. Finally, a ranking
that uses the h-index is very similar to one that uses total citations, making it less than ideal
for assessing the typical impact of articles within a journal. (JEL A11)
JEL classification: A11
Keywords: Journal rankings
∗ Federal Reserve Bank of St. Louis, P.O. Box 442, St. Louis, MO 63166-0442 (mailto:wall@stls.frb.org).
The views expressed are those of the author and do not necessarily represent official positions of the Federal
Reserve Bank of St. Louis or the Federal Reserve System. I would like to thank Kristie Engemann, Cletus
Coughlin, and Rajdeep Sengupta for their comments and suggestions.
1
I. Introduction
Journal rankings are used to compare institutions and evaluate individual
economists by weighting the journals in which they publish their research (Kalaitzidakis,
Mamuneas, and Stengos, 2003; Dusansky and Vernon, 1998). Because a journal’s ranking
is seen as a good indicator of the research quality of individual papers in the journal many
institutions use rankings either formally or informally to evaluate job seekers, current staff,
etc. Rankings are used in this way because one typically has little more than the journal
name to use as an indicator of a paper’s quality.
For an older paper, citations provide a good measure of its impact and are readily
available from a number of sources. Even so, it is doubtful that evaluators find it worth
their time to collect the necessary data for the many papers and people that they evaluate.
For a more-recent paper, the citation record is probably not long enough to gauge impact,
unless the paper is extremely successful early on. Given this information lag, perhaps the
best (or at least most practical) predictor of future impact is the reputation of the journal in
which the paper is published. Nonetheless, the purpose of this paper is to suggest that
great care should be taken when using the a journal’s ranking is used as a predictor of the
future impact of a specific article.
Although there are many ranking methodologies, and rankings have been used to
make important decisions regarding grants, salary, tenure, promotion, and hiring, little
work has been done to assess how seriously any ranking should be taken. Error bands are
de rigueur in economics, but, somehow, providers of journal rankings have gotten away
with producing little more than point estimates. The average number of citations, however
calculated and adjusted, has been taken as a useful representation of the many articles
2
within a journal. But how confidently can we say that an article in journal A is better than
an article in journal B simply because the average article in A has tended to receive more
citations than the average article in B? This paper follows Oswald (2007) by comparing
the distributions of citations across the articles within journals, with the aim of helping
decisionmakers derive mental error bands around journal rankings. My updated analysis,
with five times as many journals as used by Oswald, should go much further in making the
case that these error bands should be fairly large.
As shown below, citation distributions for journals tend to be skewed heavily by a
small number of high performers. Further, there is significant overlap in citation
distributions across journals to the extent that large percentages of articles published in
low-ranked journals outperform the median of the four most-prestigious journals.
Similarly, large percentages of articles in the most prestigious journals underperform the
median of much-lower-ranked journals. Finally, I show that a ranking that uses the hindex
is very similar to a simple ranking according to total citations, making it unsuitable
as a predictor of the impact of individual articles.
II. The Data and a Simple Mean Ranking
I start with a list of 30 journals that correspond roughly to the top 30 or so journals
identified by Engemann and Wall (2009), which excludes non-refereed or invitation-only
journals (Journal of Economic Literature, Brookings Papers on Economic Activity, and
Journal of Economic Perspectives). I also exclude journals that are primarily finance
because a large percentage of their citations would come from journals outside of
economics and thus might not be suitable for the present analysis. The May Papers and
3
Proceedings issue of the American Economic Review (AER) is treated separately from the
rest of the journal because, as shown below, it is much less selective than the regular issues
of the AER. Almost any list of the top 30 journals includes most of these journals and any
differences are at the low end; however, for the purposes of this paper, it is not important
exactly which journals are in the list. In fact, as will be apparent, my points are
strengthened if I have included some journals that do not belong in the top 30.
I compiled all articles published in 2001 in these 30 journals and tallied the total
number of citations to each article between 2001 and 2008 from all articles included in the
Institute for Scientific Information (ISI) Web of Science database. Note that this excludes
proceedings, editorial material, book reviews, corrections, reviews, meeting abstracts,
biographical items, software reviews, letters, news items, and reprints as identified by the
database. Also note that the citations in 2008 are all those included in the database as of
the day that the data were collected: September 23, 2008.
Table 1 summarizes the citation records of articles in the 30 included journals
relative to the rest of economics. The ISI database lists 6,373 articles published in
economics in 2001, and the 30 included journals represent 1,543 articles, or 24 percent, of
the total. These journals received a disproportionate share of citations: Of the 49,670
citations recorded by the database, 22,805 of them, or 46 percent, are to the included
journals. Put another way, the average number of citations across all of economics was
7.8, the average across the included journals was 14.8, and the average across the excluded
journals was 5.6.1
1 Some additional curiosities: Nineteen percent of articles in economics received no citations at all, while 4
percent of articles in the included journals and 24 percent of articles in the excluded journals were not cited.
4
Journal-level summary statistics are provided in Table 2, where journals are ranked
in declining order of mean citations. This is an extremely simple ranking method in that it
addresses few of the usual measurement concerns. Nearly all of the journal rankings in
economics are based on a quality-weighted average of citations, often using a variant of the
iterative method of Liebowitz and Palmer (1984) and correcting for some combination of
self-citations, article age, and journal size. Also, following Palacios-Huerta and Volij
(2004), some rankings control for differences in reference intensity—that is, differences
across journals in the average number of references.2 The ranking in Table 2 controls only
for journal size and article age, but not for self-citations, the quality of the citing journal, or
reference intensity. Nonetheless, because the purpose of this paper is to shed light on the
properties of journal citations as raw inputs into the various ranking procedures, and not to
provide a journal ranking, a simple ranking is preferable. It is worth noting, however, that
this simple ranking is highly correlated with one that follows all of the best measurement
practices.3
There is nothing surprising about the identities of the journals with the highest
mean number of citations—Quarterly Journal of Economics (QJE), AER, Econometrica,
and Journal of Political Economy (JPE)—although the ranking among them and the
differences between them might be. Specifically, the average article in the QJE was cited
much more frequently (45 times) than were articles in the other three journals (32, 28, and
21 times, respectively). The relative mean impact, which is the number of cites a journal
2 To date, this correction has yet to be adopted widely. Kodrzycki and Yu (2006) have produced the only
other ranking that includes reference intensity, although Engemann and Wall (2009) control for it indirectly.
3 Specifically, the present ranking and that of Engemann and Wall (2009) have a Spearman rank correlation
coefficient of 0.75. In general, journals with exclusively or primarily theoretical papers, such as Review of
Economic Studies, International Economic Review, and Journal of Economic Theory, are the ones whose
ranking is improved the most when best practices are used.
5
received relative to the number of cites to the average article published in the AER, the
profession’s flagship journal, provides a useful number to compare journals to one another.
According to this measure, the average article in the QJE was cited 43 percent more
frequently than was the average article in the AER, whereas the average articles in
Econometrica and the JPE were cited, respectively, 87 and 65 percent as frequently. There
were large differences in citation frequency between the high and low ends of the list. On
average, articles in journals ranked 22nd to 30th were cited between 27 and 31 percent as
frequently as the average AER article. Also, there was a vast middle ranking of journals
(ranked 5th to 21st) with relative mean impacts between 34 and 54 percent.
III. Skewness and a Median Ranking
How well does the mean number of citations summarize the citation tendency of
the typical article within a journal? A look at the maximum and minimum citations
reported in Table 2 provides a hint that the answer to that question might be “not so well.”
The maxima across journals range from 30 to 433, and this top article usually accounted
for 10 percent or more of the total cites received by the journal. In fact, it is not
uncommon for the share of the most-cited article to have exceeded 20 percent of the total.
For example, 38 percent of all citations to the Oxford Economic Papers were to a single
article, while the analogous numbers are 35 percent for the Journal of Applied
Econometrics and 25 percent for the International Economic Review. This phenomenon is
true even for a journal as large and prestigious as the AER, for which 15 percent of its
citations were to just one of its 93 articles. Further, for most journals there were decent
numbers of articles that received no citations at all during the period. The journals with the
6
highest shares of such articles were Oxford Economic Papers (12 percent), International
Economic Review (11 percent), the Journal of Business and Economic Statistics (11
percent), and the Journal of Economic Theory (9 percent).
This quick examination of the high and low ends of citation distributions illustrates
the high variance and the positive skew of any journal’s citations: Citations to a journal are
heavily weighted to a few articles, and many articles are cited relatively infrequently.
Although all journals share these characteristics, there are large cross-journal differences in
variance (which measures the degree of atypicality of the average number of cites). The
five journals with the highest variance relative to mean are the AER, Journal of Applied
Econometrics, AER Papers and Proceedings, Oxford Economic Papers, and
Econometrica. At the other end are four journals with a relative variance below 10:
Journal of Human Resources; Journal of Risk and Uncertainty; Journal of Money, Credit,
and Banking; and Journal of Law and Economic Organization. Skewness, which
measures the extent to which the mean is driven by the high end of the distribution, also
differs a great deal across journals, and not just between high- and low-ranked journals.
For example, the citation distribution of the QJE is the least skewed, while that of the AER
is the most skewed.
The high relative variance and extreme skewness of journals’ citation distributions
indicate that the mean is not the appropriate measure of the central tendency of the
citations received by articles in a journal. The median, which would eliminate the large
effect that a single article can have on the mean, is more appropriate. Accordingly, in
Table 3 the journals are ranked according to their median number of cites.4 As one can
4 As a practical matter, rankings based on the median are almost necessarily outdated because one must wait
several years after an article is published to collect the article-level information, meaning that the ranking is
7
see, it turns out that the two rankings are not really that different from one another.5 There
are, however, some large movements in rank for some journals as we move from Table 2
to Table 3. For example, the Journal of Applied Econometrics is ranked 11 places lower in
the median ranking, while the Journal of Industrial Economics and the AER Papers and
Proceedings fall by seven and five places, respectively. The two journals that improve
their rankings the most are the Journal of Risk and Uncertainty and the Journal of Law and
Economic Organization, which rise by seven and five places, respectively.
Because it is an indicator of how different the journals are from one another,
perhaps the difference between the relative mean and median impacts is more interesting.
Recall that the relative mean impact indicated that the average article in the QJE was cited
43 percent more frequently than the average article in the AER. According to the relative
median impact, however, the distance between the QJE and the AER, as well as the rest of
the journals, is much greater: The median article in the QJE was cited more than twice as
often as the median article in the AER. The use of medians also tends to provide higher
relative impacts for the journals ranked below the AER. Although the journals ranked 3rd
to 30th had an average relative mean impact of 0.4, their average relative median impact
was 0.47. Thus, use of the median reveals that the differences between the AER and the
journals ranked below it are not as large as suggested when one uses the mean, although
the difference between the QJE and all other journals is revealed to be even greater.
based on articles published several years prior to when the ranking is done. In contrast, the information
demands of a mean-based ranking are much less restrictive because one can pool data for recent years. For
example, Engemann and Wall (2009) look at citations received during 2001-07 to articles published during
the same period. Mean-based rankings are, therefore, probably more practical than median-based ones.
Nonetheless, it is useful to know the differences between the two ranking methods so that one has an idea of
how seriously to take the mean-based rankings that will necessarily be done.
5 Their Spearman rank-correlation coefficient is 0.91.
8
IV. Overlapping Distributions
The previous two sections demonstrated the unsuitability of the mean as an
indicator of the central tendency of citations to articles in a journal and showed how the
relative ranking of journals might differ if the median were used instead. This section
moves from a discussion of how to best perform a ranking to one of how seriously to take
any citation-based ranking. Although the median or the mean might be appropriate for
ranking journals, a look at the ends of each journal’s citation distribution provides some
notion of how large one’s mental error bands should be when using such a ranking.
Notice from Table 2 that the most-cited article for every one of the 30 journals
received more citations than did the median article across the four top-ranked journals (19).
In other words, at least one article from each journal had a greater impact than did half of
the articles in the top four journals. So how many other such articles are there for each
journal? And, conversely, given that some articles from even the top journals received
very few or even zero citations, how many articles published in each journal were cited
less often than the median of the bottom four journals? To answer these and related
questions, I split the journals into four tiers: Tiers 1 and 4 are composed of the top and
bottom four journals, respectively. Tier 1 journals all have a relative median impact
greater than 0.7, and Tier 4 journals all have a relative median impact less than 0.3. The
rest of the journals are split into Tiers 2 and 3, which, respectively, have relative median
impacts between 0.5 and 0.7 and between 0.3 and 0.5. The assignment of journals to tiers
is shown in Table 3.
It is also possible to assign individual articles from all the journals to each of these
tiers. Specifically, I define a Tier 1 article as one that was cited at least 19 times (the
9
median article across Tier 1 journals), a Tier 2 article as one that was cited at least 10 times
but fewer than 19 times, a Tier 3 article as one that was cited at least 6 times but fewer than
10 times, and a Tier 4 article as one that was cited no more than 5 times (the median across
the bottom four journals). The shares of each of these four types of article published in
every journal are provided in Table 3.
The first things to note are that every journal has articles in every tier and that there
is a strong tendency for a journal’s ranking to be related to the shares of its articles in the
highest or lowest tiers. This relationship is far from monotonic, however. For example,
nearly 20 percent of the articles in Econometrica and 14 percent of the articles in the AER
were Tier 4 articles, although Tier 4 articles were extremely rare for the QJE, which had
only one. As one would expect, Tier 4 articles were more common in Tier 2 journals than
they were in Tier 1 journals and the frequency might come as a surprise: Thirty-eight
percent of the articles in the Journal of International Economics and 35 percent of the
articles in the Journal of Monetary Economics were Tier 4 articles. Conversely, Tier 1
articles were not uncommon at the low end of the journal ranking: About 12 percent of the
articles in Games and Economic Behavior (from Tier 4) and 27 percent of the articles in
the Journal of Industrial Economics (from Tier 3) were Tier 1 articles.
Figure 1 summarizes the overall tendencies of journals from any tier to have
articles across all four tiers. One lesson from the figure is that the tiers are fairly similar in
the extent to which their articles fall into the middle two tiers—between 33 and 44 percent.
Put another way, the big difference between the journal tiers is in the prevalence of Tier 1
and Tier 4 articles. For another perspective, note that Tier 1 journals publish substantial
numbers of papers at least two tiers below the journals, whereas Tier 4 journals publish
10
substantial numbers of papers at least two tiers above the journals. Specifically, Tier 3 and
4 articles made up 29 percent of the articles published in Tier 1 journals, while Tier 1 and 2
articles made up 27 percent of the articles published in Tier 4 journals.
V. The h-Index
Hirsch (2005), responding to the perceived need for a useful quantitative measure
of individual researchers’ cumulative impact on their fields of research, proposed a simple
index, h, which is defined as the number of papers with citations greater than or equal to h.
To find an individual’s h, one need only to order the person’s papers from the most-cited to
the least-cited. Going down the list, h is where the number of papers is greater than or
equal to the number of citations to the hth paper. Recently, Braun, Glänzel, and Schubert
(2006) proposed using article-level information to derive an h-index for journals. In sync
with the previously noted observations about the skewed distribution of citations, the
advantage of the h-index applied to journals is thought to be that a simple way to combine
both quality and quantity considerations while eliminating the effect that a small number of
articles might have on mean-based measures.
The h-index has become increasingly popular: The Web of Science now provides
h-indices for all journals in its database, and the calculations are dependent on the time
frame chosen by the user. Similarly, RePEc uses the h-index for an experimental journal
ranking, although its database is drawn primarily from so-called gray literature, such as
working papers, and a fairly small and unrepresentative sample of journals.
Because all articles in my data have roughly the same amount of time to
accumulate citations, my data are ideal for ranking journals by their h-index and comparing
11
it with alternatives.6 Keep in mind that because the h-index is a measure of the overall
impact of a journal, whereas the median ranking is meant to measure the impact of a
typical article in a journal, the two rankings are not comparable. The h-index ranking
should instead be compared to a ranking by total citations across all articles in a journal.
Put another way, the h-index can be used to answer a question such as, “What journals
have the greatest total impact in economics?” Median or mean rankings, on the other
hand, address the question, “What kind of an impact is typical for an article in journal X?”
There is no evidence that this important distinction is appreciated by either of the two
sources that use the h-index for economics journals. On the RePEc site, for example, the
h-index stands without qualifications alongside several mean-based rankings.
Table 4 contains the h-index for each of the 30 included journals, along with their
total citations, their total-citations rank, and the difference between the two ranking
methods. Note first how journal size affects the difference between the median rank from
Table 3 and the h-index rank. Most obviously, the AER, AER Papers and Proceedings, the
European Economic Review, and the Journal of Economic Theory (the four largest
journals) leapfrog over journals with substantially higher median or mean citations. Again,
the link between size and the h-index is deliberate in that the index was designed originally
to measure cumulative impact over a researcher’s career, thereby making it less than
desirable for assessing the potential impact of an individual article by looking at the
journal in which it is published.
As is apparent from Table 4, there is little difference between the h-index ranking
and the total-citations ranking. This is not necessarily surprising because they are both
meant to capture total impact. In fact, given that the Spearman rank-correlation coefficient
6 See Braun, Glänzel, and Schubert (2006).
12
for the two methods is 0.98, it does not appear to have been worth the effort to gather
article-level data and calculate the h-index for each journal.
VI. Conclusions
In and of themselves, the rankings in this paper are not intended for actual use
because they use only one year’s publications and do not correct or control for a number of
important factors. Nevertheless, several general tendencies revealed by the analysis should
be helpful in determining the amount of care to take when employing journal rankings.
First, the appropriate measure of central tendency is the median rather than the mean, a
correction that can lead to substantial changes in the ranking of individual journals, but
which yields a ranking that differs little overall. Second, large percentages of articles in
the highest-ranked journals are cited less frequently than are typical articles in muchlower-
ranked journals. Similarly, large percentages of articles in the lowest-ranked
journals are cited more frequently than are typical articles in much-higher-ranked journals.
Finally, a ranking that uses the h-index is very similar to one that uses total citations,
meaning that it is of little use for assessing the relative quality of research published in a
journal.
13
References
Braun, Tibor; Glänzel, Wolfgang and Schubert, András. “A Hirsch-Type Index for
Journals.” Scientometrics, April 2006, 69(1), pp. 169-73.
Dusansky, Richard and Vernon, Clayton J. “Rankings of U.S. Economics Departments.”
Journal of Economic Perspectives, Winter 1998, 12(1), pp. 157-70.
Engemann, Kristie M. and Wall, Howard J. “A Journal Ranking for the Ambitious
Economist.” Federal Reserve Bank of St. Louis Review, May/June 2009,
forthcoming.
Hirsch, Jorge E. “An Index to Quantify an Individual’s Scientific Research Output.”
Proceedings of the National Academy of Sciences, November 2005, 102(46), pp.
16569-72.
Kalaitzidakis, Pantelis; Mamuneas, Theofanis P. and Stengos, Thanasis. “Rankings of
Academic Journals and Institutions in Economics.” Journal of the European
Economic Association, December 2003, 1(6), pp. 1346-66.
Kodrzycki, Yolanda K. and Yu, Pingkang D. “New Approaches to Ranking Economics
Journals.” Contributions to Economic Analysis and Policy, 2006, 5(1), Article 24.
Liebowitz, Stanley J. and Palmer, John P. “Assessing the Relative Impacts of Economics
Journals.” Journal of Economic Literature, March 1984, 22(1), pp. 77-88.
Oswald, Andrew J. “An Examination of the Reliability of Prestigious Scholarly Journals:
Evidence and Implications for Decision-Makers.” Economica, February 2007,
74(293), pp. 21-31.
Palacios-Huerta, Ignacio and Volij, Oscar. “The Measurement of Intellectual Influence.”
Econometrica, May 2004, 72(3), pp. 963-77.
14
Table 1. Summary Citation Characteristics of Journals
Number
of
Articles
Number
of
Citations
Mean
Citations
Article
Share
Citation
Share
Percent
Never
Cited
All journals (230) 6,373 49,670 7.8 1.00 1.00 0.19
30 included journals 1,543 22,805 14.8 0.24 0.46 0.04
200 excluded journals 4,830 26,865 5.6 0.76 0.54 0.24
Data are for citations in 2001-2008 to articles published in 2001, according to the ISI Web of
Science® as of September 23, 2008.
15
Table 2. The Numbers and Distributions of Journal Citations
Mean
Rank Journal
Number of
Articles
Mean
Citations
Relative
Mean
Impact
Citation
Maximum
Max
Article’s
Share
Citation
Minimum
Share
with Zero
Citations
Relative
Variance Skewness
1 Quarterly Journal of Economics 42 45.4 1.43 136 0.07 1 0.00 20.5 0.82
2 American Economic Review 93 31.7 1.00 433 0.15 0 0.02 74.3 6.34
3 Econometrica 66 27.6 0.87 194 0.11 0 0.03 38.1 2.65
4 Journal of Political Economy 44 20.6 0.65 76 0.08 3 0.00 16.2 1.31
5 Journal of International Economics 61 17.1 0.54 91 0.09 0 0.02 21.6 2.04
6 Rand Journal of Economics 37 16.6 0.52 127 0.21 0 0.03 30.9 3.60
7 Journal of Economic Growth 14 16.4 0.52 46 0.20 1 0.00 11.3 0.92
8 Journal of Labor Economics 32 15.0 0.47 70 0.15 3 0.00 14.5 2.49
9 American Economic Review, Pap. and Proc. 84 14.7 0.46 194 0.16 0 0.02 39.5 5.43
10 Review of Economic Studies 38 13.8 0.44 93 0.18 1 0.00 19.8 3.37
11 Journal of Monetary Economics 51 13.6 0.43 54 0.08 0 0.02 11.7 1.37
12 Journal of Applied Econometrics 35 13.4 0.42 164 0.35 0 0.06 57.4 4.98
13 Journal of Econometrics 77 13.1 0.41 97 0.10 0 0.03 22.4 2.86
14 Review of Economics and Statistics 67 13.1 0.41 69 0.08 0 0.03 12.7 2.10
15 Journal of Industrial Economics 26 12.6 0.40 54 0.16 0 0.08 17.0 1.45
16 Journal of Business and Economic Statistics 45 12.0 0.38 54 0.10 0 0.11 16.6 2.03
17 European Economic Review 98 12.0 0.38 112 0.10 0 0.04 24.4 3.96
18 Journal of Law and Economic Organization 19 11.9 0.38 46 0.20 0 0.05 9.8 1.81
19 Journal of Risk and Uncertainty 27 11.3 0.36 39 0.13 1 0.00 8.1 1.29
20 Economic Journal 66 11.3 0.36 106 0.14 0 0.06 27.8 4.02
21 Journal of Law and Economics 31 10.6 0.34 42 0.13 0 0.03 10.5 2.03
22 Journal of Public Economics 75 9.9 0.31 56 0.08 0 0.07 10.1 2.14
23 Journal of Urban Economics 48 9.5 0.30 61 0.13 0 0.02 11.6 3.03
24 Games and Economic Behavior 59 9.3 0.29 61 0.11 0 0.07 15.8 2.48
25 Journal of Human Resources 30 9.1 0.29 30 0.11 0 0.03 7.3 1.08
26 International Economic Review 45 8.9 0.28 102 0.25 0 0.11 29.1 4.75
27 Journal of Development Economics 69 8.7 0.27 68 0.11 0 0.06 12.7 3.26
28 Oxford Economic Papers 33 8.5 0.27 105 0.38 0 0.12 38.9 4.98
29 Journal of Economic Theory 87 8.4 0.27 54 0.07 0 0.09 10.2 2.37
30 Journal of Money, Credit, and Banking 44 8.4 0.27 32 0.09 0 0.07 8.6 1.46
Citations are from 2001-2008 to articles published in 2001 according to the ISI Web of Science® as of September 23, 2008.
16
Table 3. Median Citations and Overlaps in Citation Distributions
Median
Rank* Journal
Median
Citations
Relative
Median
Impact
Tier 1
Share
Tier 2
Share
Tier 3
Share
Tier 4
Share
1 Quarterly Journal of Economics 36.5 2.15 0.786 0.143 0.048 0.024
2 American Economic Review 17 1.00 0.495 0.183 0.183 0.140
3 Econometrica 16.5 0.97 0.470 0.197 0.136 0.197
Tier
1
4 Journal of Political Economy 12.5 0.74 0.341 0.273 0.273 0.114
5 Journal of Economic Growth 11.5 0.68 0.429 0.143 0.214 0.214
6 Journal of International Economics 11 0.65 0.311 0.197 0.115 0.377
7 Journal of Labor Economics 10.5 0.62 0.219 0.344 0.219 0.219
8 Rand Journal of Economics 10 0.59 0.297 0.243 0.189 0.270
9 Review of Economic Studies 10 0.59 0.184 0.316 0.158 0.342
10 Journal of Monetary Economics 9 0.53 0.255 0.235 0.157 0.353
11 Review of Economics and Statistics 9 0.53 0.194 0.299 0.224 0.284
12 Journal of Risk and Uncertainty 9 0.53 0.222 0.222 0.222 0.333
Tier
2
13 Journal of Law and Economic Organization 9 0.53 0.158 0.263 0.263 0.316
14 American Economic Review, Pap. and Proc. 8 0.47 0.190 0.250 0.190 0.369
15 Journal of Business and Economic Statistics 7 0.41 0.178 0.222 0.244 0.356
16 Journal of Econometrics 7 0.41 0.182 0.195 0.247 0.377
17 Journal of Law and Economics 7 0.41 0.129 0.194 0.387 0.290
18 Journal of Public Economics 7 0.41 0.173 0.213 0.200 0.413
19 European Economic Review 7 0.41 0.173 0.245 0.153 0.429
20 Journal of Urban Economics 7 0.41 0.146 0.146 0.333 0.375
21 Journal of Human Resources 6.5 0.38 0.167 0.233 0.133 0.467
22 Journal of Industrial Economics 6 0.35 0.269 0.115 0.154 0.462
23 Journal of Applied Econometrics 6 0.35 0.229 0.143 0.143 0.486
24 Economic Journal 6 0.35 0.121 0.197 0.212 0.470
25 Journal of Economic Theory 6 0.35 0.115 0.195 0.207 0.483
Tier
3
26 Journal of Development Economics 6 0.35 0.101 0.217 0.203 0.478
27 Games and Economic Behavior 5 0.29 0.119 0.186 0.153 0.542
28 Journal of Money, Credit, and Banking 5 0.29 0.114 0.205 0.159 0.523
29 International Economic Review 5 0.29 0.111 0.111 0.200 0.578
Tier
4
30 Oxford Economic Papers 4 0.24 0.061 0.152 0.152 0.636
* The ratio of Tier 1 and Tier 4 shares is used as a tiebreaker.
17
Table 4. Rankings Based on Total Citations and the h-Index
Total
Citations
Rank by
Total
Citations h-index
h-Index
Rank*
Difference
in Rank**
American Economic Review 2,948 1 33 1 0
Quarterly Journal of Economics 1,907 2 28 2 0
Econometrica 1,822 3 23 3 0
Journal of International Economics 1,043 6 19 4 -2
American Economic Review, Pap. and Proc. 1,235 4 17 5 1
European Economic Review 1,176 5 17 6 1
Journal of Econometrics 1,009 7 17 7 0
Journal of Political Economy 906 8 16 8 0
Review of Economics and Statistics 878 9 16 9 0
Journal of Public Economics 743 11 16 10 -1
Journal of Monetary Economics 694 13 16 11 -2
Journal of Economic Theory 731 12 14 12 0
Economic Journal 746 10 13 13 3
Rand Journal of Economics 614 14 13 14 0
Review of Economic Studies 524 18 13 15 -3
Journal of Labor Economics 480 19 13 16 -3
Journal of Development Economics 600 15 12 17 2
Games and Economic Behavior 549 16 12 18 2
Journal of Business and Economic Statistics 540 17 12 19 2
Journal of Applied Econometrics 469 20 12 20 0
Journal of Urban Economics 456 21 11 21 0
Journal of Money, Credit, and Banking 370 23 11 22 -1
Journal of Risk and Uncertainty 305 26 11 23 -3
International Economic Review 401 22 10 24 2
Journal of Law and Economics 329 24 10 25 1
Journal of Industrial Economics 328 25 10 26 1
Journal of Human Resources 273 28 10 27 -1
Journal of Economic Growth 230 29 9 28 -1
Journal of Law and Economic Organization 226 30 9 29 -1
Oxford Economic Papers 281 27 8 30 3
* Ties are broken by total citations.
** The difference between the h-index rank and the total-citations rank.
Figure 1. Shares of Article Types Across Journal Tiers
0.51
0.20
0.16
0.13
0.25 0.26
0.18
0.31
0.16
0.21 0.21
0.42
0.10
0.17 0.17
0.56
0.00
0.10
0.20
0.30
0.40
0.50
0.60
Tier 1 Articles Tier 2 Articles Tier 3 Articles Tier 4 Articles
Shares of Article Types
Tier 1 Journals Tier 2 Journals Tier 3 Journals Tier 4 Journals
Nearly all journal rankings in economics use some weighted average of citations to
calculate a journal’s impact. These rankings are often used, formally or informally, to help
assess the publication success of individual economists or institutions. Although ranking
methods and opinions are legion, scant attention has been paid to the usefulness of any
ranking as representative of the many articles published in a journal. First, because the
distributions of citations across articles within a journal are seriously skewed, and the
skewness differs across journals, the appropriate measure of central tendency is the median
rather than the mean. Second, large shares of articles in the highest-ranked journals are
cited less frequently than typical articles in much-lower-ranked journals. Finally, a ranking
that uses the h-index is very similar to one that uses total citations, making it less than ideal
for assessing the typical impact of articles within a journal. (JEL A11)
JEL classification: A11
Keywords: Journal rankings
∗ Federal Reserve Bank of St. Louis, P.O. Box 442, St. Louis, MO 63166-0442 (mailto:wall@stls.frb.org).
The views expressed are those of the author and do not necessarily represent official positions of the Federal
Reserve Bank of St. Louis or the Federal Reserve System. I would like to thank Kristie Engemann, Cletus
Coughlin, and Rajdeep Sengupta for their comments and suggestions.
1
I. Introduction
Journal rankings are used to compare institutions and evaluate individual
economists by weighting the journals in which they publish their research (Kalaitzidakis,
Mamuneas, and Stengos, 2003; Dusansky and Vernon, 1998). Because a journal’s ranking
is seen as a good indicator of the research quality of individual papers in the journal many
institutions use rankings either formally or informally to evaluate job seekers, current staff,
etc. Rankings are used in this way because one typically has little more than the journal
name to use as an indicator of a paper’s quality.
For an older paper, citations provide a good measure of its impact and are readily
available from a number of sources. Even so, it is doubtful that evaluators find it worth
their time to collect the necessary data for the many papers and people that they evaluate.
For a more-recent paper, the citation record is probably not long enough to gauge impact,
unless the paper is extremely successful early on. Given this information lag, perhaps the
best (or at least most practical) predictor of future impact is the reputation of the journal in
which the paper is published. Nonetheless, the purpose of this paper is to suggest that
great care should be taken when using the a journal’s ranking is used as a predictor of the
future impact of a specific article.
Although there are many ranking methodologies, and rankings have been used to
make important decisions regarding grants, salary, tenure, promotion, and hiring, little
work has been done to assess how seriously any ranking should be taken. Error bands are
de rigueur in economics, but, somehow, providers of journal rankings have gotten away
with producing little more than point estimates. The average number of citations, however
calculated and adjusted, has been taken as a useful representation of the many articles
2
within a journal. But how confidently can we say that an article in journal A is better than
an article in journal B simply because the average article in A has tended to receive more
citations than the average article in B? This paper follows Oswald (2007) by comparing
the distributions of citations across the articles within journals, with the aim of helping
decisionmakers derive mental error bands around journal rankings. My updated analysis,
with five times as many journals as used by Oswald, should go much further in making the
case that these error bands should be fairly large.
As shown below, citation distributions for journals tend to be skewed heavily by a
small number of high performers. Further, there is significant overlap in citation
distributions across journals to the extent that large percentages of articles published in
low-ranked journals outperform the median of the four most-prestigious journals.
Similarly, large percentages of articles in the most prestigious journals underperform the
median of much-lower-ranked journals. Finally, I show that a ranking that uses the hindex
is very similar to a simple ranking according to total citations, making it unsuitable
as a predictor of the impact of individual articles.
II. The Data and a Simple Mean Ranking
I start with a list of 30 journals that correspond roughly to the top 30 or so journals
identified by Engemann and Wall (2009), which excludes non-refereed or invitation-only
journals (Journal of Economic Literature, Brookings Papers on Economic Activity, and
Journal of Economic Perspectives). I also exclude journals that are primarily finance
because a large percentage of their citations would come from journals outside of
economics and thus might not be suitable for the present analysis. The May Papers and
3
Proceedings issue of the American Economic Review (AER) is treated separately from the
rest of the journal because, as shown below, it is much less selective than the regular issues
of the AER. Almost any list of the top 30 journals includes most of these journals and any
differences are at the low end; however, for the purposes of this paper, it is not important
exactly which journals are in the list. In fact, as will be apparent, my points are
strengthened if I have included some journals that do not belong in the top 30.
I compiled all articles published in 2001 in these 30 journals and tallied the total
number of citations to each article between 2001 and 2008 from all articles included in the
Institute for Scientific Information (ISI) Web of Science database. Note that this excludes
proceedings, editorial material, book reviews, corrections, reviews, meeting abstracts,
biographical items, software reviews, letters, news items, and reprints as identified by the
database. Also note that the citations in 2008 are all those included in the database as of
the day that the data were collected: September 23, 2008.
Table 1 summarizes the citation records of articles in the 30 included journals
relative to the rest of economics. The ISI database lists 6,373 articles published in
economics in 2001, and the 30 included journals represent 1,543 articles, or 24 percent, of
the total. These journals received a disproportionate share of citations: Of the 49,670
citations recorded by the database, 22,805 of them, or 46 percent, are to the included
journals. Put another way, the average number of citations across all of economics was
7.8, the average across the included journals was 14.8, and the average across the excluded
journals was 5.6.1
1 Some additional curiosities: Nineteen percent of articles in economics received no citations at all, while 4
percent of articles in the included journals and 24 percent of articles in the excluded journals were not cited.
4
Journal-level summary statistics are provided in Table 2, where journals are ranked
in declining order of mean citations. This is an extremely simple ranking method in that it
addresses few of the usual measurement concerns. Nearly all of the journal rankings in
economics are based on a quality-weighted average of citations, often using a variant of the
iterative method of Liebowitz and Palmer (1984) and correcting for some combination of
self-citations, article age, and journal size. Also, following Palacios-Huerta and Volij
(2004), some rankings control for differences in reference intensity—that is, differences
across journals in the average number of references.2 The ranking in Table 2 controls only
for journal size and article age, but not for self-citations, the quality of the citing journal, or
reference intensity. Nonetheless, because the purpose of this paper is to shed light on the
properties of journal citations as raw inputs into the various ranking procedures, and not to
provide a journal ranking, a simple ranking is preferable. It is worth noting, however, that
this simple ranking is highly correlated with one that follows all of the best measurement
practices.3
There is nothing surprising about the identities of the journals with the highest
mean number of citations—Quarterly Journal of Economics (QJE), AER, Econometrica,
and Journal of Political Economy (JPE)—although the ranking among them and the
differences between them might be. Specifically, the average article in the QJE was cited
much more frequently (45 times) than were articles in the other three journals (32, 28, and
21 times, respectively). The relative mean impact, which is the number of cites a journal
2 To date, this correction has yet to be adopted widely. Kodrzycki and Yu (2006) have produced the only
other ranking that includes reference intensity, although Engemann and Wall (2009) control for it indirectly.
3 Specifically, the present ranking and that of Engemann and Wall (2009) have a Spearman rank correlation
coefficient of 0.75. In general, journals with exclusively or primarily theoretical papers, such as Review of
Economic Studies, International Economic Review, and Journal of Economic Theory, are the ones whose
ranking is improved the most when best practices are used.
5
received relative to the number of cites to the average article published in the AER, the
profession’s flagship journal, provides a useful number to compare journals to one another.
According to this measure, the average article in the QJE was cited 43 percent more
frequently than was the average article in the AER, whereas the average articles in
Econometrica and the JPE were cited, respectively, 87 and 65 percent as frequently. There
were large differences in citation frequency between the high and low ends of the list. On
average, articles in journals ranked 22nd to 30th were cited between 27 and 31 percent as
frequently as the average AER article. Also, there was a vast middle ranking of journals
(ranked 5th to 21st) with relative mean impacts between 34 and 54 percent.
III. Skewness and a Median Ranking
How well does the mean number of citations summarize the citation tendency of
the typical article within a journal? A look at the maximum and minimum citations
reported in Table 2 provides a hint that the answer to that question might be “not so well.”
The maxima across journals range from 30 to 433, and this top article usually accounted
for 10 percent or more of the total cites received by the journal. In fact, it is not
uncommon for the share of the most-cited article to have exceeded 20 percent of the total.
For example, 38 percent of all citations to the Oxford Economic Papers were to a single
article, while the analogous numbers are 35 percent for the Journal of Applied
Econometrics and 25 percent for the International Economic Review. This phenomenon is
true even for a journal as large and prestigious as the AER, for which 15 percent of its
citations were to just one of its 93 articles. Further, for most journals there were decent
numbers of articles that received no citations at all during the period. The journals with the
6
highest shares of such articles were Oxford Economic Papers (12 percent), International
Economic Review (11 percent), the Journal of Business and Economic Statistics (11
percent), and the Journal of Economic Theory (9 percent).
This quick examination of the high and low ends of citation distributions illustrates
the high variance and the positive skew of any journal’s citations: Citations to a journal are
heavily weighted to a few articles, and many articles are cited relatively infrequently.
Although all journals share these characteristics, there are large cross-journal differences in
variance (which measures the degree of atypicality of the average number of cites). The
five journals with the highest variance relative to mean are the AER, Journal of Applied
Econometrics, AER Papers and Proceedings, Oxford Economic Papers, and
Econometrica. At the other end are four journals with a relative variance below 10:
Journal of Human Resources; Journal of Risk and Uncertainty; Journal of Money, Credit,
and Banking; and Journal of Law and Economic Organization. Skewness, which
measures the extent to which the mean is driven by the high end of the distribution, also
differs a great deal across journals, and not just between high- and low-ranked journals.
For example, the citation distribution of the QJE is the least skewed, while that of the AER
is the most skewed.
The high relative variance and extreme skewness of journals’ citation distributions
indicate that the mean is not the appropriate measure of the central tendency of the
citations received by articles in a journal. The median, which would eliminate the large
effect that a single article can have on the mean, is more appropriate. Accordingly, in
Table 3 the journals are ranked according to their median number of cites.4 As one can
4 As a practical matter, rankings based on the median are almost necessarily outdated because one must wait
several years after an article is published to collect the article-level information, meaning that the ranking is
7
see, it turns out that the two rankings are not really that different from one another.5 There
are, however, some large movements in rank for some journals as we move from Table 2
to Table 3. For example, the Journal of Applied Econometrics is ranked 11 places lower in
the median ranking, while the Journal of Industrial Economics and the AER Papers and
Proceedings fall by seven and five places, respectively. The two journals that improve
their rankings the most are the Journal of Risk and Uncertainty and the Journal of Law and
Economic Organization, which rise by seven and five places, respectively.
Because it is an indicator of how different the journals are from one another,
perhaps the difference between the relative mean and median impacts is more interesting.
Recall that the relative mean impact indicated that the average article in the QJE was cited
43 percent more frequently than the average article in the AER. According to the relative
median impact, however, the distance between the QJE and the AER, as well as the rest of
the journals, is much greater: The median article in the QJE was cited more than twice as
often as the median article in the AER. The use of medians also tends to provide higher
relative impacts for the journals ranked below the AER. Although the journals ranked 3rd
to 30th had an average relative mean impact of 0.4, their average relative median impact
was 0.47. Thus, use of the median reveals that the differences between the AER and the
journals ranked below it are not as large as suggested when one uses the mean, although
the difference between the QJE and all other journals is revealed to be even greater.
based on articles published several years prior to when the ranking is done. In contrast, the information
demands of a mean-based ranking are much less restrictive because one can pool data for recent years. For
example, Engemann and Wall (2009) look at citations received during 2001-07 to articles published during
the same period. Mean-based rankings are, therefore, probably more practical than median-based ones.
Nonetheless, it is useful to know the differences between the two ranking methods so that one has an idea of
how seriously to take the mean-based rankings that will necessarily be done.
5 Their Spearman rank-correlation coefficient is 0.91.
8
IV. Overlapping Distributions
The previous two sections demonstrated the unsuitability of the mean as an
indicator of the central tendency of citations to articles in a journal and showed how the
relative ranking of journals might differ if the median were used instead. This section
moves from a discussion of how to best perform a ranking to one of how seriously to take
any citation-based ranking. Although the median or the mean might be appropriate for
ranking journals, a look at the ends of each journal’s citation distribution provides some
notion of how large one’s mental error bands should be when using such a ranking.
Notice from Table 2 that the most-cited article for every one of the 30 journals
received more citations than did the median article across the four top-ranked journals (19).
In other words, at least one article from each journal had a greater impact than did half of
the articles in the top four journals. So how many other such articles are there for each
journal? And, conversely, given that some articles from even the top journals received
very few or even zero citations, how many articles published in each journal were cited
less often than the median of the bottom four journals? To answer these and related
questions, I split the journals into four tiers: Tiers 1 and 4 are composed of the top and
bottom four journals, respectively. Tier 1 journals all have a relative median impact
greater than 0.7, and Tier 4 journals all have a relative median impact less than 0.3. The
rest of the journals are split into Tiers 2 and 3, which, respectively, have relative median
impacts between 0.5 and 0.7 and between 0.3 and 0.5. The assignment of journals to tiers
is shown in Table 3.
It is also possible to assign individual articles from all the journals to each of these
tiers. Specifically, I define a Tier 1 article as one that was cited at least 19 times (the
9
median article across Tier 1 journals), a Tier 2 article as one that was cited at least 10 times
but fewer than 19 times, a Tier 3 article as one that was cited at least 6 times but fewer than
10 times, and a Tier 4 article as one that was cited no more than 5 times (the median across
the bottom four journals). The shares of each of these four types of article published in
every journal are provided in Table 3.
The first things to note are that every journal has articles in every tier and that there
is a strong tendency for a journal’s ranking to be related to the shares of its articles in the
highest or lowest tiers. This relationship is far from monotonic, however. For example,
nearly 20 percent of the articles in Econometrica and 14 percent of the articles in the AER
were Tier 4 articles, although Tier 4 articles were extremely rare for the QJE, which had
only one. As one would expect, Tier 4 articles were more common in Tier 2 journals than
they were in Tier 1 journals and the frequency might come as a surprise: Thirty-eight
percent of the articles in the Journal of International Economics and 35 percent of the
articles in the Journal of Monetary Economics were Tier 4 articles. Conversely, Tier 1
articles were not uncommon at the low end of the journal ranking: About 12 percent of the
articles in Games and Economic Behavior (from Tier 4) and 27 percent of the articles in
the Journal of Industrial Economics (from Tier 3) were Tier 1 articles.
Figure 1 summarizes the overall tendencies of journals from any tier to have
articles across all four tiers. One lesson from the figure is that the tiers are fairly similar in
the extent to which their articles fall into the middle two tiers—between 33 and 44 percent.
Put another way, the big difference between the journal tiers is in the prevalence of Tier 1
and Tier 4 articles. For another perspective, note that Tier 1 journals publish substantial
numbers of papers at least two tiers below the journals, whereas Tier 4 journals publish
10
substantial numbers of papers at least two tiers above the journals. Specifically, Tier 3 and
4 articles made up 29 percent of the articles published in Tier 1 journals, while Tier 1 and 2
articles made up 27 percent of the articles published in Tier 4 journals.
V. The h-Index
Hirsch (2005), responding to the perceived need for a useful quantitative measure
of individual researchers’ cumulative impact on their fields of research, proposed a simple
index, h, which is defined as the number of papers with citations greater than or equal to h.
To find an individual’s h, one need only to order the person’s papers from the most-cited to
the least-cited. Going down the list, h is where the number of papers is greater than or
equal to the number of citations to the hth paper. Recently, Braun, Glänzel, and Schubert
(2006) proposed using article-level information to derive an h-index for journals. In sync
with the previously noted observations about the skewed distribution of citations, the
advantage of the h-index applied to journals is thought to be that a simple way to combine
both quality and quantity considerations while eliminating the effect that a small number of
articles might have on mean-based measures.
The h-index has become increasingly popular: The Web of Science now provides
h-indices for all journals in its database, and the calculations are dependent on the time
frame chosen by the user. Similarly, RePEc uses the h-index for an experimental journal
ranking, although its database is drawn primarily from so-called gray literature, such as
working papers, and a fairly small and unrepresentative sample of journals.
Because all articles in my data have roughly the same amount of time to
accumulate citations, my data are ideal for ranking journals by their h-index and comparing
11
it with alternatives.6 Keep in mind that because the h-index is a measure of the overall
impact of a journal, whereas the median ranking is meant to measure the impact of a
typical article in a journal, the two rankings are not comparable. The h-index ranking
should instead be compared to a ranking by total citations across all articles in a journal.
Put another way, the h-index can be used to answer a question such as, “What journals
have the greatest total impact in economics?” Median or mean rankings, on the other
hand, address the question, “What kind of an impact is typical for an article in journal X?”
There is no evidence that this important distinction is appreciated by either of the two
sources that use the h-index for economics journals. On the RePEc site, for example, the
h-index stands without qualifications alongside several mean-based rankings.
Table 4 contains the h-index for each of the 30 included journals, along with their
total citations, their total-citations rank, and the difference between the two ranking
methods. Note first how journal size affects the difference between the median rank from
Table 3 and the h-index rank. Most obviously, the AER, AER Papers and Proceedings, the
European Economic Review, and the Journal of Economic Theory (the four largest
journals) leapfrog over journals with substantially higher median or mean citations. Again,
the link between size and the h-index is deliberate in that the index was designed originally
to measure cumulative impact over a researcher’s career, thereby making it less than
desirable for assessing the potential impact of an individual article by looking at the
journal in which it is published.
As is apparent from Table 4, there is little difference between the h-index ranking
and the total-citations ranking. This is not necessarily surprising because they are both
meant to capture total impact. In fact, given that the Spearman rank-correlation coefficient
6 See Braun, Glänzel, and Schubert (2006).
12
for the two methods is 0.98, it does not appear to have been worth the effort to gather
article-level data and calculate the h-index for each journal.
VI. Conclusions
In and of themselves, the rankings in this paper are not intended for actual use
because they use only one year’s publications and do not correct or control for a number of
important factors. Nevertheless, several general tendencies revealed by the analysis should
be helpful in determining the amount of care to take when employing journal rankings.
First, the appropriate measure of central tendency is the median rather than the mean, a
correction that can lead to substantial changes in the ranking of individual journals, but
which yields a ranking that differs little overall. Second, large percentages of articles in
the highest-ranked journals are cited less frequently than are typical articles in muchlower-
ranked journals. Similarly, large percentages of articles in the lowest-ranked
journals are cited more frequently than are typical articles in much-higher-ranked journals.
Finally, a ranking that uses the h-index is very similar to one that uses total citations,
meaning that it is of little use for assessing the relative quality of research published in a
journal.
13
References
Braun, Tibor; Glänzel, Wolfgang and Schubert, András. “A Hirsch-Type Index for
Journals.” Scientometrics, April 2006, 69(1), pp. 169-73.
Dusansky, Richard and Vernon, Clayton J. “Rankings of U.S. Economics Departments.”
Journal of Economic Perspectives, Winter 1998, 12(1), pp. 157-70.
Engemann, Kristie M. and Wall, Howard J. “A Journal Ranking for the Ambitious
Economist.” Federal Reserve Bank of St. Louis Review, May/June 2009,
forthcoming.
Hirsch, Jorge E. “An Index to Quantify an Individual’s Scientific Research Output.”
Proceedings of the National Academy of Sciences, November 2005, 102(46), pp.
16569-72.
Kalaitzidakis, Pantelis; Mamuneas, Theofanis P. and Stengos, Thanasis. “Rankings of
Academic Journals and Institutions in Economics.” Journal of the European
Economic Association, December 2003, 1(6), pp. 1346-66.
Kodrzycki, Yolanda K. and Yu, Pingkang D. “New Approaches to Ranking Economics
Journals.” Contributions to Economic Analysis and Policy, 2006, 5(1), Article 24.
Liebowitz, Stanley J. and Palmer, John P. “Assessing the Relative Impacts of Economics
Journals.” Journal of Economic Literature, March 1984, 22(1), pp. 77-88.
Oswald, Andrew J. “An Examination of the Reliability of Prestigious Scholarly Journals:
Evidence and Implications for Decision-Makers.” Economica, February 2007,
74(293), pp. 21-31.
Palacios-Huerta, Ignacio and Volij, Oscar. “The Measurement of Intellectual Influence.”
Econometrica, May 2004, 72(3), pp. 963-77.
14
Table 1. Summary Citation Characteristics of Journals
Number
of
Articles
Number
of
Citations
Mean
Citations
Article
Share
Citation
Share
Percent
Never
Cited
All journals (230) 6,373 49,670 7.8 1.00 1.00 0.19
30 included journals 1,543 22,805 14.8 0.24 0.46 0.04
200 excluded journals 4,830 26,865 5.6 0.76 0.54 0.24
Data are for citations in 2001-2008 to articles published in 2001, according to the ISI Web of
Science® as of September 23, 2008.
15
Table 2. The Numbers and Distributions of Journal Citations
Mean
Rank Journal
Number of
Articles
Mean
Citations
Relative
Mean
Impact
Citation
Maximum
Max
Article’s
Share
Citation
Minimum
Share
with Zero
Citations
Relative
Variance Skewness
1 Quarterly Journal of Economics 42 45.4 1.43 136 0.07 1 0.00 20.5 0.82
2 American Economic Review 93 31.7 1.00 433 0.15 0 0.02 74.3 6.34
3 Econometrica 66 27.6 0.87 194 0.11 0 0.03 38.1 2.65
4 Journal of Political Economy 44 20.6 0.65 76 0.08 3 0.00 16.2 1.31
5 Journal of International Economics 61 17.1 0.54 91 0.09 0 0.02 21.6 2.04
6 Rand Journal of Economics 37 16.6 0.52 127 0.21 0 0.03 30.9 3.60
7 Journal of Economic Growth 14 16.4 0.52 46 0.20 1 0.00 11.3 0.92
8 Journal of Labor Economics 32 15.0 0.47 70 0.15 3 0.00 14.5 2.49
9 American Economic Review, Pap. and Proc. 84 14.7 0.46 194 0.16 0 0.02 39.5 5.43
10 Review of Economic Studies 38 13.8 0.44 93 0.18 1 0.00 19.8 3.37
11 Journal of Monetary Economics 51 13.6 0.43 54 0.08 0 0.02 11.7 1.37
12 Journal of Applied Econometrics 35 13.4 0.42 164 0.35 0 0.06 57.4 4.98
13 Journal of Econometrics 77 13.1 0.41 97 0.10 0 0.03 22.4 2.86
14 Review of Economics and Statistics 67 13.1 0.41 69 0.08 0 0.03 12.7 2.10
15 Journal of Industrial Economics 26 12.6 0.40 54 0.16 0 0.08 17.0 1.45
16 Journal of Business and Economic Statistics 45 12.0 0.38 54 0.10 0 0.11 16.6 2.03
17 European Economic Review 98 12.0 0.38 112 0.10 0 0.04 24.4 3.96
18 Journal of Law and Economic Organization 19 11.9 0.38 46 0.20 0 0.05 9.8 1.81
19 Journal of Risk and Uncertainty 27 11.3 0.36 39 0.13 1 0.00 8.1 1.29
20 Economic Journal 66 11.3 0.36 106 0.14 0 0.06 27.8 4.02
21 Journal of Law and Economics 31 10.6 0.34 42 0.13 0 0.03 10.5 2.03
22 Journal of Public Economics 75 9.9 0.31 56 0.08 0 0.07 10.1 2.14
23 Journal of Urban Economics 48 9.5 0.30 61 0.13 0 0.02 11.6 3.03
24 Games and Economic Behavior 59 9.3 0.29 61 0.11 0 0.07 15.8 2.48
25 Journal of Human Resources 30 9.1 0.29 30 0.11 0 0.03 7.3 1.08
26 International Economic Review 45 8.9 0.28 102 0.25 0 0.11 29.1 4.75
27 Journal of Development Economics 69 8.7 0.27 68 0.11 0 0.06 12.7 3.26
28 Oxford Economic Papers 33 8.5 0.27 105 0.38 0 0.12 38.9 4.98
29 Journal of Economic Theory 87 8.4 0.27 54 0.07 0 0.09 10.2 2.37
30 Journal of Money, Credit, and Banking 44 8.4 0.27 32 0.09 0 0.07 8.6 1.46
Citations are from 2001-2008 to articles published in 2001 according to the ISI Web of Science® as of September 23, 2008.
16
Table 3. Median Citations and Overlaps in Citation Distributions
Median
Rank* Journal
Median
Citations
Relative
Median
Impact
Tier 1
Share
Tier 2
Share
Tier 3
Share
Tier 4
Share
1 Quarterly Journal of Economics 36.5 2.15 0.786 0.143 0.048 0.024
2 American Economic Review 17 1.00 0.495 0.183 0.183 0.140
3 Econometrica 16.5 0.97 0.470 0.197 0.136 0.197
Tier
1
4 Journal of Political Economy 12.5 0.74 0.341 0.273 0.273 0.114
5 Journal of Economic Growth 11.5 0.68 0.429 0.143 0.214 0.214
6 Journal of International Economics 11 0.65 0.311 0.197 0.115 0.377
7 Journal of Labor Economics 10.5 0.62 0.219 0.344 0.219 0.219
8 Rand Journal of Economics 10 0.59 0.297 0.243 0.189 0.270
9 Review of Economic Studies 10 0.59 0.184 0.316 0.158 0.342
10 Journal of Monetary Economics 9 0.53 0.255 0.235 0.157 0.353
11 Review of Economics and Statistics 9 0.53 0.194 0.299 0.224 0.284
12 Journal of Risk and Uncertainty 9 0.53 0.222 0.222 0.222 0.333
Tier
2
13 Journal of Law and Economic Organization 9 0.53 0.158 0.263 0.263 0.316
14 American Economic Review, Pap. and Proc. 8 0.47 0.190 0.250 0.190 0.369
15 Journal of Business and Economic Statistics 7 0.41 0.178 0.222 0.244 0.356
16 Journal of Econometrics 7 0.41 0.182 0.195 0.247 0.377
17 Journal of Law and Economics 7 0.41 0.129 0.194 0.387 0.290
18 Journal of Public Economics 7 0.41 0.173 0.213 0.200 0.413
19 European Economic Review 7 0.41 0.173 0.245 0.153 0.429
20 Journal of Urban Economics 7 0.41 0.146 0.146 0.333 0.375
21 Journal of Human Resources 6.5 0.38 0.167 0.233 0.133 0.467
22 Journal of Industrial Economics 6 0.35 0.269 0.115 0.154 0.462
23 Journal of Applied Econometrics 6 0.35 0.229 0.143 0.143 0.486
24 Economic Journal 6 0.35 0.121 0.197 0.212 0.470
25 Journal of Economic Theory 6 0.35 0.115 0.195 0.207 0.483
Tier
3
26 Journal of Development Economics 6 0.35 0.101 0.217 0.203 0.478
27 Games and Economic Behavior 5 0.29 0.119 0.186 0.153 0.542
28 Journal of Money, Credit, and Banking 5 0.29 0.114 0.205 0.159 0.523
29 International Economic Review 5 0.29 0.111 0.111 0.200 0.578
Tier
4
30 Oxford Economic Papers 4 0.24 0.061 0.152 0.152 0.636
* The ratio of Tier 1 and Tier 4 shares is used as a tiebreaker.
17
Table 4. Rankings Based on Total Citations and the h-Index
Total
Citations
Rank by
Total
Citations h-index
h-Index
Rank*
Difference
in Rank**
American Economic Review 2,948 1 33 1 0
Quarterly Journal of Economics 1,907 2 28 2 0
Econometrica 1,822 3 23 3 0
Journal of International Economics 1,043 6 19 4 -2
American Economic Review, Pap. and Proc. 1,235 4 17 5 1
European Economic Review 1,176 5 17 6 1
Journal of Econometrics 1,009 7 17 7 0
Journal of Political Economy 906 8 16 8 0
Review of Economics and Statistics 878 9 16 9 0
Journal of Public Economics 743 11 16 10 -1
Journal of Monetary Economics 694 13 16 11 -2
Journal of Economic Theory 731 12 14 12 0
Economic Journal 746 10 13 13 3
Rand Journal of Economics 614 14 13 14 0
Review of Economic Studies 524 18 13 15 -3
Journal of Labor Economics 480 19 13 16 -3
Journal of Development Economics 600 15 12 17 2
Games and Economic Behavior 549 16 12 18 2
Journal of Business and Economic Statistics 540 17 12 19 2
Journal of Applied Econometrics 469 20 12 20 0
Journal of Urban Economics 456 21 11 21 0
Journal of Money, Credit, and Banking 370 23 11 22 -1
Journal of Risk and Uncertainty 305 26 11 23 -3
International Economic Review 401 22 10 24 2
Journal of Law and Economics 329 24 10 25 1
Journal of Industrial Economics 328 25 10 26 1
Journal of Human Resources 273 28 10 27 -1
Journal of Economic Growth 230 29 9 28 -1
Journal of Law and Economic Organization 226 30 9 29 -1
Oxford Economic Papers 281 27 8 30 3
* Ties are broken by total citations.
** The difference between the h-index rank and the total-citations rank.
Figure 1. Shares of Article Types Across Journal Tiers
0.51
0.20
0.16
0.13
0.25 0.26
0.18
0.31
0.16
0.21 0.21
0.42
0.10
0.17 0.17
0.56
0.00
0.10
0.20
0.30
0.40
0.50
0.60
Tier 1 Articles Tier 2 Articles Tier 3 Articles Tier 4 Articles
Shares of Article Types
Tier 1 Journals Tier 2 Journals Tier 3 Journals Tier 4 Journals
Tidak ada komentar:
Posting Komentar