As part of the MOOC Decision Making Under Uncertainty: Applying Structured Expert Judgment I had to conduct an actual elicitation with experts. Structured Expert Judgment is designed for and implemented in areas in which there is an evident lack of data and hence, we are forced to rely on experts to make predictions. This is not the case of scientometrics, where the volume of data is actually increasing exponentially with the transition to online collaboration, commenting and dissemination of scientific work.
However, we do have a problem when deciding which data and how such data should be used in research assessment. An area in which the use of scientometrics is especially troublesome is that of the Humanities. For this reason, in fields or when responding to issues in which there is a lack of consensus on the use of scientometrics, conducted structured expert judgment elicitations may come in handy. For this reason, in my elicitation I tried to focus on the development of university rankings (another controversial area) in the Humanities, where lack of consensus would be all over the place. The actual submitted report is here, and of course, this is an exercise and hence none of the results should actually be taken seriously, beware!
If you find it useful in whichever way and want to reference it feel free to do so! The paper is uploaded in Zenodo and you can find it here:
Nicolas Robinson-Garcia. (2020, April 27). Using Structured Expert Judgment to predict a university ranking in the Humanities (Version 1). Zenodo. http://doi.org/10.5281/zenodo.377039