Program

Many steps have been taken to develop and implement broader approaches to assessing research and knowledge exchange, as exemplified by the EU’s Reform of Research Impact Assessment and the worldwide Open Science movement. In line with CoARA (Coalition for Advancing Research Assessment) recommendations, how can research intelligence approaches capture a diversity of contribution to science and society? What are the challenges and gaps? What are the opportunities? We will discuss how SciVal combines a wider range of indicators and insights with peer review and expert knowledge to provide a broader view of research and its impact.

We propose an establishment of a Nordic Network for Advanced Bibliometrics that will promote the development and application of innovative bibliometric techniques among researchers and practitioners in the Nordic countries. By leveraging algorithmic classification, machine learning, citation context analysis, linked data, APIs, and visualization techniques, the network aims to enhance the understanding of scholarly communication, research impact, and knowledge dynamics. The network will foster a community of researchers and professionals working towards advancing bibliometrics in the Nordic region and beyond through collaboration, training, and research projects. In this workshop, we will showcase a number of cases by colleagues and then stage a panel discussion about how to set up such a network in the best way. We propose to develop training activities, hackathons, as well as online activities A dedicated communication platform based on Teams will be established to facilitate communication, knowledge sharing, and resource dissemination among network members. Regular updates, newsletters, and mailing lists will inform the network about ongoing activities and opportunities. The network will be open to both researchers and professionals working in bibliometrics and research policy. It will focus on open, transparent scholarly practices in line with the current state of knowledge in the field and ongoing policy discussions.

Open Science has been steadily on the rise in the past decade. This includes Open Science practices, infrastructures and policies, and touches on virtually all aspects of the research cycle and the academic system. The impacts of Open Science are often in the spotlight, but we should clearly distinguish it from the effects of Open Science on impacts. In the PathOS project, we are interested in understanding the causal effects of Open Science. Causal effects are difficult to infer in general. A fundamental problem is that many of the concepts of interest are not directly observable. Scientometric studies typically use various metrics as indicators of the (unobserved) concepts of interest, but this also raises some thorny questions around causality. We will review some of the challenges around causality that emerge when studying open science and indicators, and suggest ways forward.

University rankings guide many aspects of universities operations including students’ and faculty recruitment, securing funding, finding collaborators and more. At the heart of any ranking is the idea of capturing research excellence and impact which is frequently focused on the number of publications and received citations. However, in the past decade there is an increasing demand to capture a university’s impact and excellence beyond the traditional paradigm of scientific output. These include societal and economical impact, support and mentorship for successful career paths for trainees, equity, inclusion and diversity and more. In this presentation we would discuss the power of using various datasets and sources to capture the multi-faceted nature of the university and how these novel impact indicators might be used in a ranking exercise.

In 2003 the Flemish Government introduced performance-based research funding for universities in Flanders (Debackere & Glänzel, 2004; Engels & Guns, 2018; Luwel, 2021). Every five years the government procures a consulting team and panels of international experts to evaluate the effects and the effectiveness of these research and innovation subsidies to the universities. During the workshop we will share and discuss our observations regarding the gradual changes of the Flemish PRFSs since 2003, the possible effects of these PRFSs (including on support offices at universities), and the feasibility in terms of complexity of data collection, curation, transparency and validation.

Bibliometric indicators are often used to help estimate the quality of academic journal articles, but with little systematic evidence about the relationship between the two. This talk summarises the findings of studies of this relationship for quality scores from the UK Research Excellence Framework (REF) 2021, the world’s most financially important systematic academic expert review exercise. It also reports the extent to which machine learning can estimate the quality of journal articles from bibliometric information and metadata. These studies drew on an almost complete set of individual article quality scores from REF2021 – the largest science-wide data set of this nature. The bibliometric analyses surprisingly suggest that there may be a positive association between quality scores and article or journal citation rates in all broad fields. This relationship is never very strong and is weak in some social sciences and in the arts and humanities. The machine learning analyses developed a range of strategies to predict quality scores alongside expert reviewers to reduce the human labour needed for reviewing, then asked a sample of the original expert reviewers to comment on the desirability of the strategies. Although a technically desirable strategy was developed, its ethical implications led to it not being recommended for future REFs.

Metrics, research quality and the research process. An exploration of their interrelations. By Fredrik Niclas Piro and Dag W. Aksnes (NIFU – Nordic Institute for Studies in Innovation, Research and Education, Norway).

A cross national comparison of evolution of co-authorship practices in Social Sciences and Humanities. By Tim C. E. Engels (University of Antwerp, Belgium) and Emanuel Kulczycki (Adam Mickiewicz University, Poland).

Recovering from the guillotine: Hydra-based construction of a new institution-level performance-based development framework for Aalborg University. By Birger Larsen, Kathrine Bjerg Bennike, Poul Meier Melchiorsen (University of Aalborg, Denmark) and Gunnar Sivertsen (NIFU -Nordic Institute for Studies in Innovation, Research and Education, Norway).

Assessing the research competitiveness of Nordic countries. By Giovanni Abramo (National Research Council, Italy) and Ciriaco Andrea D’Angelo (University of Rome, Italy).

The use of patent analysis in foresight, insights and assessments of methods and approaches. By Melanie Martini and Marcus John (Fraunhofer Institute for Technological Trend Analysis INT, Germany).

Universities accelerate the use of knowledge in technological development for patenting. By Alvin Shijie Ding (Elsevier, China) and Rachel Herbert (Elsevier, UK).

Who funds Nordic research? By Ross W.  K. Potter (Clarivate, UK).

Bibliographic coupling as a sleeping beauty: Recent developments around the original similarity measure. By Jeffrey Demaine (McMaster University, Canada).

1. Broadening the conception of ‘what counts’ – example of a narrative CV in a university alliance. By Maria Pietilä, Katri Rintamäki and Jouni Kekäle (University of Eastern Finland, Finland).

2. Researcher mobility in Sweden: Bibliometric analysis using a newly developed model and a customizable research tool. By Silvia Dobre, Rachel Herbert (Elsevier, the Netherlands) and Hans Pohl (Lindholmen Science Park AB, Sweden).

3. Predicting the future of the research trends – on examples of OSH publications from 2010-22. By Witold Sygocki (Central Institute for Labour Protection – National Research Institute, Poland) and Aneta Drabek (University of Silesia in Katowice, Poland).

4. The SDGs and publications of University of Helsinki: Tracking contributions responsibly? By Petri Turunen, Tuula Huuskonen and Terhi Sandgren (Helsinki University Library, Finland).

5. Interdisciplinary research classification based on a combined conceptual-empirical framework. By Shunshun Shi and Lin Zhang (Wuhan University, China).

6. Collecting author affiliation data for Flemish non-Web of Science SSH publications: Process, results and lessons learned. By Peter Aspeslagh (University of Antwerp, Belgium).

7. More than our rank: A preliminary analysis of signatories’ narratives. By Marianne Gauffriau (IT University of Copenhagen, Denmark), Elizabeth Gadd (Loughborough University, UK) and Laura Himanen (CSC – IT Center for Science, Finland).

8. Introducing ReformScape: An online tool mapping the global research assessment reform landscape. By Alex Rushforth, Marta Sienkiewicz (Leiden University, The Netherlands), Haley Hazlett (Declaration on Research Assessment, USA), Ruth Schmidt (Illinois Institute of Technology, USA), Sarah de Rijcke (Leiden University, Netherlands), Stephen Curry (Imperial College, UK).

9. Extraction and analysis of citation data from student output in order to improve library instruction. By John Holmberg Runsten, Lars Våge and Daniel Fahlén (Mid Sweden University Library, Sweden).

10. Bibliometric analysis of traditional and emerging computational techniques for computer-aided design of TFOs. By Martha Hincapié-López (Universidad Autónoma de Bucaramanga; Colombia), Efrén Romero-Riaño (Colombiano de Ciencia y Tecnología, Colombia), Efraín Pinzón-Reyes (Universidad de Santander, Colombia) and Y. Vladimir Pabón-Martínez (Universidad Autónoma de Bucaramanga, Colombia).

11. Creating synergies in methods of review research and bibliometric analysis. By Sabine Wollscheid, Dag W. Aksnes, Henrik Karlstrøm, Lone W. Fossum and Fride Flobakk-Sitter (NIFU – Nordic Institute for Studies in Research Innovation and Education, Norway).

12. The sex of Nordic cancer researchers who write international and domestic papers. By Grant Lewison and Richard Sullivan (King’s College London, UK).

13. On the performativity of SDG classifications in large bibliometric databases. By Matteo Ottaviani and Stephan Stahlschmidt (German Centre for Higher Education Research and Science Studies (DZHW), Germany).

14. Faculty perceptions of research assessment in Social Studies, Arts & Humanities: A case study from Chile. By Erwin Krauskopf (Universidad de Las Américas, Chile) and Mauricio Salgado (Centro de Estudios Públicos, Chile).

15. Publishing by PhD candidates at the University of Bergen, Norway. By Caroline S. Armitage and Eli Heldaas Seland (University of Bergen Library, Norway).

Policy citing Science: A study of references in Swedish governmental reports (SOUs). By Linus Salö (Stockholm University, Sweden), Björn Hammarfelt and Gustaf Nelhans (University of Borås, Sweden).

An Investigation of policy citations to Nordic scientific publication. By Ashraf Maleki and Kim Holmberg (University of Turku, Finland).

Mapping the research-policy ecosystem: Who does what, when, where and why? By Katherine Hart, Nathalie Cornée and Euan Adie (Overton, UK).

Why do we still know so little about the total landscape of scholarly journals? Leveraging public data for building a common foundation. By Mikael Laakso (Hanken School of Economics, Finland) and Janne Pölönen (Federation of Finnish Learned Societies, Finland).

Systematic development and documentation of cross-disciplinary bibliometric search queries. By Mette Venås Overballe-Petersen (Danish Agency for Higher Education and Science, Denmark).

The ripple effect of scientific misconduct on an author’s career. By Kiran Sharma (BML Munjal University, India) and Satyam Mukherjee (Shiv Nadar University, India).

Narrowing the gap between demand and supply of peer review. By Gustaf Nelhans (University of Borås, Sweden) and Charlotte Wien (Elsevier, Denmark).

Trade press as pathways to impact. By Silvia Dobre, Rachel Herbert (Elsevier, the Netherlands) and Diana Hicks (Georgia Institute of Technology, USA).

Cancel.

Exploring the thematic landscape of the circular economy: A comparative study of publication and Wikipedia data. By Daniel Richter, Philipp Baaden and Mohammad Saknini (Fraunhofer Institute for Technological Trend Anlaysis, Germany).

Evaluating the diversity of scientific discourse on twenty-one multilingual Wikipedias using citation analysis. By Michael Taylor (Digital Science and University of Wolverhampton, UK), Roisi Proven (Digital Science, UK) and Carlos Areia (Digital Science and University of Coventry, UK).

Field affinity. By Henrik Karlstrøm and Dag W. Aksnes (NIFU – Nordic Institute for Studies in Innovation, Research and Education, Norway).

Responsible research assessment (RRA) entails balancing qualitative and quantitative methods, recognizing diversity of academic work and fields, and rewarding open science practices. CoARA agreement for Reforming Research Assessment reinforces the DORA (The San Francisco Declaration on Research Assessment) recommendation against inappropriate uses of journal- and publication-based metrics (especially JIF and h-index). While the European Council supports the RRA agenda, it recently invited member states to address the issue of predatory publishing practices. 
 
The Nordic bibliometric indicator relies on evaluation of journals and book publishers by national field-specific panels of experts. Since 2005, the indicator has been adapted in some form at national or institutional level in all Nordic countries but now its uses are reconsidered. Denmark and Norway have decided to stop using the indicator in the performance-based funding of universities, and Finland is reconsidering its funding model. Many Swedish universities use the Norwegian list for internal funding allocations, and Iceland uses the Finnish list in the new evaluation system for universities.

The panel discusses the following three questions from the national and/or their institution’s perspective:
1.    What are the most important recent changes in the use of the Nordic bibliometric indicator in your country/institution?
2.    What have been the main concerns about the uses of the indicator with respect to the RRA agenda and the CoARA Agreement?
3.    What is the role of journal evaluation (by citation metrics or experts) in addressing predatory, questionable, deceptive and low-quality publishing practices?  

The panel comprises experts in research assessment and bibliometrics from five Nordic countries:
Denmark: Marianne Gauffriau (IT University of Copenhagen)
Finland: Laura Niemi (University of Turku)
Iceland: Baldvin Zarioh (University of Iceland)
Norway: Gunnar Sivertsen (NIFU – Nordic Institute for Studies in Innovation, Research and Education)
Sweden: Björn Hammarfelt (University of Borås)

Locally relevant, globally visible? Geographic orientation of social sciences and humanities. By Raf Guns, Cristina Arhiliuc and Hongyu Zhou (University of Antwerp, Belgium).

Geopolitical influences on global collaboration patterns in science and how they affect the Nordic countries. By Gunnar Sivertsen (NIFU – Nordic Institute for Studies in Innovation, Research and Education, Norway).

Researchers’ mobility in Nordic countries. By Milovan Kovač (Clarivate, Serbia) and Ross W. K. Potter (Clarivate, UK).

A preliminary exploration on the dynamics of international scientific mobility. By Jialin Liu and Yi Bu (Peking University, China).

Science is an increasingly global activity, as demonstrated through the exchange of knowledge, resources, and people across national borders. Exchanges, however, can often be asymmetrical, leading to concentrations within science. Using bibliometric data, this presentation will explore the scientific product space of nations, how these develop, and the impact of scientific mobility, global funding, and collaboration on national science portfolios. The presentation will discuss threats to an equitable global science system (e.g., helicopter science and isolationism), challenges (e.g., language barriers and the monopoly of a lingua franca), and opportunities (e.g., science diplomacy and collective infrastructure). The presentation will close by a review of relevant global science policies and recommendations for the use of bibliometrics in generating and evaluating national and international science policy.

Cocreating open science ecosystem to foster the reform of research and researchers assessment. By Susanna Nykyri (Tampere University, Finland),  Ilmari Jauhiainen (Federation of Finnish Learned Societies, Finland), Laura Niemi (University of Turku, Finland) and Ari Rouvari (CSC – IT Center for Science, Finland).

An analysis of publication trends by income level in the Directory of Open Access Journals 1987–2020. By David Druelinger (Touro University, USA) and Lai Ma (University College Dublin, Ireland).

The Open access citation advantage in the context of scholarly publishing at a higher education institution. By Šárka Erben Johansson and Hampus Rabow (Malmö University, Sweden).