Conference Agenda (All times are shown in Eastern Daylight Time)

This study constructed an integrated framework for measuring LLMs’ knowledge breadth (able to cover multiple fields) and depth (able to handle complex problems) and perform calculations based on evidence. The current research evidence on measuring LLMs was obtained through a systematic review. Then, an integrated framework based on the matching of capability and task was constructed, where the generation capability and knowledge management task reflect the knowledge breadth, while other capabilities and tasks reflect the knowledge depth. Knowledge breadth and depth were measured through Coverage and Revealed Technological Advantage. The results showed that 36 matches of capability and task were selected from representative papers, and the knowledge breadth and depth panorama of LLMs was displayed. Furthermore, there is no absolute advantage or disadvantage in LLMs. This study clarifies the boundaries of the LLM’s measurement, and the framework further ensures the diversity of benchmark models and avoids redundancy and misalignment biases.

The field of scientific and technical information (STI) is characterized by limited open-source training data, strong timeliness requirements, and high demands for professional expertise, necessitating the combination of general capabilities and domain-specific capabilities of large language models (LLMs). Based on common STI research tasks, this study establishes benchmark datasets for evaluating LLMs in the STI field, comprising basic knowledge ability, dynamic research ability, and thematic research ability. A total of 1,557 objective and subjective questions were selected to evaluate the performance of eight LLMs developed by commercial organizations, research institutions, and universities in the STI field. The results indicate that LLMs perform well in terms of STI domain knowledge but still exhibit significant gaps in dynamic and thematic information research. There is a need to actively explore and promote the integration, adaptation, and application of LLM technologies in the STI field to provide robust support for high-quality and high-efficiency STI services.