Appendix C

Definitions of Digital Products

DATA

Data are digital objects broadly defined as observations or measurements (processed or unprocessed) collected in the course of scientific investigations. Data may take many forms, including numbers, text, images, and videos. Data produced by federal laboratories often have unique qualities, such as being generated by a unique piece of scientific equipment (e.g., colliders) or in massive quantities not available elsewhere (e.g., satellite data). Because of their unique and often irreplaceable qualities, these types of raw data are generally retained for archival purposes. When stored in a form amenable to external analysis, raw and/or preprocessed data may be distributed outside of the lab to ensure reproducibility of scientific results and allow external researchers to advance new use cases and improve data processing pipelines.

Datasets and Databases

While raw data may sometimes be valuable to researchers outside of the federal laboratories, it is often more beneficial for the labs to release datasets: structured collections of data pertaining to a particular endeavor or measurement. Like data, datasets make take many forms, including tables of numeric values, collections of images or videos, or digital formats. Datasets also can include engineering studies, such as characterization of materials, performance measurements of software systems, and failure rates for different devices.

Especially in the latter case, it may be useful to house datasets within a database. A database is a collection of datasets that includes a software framework that allows those datasets to be electronically accessed, analyzed, manipulated, and updated. Databases may enable not just data access but efficient data access, for example, by allowing access to manageable parts of datasets that are too large to retrieve using traditional techniques.

Metadata and Data Curation

Particularly when releasing data to people who have not been involved in the data’s creation, it is helpful to include metadata—information on the structure and provenance of a dataset. Metadata, increasingly considered a digital product on its own, is often essential to locating and retrieving datasets of interest, understanding the context in which the data were produced, and making appropriate use of the data. Accurate and informative metadata is critical to good data curation—the process of managing datasets and the corresponding metadata to ensure that no information about the structure, provenance, or quality of data is lost. Good data curation, in turn, is an essential component of open and shareable data.

SOFTWARE

Data can often be of limited use in their raw form. Algorithms are computational procedures that take data as inputs and calculate outputs that answer questions and solve problems. Software is a human- and machine-readable set of instructions that implement an algorithm or set of algorithms. Software can be packaged for wider use as an application or app, allowing users to make use of the functionality of a piece of software without having expertise in the underlying code powering the app.

Code Snippets

Code snippets are small subsets of software with lightweight documentation intended to be reused. They allow a programmer to use a standard implementation of a specific statistical technique, reducing error and increasing replicability. Snippets are broadly available online under open-source licenses and are often deeply woven into scientific software.

Scientific Software Artifacts

A particularly important type of code snippet is a scientific software artifact. Such artifacts, which may take the form of data readers or calibration routines, are often an integral part of data, such that they are needed to understand the context of the data. When data are shared, the inclusion of scientific software artifacts greatly increases the usability and portability of the shared data.

STATISTICAL MODELS

Statistical models are a type of digital product that bring together data and software to enable prediction. They are a mathematical framework for predicting the outcome of empirical phenomena based on underlying parameters. Software can be used to define these models and optimize the value of their

underlying parameters based on input data. A model with optimized parameters is known as a “fitted” model, and can be used for making predictions.

Artificial Intelligence/Machine Learning (AI/ML)

AI/ML is an increasingly popular subfield of statistical modeling. AI/ML employs highly flexible models that can identify complex or unexpected patterns in data, thereby fueling increased predictive performance, instead of prescribing a specific mathematical description of a phenomenon, such as is done in traditional statistical modeling. Deep learning, one popular subtype of AI/ML, uses neural networks to “understand” very complex data, including images, text, speech, and videos.

Training Datasets

The datasets used to “fit” or “train” AI/ML models are known as training datasets. These large datasets often consist of many pairs of inputs and outputs (e.g., input images together with a label indicating the presence or absence of a face), and in the training process, model parameters are optimized to provide the correct output in response to a given input. It is important to provide training datasets that are both large and unbiased (as representative as possible of the full range of inputs that the model may encounter).

Software Notebooks

Software notebooks are web-based tools that support all aspects of data transformation, including workflows, code, data, equations, and visualizations. They are both human- and machine-readable, and can serve as a form of publication as well as a form of software since instructions can be executed within the document. Notebooks can live online as self-contained toolkits or as documentation of a specific data analysis and modeling attempt.¹ They can also link to other kinds of digital products, including datasets, external software, workflows, publications, and patents.²

___________________

¹ An empirical study of references to notebooks in astronomy over a 5-year period (2014–2018) found that notebooks appear better suited to supporting reuse of machine learning products than to providing direct access to software code and data. The study authors further recommend that any notebooks cited in publications be “stabilized”—frozen in time so the next user can start from the same place where the conclusions claimed were drawn. Indeed, these notebooks even serve as a discovery mechanism for reuse. Wofford, M., B. Boscoe, C. Borgman, I. Pasquetto, and M. Golshan, Milena. 2020. Jupyter notebooks as discovery mechanisms for open science: Citation practices in the astronomy community. Computing in Science & Engineering 22(1).

² Influenced by and derived from Randles, Bernadette M., et al. 2017. Using the Jupyter notebook as a tool for open science: An empirical study. 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE.

OTHER DIGITAL PRODUCTS

Electronic Media

Many documents and other materials that were previously disseminated through print or other analog media are now distributed by federal laboratories in digital form. These include electronic versions of scientific publications (one vehicle for communicating scientific discoveries from federal labs); electronic manufacturing designs of physical objects; and digital images and videos for directly communicating results to the scientific community, policy makers, the media, and the general public.

Digital Services

Many scientific analyses and digital products developed at federal laboratories require more computational power than is available in a single computer. Cloud computing allows individual users to remotely access the computational power of many computers at once, which can be essential to harness and analyze the vast datasets, complex software, and powerful statistical models routinely produced by federal labs and the private sector. Computing power may be provided by a cluster of computers at an individual lab or a centralized scientific computing facility (e.g., the National Energy Research Supercomputing Center). These facilities, along with the substantial in-house computer processing and analytical expertise at federal labs, can be made available to external researchers.

In addition, federal labs support a wide range of other public-facing digital services, including time.gov (through which the National Institute of Standards and Technology [NIST] provides digital access to the official time) and weather.gov (through which the National Weather Service provides digital access to weather data). Federal labs also maintain an array of software repositories through which they can provide public access to software and code produced in the lab (e.g. osti.gov/doecode, code.mil, code.nasa.gov, software.nasa.gov, and code.nsa.gov).