Is AI software equal to or better than humans at cataloging science data? 2023

Source: Barendse, C. (2023) How generative AI will revolutionize data catalogs

Data catalogs are critical tools for managing and governing data, and they have come a long way from their humble beginnings as simple spreadsheets or databases. Today’s data catalogs offer rich features, such as automatic ingestion, data discovery, lineage tracking, and data quality management, making them powerful tools for any data-driven organization. But what does the future hold for these tools, and how will they evolve to meet the needs of tomorrow?

The limitations of traditional data catalogs

Before we explore the future of data catalogs, let’s first consider some of their limitations. Traditional data catalogs require manual cataloging, which is a labor-intensive process that can be time-consuming and discouraging for users. They also struggle to keep up with changing data landscapes, resulting in incomplete and outdated information. Furthermore, traditional data catalogs are designed primarily for structured data, which means they struggle to manage unstructured data sources such as text documents, images, and videos.

Below is an overview of some of the key limitations of traditional data catalogs:

1. Time-consuming manual curation: Manually curating data assets is a labor-intensive process that requires a significant amount of time and effort from data stewards and other team members.
2. Difficulty staying up-to-date: As data landscapes change quickly, traditional data catalogs struggle to keep up with new and evolving data sources, schemas, and relationships, resulting in outdated and incomplete information.
3. User friction: Logging into a separate data catalog interface can be cumbersome, discouraging users from leveraging the catalog to its fullest potential.
4. Complexity in tracking data lineage: Traditional data catalogs often struggle to effectively track and visualize data lineage, which is crucial for understanding how data flows through an organization’s systems, as well as for maintaining data quality and compliance.
5. No support for unstructured data: Traditional catalogs are primarily designed for structured data, which means they struggle to catalog and manage unstructured data sources like text documents, images, and videos.

The next generation of data catalogs will be fully powered by generative AI.

What will the next generation data catalogs look like?

Enter the next generation of data catalogs — fully powered by generative AI. These advanced tools will automate data management activities such as curation and data quality, saving time and resources for data stewards and other team members. Imagine having a chatbot that knows everything about your data, from where to find tables, to creating charts, and monitoring data quality. It will be like having your own expert team member dedicated solely to helping you to complete your work.

Data Catalogs 3.0 will offer several key features that set them apart from traditional data catalogs, including a chatbot-style interface for data discovery that facilitates easy data exploration and democratization of data. These tools will also integrate with collaboration platforms and data creation tools, such as Microsoft Word, Slack, Snowflake, and dbt, enabling users to access the catalog without having to leave what they’re doing.

Key features of data catalogs 3.0

Data Catalogs 3.0 will offer several key features that set them apart from traditional data catalogs, including:

1. Chatbot-style interface for data discovery: Users can access and explore their organization’s data through a conversational chat interface, facilitating easy data discovery and enabling true democratization of data throughout the organization.
2. Embedded integration: Future catalogs will integrate with data creation tools and collaboration platforms, such as Microsoft Word, Slack, Snowflake, and DBT, so users never have to leave what they are doing to access the catalog.
3. Ingestion of data and business domain knowledge: Future data catalogs will ingest all of the organization’s data, including documents, business rules, business strategy, databases, and other digital assets. This will enable organizations to gain a holistic view of their data landscape.
4. Advanced security and privacy: Data catalogs will be able to identify and classify data (structured and unstructured) across the entire organization and automatically implement, security and retention processes.
5. AI-driven curation and cataloging: Gone will be the need for manual curation. By having access to metadata and the data itself, Data Catalogs 3.0 will automatically learn and curate the catalog themselves.
6. AI-driven data quality monitoring: By combining traditional data quality management, business domain knowledge, data observability, and AI, data catalogs will be able to automatically develop and implement data quality monitoring across all dimensions of data quality (completeness, accuracy, validity, uniqueness, timeliness, consistency).

Source: Barendse, C. (2023) How generative AI will revolutionize data catalogs

The future of data catalogs is bright and full of promise. The advancements in generative AI technology will transform these tools into even more powerful assets for data management and governance. The chatbot-style interface for data discovery, the integration with collaboration platforms and data creation tools, and the ability to ingest all types of data will make these tools even more accessible and user-friendly.