Data Governance with Semarchy and Purview

Modern organizations generate large quantities of data, often from numerous, disparate sources. The Semarchy Data Platform is the intelligent data hub for data integration, master data management (MDM), reference data management (RDM), application data management (ADM), data integration, quality, and governance. Semarchy Data Management (xDM) brings extreme agility for defining and implementing data management applications and releasing them to production.

This architecture demonstrates how to include Master Data Management (MDM) into the Azure ecosystem and combine it with Microsoft Purview to enable governance for your critical data assets.

Architecture

The following architecture illustrates the data flow of the integration of Semarchy xDM with Microsoft Purview for data governance.

Data Flow

Metadata and data flow include the following steps:

• Source data integration from identified source systems: This integration is built with Azure Data Factory, Semarchy xDI (Semarchy’s data integration component), or your integration solution, and uses SQL or REST endpoints exposed by Semarchy xDM for batch and real-time integration.

• Master Data Management in Semarchy xDM

  • Data is enriched, standardized, and validated for quality using a combination of rules, plugins, AI models (such as Azure Machine Learning and Azure OpenAI), and third-party services. Semarchy xDM enhances these automated quality processes by supporting data recycling and integrating user corrections to maintain high data quality.
  • Matching, merging, and survivorship processes are executed automatically by blending advanced automation with user input, resulting in precise and dependable consolidated golden records.
  • Users access fully customized Data Management Applications to browse, search, manage data, and Collaborate with Data-Driven Workflows.

• Golden data distribution to consumer operational and analytical applications such as Azure Synapse Analytics, Power BI, Azure Machine Learning, and Azure OpenAI, ensures uninterrupted integration and usage across platforms. Semarchy xDM provides built-in SQL and REST endpoints, and Data Notifications for event-based propagation in Azure Service Bus.

• Semarchy xDM Metadata Synchronization with Microsoft Purview to gain comprehensive visibility and lineage of the entire master data flow.

• Provide business users and data stewards with full visibility and lineage of the overall master data flow, and expose the master data products to the organization as part of your catalog and governance initiatives.

Components

This architecture involves the following components.

Core Components

• Semarchy xDM is a no-code platform that allows data teams to quickly develop customized master data management solutions, offering a wide range of capabilities for complex data ecosystems.
• Microsoft Purview is a data governance solution that provides broad visibility into on-premises and cloud data estates. Semarchy xDM integrates with Microsoft Purview to provide insights into Semarchy data hubs as data products and end-to-end master data lineage.
• Semarchy Purview Connector synchronizes metadata from Semarchy data hubs with Microsoft Purview, linking logical model assets (that is, entities and attributes) and their corresponding physical assets (that is, tables and columns), and thereby enabling end-to-end data lineage.

Infrastructure

• Microsoft Entra ID, for user authentication and single sign-on to the Semarchy platform.
• Azure Key Vault is a cloud service that provides secure storage for secrets. You can use it to encrypt, decrypt, and store secrets (passwords, for example) used in xDM.

Integration Components

• Azure Data Factory is a hybrid data integration service. You can use Data Factory for inbound and outbound dataflows to the Semarchy data hubs.
• Azure Service Bus is a fully managed enterprise message broker, which you can use to propagate all changes occurring in the Semarchy Data Hubs using built-in Data Notifications.

AI Components

• Azure Machine Learning is a cloud service for accelerating and managing machine learning (ML) projects. Semarchy xDM can use customized Azure Machine Learning models in the data hub certification processes.
• Azure OpenAI is a suite of AI services providing access to OpenAI's powerful language models. Semarchy xDM includes built-in plug-ins using these language models to enrich and certify data, for example for content generation, summarization, or translation.

Source and Consumer Systems

Among others, this architecture includes the following systems from which you collect master data to be managed in Semarchy xDM, or to which you send golden data produced by Semarchy xDM.

• Azure Synapse Analytics is a fast, flexible, and trusted cloud data warehouse that uses a massively parallel processing architecture. Semarchy Data Hubs act as providers of certified metadata for Azure Synapse.
• Power BI is a business analytics suite that delivers insights throughout your organization. You can use Power BI to build dashboards and reports on top of Semarchy Discovery metrics and the Semarchy Data Hubs.

Scenario Details

Efficient integration of master data management (MDM) and data governance tools is crucial for organizations striving to maintain high-quality, consistent, and secure data across their operations. The combination of Semarchy xDM and Microsoft Purview provides a powerful solution for comprehensive data governance and master data management.

Comprehensive Data Governance and Master Data Management

*Unified Unified Cataloging and Governance

Microsoft Purview’s Unified Cataloging and discovery features automatically scan, classify, and map data, providing organizations with a comprehensive view of their data landscape. Integrated with Microsoft Purview, Semarchy xDM publishes detailed metadata for both logical and physical artifacts, simplifying data discovery and fostering trust in the data:

• Semarchy xDM instances, data locations, entities, attributes, and relationships are converted into Microsoft Purview entities using Semarchy-specific asset types.
• Processes associate the physical tables and depict the certification process for each entity.
• xDM entities are organized into Data Product as golden-record data.

This integration ensures that users can easily understand and utilize high-quality data for their analytical and operational needs, enhancing overall data governance and management efficiency.

Enhanced Data Lineage and Traceability

Microsoft Purview’s robust data lineage tracking feature illustrates how data moves and changes across different systems, providing a clear view of its journey. When integrated with Microsoft Purview, Semarchy xDM harnesses this lineage data to offer detailed insights into the origins and transformations of master data. This integration enhances transparency and traceability, ensuring that organizations have a comprehensive understanding of their data's history and evolution.

Improved Data Quality and Consistency

Data Quality Enforcement

Semarchy xDM prioritizes data quality through its data enrichment, standardization, and validation capabilities, ensuring that data is accurate, consistent, and reliable. When integrated with Microsoft Purview, Semarchy xDM extends these quality measures across all data assets, ensuring uniformity and reliability throughout the organization’s data landscape. This integration enhances overall data quality, reliability, and consistency, leading to better-informed decision-making and operational efficiency.

Holistic Data Governance

Microsoft Purview plays a pivotal role in establishing and enforcing data governance policies organization-wide. This ensures that data is managed and utilized in accordance with regulatory requirements and internal standards. When seamlessly integrated with Semarchy xDM, these governance policies can be directly applied to master data management processes. This integration ensures that compliance and governance principles are upheld throughout the entire data lifecycle, promoting data integrity, security, and regulatory compliance across the organization.

Operational Efficiency and Scalability

*Streamlined Data Integration and Management

Semarchy xDM streamlines the integration and management of master data from diverse sources, ensuring a unified and comprehensive view of data assets. Integrated with Microsoft Purview, organizations can govern these integrations effectively, ensuring that data flows adhere to governance policies and are optimized for efficiency and performance. This integration enhances data management capabilities, enabling organizations to harness the full potential of their data assets while maintaining governance and compliance standards.

Scalability and Flexibility

Both Microsoft Purview and Semarchy xDM are built to accommodate the evolving data needs of organizations. Their scalability ensures that they can effectively manage to grow volumes of data and adapt to changing business demands. Through seamless integration, organizations benefit from scalable and flexible data management and governance solutions. This integration enables them to meet current and future data challenges while maintaining efficiency and compliance with evolving business requirements.

Enhanced Collaboration and Data Utilization

Improved Collaboration

Microsoft Purview offers a centralized tool for effective data governance, facilitating seamless collaboration among teams working on data-related initiatives. Integrated with Semarchy xDM, this collaboration is further enhanced through advanced data stewardship and user collaboration tools. This integration ensures that data governance and master data management (MDM) efforts are closely aligned and well-coordinated, promoting synergy and efficiency across data governance initiatives within the organization.

Maximized Data Value

By integrating Semarchy xDM with Microsoft Purview, organizations ensure the availability of high-quality, well-governed master data for a wide range of applications, including analytics, business intelligence, and operational use. This integration maximizes the value derived from data assets, leading to improved business outcomes, and better-informed decision-making processes. Ultimately, it enables organizations to apply their data assets more effectively to drive growth, innovation, and competitive advantage.

Considerations

The Semarchy Data Platform has several features that address the issues of reliability, security, cost optimization, operational excellence, and performance efficiency. Further information about architectural excellence can be found in this article on the ** pillars of well-architected Azure frameworks.**

Reliability

Reliability ensures your application fulfills the promises you make to your customers. For more information, see Overview of the reliability pillar.

Semarchy xDM runs natively on Azure Kubernetes Service and Azure SQL Database, which offers out-of-the-box capabilities to support high availability.

Security

Security shields against intentional attacks and misuse of your valuable data and systems. For more information, see Security overview in Microsoft learn page.

Semarchy xDM authenticates users via its identity management layer that supports role mapping, lookup, and profile synchronization. It provides native support for multiple IDPs, including Microsoft Entra ID. It also includes advanced security features such as fine-grained privileges to secure access and operations.

Performance Efficiency

Performance efficiency is the capability of your system to scale and effectively meet user needs. For more information, see Performance Efficiency learn page in microsoft learn.

Semarchy xDM runs natively on Azure Kubernetes Service and Azure SQL Database. You can configure Azure Kubernetes Service to scale up and out. You can deploy and configure Azure SQL Database to balance performance, scalability, and costs.

Cost Optimization

Cost optimization involves finding ways to reduce unnecessary expenses and enhance operational efficiency. For more information, see Cost Optimization learn page in Microsoft learn.

Running costs consist of the Semarchy software subscription license and Azure consumption. Contact Semarchy for more information.

Deploy this scenario

For detailed instructions on deploying a Semarchy Data Management instance in Azure refer to Semarchy’s documentation.

To deploy this scenario:

1. Deploy Semarchy xDM using Azure Kubernetes Services.
2. Configure Secrets Management to use Azure Key Vault.
3. Configure Authentication with Microsoft Entra ID.
4. Design and deploy your customized master data model in xDM
5. Integrate your data into xDM using Azure Data Factory.

Contributors to this document

• David Cox
• Cedric Blanc
• François-Xavier Nicolas (FX)

Reference document

• Deploy Semarchy xDM
• Copy Secrets Management
• Configure Authentication with Microsoft Entra ID
• Integrate your data into xDM using Azure Data Factory
• Configure, deploy, and execute the Semarchy Purview connector as an Azure Functions application.