Data Mesh and the City Planner

City planning is a complex beast. Planners must predict population growth and its impact on zoning, utilities, transportation, social services, and regulations to name a few. A city planner rarely, if ever, gets a chance to start from scratch. They must take the current architecture and infrastructure into consideration when planning the future. 

Data mesh planning is a lot like city planning. Planners want to provide as much freedom and flexibility as possible to encourage business growth. To do so, they must provide zoning regulations, common infrastructure services and governance capability, which promote both efficient and sustainable growth. 

Zoning Regulations

 
Cities provide borders and rules for land management via zoning regulations. Likewise, data mesh provides borders for data products via business domain boundaries and rules for data products via data management practices. Each domain team is responsible for providing reliable, high-quality information to everyone within the data mesh community. They must instill confidence of their data products by ensuring good data management practices such as data quality, security, discoverability, accessibility, usability, and interoperability, are consistently applied.

Infrastructure Services

 
Cities provide common services in the form of public utilities, roads, mass transportation, and emergency services, e.g., local fire, police and medical. Shared infrastructure and social services facilitate more efficient development and maintenance of businesses and homes. This in turn enables citizens and businesses to be more productive. 

Data mesh provides infrastructure services which support data product development and operations via a self-serve data platform. Each organization can subscribe to common services from the cloud marketplace or build their own. An example of a common need is metadata management. Data product teams are responsible for the creation and maintenance of all their technical, operational, and business metadata. Collaborative metadata, which captures information like consumer ratings and feedback, is created and maintained by the data product consumers. Planners must provide clarity on how metadata will be managed, internally within and externally from data products, and how those services will be provided. Other potential candidates for common services might be around information security, observability, and reference data management.   

Governance

 
The city planner must also consider governance polices and compliance. The city planner must be aware of federal, state and city policies and regulations, clarify the rationale behind new or modified city regulation requests, and understand their enforcement capabilities. If there are holes in the regulations or gaps within the enforcement capabilities, regulations will be ineffective, and issues will emerge. For example, unsafe drinking water, zone misuse, transportation gridlock and city fund mismanagement erode public trust and can result in costly litigation.

Data mesh, likewise, requires governance to ensure the trust and confidence that the data products are being delivered via federated governance. This means regulations, standards and best practices must be clearly defined and published to ensure everyone is keenly aware of overall governing principles and their roles and responsibilities; and secondly, but just as important, processes to verify their compliance. For example, the ability to detect PII, PCI, PHI and other regulator violations is critical to organizations financially, socially, and ethically.

Organizations, just as cities, also have financial constraints. In the subscription services model, financial governance is critical to continuously track resource spend, identify potential offenders, and proactively address them before they cause an overrun of the budget. For example, automated monitoring and management of resources via workload management capabilities can dynamically prioritize the needs of the user community without negatively impacting their productivity. Observability capabilities can identify biggest offenders, which drive opportunities to refactor code or redesign data structures to be more performant. Without these types of capabilities, resources can dynamically scale up and out of control.

Like city planners, data mesh planners are not starting from scratch. Within the analytic ecosystem, most of these concepts have existed for years. Some organizations are leveraging these mature data management concepts effectively and may not choose to change. For those who do decide to transition to data mesh, there is no data mesh easy button. And mergers and acquisitions may also find themselves straddling a traditional and data mesh approach. In either case, both approaches will co-exist within organizations for some time. 

Transitioning to data mesh is a journey based on business-driven initiatives and delivering business value. Planners must provide clear guidelines on zone regulations, infrastructure services and governance to ensure efficient and sustainable growth within the overall community.