Document canonical plugin architecture

README.md

@@ -2,19 +2,374 @@
 
 A plugin-based DNS reconciliation service.
 
+## Architecture
+
+dns-sync separates generic reconciliation from application-specific endpoint
+behavior.
+
+    native endpoint
+          |
+          v
+    <App>Client
+          |
+          | native objects
+          v
+    Plugin
+          |
+          | DiscoveredObject
+          v
+    dns-sync core
+          |
+          | normalization
+          | observations
+          | desired state
+          | persistence
+          | reconciliation
+          | planning
+          v
+    PlanAction
+          |
+          v
+    Plugin
+          |
+          | native translation
+          v
+    <App>Client
+          |
+          v
+    native endpoint
+
+The core owns orchestration truth.
+
+Plugins own application semantics.
+
+Clients own endpoint-native communication.
+
 ## Plugin contract
 
 The reconciliation runtime is adapter-based.
 
-Adapter plugins expose:
+Every adapter package must export:
 
-- `Plugin`
 - `configured_endpoints(config)`
+- `Plugin`
 
-Each adapter declares object-specific discovery, apply, and verification
+The registry calls:
-capabilities. Built-in adapters live under `dns_sync/plugins/adapters/`.
+
-External adapters are loaded from `DNS_SYNC_PLUGIN_PATH` and must be
+    configured_endpoints(config)
-explicitly enabled through `ENABLED_PLUGINS`.
+
+to obtain the endpoints owned by the plugin.
+
+For each configured endpoint, the registry constructs:
+
+    Plugin(config, endpoint)
+
+Every endpoint returned by `configured_endpoints(config)` must identify the
+plugin that returned it. The registry rejects cross-plugin endpoint ownership.
+
+Plugins declare object-specific capabilities using the generic
+`Capability` model from `dns_sync.adapters`.
+
+Capabilities currently describe:
+
+- discovery
+- apply
+- verification
+
+for a particular generic object type.
+
+Built-in adapters live under:
+
+    dns_sync/plugins/adapters/
+
+External adapters are loaded from:
+
+    DNS_SYNC_PLUGIN_PATH
+
+and must be explicitly enabled through:
+
+    ENABLED_PLUGINS
+
+## Canonical plugin structure
+
+Application integrations should use this package structure:
+
+    <plugin_name>/
+    ├── __init__.py
+    └── client.py
+
+The repository contains a copyable example under:
+
+    examples/plugin_template/
+
+The example is documentation only and is not loaded from that location.
+
+### Adapter boundary: __init__.py
+
+`__init__.py` is the dns-sync-facing plugin boundary.
+
+It owns:
+
+- endpoint configuration through `configured_endpoints(config)`
+- generic capability declarations
+- validation of the supplied endpoint
+- native-to-generic discovery translation
+- generic-to-native apply translation
+- verification
+- optional retirement behavior
+- application-specific semantic validation
+
+The adapter communicates with the core using generic types such as:
+
+- `Endpoint`
+- `DiscoveredObject`
+- `NormalizedObject`
+- `ObjectReference`
+- `PlanAction`
+- `ApplyResult`
+- `RetireRequest`
+
+### Native boundary: client.py
+
+`client.py` is private implementation machinery for that application.
+
+It owns:
+
+- authentication
+- HTTP sessions, sockets, commands, or other transport
+- application-native request and response handling
+- application-native models
+- endpoint-local mutation primitives
+- endpoint-specific verification mechanics
+
+A client represents one explicit `Endpoint`.
+
+The client should not independently rediscover which endpoint it is supposed
+to contact from another global endpoint setting.
+
+There is deliberately no generic dns-sync `Client` base class. Different
+applications may communicate through completely different mechanisms. The
+shared contract is the adapter plugin boundary, not the transport layer.
+
+Application-specific models belong inside the plugin package rather than the
+dns-sync core unless they represent a genuinely application-independent core
+concept.
+
+## Discovery
+
+`Plugin.discover(object_type)` reads native endpoint state through the private
+client and returns `DiscoveredObject` instances.
+
+A discovered object carries:
+
+    object_type
+    local_key
+    payload
+    references
+
+`local_key` is endpoint-local identity. It is not assumed to be portable to
+another endpoint.
+
+The plugin owns translation from the native application's representation into
+this generic discovery representation.
+
+Unsupported object types should fail explicitly rather than silently returning
+an empty or misleading result.
+
+## Normalized state
+
+The core converts discoveries into `NormalizedObject` instances and owns the
+reconciliation state built from them.
+
+A normalized object carries:
+
+    object_type
+    logical_key
+    payload
+    local_key
+    references
+
+`logical_key` represents portable identity.
+
+`local_key` represents endpoint-local identity.
+
+Plugins do not persist their own reconciliation truth. The core owns:
+
+- observations
+- observation status
+- desired state
+- applied and verified state
+- reconciliation
+- planning
+- dependency resolution
+
+## Object references
+
+Dependencies between objects use `ObjectReference`.
+
+A reference identifies another object by exactly one of:
+
+    local_key
+
+or:
+
+    logical_key
+
+but never both.
+
+Endpoint discovery may expose local references because those are the identities
+native applications actually use.
+
+Central reconciliation can convert those into portable logical references.
+
+During execution, dns-sync resolves portable references back to the correct
+endpoint-local identity before invoking the target adapter.
+
+Plugins should never treat an endpoint-local ID from one endpoint as portable
+desired state for another endpoint.
+
+## Apply
+
+`Plugin.apply(action)` receives a generic `PlanAction`.
+
+Before mutation, the adapter should validate that the action belongs to its
+endpoint.
+
+Shared validation helpers are available in:
+
+    dns_sync.adapter_support
+
+including:
+
+    require_endpoint_plugin(...)
+    require_action_endpoint(...)
+    require_desired(...)
+
+`NOOP` and `SKIP` actions must not mutate the endpoint.
+
+For a mutating operation, the adapter should:
+
+1. validate the requested operation;
+2. validate the desired normalized object;
+3. translate desired state into an application-native object;
+4. invoke the private client;
+5. re-fetch or rediscover the actual resulting native object;
+6. obtain its real endpoint-local identity;
+7. return an `ApplyResult`.
+
+Do not guess or predict endpoint-local IDs when the endpoint assigns them.
+
+Unsupported operations should fail explicitly.
+
+## Apply results
+
+A successful mutation returns an `ApplyResult` whose normalized object contains
+the actual endpoint-local key.
+
+Conceptually:
+
+    ApplyResult(
+        object=normalized_applied_object,
+        detail=...,
+    )
+
+The returned object's `local_key` is required.
+
+An apply result may also include deferred `RetireRequest` objects when a
+replacement workflow requires the old object to remain temporarily while
+dependencies are rebound.
+
+## Verification
+
+`Plugin.verify(object)` checks that the actual endpoint state matches the
+normalized object dns-sync expects to exist.
+
+Verification is intentionally separate from mutation.
+
+A successful API response is not sufficient evidence that endpoint state
+converged.
+
+The executor requires supported mutations to pass verification before the
+result is treated as successfully applied.
+
+## Retirement
+
+`retire()` is optional.
+
+A plugin should implement retirement only when it can safely remove or retire
+endpoint-local objects.
+
+Retirement is used by deletion and replacement workflows.
+
+The executor performs preflight checks before mutation. If execution requires
+retirement and the adapter does not provide it, the operation fails before the
+unsafe mutation begins.
+
+Retirement should use the endpoint-local identity supplied by dns-sync and fail
+closed when the object cannot be identified safely.
+
+## Fail-closed behavior
+
+Plugins should reject ambiguous or unsupported behavior.
+
+In particular:
+
+- reject endpoints owned by another plugin;
+- reject actions intended for another endpoint;
+- reject unsupported object types;
+- reject unsupported plan operations;
+- reject malformed native state;
+- reject ambiguous endpoint-local identities;
+- never guess missing IDs;
+- never report successful application until the resulting object can be
+  identified;
+- never treat mutation success alone as verification.
+
+The generic executor adds additional preflight and dependency safety checks.
+
+## Configuration ownership
+
+Configuration should live at the lowest layer that actually understands it.
+
+Use this rule:
+
+    Does generic reconciliation need to understand this value?
+
+    yes -> core configuration may be appropriate
+    no  -> plugin/client configuration
+
+Examples of generic reconciliation policy include:
+
+    DNS_TARGET_IP
+    DOMAIN_MATCHING
+    DOMAIN
+
+Examples of plugin/client configuration include:
+
+    NPM_ENDPOINTS
+    PROXY_API_PORT
+    PROXY_API_SCHEME
+    NPM_IDENTITY
+    NPM_SECRET
+    VERIZON_ENDPOINTS
+    ROUTER_PASSWORD
+    ROUTER_TIMEOUT
+
+`configured_endpoints(config)` owns endpoint discovery for the plugin.
+
+The client receives the resulting explicit `Endpoint`; it should not implement
+a second independent endpoint-selection mechanism.
+
+## Secret handling
+
+Secret material should remain below the normalized and persistence boundary
+unless the core has an explicit safe representation for it.
+
+Plugins must not place credentials, authentication tokens, private keys, or
+other secret material into generic persisted payloads.
+
+Runtime credentials should normally be provided through environment variables,
+container secrets, or another external secret-management mechanism.
 
 ## Plugins
 
@@ -38,6 +393,125 @@ The Verizon adapter lives under `external_plugins/verizon` and is deliberately
 not copied into the dns-sync container image. The external plugin directory is
 mounted read-only at runtime.
 
+## Plugin loading
+
+Only explicitly enabled plugins are loadable.
+
+The registry first checks for a built-in adapter under:
+
+    dns_sync/plugins/adapters/<plugin_name>/
+
+If no built-in adapter exists, it looks for an external package at:
+
+    $DNS_SYNC_PLUGIN_PATH/<plugin_name>/__init__.py
+
+External plugins are loaded as packages, so they may use relative imports for
+private modules such as:
+
+    from .client import ExampleClient
+
+Imports from dns-sync itself should remain absolute, for example:
+
+    from dns_sync.adapters import Endpoint
+    from dns_sync.adapter_support import require_endpoint_plugin
+
+The external plugin directory does not need to be added globally to
+`PYTHONPATH`.
+
+## Adding a new plugin
+
+Start from:
+
+    examples/plugin_template/
+
+For an external plugin:
+
+1. copy the template into the configured external plugin root;
+2. rename the directory to the plugin name;
+3. rename `ExampleClient` and `ExampleRecord`;
+4. implement `configured_endpoints(config)`;
+5. implement native authentication and transport in `client.py`;
+6. declare only the generic capabilities actually supported;
+7. implement native-to-`DiscoveredObject` translation;
+8. implement `NormalizedObject`-to-native translation;
+9. implement `apply()` and `verify()`;
+10. implement `retire()` only when safe retirement is supported;
+11. add focused client and adapter tests;
+12. enable the plugin through `ENABLED_PLUGINS`;
+13. run read-only reconciliation against the real endpoint before permitting
+    mutations.
+
+Do not begin a new integration by adding application-specific models or
+transport behavior to the core.
+
+## Reference implementations
+
+Use both existing plugins when designing a new integration.
+
+### NPM
+
+The NPM plugin demonstrates:
+
+- a built-in adapter;
+- multiple configured endpoints;
+- proxy-host and certificate object types;
+- endpoint-local object IDs;
+- cross-object certificate references;
+- portable logical references;
+- replacement workflows;
+- deferred retirement;
+- certificate material handling.
+
+Its package lives at:
+
+    dns_sync/plugins/adapters/npm/
+
+### Verizon
+
+The Verizon plugin demonstrates:
+
+- an external adapter;
+- plugin-owned endpoint configuration;
+- a private native record model;
+- a private endpoint client;
+- DNS record discovery;
+- create/update operations;
+- verification;
+- a read-only mounted external plugin package.
+
+Its package lives at:
+
+    external_plugins/verizon/
+
+Together they demonstrate the preferred boundary:
+
+    dns-sync core
+        |
+        +-- generic state and orchestration
+        |
+        +-- Plugin
+              |
+              +-- native / normalized translation
+              |
+              +-- <App>Client
+                    |
+                    +-- application-native implementation
+
+## Plugin development validation
+
+Before committing a plugin change, run at least:
+
+    git diff --check
+    python3 -m compileall -q dns_sync external_plugins tests examples
+    python3 -m unittest discover -s tests -v
+
+Changes involving plugin loading, native clients, endpoint configuration, or
+mutation behavior should also be validated with a real read-only container
+reconciliation against the intended endpoints.
+
+A clean read-only plan does not by itself authorize mutation. Review unexpected
+planned changes before enabling apply mode.
+
 ## Container runtime
 
 The container periodically executes the generic reconciliation engine:

examples/plugin_template/README.md

@@ -0,0 +1,44 @@
+# dns-sync plugin template
+
+This directory is a documentation/example package. dns-sync does not load it
+from this location.
+
+Copy it to one of these locations and rename it for the application:
+
+    dns_sync/plugins/adapters/<plugin_name>/
+
+for a built-in plugin, or:
+
+    $DNS_SYNC_PLUGIN_PATH/<plugin_name>/
+
+for an external plugin.
+
+The canonical package layout is:
+
+    <plugin_name>/
+    ├── __init__.py
+    └── client.py
+
+`__init__.py` is the dns-sync adapter boundary. It owns endpoint discovery,
+generic capabilities, native-to-normalized translation, apply, verify, and
+optional retirement behavior.
+
+`client.py` is private to the plugin. It owns authentication, transport,
+application-specific models, native request/response handling, and endpoint
+operations.
+
+## Before enabling a copied plugin
+
+1. Rename `example`, `ExampleClient`, and `ExampleRecord`.
+2. Implement `configured_endpoints(config)`.
+3. Implement the native client methods.
+4. Declare only capabilities the plugin really supports.
+5. Implement discovery for those object types.
+6. Translate normalized desired state into native state in `apply()`.
+7. Re-discover mutations and return the real endpoint-local key.
+8. Implement `verify()`.
+9. Add `retire()` only when retirement/deletion is safely supported.
+10. Add focused adapter/client tests.
+11. Validate using read-only reconciliation before enabling mutations.
+
+See the root `README.md` for the complete plugin contract and architecture.

examples/plugin_template/__init__.py

@@ -0,0 +1,208 @@
+"""Canonical dns-sync adapter plugin template."""
+
+import os
+
+from dns_sync.adapter_support import (
+    require_action_endpoint,
+    require_desired,
+    require_endpoint_plugin,
+)
+from dns_sync.adapters import (
+    DNS_RECORDS,
+    ApplyResult,
+    Capability,
+    CapabilityOperation,
+    DiscoveredObject,
+    NormalizedObject,
+    PlanAction,
+    PlanOperation,
+)
+from dns_sync.config import parse_endpoint_list
+
+from .client import ExampleClient, ExampleRecord
+
+
+PLUGIN_NAME = "example"
+
+
+def configured_endpoints(config):
+    """Return every endpoint owned by this plugin."""
+    return parse_endpoint_list(
+        PLUGIN_NAME,
+        os.getenv(
+            "EXAMPLE_ENDPOINTS",
+            "example.test",
+        ),
+        443,
+    )
+
+
+class Plugin:
+    """dns-sync adapter for one configured endpoint."""
+
+    name = PLUGIN_NAME
+
+    capabilities = frozenset(
+        {
+            Capability(
+                DNS_RECORDS,
+                CapabilityOperation.DISCOVER,
+            ),
+            Capability(
+                DNS_RECORDS,
+                CapabilityOperation.APPLY,
+            ),
+            Capability(
+                DNS_RECORDS,
+                CapabilityOperation.VERIFY,
+            ),
+        }
+    )
+
+    def __init__(self, config, endpoint):
+        require_endpoint_plugin(
+            endpoint,
+            self.name,
+        )
+
+        self.endpoint = endpoint
+        self.client = ExampleClient(endpoint)
+
+    def connect(self) -> None:
+        self.client.connect()
+
+    def discover(
+        self,
+        object_type: str,
+    ) -> list[DiscoveredObject]:
+        if object_type != DNS_RECORDS:
+            raise RuntimeError(
+                f"{self.endpoint.identity} cannot discover "
+                f"object type {object_type!r}"
+            )
+
+        return [
+            DiscoveredObject(
+                object_type=DNS_RECORDS,
+                local_key=record.local_key,
+                payload={
+                    "hostname": record.hostname,
+                    "ip": record.ip,
+                    "record_type": record.record_type,
+                },
+            )
+            for record in self.client.list_records()
+        ]
+
+    def _native_record(
+        self,
+        object: NormalizedObject,
+    ) -> ExampleRecord:
+        if object.object_type != DNS_RECORDS:
+            raise RuntimeError(
+                f"{self.endpoint.identity} cannot handle "
+                f"object type {object.object_type!r}"
+            )
+
+        payload = object.payload
+        hostname = str(payload.get("hostname") or "")
+
+        if hostname != object.logical_key:
+            raise RuntimeError(
+                "DNS object hostname does not match its logical key"
+            )
+
+        record_type = str(
+            payload.get("record_type") or ""
+        ).upper()
+
+        if record_type != "A":
+            raise RuntimeError(
+                f"Example plugin does not support "
+                f"{record_type!r} records"
+            )
+
+        if payload.get("wildcard") is True:
+            raise RuntimeError(
+                "Example plugin does not support wildcard records"
+            )
+
+        return ExampleRecord(
+            local_key=object.local_key or "",
+            hostname=hostname,
+            ip=str(payload.get("ip") or ""),
+            record_type=record_type,
+        )
+
+    def apply(
+        self,
+        action: PlanAction,
+    ) -> ApplyResult | None:
+        require_action_endpoint(
+            action,
+            self.endpoint,
+        )
+
+        if action.operation in {
+            PlanOperation.NOOP,
+            PlanOperation.SKIP,
+        }:
+            return None
+
+        desired = require_desired(
+            action,
+            context="Example apply action",
+        )
+
+        record = self._native_record(desired)
+
+        if action.operation == PlanOperation.CREATE:
+            detail = self.client.add_record(record)
+        elif action.operation == PlanOperation.UPDATE:
+            detail = self.client.update_record(record)
+        else:
+            raise RuntimeError(
+                f"Example adapter does not support "
+                f"{action.operation.value} actions"
+            )
+
+        records = self.client.list_records()
+        applied = next(
+            (
+                candidate
+                for candidate in records
+                if candidate.hostname.lower()
+                == record.hostname.lower()
+            ),
+            None,
+        )
+
+        if applied is None:
+            raise RuntimeError(
+                "Example mutation succeeded but the object "
+                "could not be rediscovered"
+            )
+
+        normalized = NormalizedObject(
+            object_type=desired.object_type,
+            logical_key=desired.logical_key,
+            local_key=applied.local_key,
+            payload=dict(desired.payload),
+            references=desired.references,
+        )
+
+        return ApplyResult(
+            object=normalized,
+            detail=detail,
+        )
+
+    def verify(
+        self,
+        object: NormalizedObject,
+    ) -> bool:
+        return self.client.verify_record(
+            self._native_record(object)
+        )
+
+    # Add retire(self, request) only when this plugin supports safe
+    # deletion/replacement retirement.

examples/plugin_template/client.py

@@ -0,0 +1,56 @@
+"""Private native client for the example dns-sync plugin."""
+
+from dataclasses import dataclass
+
+from dns_sync.adapters import Endpoint
+
+
+@dataclass(frozen=True)
+class ExampleRecord:
+    """Endpoint-native record used only by this plugin."""
+
+    local_key: str
+    hostname: str
+    ip: str
+    record_type: str = "A"
+
+
+class ExampleClient:
+    """Client for one explicit example endpoint."""
+
+    def __init__(self, endpoint: Endpoint):
+        if endpoint.plugin != "example":
+            raise RuntimeError(
+                "ExampleClient requires an example endpoint"
+            )
+
+        self.endpoint = endpoint
+
+    def connect(self) -> None:
+        """Authenticate or initialize the native connection."""
+        raise NotImplementedError
+
+    def list_records(self) -> list[ExampleRecord]:
+        """Return endpoint-native records."""
+        raise NotImplementedError
+
+    def add_record(
+        self,
+        record: ExampleRecord,
+    ) -> str | None:
+        """Create one native record."""
+        raise NotImplementedError
+
+    def update_record(
+        self,
+        record: ExampleRecord,
+    ) -> str | None:
+        """Update one native record."""
+        raise NotImplementedError
+
+    def verify_record(
+        self,
+        record: ExampleRecord,
+    ) -> bool:
+        """Verify endpoint-native state."""
+        raise NotImplementedError

tests/test_plugin_template.py

@@ -0,0 +1,73 @@
+import os
+import shutil
+import tempfile
+import unittest
+from pathlib import Path
+from types import SimpleNamespace
+from unittest.mock import patch
+
+from dns_sync.registry import (
+    configured_adapter_endpoints,
+    load_adapter,
+)
+
+
+class PluginTemplateTests(unittest.TestCase):
+    def test_template_loads_as_external_plugin(self):
+        source = Path("examples/plugin_template")
+
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            plugin = root / "example"
+
+            shutil.copytree(
+                source,
+                plugin,
+            )
+
+            with patch.dict(
+                os.environ,
+                {
+                    "ENABLED_PLUGINS": "example",
+                    "DNS_SYNC_PLUGIN_PATH": directory,
+                    "EXAMPLE_ENDPOINTS": "example.test",
+                },
+                clear=True,
+            ):
+                config = SimpleNamespace()
+
+                endpoints = configured_adapter_endpoints(
+                    "example",
+                    config,
+                )
+
+                self.assertEqual(
+                    tuple(
+                        endpoint.identity
+                        for endpoint in endpoints
+                    ),
+                    ("example://example.test:443",),
+                )
+
+                adapter = load_adapter(
+                    "example",
+                    config,
+                    endpoints[0],
+                )
+
+                self.assertEqual(
+                    adapter.name,
+                    "example",
+                )
+                self.assertEqual(
+                    adapter.endpoint,
+                    endpoints[0],
+                )
+                self.assertEqual(
+                    adapter.client.endpoint,
+                    endpoints[0],
+                )
+
+
+if __name__ == "__main__":
+    unittest.main()