Plugin Architecture & Registration

Plugin Types

Plugin (plugins/traits.rs) is Send + Sync and requires name(); version(), initialize(), shutdown(), description(), and author() have defaults. There is no 'static trait bound; registry-owned Arc<dyn Trait> supplies the necessary lifetime.

Native Rust extractors implement InternalDocumentExtractor

DocumentExtractor is the binding-facing surface. In-crate extractors implement InternalDocumentExtractor and get DocumentExtractor from a blanket impl. Implementing DocumentExtractor directly in this crate is the wrong layer.

#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
impl InternalDocumentExtractor for MyExtractor {
    async fn extract_content(&self, content: &[u8], mime_type: &str, config: &ExtractionConfig)
        -> Result<InternalDocument> { /* ... */ }

    fn supported_mime_types(&self) -> &[&str] { &["application/x-custom"] }
    fn priority(&self) -> i32 { 50 }
}

extract_path has a default that reads the file and delegates to extract_content (and errors without tokio-runtime).

Always use the two-arm cfg_attr form for async_trait. A bare #[async_trait] does not match the trait declaration on wasm32.

The public trait has exactly four items — extract, supported_mime_types, priority, can_handle. There is no as_sync_extractor; writing one is a compile error. WASM sync support is the separate SyncExtractor trait — see wasm-constraints.

Priority System

The registry selects the highest priority extractor for each MIME type. The ranges are conventions over an unclamped i32; negative and values above 100 are representable. Equal MIME and priority is a collision: the later registration replaces the earlier entry and warns. Give competing plugins distinct priorities.

Registration

// crates/xberg/src/extractors/mod.rs -> register_default_extractors()
let registry = get_document_extractor_registry();
let mut registry = registry.write();
registry.register(Arc::new(MyExtractor::new()))?;

Feature-gate optional formats:

#[cfg(feature = "office")]
{
    registry.register(Arc::new(DocxExtractor::new()))?;
    registry.register(Arc::new(PptxExtractor::new()))?;
}

PostProcessor Pattern

#[cfg_attr(not(target_arch = "wasm32"), async_trait)]
#[cfg_attr(target_arch = "wasm32", async_trait(?Send))]
impl PostProcessor for MyProcessor {
    async fn process(&self, result: &mut ExtractedDocument, config: &ExtractionConfig)
        -> Result<()> {
        result.content = process_content(&result.content);
        Ok(())
    }
    fn processing_stage(&self) -> ProcessingStage { ProcessingStage::Middle }
}

The enum is ProcessingStage and the accessor is processing_stage(). Stages: Early (default) → Middle → Late. process takes &mut ExtractedDocument, not an owned result.

Critical Rules

1. All plugins MUST be Send + Sync — Plugin requires it.
2. In-crate extractors implement InternalDocumentExtractor, never DocumentExtractor.
3. Use the two-arm cfg_attr async_trait form on every plugin trait impl.
4. Feature-gate optional formats with #[cfg(feature = "...")] at the registration site.
5. Initialization is lazy via ensure_initialized() (extractors/mod.rs), called before first extraction.
6. Plugin names are kebab-case (e.g. "pdf-extractor").
7. A new extractor struct needs #[cfg_attr(alef, alef(skip))] or the binding regen aborts — see alef-generated-bindings.

Registry and lifecycle invariants

• Eight plugin types have eight process-global typed registries in plugins/registry/mod.rs. There is no universal PluginRegistry.
• Registries use Arc<parking_lot::RwLock<_>>. Their guards are not poisoned and .read()/.write() return guards directly.
• Extractor lookup is HashMap<mime, BTreeMap<priority, entry>>: exact MIME lookup is constant-time on the outer map; wildcard-family lookup scans registered MIME keys.
• Registration calls initialize() and rejects a plugin whose initialization fails. Registries support register, remove, clear, and shutdown_all; there is no hot reload.
• All eight plugin types can be registered from language bindings. Plugin interfaces are public APIs, so breaking changes follow the public compatibility policy.
• Return errors rather than panicking. Test lifecycle, collision/replacement, concurrent access, and failure paths with test doubles; use real backends for integration coverage. No dispatch-overhead benchmark exists unless one is explicitly added.

Alef-generated Python bridge

The Python bridge is generated into crates/xberg-py/src/lib.rs; there is no hand-written plugins.rs. Change Alef/configuration and regenerate rather than editing the bridge.

• PyO3 0.29 uses Python::attach. Async host calls enter Python from tokio::task::spawn_blocking and propagate the caller's contextvars context.
• Cache frequently accessed host data such as plugin names in Rust fields so infallible methods do not need repeated GIL acquisition. Do not assume allow_threads is in use.
• Every trait method return crosses the bridge through native extraction or JSON fallback. Crossing types therefore need Serialize + Deserialize + Default, including unit enums.
• Host exceptions become XbergError::Other with plugin and method context; the original Python exception type and traceback are not retained. Infallible methods can only warn and return Default::default(), so a default may indicate bridge failure rather than real data.
• Rust-side extractor plugin failures may use XbergError::Plugin, which is fallback-eligible; do not assume Python bridge errors have the same fallback behavior.
• Validate the Python protocol at registration. Do not quote GIL overhead without a current benchmark.