尧图网站建设 
FiftyOne × Databricks Mosaic AI Search为视觉数据构建可查询的向量相似性索引【免费下载链接】fiftyoneRefine high-quality datasets and visual AI models项目地址: https://gitcode.com/GitHub_Trending/fi/fiftyoneDatabricks Mosaic AI Search 是内置于 Databricks Data Intelligence Platform 的向量数据库与平台的数据治理与生产力工具深度集成。本文以 FiftyOne 官方集成文档为主线系统讲解如何在 FiftyOne 中一键创建 Mosaic 向量索引、上传 embeddings、执行相似性查询Python API 与 App 点选均可并覆盖环境准备、三种认证方式、配置参数、索引管理与常见查询范式帮助你在计算机视觉数据上直接复用 Mosaic 的向量检索能力。本文内容对应 FiftyOne OSS 1.4.0 与 FiftyOne Enterprise 2.7.0 及以上版本见 docs/source/integrations/mosaic.rst 中的版本声明适用的核心 API 位于fiftyone.brain.compute_similarity与SampleCollection.sort_by_similarity。提示借助 Mosaic 相似性索引你还可以直接用自然语言搜索数据当底层模型支持文本提示时详见 FiftyOne 的文本相似性搜索能力。一、基本工作流Basic Recipe在 FiftyOne 数据集上使用 Mosaic 建立相似性索引并查询数据完整流程如下连接你的 Databricks workspace并创建一个 AI Search endpoint向量搜索端点。将数据集加载到 FiftyOne。为数据集中的样本samples或对象块patches计算 embedding 向量或选择一个模型来生成 embeddings。调用compute_similarity()方法设置backendmosaic并指定自定义的brain_key为样本或对象块生成 Mosaic 相似性索引。使用该 Mosaic 索引通过sort_by_similarity()查询数据。可选删除索引。运行上述流程前你需要一个已启用 AI Search 的 Databricks 账号并安装 Databricks AI Search Python 包pip install databricks-ai-search同时按下文「认证」一节配置凭据避免每次与 Mosaic 索引交互时手动输入。下面是最小可运行示例——先加载数据集并直接由 FiftyOne 计算 embeddings、创建索引import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz # Step 1: Load your data into FiftyOne dataset foz.load_zoo_dataset(quickstart) # Steps 2 and 3: Compute embeddings and create a similarity index mosaic_index fob.compute_similarity( dataset, brain_keymosaic_index, backendmosaic, )索引生成后通过brain_key查询数据# Step 4: Query your data query dataset.first().id # query by sample ID view dataset.sort_by_similarity( query, brain_keymosaic_index, k10, # limit to 10 most similar samples ) # Step 5 (optional): Cleanup # Delete the Mosaic index mosaic_index.cleanup() # Delete run record from FiftyOne dataset.delete_brain_run(mosaic_index)可以看到compute_similarity()在不指定model或embeddings时会自动用默认模型为全部样本计算向量并建索引sort_by_similarity()是定义在 fiftyone/core/collections.py 上的视图阶段签名如下def sort_by_similarity(self, query, kNone, reverseFalse, dist_fieldNone, brain_keyNone)其中query可以是样本/块的 ID、与索引同维度的向量、ID 列表或文本提示k控制返回的最近邻数量默认对整个集合排序reverseTrue则按最不相似排序部分后端可能不支持。二、环境准备与安装Setup使用 Mosaic AI Search 的前提拥有满足 AI Search 要求的 Databricks workspace已创建 AI Search endpoint向量搜索端点在 Databricks 中准备好用于创建 AI Search 索引的 catalog 与 schema命名空间。安装 Mosaic AI Vector Search 客户端要使用 Mosaic 后端必须安装 Databricks Vector Search Python 包pip install databricks-ai-search切换到 Mosaic 后端默认情况下compute_similarity()与sort_by_similarity()使用 sklearn 后端。要改用 Mosaic只需将compute_similarity()的可选参数backend设为mosaicimport fiftyone.brain as fob fob.compute_similarity(..., backendmosaic, ...)也可以永久性地将 FiftyOne 默认相似性后端切换为 Mosaic两种方式任选其一方式一环境变量export FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKENDmosaic方式二brain 配置文件编辑位于~/.fiftyone/brain_config.json的 brain 配置设置default_similarity_backend{ default_similarity_backend: mosaic }关于默认后端的三种设置层级单次调用传参 会话级环境变量 持久化配置文件在 docs/source/user_guide/similarity.rst 中有完整说明该文档同时列出了所有内置后端的配置类其中 Mosaic 对应fiftyone.brain.internal.core.mosaic.MosaicSimilarityConfig实现位于独立的fiftyone-brain包。三、认证AuthenticationDatabricks 凭据可以通过多种方式提供以下三种由官方文档明确支持。环境变量推荐把凭据存入环境变量FiftyOne 每次建立 Databricks 连接时自动读取export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_WORKSPACE_URLhttps://unique-url.cloud.databricks.com/ export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_PERSONAL_ACCESS_TOKENXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_CATALOG_NAMEXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_SCHEMA_NAMEXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_ENDPOINT_NAMEXXXXXXXX这是认证 Mosaic 客户端的一组示例变量Databricks 还支持更多认证方式如服务主体、OAuth 等可按需扩展。FiftyOne Brain 配置在~/.fiftyone/brain_config.json中存放凭据注意该文件默认不存在需自行创建{ similarity_backends: { mosaic: { workspace_url: https://unique-url.cloud.databricks.com/, personal_access_token: XXXXXXXX, catalog_name: XXXXXXXX, schema_name: XXXXXXXX, endpoint_name: XXXXXXXX } } }关键字参数也可以在每次调用需要连接 Databricks 的方法如compute_similarity()时手动传入凭据import fiftyone.brain as fob mosaic_index fob.compute_similarity( ... backendmosaic, brain_keymosaic_index, workspace_url https://unique-url.cloud.databricks.com/, personal_access_token XXXXXXXX, catalog_name XXXXXXXX, schema_name XXXXXXXX, endpoint_name XXXXXXXX )需要特别注意的是采用这种策略时之后通过load_brain_results()加载索引时也必须手动再次提供全部凭据mosaic_index dataset.load_brain_results( mosaic_index, workspace_url https://unique-url.cloud.databricks.com/, personal_access_token XXXXXXXX, catalog_name XXXXXXXX, schema_name XXXXXXXX, endpoint_name XXXXXXXX )四、Mosaic 配置参数Mosaic 后端支持以下参数定制相似性查询index_nameNone要使用或创建的 Mosaic AI Search 索引名称。若不指定会自动生成一个新的唯一名称。该参数可通过上文任一策略指定。以下是一个包含全部可用参数的 brain 配置示例{ similarity_backends: { mosaic: { index_name: your-index } } }不过实践中更常见的做法是直接把它传给compute_similarity()用于配置某个特定的新索引mosaic_index fob.compute_similarity( ... backendmosaic, brain_keymosaic_index, index_nameyour-index, )从实现角度看这类参数最终会汇聚到MosaicSimilarityConfigfiftyone.brain.internal.core.mosaic.MosaicSimilarityConfig中每次调用compute_similarity()传入的关键字参数、brain_config.json中similarity_backends.mosaic段落的配置以及对应的FIFTYONE_BRAIN_SIMILARITY_MOSAIC_*环境变量共同决定了后端实例化时使用的连接信息与索引标识。五、管理 Brain RunsFiftyOne 提供一系列方法管理 brain runs即相似性计算在数据集上留下的运行记录。列出 brain runs——查看数据集上可用的 brain keysimport fiftyone.brain as fob # List all brain runs dataset.list_brain_runs() # Only list similarity runs dataset.list_brain_runs(typefob.Similarity) # Only list specific similarity runs dataset.list_brain_runs( typefob.Similarity, patches_fieldground_truth, supports_promptsTrue, )获取 brain run 信息——查看某个 brain run 的配置info dataset.get_brain_info(brain_key) print(info)加载索引实例——用load_brain_results()取回该 brain run 对应的SimilarityIndex实例。重命名 brain keydataset.rename_brain_run(brain_key, new_brain_key)删除运行记录dataset.delete_brain_run(brain_key)上述方法均定义在 fiftyone/core/collections.py 中例如list_brain_runsL4927、rename_brain_runL4949、get_brain_infoL4958、load_brain_resultsL4970、delete_brain_runL5008它们对所有相似性后端统一生效。注意delete_brain_run()只删除 FiftyOne 数据集上的 brain run记录不会删除关联的 Mosaic 索引。删除 Mosaic 索引需显式执行# Delete the Mosaic index mosaic_index dataset.load_brain_results(brain_key) mosaic_index.cleanup()六、实战示例常见向量检索工作流以下示例均假设已按上文配置好 Databricks 账号与凭据。创建相似性索引创建新的 Mosaic 相似性索引需为compute_similarity()提供embeddings或model参数共有四种常见写法import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) model_name clip-vit-base32-torch model foz.load_zoo_model(model_name) brain_key mosaic_index # Option 1: Compute embeddings on the fly from model name fob.compute_similarity( dataset, modelmodel_name, backendmosaic, brain_keybrain_key, ) # Option 2: Compute embeddings on the fly from model instance fob.compute_similarity( dataset, modelmodel, backendmosaic, brain_keybrain_key, ) # Option 3: Pass pre-computed embeddings as a numpy array embeddings dataset.compute_embeddings(model) fob.compute_similarity( dataset, embeddingsembeddings, backendmosaic, brain_keybrain_key, ) # Option 4: Pass pre-computed embeddings by field name dataset.compute_embeddings(model, embeddings_fieldembeddings) fob.compute_similarity( dataset, embeddingsembeddings, backendmosaic, brain_keybrain_key, )四种方式的差异在于 embeddings 的供给途径前两种由 FiftyOne 在创建索引时实时计算按模型名或模型实例后两种复用已算好的向量numpy 数组或数据集字段。创建对象块patch相似性索引通过patches_field参数可以为数据集内的目标检测对象块建立相似性索引每个检测框对应的图像块作为一个检索单元import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, patches_fieldground_truth, modelclip-vit-base32-torch, backendmosaic, brain_keymosaic_patches, )连接已有索引如果 Mosaic 中已存在存储了样本/块向量的索引可通过index_name直接连接并用embeddingsFalse跳过向量计算import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, modelclip-vit-base32-torch, # zoo model used (if applicable) embeddingsFalse, # dont compute embeddings index_nameyour-index, # the existing Mosaic index brain_keymosaic_index, backendmosaic, )向索引增删 embeddings修改数据集后可用add_to_index()与remove_from_index()同步更新 Mosaic 索引import numpy as np import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) print(mosaic_index.total_index_size) # 200 view dataset.take(10) ids view.values(id) # Delete 10 samples from a dataset dataset.delete_samples(view) # Delete the corresponding vectors from the index mosaic_index.remove_from_index(sample_idsids) # Add 20 samples to a dataset samples [fo.Sample(filepathtmp%d.jpg % i) for i in range(20)] sample_ids dataset.add_samples(samples) # Add corresponding embeddings to the index embeddings np.random.rand(20, 512) mosaic_index.add_to_index(embeddings, sample_ids) print(mosaic_index.total_index_size) # 210注意示例中的 embeddings 维度为 512需与索引所用模型这里是 CLIP的输出维度保持一致。从索引取回 embeddings用get_embeddings()按 ID 从 Mosaic 索引取回向量import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) # Retrieve embeddings for the entire dataset ids dataset.values(id) embeddings, sample_ids, _ mosaic_index.get_embeddings(sample_idsids) print(embeddings.shape) # (200, 512) print(sample_ids.shape) # (200,) # Retrieve embeddings for a view ids dataset.take(10).values(id) embeddings, sample_ids, _ mosaic_index.get_embeddings(sample_idsids) print(embeddings.shape) # (10, 512) print(sample_ids.shape) # (10,)七、查询 Mosaic 索引通过在任意数据集或视图上追加sort_by_similarity()阶段即可查询 Mosaic 索引。查询可以是以下任意一种一个 ID样本或块一个与索引同维度的查询向量一个 ID 列表样本或块一段文本提示当模型支持时import numpy as np import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) # Query by vector query np.random.rand(512) # matches the dimension of CLIP embeddings view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by sample ID query dataset.first().id view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by a list of IDs query [dataset.first().id, dataset.last().id] view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by text prompt query a photo of a dog view dataset.sort_by_similarity(query, k10, brain_keymosaic_index)关键语义对DatasetView执行相似性搜索时只会返回该视图内的结果如果视图中包含未纳入索引的样本它们永远不会出现在结果里。这意味着你可以对整个Dataset建立一次索引之后通过构造包含目标图片的视图在数据集任意子集上反复检索——这正是 docs/source/user_guide/similarity.rst 所描述的通用相似性 API 与视图系统协同的最佳实践。八、访问底层 Mosaic 客户端每个 Mosaic 索引都暴露client属性可直接访问底层的 Mosaic 客户端实例并调用其方法import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) mosaic_client mosaic_index.client print(mosaic_client)这为高级场景如直接使用 Databricks Vector Search 原生 API 做索引管理、元数据过滤等留出了充分的扩展空间。九、高级用法如「Mosaic 配置参数」一节所述通过向compute_similarity()传入可选参数可以定制 Mosaic 索引。下面是一个综合示例指定自定义索引名并且只对数据集的一个子集填充向量先用embeddingsFalse创建空索引再分批add_to_indeximport fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) # Create a custom Mosaic index mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, embeddingsFalse, # well add embeddings below brain_keymosaic_index, backendmosaic, index_namecustom-quickstart-index, ) # Add embeddings for a subset of the dataset view dataset.take(10) embeddings, sample_ids, _ mosaic_index.compute_embeddings(view) mosaic_index.add_to_index(embeddings, sample_ids)这里的mosaic_index.compute_embeddings(view)是SimilarityIndex实例提供的能力与add_to_index()组合即可实现先建索引、后分批灌入向量的增量式构建。十、实现要点与源码对照从本仓库源码可以进一步确认 Mosaic 集成在 FiftyOne 中的落点sort_by_similarity()是SampleCollection上的视图阶段方法完整签名与 docstring 见 fiftyone/core/collections.py它要求先通过fiftyone.brain.compute_similarity建立相似性索引并支持 ID、向量、向量数组与文本提示四类查询输入。所有 brain run 管理方法list_brain_runs、get_brain_info、load_brain_results、rename_brain_run、delete_brain_run同样定义于 fiftyone/core/collections.py是后端无关的统一 API。Mosaic 后端的实现MosaicSimilarityConfig等位于独立的fiftyone-brain包fiftyone.brain.internal.core.mosaic仓库侧的相似性总览文档 docs/source/user_guide/similarity.rst 列出了包括 sklearn、Qdrant、Redis、Pinecone、MongoDB、Elasticsearch、pgvector、Mosaic、Milvus、LanceDB 在内的全部内置后端及其配置类便于横向对比。默认后端的切换机制backend参数、FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKEND环境变量、~/.fiftyone/brain_config.json中的default_similarity_backend在 docs/source/user_guide/similarity.rst 中有完整定义Mosaic 集成文档中给出的配置示例与之完全一致。综上Mosaic 集成遵循 FiftyOne 统一的相似性后端抽象你既可以用最小参数一键建索引也可以通过index_name、patches_field、model/embeddings与 brain config 实现精细控制查询侧统一走sort_by_similarity()配合视图系统即可在任意数据子集上执行向量检索与自然语言检索。【免费下载链接】fiftyoneRefine high-quality datasets and visual AI models项目地址: https://gitcode.com/GitHub_Trending/fi/fiftyone创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 FiftyOne × Databricks Mosaic AI Search：为视觉数据构建可查询的向量相似性索引
FiftyOne × Databricks Mosaic AI Search为视觉数据构建可查询的向量相似性索引【免费下载链接】fiftyoneRefine high-quality datasets and visual AI models项目地址: https://gitcode.com/GitHub_Trending/fi/fiftyoneDatabricks Mosaic AI Search 是内置于 Databricks Data Intelligence Platform 的向量数据库与平台的数据治理与生产力工具深度集成。本文以 FiftyOne 官方集成文档为主线系统讲解如何在 FiftyOne 中一键创建 Mosaic 向量索引、上传 embeddings、执行相似性查询Python API 与 App 点选均可并覆盖环境准备、三种认证方式、配置参数、索引管理与常见查询范式帮助你在计算机视觉数据上直接复用 Mosaic 的向量检索能力。本文内容对应 FiftyOne OSS 1.4.0 与 FiftyOne Enterprise 2.7.0 及以上版本见 docs/source/integrations/mosaic.rst 中的版本声明适用的核心 API 位于fiftyone.brain.compute_similarity与SampleCollection.sort_by_similarity。提示借助 Mosaic 相似性索引你还可以直接用自然语言搜索数据当底层模型支持文本提示时详见 FiftyOne 的文本相似性搜索能力。一、基本工作流Basic Recipe在 FiftyOne 数据集上使用 Mosaic 建立相似性索引并查询数据完整流程如下连接你的 Databricks workspace并创建一个 AI Search endpoint向量搜索端点。将数据集加载到 FiftyOne。为数据集中的样本samples或对象块patches计算 embedding 向量或选择一个模型来生成 embeddings。调用compute_similarity()方法设置backendmosaic并指定自定义的brain_key为样本或对象块生成 Mosaic 相似性索引。使用该 Mosaic 索引通过sort_by_similarity()查询数据。可选删除索引。运行上述流程前你需要一个已启用 AI Search 的 Databricks 账号并安装 Databricks AI Search Python 包pip install databricks-ai-search同时按下文「认证」一节配置凭据避免每次与 Mosaic 索引交互时手动输入。下面是最小可运行示例——先加载数据集并直接由 FiftyOne 计算 embeddings、创建索引import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz # Step 1: Load your data into FiftyOne dataset foz.load_zoo_dataset(quickstart) # Steps 2 and 3: Compute embeddings and create a similarity index mosaic_index fob.compute_similarity( dataset, brain_keymosaic_index, backendmosaic, )索引生成后通过brain_key查询数据# Step 4: Query your data query dataset.first().id # query by sample ID view dataset.sort_by_similarity( query, brain_keymosaic_index, k10, # limit to 10 most similar samples ) # Step 5 (optional): Cleanup # Delete the Mosaic index mosaic_index.cleanup() # Delete run record from FiftyOne dataset.delete_brain_run(mosaic_index)可以看到compute_similarity()在不指定model或embeddings时会自动用默认模型为全部样本计算向量并建索引sort_by_similarity()是定义在 fiftyone/core/collections.py 上的视图阶段签名如下def sort_by_similarity(self, query, kNone, reverseFalse, dist_fieldNone, brain_keyNone)其中query可以是样本/块的 ID、与索引同维度的向量、ID 列表或文本提示k控制返回的最近邻数量默认对整个集合排序reverseTrue则按最不相似排序部分后端可能不支持。二、环境准备与安装Setup使用 Mosaic AI Search 的前提拥有满足 AI Search 要求的 Databricks workspace已创建 AI Search endpoint向量搜索端点在 Databricks 中准备好用于创建 AI Search 索引的 catalog 与 schema命名空间。安装 Mosaic AI Vector Search 客户端要使用 Mosaic 后端必须安装 Databricks Vector Search Python 包pip install databricks-ai-search切换到 Mosaic 后端默认情况下compute_similarity()与sort_by_similarity()使用 sklearn 后端。要改用 Mosaic只需将compute_similarity()的可选参数backend设为mosaicimport fiftyone.brain as fob fob.compute_similarity(..., backendmosaic, ...)也可以永久性地将 FiftyOne 默认相似性后端切换为 Mosaic两种方式任选其一方式一环境变量export FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKENDmosaic方式二brain 配置文件编辑位于~/.fiftyone/brain_config.json的 brain 配置设置default_similarity_backend{ default_similarity_backend: mosaic }关于默认后端的三种设置层级单次调用传参 会话级环境变量 持久化配置文件在 docs/source/user_guide/similarity.rst 中有完整说明该文档同时列出了所有内置后端的配置类其中 Mosaic 对应fiftyone.brain.internal.core.mosaic.MosaicSimilarityConfig实现位于独立的fiftyone-brain包。三、认证AuthenticationDatabricks 凭据可以通过多种方式提供以下三种由官方文档明确支持。环境变量推荐把凭据存入环境变量FiftyOne 每次建立 Databricks 连接时自动读取export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_WORKSPACE_URLhttps://unique-url.cloud.databricks.com/ export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_PERSONAL_ACCESS_TOKENXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_CATALOG_NAMEXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_SCHEMA_NAMEXXXXXXXX export FIFTYONE_BRAIN_SIMILARITY_MOSAIC_ENDPOINT_NAMEXXXXXXXX这是认证 Mosaic 客户端的一组示例变量Databricks 还支持更多认证方式如服务主体、OAuth 等可按需扩展。FiftyOne Brain 配置在~/.fiftyone/brain_config.json中存放凭据注意该文件默认不存在需自行创建{ similarity_backends: { mosaic: { workspace_url: https://unique-url.cloud.databricks.com/, personal_access_token: XXXXXXXX, catalog_name: XXXXXXXX, schema_name: XXXXXXXX, endpoint_name: XXXXXXXX } } }关键字参数也可以在每次调用需要连接 Databricks 的方法如compute_similarity()时手动传入凭据import fiftyone.brain as fob mosaic_index fob.compute_similarity( ... backendmosaic, brain_keymosaic_index, workspace_url https://unique-url.cloud.databricks.com/, personal_access_token XXXXXXXX, catalog_name XXXXXXXX, schema_name XXXXXXXX, endpoint_name XXXXXXXX )需要特别注意的是采用这种策略时之后通过load_brain_results()加载索引时也必须手动再次提供全部凭据mosaic_index dataset.load_brain_results( mosaic_index, workspace_url https://unique-url.cloud.databricks.com/, personal_access_token XXXXXXXX, catalog_name XXXXXXXX, schema_name XXXXXXXX, endpoint_name XXXXXXXX )四、Mosaic 配置参数Mosaic 后端支持以下参数定制相似性查询index_nameNone要使用或创建的 Mosaic AI Search 索引名称。若不指定会自动生成一个新的唯一名称。该参数可通过上文任一策略指定。以下是一个包含全部可用参数的 brain 配置示例{ similarity_backends: { mosaic: { index_name: your-index } } }不过实践中更常见的做法是直接把它传给compute_similarity()用于配置某个特定的新索引mosaic_index fob.compute_similarity( ... backendmosaic, brain_keymosaic_index, index_nameyour-index, )从实现角度看这类参数最终会汇聚到MosaicSimilarityConfigfiftyone.brain.internal.core.mosaic.MosaicSimilarityConfig中每次调用compute_similarity()传入的关键字参数、brain_config.json中similarity_backends.mosaic段落的配置以及对应的FIFTYONE_BRAIN_SIMILARITY_MOSAIC_*环境变量共同决定了后端实例化时使用的连接信息与索引标识。五、管理 Brain RunsFiftyOne 提供一系列方法管理 brain runs即相似性计算在数据集上留下的运行记录。列出 brain runs——查看数据集上可用的 brain keysimport fiftyone.brain as fob # List all brain runs dataset.list_brain_runs() # Only list similarity runs dataset.list_brain_runs(typefob.Similarity) # Only list specific similarity runs dataset.list_brain_runs( typefob.Similarity, patches_fieldground_truth, supports_promptsTrue, )获取 brain run 信息——查看某个 brain run 的配置info dataset.get_brain_info(brain_key) print(info)加载索引实例——用load_brain_results()取回该 brain run 对应的SimilarityIndex实例。重命名 brain keydataset.rename_brain_run(brain_key, new_brain_key)删除运行记录dataset.delete_brain_run(brain_key)上述方法均定义在 fiftyone/core/collections.py 中例如list_brain_runsL4927、rename_brain_runL4949、get_brain_infoL4958、load_brain_resultsL4970、delete_brain_runL5008它们对所有相似性后端统一生效。注意delete_brain_run()只删除 FiftyOne 数据集上的 brain run记录不会删除关联的 Mosaic 索引。删除 Mosaic 索引需显式执行# Delete the Mosaic index mosaic_index dataset.load_brain_results(brain_key) mosaic_index.cleanup()六、实战示例常见向量检索工作流以下示例均假设已按上文配置好 Databricks 账号与凭据。创建相似性索引创建新的 Mosaic 相似性索引需为compute_similarity()提供embeddings或model参数共有四种常见写法import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) model_name clip-vit-base32-torch model foz.load_zoo_model(model_name) brain_key mosaic_index # Option 1: Compute embeddings on the fly from model name fob.compute_similarity( dataset, modelmodel_name, backendmosaic, brain_keybrain_key, ) # Option 2: Compute embeddings on the fly from model instance fob.compute_similarity( dataset, modelmodel, backendmosaic, brain_keybrain_key, ) # Option 3: Pass pre-computed embeddings as a numpy array embeddings dataset.compute_embeddings(model) fob.compute_similarity( dataset, embeddingsembeddings, backendmosaic, brain_keybrain_key, ) # Option 4: Pass pre-computed embeddings by field name dataset.compute_embeddings(model, embeddings_fieldembeddings) fob.compute_similarity( dataset, embeddingsembeddings, backendmosaic, brain_keybrain_key, )四种方式的差异在于 embeddings 的供给途径前两种由 FiftyOne 在创建索引时实时计算按模型名或模型实例后两种复用已算好的向量numpy 数组或数据集字段。创建对象块patch相似性索引通过patches_field参数可以为数据集内的目标检测对象块建立相似性索引每个检测框对应的图像块作为一个检索单元import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, patches_fieldground_truth, modelclip-vit-base32-torch, backendmosaic, brain_keymosaic_patches, )连接已有索引如果 Mosaic 中已存在存储了样本/块向量的索引可通过index_name直接连接并用embeddingsFalse跳过向量计算import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, modelclip-vit-base32-torch, # zoo model used (if applicable) embeddingsFalse, # dont compute embeddings index_nameyour-index, # the existing Mosaic index brain_keymosaic_index, backendmosaic, )向索引增删 embeddings修改数据集后可用add_to_index()与remove_from_index()同步更新 Mosaic 索引import numpy as np import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) print(mosaic_index.total_index_size) # 200 view dataset.take(10) ids view.values(id) # Delete 10 samples from a dataset dataset.delete_samples(view) # Delete the corresponding vectors from the index mosaic_index.remove_from_index(sample_idsids) # Add 20 samples to a dataset samples [fo.Sample(filepathtmp%d.jpg % i) for i in range(20)] sample_ids dataset.add_samples(samples) # Add corresponding embeddings to the index embeddings np.random.rand(20, 512) mosaic_index.add_to_index(embeddings, sample_ids) print(mosaic_index.total_index_size) # 210注意示例中的 embeddings 维度为 512需与索引所用模型这里是 CLIP的输出维度保持一致。从索引取回 embeddings用get_embeddings()按 ID 从 Mosaic 索引取回向量import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) # Retrieve embeddings for the entire dataset ids dataset.values(id) embeddings, sample_ids, _ mosaic_index.get_embeddings(sample_idsids) print(embeddings.shape) # (200, 512) print(sample_ids.shape) # (200,) # Retrieve embeddings for a view ids dataset.take(10).values(id) embeddings, sample_ids, _ mosaic_index.get_embeddings(sample_idsids) print(embeddings.shape) # (10, 512) print(sample_ids.shape) # (10,)七、查询 Mosaic 索引通过在任意数据集或视图上追加sort_by_similarity()阶段即可查询 Mosaic 索引。查询可以是以下任意一种一个 ID样本或块一个与索引同维度的查询向量一个 ID 列表样本或块一段文本提示当模型支持时import numpy as np import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) # Query by vector query np.random.rand(512) # matches the dimension of CLIP embeddings view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by sample ID query dataset.first().id view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by a list of IDs query [dataset.first().id, dataset.last().id] view dataset.sort_by_similarity(query, k10, brain_keymosaic_index) # Query by text prompt query a photo of a dog view dataset.sort_by_similarity(query, k10, brain_keymosaic_index)关键语义对DatasetView执行相似性搜索时只会返回该视图内的结果如果视图中包含未纳入索引的样本它们永远不会出现在结果里。这意味着你可以对整个Dataset建立一次索引之后通过构造包含目标图片的视图在数据集任意子集上反复检索——这正是 docs/source/user_guide/similarity.rst 所描述的通用相似性 API 与视图系统协同的最佳实践。八、访问底层 Mosaic 客户端每个 Mosaic 索引都暴露client属性可直接访问底层的 Mosaic 客户端实例并调用其方法import fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, brain_keymosaic_index, backendmosaic, ) mosaic_client mosaic_index.client print(mosaic_client)这为高级场景如直接使用 Databricks Vector Search 原生 API 做索引管理、元数据过滤等留出了充分的扩展空间。九、高级用法如「Mosaic 配置参数」一节所述通过向compute_similarity()传入可选参数可以定制 Mosaic 索引。下面是一个综合示例指定自定义索引名并且只对数据集的一个子集填充向量先用embeddingsFalse创建空索引再分批add_to_indeximport fiftyone as fo import fiftyone.brain as fob import fiftyone.zoo as foz dataset foz.load_zoo_dataset(quickstart) # Create a custom Mosaic index mosaic_index fob.compute_similarity( dataset, modelclip-vit-base32-torch, embeddingsFalse, # well add embeddings below brain_keymosaic_index, backendmosaic, index_namecustom-quickstart-index, ) # Add embeddings for a subset of the dataset view dataset.take(10) embeddings, sample_ids, _ mosaic_index.compute_embeddings(view) mosaic_index.add_to_index(embeddings, sample_ids)这里的mosaic_index.compute_embeddings(view)是SimilarityIndex实例提供的能力与add_to_index()组合即可实现先建索引、后分批灌入向量的增量式构建。十、实现要点与源码对照从本仓库源码可以进一步确认 Mosaic 集成在 FiftyOne 中的落点sort_by_similarity()是SampleCollection上的视图阶段方法完整签名与 docstring 见 fiftyone/core/collections.py它要求先通过fiftyone.brain.compute_similarity建立相似性索引并支持 ID、向量、向量数组与文本提示四类查询输入。所有 brain run 管理方法list_brain_runs、get_brain_info、load_brain_results、rename_brain_run、delete_brain_run同样定义于 fiftyone/core/collections.py是后端无关的统一 API。Mosaic 后端的实现MosaicSimilarityConfig等位于独立的fiftyone-brain包fiftyone.brain.internal.core.mosaic仓库侧的相似性总览文档 docs/source/user_guide/similarity.rst 列出了包括 sklearn、Qdrant、Redis、Pinecone、MongoDB、Elasticsearch、pgvector、Mosaic、Milvus、LanceDB 在内的全部内置后端及其配置类便于横向对比。默认后端的切换机制backend参数、FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKEND环境变量、~/.fiftyone/brain_config.json中的default_similarity_backend在 docs/source/user_guide/similarity.rst 中有完整定义Mosaic 集成文档中给出的配置示例与之完全一致。综上Mosaic 集成遵循 FiftyOne 统一的相似性后端抽象你既可以用最小参数一键建索引也可以通过index_name、patches_field、model/embeddings与 brain config 实现精细控制查询侧统一走sort_by_similarity()配合视图系统即可在任意数据子集上执行向量检索与自然语言检索。【免费下载链接】fiftyoneRefine high-quality datasets and visual AI models项目地址: https://gitcode.com/GitHub_Trending/fi/fiftyone创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考