尧图网站建设 
Crawl4AI 分块策略Chunking Strategy深度解析从正则切分到滑动窗口的完整实践【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai本文围绕 Crawl4AI 的分块策略Chunking Strategy展开分块是将网页抓取到的大段文本切分为可管理片段的核心技术是余弦相似度提取、语义检索与 RAG检索增强生成流水线的地基。读完本文你将掌握 Crawl4AI 提供的全部内置分块类正则、NLP 句子、主题分段、定长、滑动窗口、重叠窗口、恒等、它们在CrawlerRunConfig中的接入方式、在抓取管线中的实际调用链以及分块 余弦相似度的组合用法。为什么要做分块Chunking在把网页内容交给 LLM、嵌入模型或相似度算法之前必须先切分。原文档chunking.md给出的三个动机同样适用于 Crawl4AI 的实际场景余弦相似度与查询相关性为语义相似度分析准备可比较的文本片段。Crawl4AI 中CosineStrategy、SemanticExtractionStrategy等策略正是建立在句子/段落级 chunk之上做聚类与过滤的RAG 系统集成分块结果是存入向量库、被检索的最小单元切分质量直接影响召回结构化处理支持按句子、按主题、按固定窗口等多种切分方式适应不同下游任务。从源码结构看Crawl4AI 把这一层抽象为独立的策略模块 chunking_strategy.py与提取策略extraction_strategy.py、内容过滤策略并列统一通过CrawlerRunConfig注入到抓取流程中。ChunkingStrategy 抽象接口与类体系所有分块策略继承自抽象基类ChunkingStrategy只约定一个方法——chunk(text) - list这是文档示例中各类chunk方法在 Crawl4AI 中的正式形态chunking_strategy.pyclass ChunkingStrategy(ABC): Abstract base class for chunking strategies. abstractmethod def chunk(self, text: str) - list: Chunk the given text. Args: text (str): The text to chunk. Returns: list: A list of chunks. pass当前仓库内置 7 个策略实现策略类切分依据关键参数默认值IdentityChunking不切分整体返回[text]无RegexChunking正则模式列表patterns[r\n\n]NlpSentenceChunkingNLTKsent_tokenize无TopicSegmentationChunkingTextTiling 主题分段num_keywords3FixedLengthWordChunking固定词数chunk_size100SlidingWindowChunking滑动窗口按 step 步进window_size100, step50OverlappingWindowChunking定长窗口 固定重叠量window_size1000, overlap100IdentityChunkingchunking_strategy.py虽然简单f(x) [x]但在管线中承担关键角色下文会说明。1. 正则分块 RegexChunking按正则表达式列表逐层切分文本是最粗粒度、零依赖的方案也是CrawlerRunConfig的默认分块策略。Crawl4AI 的实现chunking_strategy.pyclass RegexChunking(ChunkingStrategy): def __init__(self, patternsNone, **kwargs): if patterns is None: patterns [r\n\n] # Default split pattern self.patterns patterns def chunk(self, text: str) - list: paragraphs [text] for pattern in self.patterns: new_paragraphs [] for paragraph in paragraphs: new_paragraphs.extend(re.split(pattern, paragraph)) paragraphs new_paragraphs return paragraphs与原文档示例的差异仓库实现采用逐模式迭代扩展的写法对每个 pattern 再切一次多模式间是叠加切分关系默认模式r\n\n即按空行切段落对 Markdown 输出非常友好。2. NLP 句子分块 NlpSentenceChunking用 NLTK 的句子切分器把文本切成句子适合以完整语句为语义单元的场景chunking_strategy.pyclass NlpSentenceChunking(ChunkingStrategy): def __init__(self, **kwargs): load_nltk_punkt() def chunk(self, text: str) - list: from nltk.tokenize import sent_tokenize sentences sent_tokenize(text) sens [sent.strip() for sent in sentences] return sens一个值得注意的实现细节构造时会显式调用load_nltk_punkt()定义于 model_loader.py负责确保 NLTK 的 punkt 数据资源可用避免在离线环境因缺数据而抛错。CosineStrategy这类以句子为聚类输入的策略常与之配合。3. 主题分段 TopicSegmentationChunking基于 NLTK 的TextTilingTokenizer做主题连贯性分段并额外提供关键词提取能力chunking_strategy.pyclass TopicSegmentationChunking(ChunkingStrategy): def __init__(self, num_keywords3, **kwargs): import nltk as nl self.tokenizer nl.tokenize.TextTilingTokenizer() self.num_keywords num_keywords def chunk(self, text: str) - list: return self.tokenizer.tokenize(text) def chunk_with_topics(self, text: str) - list: segments self.chunk(text) return [(segment, self.extract_keywords(segment)) for segment in segments]相比原文档示例仓库版本多出一个num_keywords参数和chunk_with_topics()方法先按 TextTiling 分段再对每段做词频统计过滤英语停用词与标点得到 top-N 关键词。extract_keywords()的实现见 chunking_strategy.py其输出(segment, keywords)列表可直接作为 RAG 片段的摘要标签。4. 定长词数分块 FixedLengthWordChunking按固定词数硬切无重叠、无依赖行为最可预测chunking_strategy.pyclass FixedLengthWordChunking(ChunkingStrategy): def __init__(self, chunk_size100, **kwargs): self.chunk_size chunk_size def chunk(self, text: str) - list: words text.split() return [ .join(words[i : i self.chunk_size]) for i in range(0, len(words), self.chunk_size) ]chunk_size单位为词数默认 100 词。原文档示例中chunk_size5的用法与仓库一致。注意其切分点不感知句子边界长句可能被拦腰截断若下游是 LLM 摘要通常建议换成下面的滑动窗口或句子分块。5. 滑动窗口分块 SlidingWindowChunking以step为步长滑动定长窗口产生重叠片段以保留上下文连贯性chunking_strategy.pyclass SlidingWindowChunking(ChunkingStrategy): def __init__(self, window_size100, step50, **kwargs): self.window_size window_size self.step step def chunk(self, text: str) - list: words text.split() chunks [] if len(words) self.window_size: return [text] for i in range(0, len(words) - self.window_size 1, self.step): chunks.append( .join(words[i : i self.window_size])) if i self.window_size len(words): chunks.append( .join(words[-self.window_size :])) return chunks两个仓库实现比文档示例更完善的边界处理文本短于窗口时直接返回整段原文末尾不足一窗的内容会补一个取最后 window_size 个词的尾块避免丢词。参数语义step window_size - overlap即重叠量等于二者之差如默认window_size100, step50即 50 词重叠。6. 重叠窗口分块 OverlappingWindowChunking这是文档未提及、但仓库中同样内置的变体直接用overlap参数声明相邻块的重叠词数比step语义更直白chunking_strategy.pyclass OverlappingWindowChunking(ChunkingStrategy): def __init__(self, window_size1000, overlap100, **kwargs): self.window_size window_size self.overlap overlap def chunk(self, text: str) - list: words text.split() chunks [] if len(words) self.window_size: return [text] start 0 while start len(words): end start self.window_size chunks.append( .join(words[start:end])) if end len(words): break start end - self.overlap return chunks默认window_size1000, overlap10010% 重叠更贴近大文档送 LLM 时的常见规格。选型建议想精细控制步进用SlidingWindowChunking想声明重叠比例用OverlappingWindowChunking。分块在 Crawl4AI 抓取管线中的接入方式默认值与配置注入CrawlerRunConfig把分块策略声明为默认RegexChunking()并在校验逻辑中强制类型约束async_configs.pychunking_strategy: ChunkingStrategy RegexChunking(),校验与兜底async_configs.py若传入的对象不是ChunkingStrategy实例会抛出chunking_strategy must be an instance of ChunkingStrategy若显式传None则回退为RegexChunking()。因此即使不做任何配置分块也始终存在只是默认按空行切段。自定义用法示例与 tests/async/test_chunking_and_extraction_strategies.py 中test_regex_chunking的写法一致from crawl4ai import AsyncWebCrawler from crawl4ai.chunking_strategy import SlidingWindowChunking chunking_strategy SlidingWindowChunking(window_size100, step50) result await crawler.arun( urlurl, chunking_strategychunking_strategy, extraction_strategyextraction_strategy, bypass_cacheTrue, )管线中的实际调用链chunking 与 extraction 的衔接在AsyncWebCrawler的_crawl流程里分块发生在 Markdown 生成之后、结构化提取之前async_webcrawler.py# Use IdentityChunking for HTML input, otherwise use provided chunking strategy chunking ( IdentityChunking() if content_format in [html, cleaned_html, fit_html] else config.chunking_strategy ) sections chunking.chunk(content) # Use async version if available for better parallelism if hasattr(config.extraction_strategy, arun): extracted_content await config.extraction_strategy.arun(_url, sections) else: extracted_content await asyncio.to_thread( config.extraction_strategy.run, url, sections )这段代码揭示了三个关键事实分块输入取决于input_format。提取策略默认input_formatmarkdownextraction_strategy.py此时你的chunking_strategy才会被调用而 HTML 类提取策略会在构造时强制input_formathtml如 extraction_strategy.py 的kwargs[input_format] html此时管线自动改用IdentityChunking()把整页作为单块送入用户配置的分块策略被旁路——这是理解为什么我配置的分块没生效的关键chunk 结果是提取策略的唯一输入。extraction_strategy.run/arun(url, sections)接收的就是chunk()的返回列表CrawlResult.extracted_content为其 JSON 序列化异步优先。若提取策略实现了arun如LLMExtractionStrategy则走原生协程否则用asyncio.to_thread在线程池中执行同步版run避免阻塞事件循环。提取策略内部的二次分块需要区分两层分块CrawlerRunConfig.chunking_strategy是管线级切分而部分提取策略还自带内部切分参数。以LLMExtractionStrategy为例extraction_strategy.pychunk_token_threshold单个 chunk 的 token 上限overlap_rate相邻 chunk 的重叠比例apply_chunking置为False时会把chunk_token_threshold设为1e9extraction_strategy.py等效关闭内部切分。其内部用merge_chunks合并句子extraction_strategy.py来组织发给 LLM 的上下文。因此实践上可理解为外层的chunking_strategy决定片段怎么切策略内部参数决定片段如何归并/过滤后再进模型两者叠加但职责不同。分块 余弦相似度组合检索工作流文档的核心组合示例是用 TF-IDF 向量与余弦相似度在分块结果上按查询词打分原文示例保留如下from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity class CosineSimilarityExtractor: def __init__(self, query): self.query query self.vectorizer TfidfVectorizer() def find_relevant_chunks(self, chunks): vectors self.vectorizer.fit_transform([self.query] chunks) similarities cosine_similarity(vectors[0:1], vectors[1:]).flatten() return [(chunks[i], similarities[i]) for i in range(len(chunks))] # Example Workflow text This is a sample document. It has multiple sentences. We are testing chunking and similarity. chunker SlidingWindowChunking(window_size5, step3) chunks chunker.chunk(text) query testing chunking extractor CosineSimilarityExtractor(query) relevant_chunks extractor.find_relevant_chunks(chunks) print(relevant_chunks)工作流要点先由任一分块策略产出chunks再对query chunks统一做 TF-IDF 向量化最后取查询向量与其余各块的余弦相似度逐块打分实现按相关性取回片段。这正是 Crawl4AI 中余弦相似度类提取的朴素版原型仓库的CosineStrategyextraction_strategy.py在此基础上进一步做了文档级嵌入过滤filter_documents_embeddingssemantic_filter与层次聚类hierarchical_clustering再合并同簇句子merge_chunks但分块 → 向量化 → 相似度筛选的骨架是一致的。选型建议与常见问题结合源码行为给出几条可直接落地的选型准则场景推荐策略理由Markdown/文档抓取 LLM 提取RegexChunking默认r\n\n按段落切分粒度适中零依赖句子级相似度聚类CosineStrategyNlpSentenceChunking句子是聚类与去重的自然单元RAG 入库需上下文连贯SlidingWindowChunking/OverlappingWindowChunking重叠保留跨块上下文主题/章节级摘要TopicSegmentationChunkingTextTiling 按主题边界切分可附关键词HTML 类提取策略无需配置管线自动走IdentityChunkingasync_webcrawler.py常见问题配置了分块策略却没生效先确认所用提取策略的input_format。HTML 系策略强制整块输入IdentityChunking旁路此时分块参数被忽略NLP 类策略环境要求NlpSentenceChunking与TopicSegmentationChunking依赖 NLTK 及 punkt 数据仓库通过 model_loader.py 的load_nltk_punkt()在构造时加载想验证自己的配置可参考 tests/async/test_chunking_and_extraction_strategies.py 的测试模式arun后检查result.extracted_content是否为多元素的 JSON 列表即可断言分块确实发生且与提取策略联动正常。小结Crawl4AI 的分块模块以单一抽象接口ChunkingStrategy.chunk()统一了 7 种切分算法默认值RegexChunking按空行切段保证了开箱即用而input_format驱动的旁路机制HTML 输入自动整块则体现了分块服务于提取策略的设计取向。掌握外层 chunking_strategy 切片段、内层策略参数管归并这两层职责后即可按需组合分块与余弦相似度/LLM 提取构建结构化的内容处理与 RAG 流水线。【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Crawl4AI 分块策略（Chunking Strategy）深度解析：从正则切分到滑动窗口的完整实践
Crawl4AI 分块策略Chunking Strategy深度解析从正则切分到滑动窗口的完整实践【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai本文围绕 Crawl4AI 的分块策略Chunking Strategy展开分块是将网页抓取到的大段文本切分为可管理片段的核心技术是余弦相似度提取、语义检索与 RAG检索增强生成流水线的地基。读完本文你将掌握 Crawl4AI 提供的全部内置分块类正则、NLP 句子、主题分段、定长、滑动窗口、重叠窗口、恒等、它们在CrawlerRunConfig中的接入方式、在抓取管线中的实际调用链以及分块 余弦相似度的组合用法。为什么要做分块Chunking在把网页内容交给 LLM、嵌入模型或相似度算法之前必须先切分。原文档chunking.md给出的三个动机同样适用于 Crawl4AI 的实际场景余弦相似度与查询相关性为语义相似度分析准备可比较的文本片段。Crawl4AI 中CosineStrategy、SemanticExtractionStrategy等策略正是建立在句子/段落级 chunk之上做聚类与过滤的RAG 系统集成分块结果是存入向量库、被检索的最小单元切分质量直接影响召回结构化处理支持按句子、按主题、按固定窗口等多种切分方式适应不同下游任务。从源码结构看Crawl4AI 把这一层抽象为独立的策略模块 chunking_strategy.py与提取策略extraction_strategy.py、内容过滤策略并列统一通过CrawlerRunConfig注入到抓取流程中。ChunkingStrategy 抽象接口与类体系所有分块策略继承自抽象基类ChunkingStrategy只约定一个方法——chunk(text) - list这是文档示例中各类chunk方法在 Crawl4AI 中的正式形态chunking_strategy.pyclass ChunkingStrategy(ABC): Abstract base class for chunking strategies. abstractmethod def chunk(self, text: str) - list: Chunk the given text. Args: text (str): The text to chunk. Returns: list: A list of chunks. pass当前仓库内置 7 个策略实现策略类切分依据关键参数默认值IdentityChunking不切分整体返回[text]无RegexChunking正则模式列表patterns[r\n\n]NlpSentenceChunkingNLTKsent_tokenize无TopicSegmentationChunkingTextTiling 主题分段num_keywords3FixedLengthWordChunking固定词数chunk_size100SlidingWindowChunking滑动窗口按 step 步进window_size100, step50OverlappingWindowChunking定长窗口 固定重叠量window_size1000, overlap100IdentityChunkingchunking_strategy.py虽然简单f(x) [x]但在管线中承担关键角色下文会说明。1. 正则分块 RegexChunking按正则表达式列表逐层切分文本是最粗粒度、零依赖的方案也是CrawlerRunConfig的默认分块策略。Crawl4AI 的实现chunking_strategy.pyclass RegexChunking(ChunkingStrategy): def __init__(self, patternsNone, **kwargs): if patterns is None: patterns [r\n\n] # Default split pattern self.patterns patterns def chunk(self, text: str) - list: paragraphs [text] for pattern in self.patterns: new_paragraphs [] for paragraph in paragraphs: new_paragraphs.extend(re.split(pattern, paragraph)) paragraphs new_paragraphs return paragraphs与原文档示例的差异仓库实现采用逐模式迭代扩展的写法对每个 pattern 再切一次多模式间是叠加切分关系默认模式r\n\n即按空行切段落对 Markdown 输出非常友好。2. NLP 句子分块 NlpSentenceChunking用 NLTK 的句子切分器把文本切成句子适合以完整语句为语义单元的场景chunking_strategy.pyclass NlpSentenceChunking(ChunkingStrategy): def __init__(self, **kwargs): load_nltk_punkt() def chunk(self, text: str) - list: from nltk.tokenize import sent_tokenize sentences sent_tokenize(text) sens [sent.strip() for sent in sentences] return sens一个值得注意的实现细节构造时会显式调用load_nltk_punkt()定义于 model_loader.py负责确保 NLTK 的 punkt 数据资源可用避免在离线环境因缺数据而抛错。CosineStrategy这类以句子为聚类输入的策略常与之配合。3. 主题分段 TopicSegmentationChunking基于 NLTK 的TextTilingTokenizer做主题连贯性分段并额外提供关键词提取能力chunking_strategy.pyclass TopicSegmentationChunking(ChunkingStrategy): def __init__(self, num_keywords3, **kwargs): import nltk as nl self.tokenizer nl.tokenize.TextTilingTokenizer() self.num_keywords num_keywords def chunk(self, text: str) - list: return self.tokenizer.tokenize(text) def chunk_with_topics(self, text: str) - list: segments self.chunk(text) return [(segment, self.extract_keywords(segment)) for segment in segments]相比原文档示例仓库版本多出一个num_keywords参数和chunk_with_topics()方法先按 TextTiling 分段再对每段做词频统计过滤英语停用词与标点得到 top-N 关键词。extract_keywords()的实现见 chunking_strategy.py其输出(segment, keywords)列表可直接作为 RAG 片段的摘要标签。4. 定长词数分块 FixedLengthWordChunking按固定词数硬切无重叠、无依赖行为最可预测chunking_strategy.pyclass FixedLengthWordChunking(ChunkingStrategy): def __init__(self, chunk_size100, **kwargs): self.chunk_size chunk_size def chunk(self, text: str) - list: words text.split() return [ .join(words[i : i self.chunk_size]) for i in range(0, len(words), self.chunk_size) ]chunk_size单位为词数默认 100 词。原文档示例中chunk_size5的用法与仓库一致。注意其切分点不感知句子边界长句可能被拦腰截断若下游是 LLM 摘要通常建议换成下面的滑动窗口或句子分块。5. 滑动窗口分块 SlidingWindowChunking以step为步长滑动定长窗口产生重叠片段以保留上下文连贯性chunking_strategy.pyclass SlidingWindowChunking(ChunkingStrategy): def __init__(self, window_size100, step50, **kwargs): self.window_size window_size self.step step def chunk(self, text: str) - list: words text.split() chunks [] if len(words) self.window_size: return [text] for i in range(0, len(words) - self.window_size 1, self.step): chunks.append( .join(words[i : i self.window_size])) if i self.window_size len(words): chunks.append( .join(words[-self.window_size :])) return chunks两个仓库实现比文档示例更完善的边界处理文本短于窗口时直接返回整段原文末尾不足一窗的内容会补一个取最后 window_size 个词的尾块避免丢词。参数语义step window_size - overlap即重叠量等于二者之差如默认window_size100, step50即 50 词重叠。6. 重叠窗口分块 OverlappingWindowChunking这是文档未提及、但仓库中同样内置的变体直接用overlap参数声明相邻块的重叠词数比step语义更直白chunking_strategy.pyclass OverlappingWindowChunking(ChunkingStrategy): def __init__(self, window_size1000, overlap100, **kwargs): self.window_size window_size self.overlap overlap def chunk(self, text: str) - list: words text.split() chunks [] if len(words) self.window_size: return [text] start 0 while start len(words): end start self.window_size chunks.append( .join(words[start:end])) if end len(words): break start end - self.overlap return chunks默认window_size1000, overlap10010% 重叠更贴近大文档送 LLM 时的常见规格。选型建议想精细控制步进用SlidingWindowChunking想声明重叠比例用OverlappingWindowChunking。分块在 Crawl4AI 抓取管线中的接入方式默认值与配置注入CrawlerRunConfig把分块策略声明为默认RegexChunking()并在校验逻辑中强制类型约束async_configs.pychunking_strategy: ChunkingStrategy RegexChunking(),校验与兜底async_configs.py若传入的对象不是ChunkingStrategy实例会抛出chunking_strategy must be an instance of ChunkingStrategy若显式传None则回退为RegexChunking()。因此即使不做任何配置分块也始终存在只是默认按空行切段。自定义用法示例与 tests/async/test_chunking_and_extraction_strategies.py 中test_regex_chunking的写法一致from crawl4ai import AsyncWebCrawler from crawl4ai.chunking_strategy import SlidingWindowChunking chunking_strategy SlidingWindowChunking(window_size100, step50) result await crawler.arun( urlurl, chunking_strategychunking_strategy, extraction_strategyextraction_strategy, bypass_cacheTrue, )管线中的实际调用链chunking 与 extraction 的衔接在AsyncWebCrawler的_crawl流程里分块发生在 Markdown 生成之后、结构化提取之前async_webcrawler.py# Use IdentityChunking for HTML input, otherwise use provided chunking strategy chunking ( IdentityChunking() if content_format in [html, cleaned_html, fit_html] else config.chunking_strategy ) sections chunking.chunk(content) # Use async version if available for better parallelism if hasattr(config.extraction_strategy, arun): extracted_content await config.extraction_strategy.arun(_url, sections) else: extracted_content await asyncio.to_thread( config.extraction_strategy.run, url, sections )这段代码揭示了三个关键事实分块输入取决于input_format。提取策略默认input_formatmarkdownextraction_strategy.py此时你的chunking_strategy才会被调用而 HTML 类提取策略会在构造时强制input_formathtml如 extraction_strategy.py 的kwargs[input_format] html此时管线自动改用IdentityChunking()把整页作为单块送入用户配置的分块策略被旁路——这是理解为什么我配置的分块没生效的关键chunk 结果是提取策略的唯一输入。extraction_strategy.run/arun(url, sections)接收的就是chunk()的返回列表CrawlResult.extracted_content为其 JSON 序列化异步优先。若提取策略实现了arun如LLMExtractionStrategy则走原生协程否则用asyncio.to_thread在线程池中执行同步版run避免阻塞事件循环。提取策略内部的二次分块需要区分两层分块CrawlerRunConfig.chunking_strategy是管线级切分而部分提取策略还自带内部切分参数。以LLMExtractionStrategy为例extraction_strategy.pychunk_token_threshold单个 chunk 的 token 上限overlap_rate相邻 chunk 的重叠比例apply_chunking置为False时会把chunk_token_threshold设为1e9extraction_strategy.py等效关闭内部切分。其内部用merge_chunks合并句子extraction_strategy.py来组织发给 LLM 的上下文。因此实践上可理解为外层的chunking_strategy决定片段怎么切策略内部参数决定片段如何归并/过滤后再进模型两者叠加但职责不同。分块 余弦相似度组合检索工作流文档的核心组合示例是用 TF-IDF 向量与余弦相似度在分块结果上按查询词打分原文示例保留如下from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity class CosineSimilarityExtractor: def __init__(self, query): self.query query self.vectorizer TfidfVectorizer() def find_relevant_chunks(self, chunks): vectors self.vectorizer.fit_transform([self.query] chunks) similarities cosine_similarity(vectors[0:1], vectors[1:]).flatten() return [(chunks[i], similarities[i]) for i in range(len(chunks))] # Example Workflow text This is a sample document. It has multiple sentences. We are testing chunking and similarity. chunker SlidingWindowChunking(window_size5, step3) chunks chunker.chunk(text) query testing chunking extractor CosineSimilarityExtractor(query) relevant_chunks extractor.find_relevant_chunks(chunks) print(relevant_chunks)工作流要点先由任一分块策略产出chunks再对query chunks统一做 TF-IDF 向量化最后取查询向量与其余各块的余弦相似度逐块打分实现按相关性取回片段。这正是 Crawl4AI 中余弦相似度类提取的朴素版原型仓库的CosineStrategyextraction_strategy.py在此基础上进一步做了文档级嵌入过滤filter_documents_embeddingssemantic_filter与层次聚类hierarchical_clustering再合并同簇句子merge_chunks但分块 → 向量化 → 相似度筛选的骨架是一致的。选型建议与常见问题结合源码行为给出几条可直接落地的选型准则场景推荐策略理由Markdown/文档抓取 LLM 提取RegexChunking默认r\n\n按段落切分粒度适中零依赖句子级相似度聚类CosineStrategyNlpSentenceChunking句子是聚类与去重的自然单元RAG 入库需上下文连贯SlidingWindowChunking/OverlappingWindowChunking重叠保留跨块上下文主题/章节级摘要TopicSegmentationChunkingTextTiling 按主题边界切分可附关键词HTML 类提取策略无需配置管线自动走IdentityChunkingasync_webcrawler.py常见问题配置了分块策略却没生效先确认所用提取策略的input_format。HTML 系策略强制整块输入IdentityChunking旁路此时分块参数被忽略NLP 类策略环境要求NlpSentenceChunking与TopicSegmentationChunking依赖 NLTK 及 punkt 数据仓库通过 model_loader.py 的load_nltk_punkt()在构造时加载想验证自己的配置可参考 tests/async/test_chunking_and_extraction_strategies.py 的测试模式arun后检查result.extracted_content是否为多元素的 JSON 列表即可断言分块确实发生且与提取策略联动正常。小结Crawl4AI 的分块模块以单一抽象接口ChunkingStrategy.chunk()统一了 7 种切分算法默认值RegexChunking按空行切段保证了开箱即用而input_format驱动的旁路机制HTML 输入自动整块则体现了分块服务于提取策略的设计取向。掌握外层 chunking_strategy 切片段、内层策略参数管归并这两层职责后即可按需组合分块与余弦相似度/LLM 提取构建结构化的内容处理与 RAG 流水线。【免费下载链接】crawl4ai Crawl4AI: Open-source LLM Friendly Web Crawler Scraper. Dont be shy, join here: https://discord.gg/jP8KfhDhyN项目地址: https://gitcode.com/GitHub_Trending/craw/crawl4ai创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考