Transformers 大语言模型提示词工程Prompt Engineering完全指南【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers提示词工程Prompt Engineering又称 Prompting通过自然语言指令引导大语言模型LLM在各类任务上输出期望结果是无需微调即可使用 LLM 的核心技能。本文基于本仓库 prompting 指南日文版见 docs/source/ja/tasks/prompting.md完整覆盖提示词设计的最佳实践、零样本与少样本提示、思维链Chain-of-Thought等高级技巧并结合 Transformers 的pipeline与源码实现给出可直接复制的任务级实战示例。读完本文你将能够为文本分类、命名实体识别、翻译、摘要、问答与推理任务设计高质量的 LLM 提示词并判断何时应该转向微调。什么是提示词工程为什么好的提示词如此重要Falcon、LLaMA 等大语言模型是预训练 Transformer 模型核心能力是根据已有输入文本预测下一个 token。它们通常拥有数十亿到数万亿的参数在超大规模语料上经过长时间训练因此极其强大且通用——通过自然语言提示词就能让模型立即解决多种 NLP 任务。但提示词工程之所以成为一门学问是因为自然语言比编程语言更灵活、更具表现力同时也更模糊。提示词对变化非常敏感对提示做微小改动输出就可能大相径庭。不存在一个万能配方能保证所有场景都得到理想结果不过研究者已经总结出一套能更稳定取得较好效果的最佳实践。[!TIP] 提示词工程只是优化 LLM 输出的一个环节。另一个关键要素是文本生成策略在不改动任何可训练参数的前提下定制 LLM 生成时选择后续 token 的方式。通过调节生成参数如max_new_tokens、do_sample、top_k可以让生成文本更一致、更接近人类表达。生成策略与参数不在本文范围内可进一步阅读 LLM 生成教程与文本生成策略。模型类型从 decoder-only 到 encoder-decoder现代 LLM 绝大多数是decoder-only仅解码器的 Transformer例如 LLaMA、Llama2、Falcon、GPT2。这类模型用text-generation管道执行推理from transformers import pipeline import torch torch.manual_seed(0) # doctest: IGNORE_RESULT generator pipeline(text-generation, modelopenai-community/gpt2) prompt Hello, Im a language model generator(prompt, max_length30) # [{generated_text: Hello, Im a language model expert, so Im a big believer in the concept that I know very well and then I try to look into}]需要留意的是也存在少量可用于提示词的 encoder-decoder编码器-解码器LLM如 Flan-T5 或 BART。按文档建议这类模型应直接用 [AutoModelForSeq2SeqLM] 类加载而非Pipeline再从模型本身调用生成方法因为它们面向的是 seq2seq 任务。从本仓库源码看text-generation管道在 SUPPORTED_TASKS 注册表中对应TextGenerationPipeline实现text_generation.py其默认模型当前为HuggingFaceTB/SmolLM3-3B。该管道的__call__文档text_generation.py说明了它支持字符串、字符串列表也支持聊天格式ChatType即带role与content键的字典列表传入聊天格式时模型会自动套用其 chat template。Base 模型 vs Instruct/Chat 模型Hugging Face Hub 上大多数较新的 LLM checkpoint 都有两个版本**base基础**与instruct指令/chat对话。例如tiiuae/falcon-7b与tiiuae/falcon-7b-instruct。base 模型擅长在给定初始提示后续写文本但在需要遵循指令或进行对话时并不理想。instruct/chat 模型是在 base 版本基础上用指令与对话数据进一步微调得到的。这段额外的微调使其成为绝大多数 NLP 任务的更优选择。因此选择模型时建议优先挑选最新、能力更强的模型以获得更优表现并注意区分 base 与 instruct 变体。环境搭建在开始实验前先安装依赖pip install -q transformers accelerate随后加载模型。Falcon 系列使用bfloat16数据类型训练因此推荐沿用同样的精度——这要求较新的 CUDA 版本并在最新显卡上表现最佳from transformers import pipeline, AutoTokenizer import torch torch.manual_seed(0) # doctest: IGNORE_RESULT model tiiuae/falcon-7b-instruct tokenizer AutoTokenizer.from_pretrained(model) pipe pipeline( text-generation, modelmodel, tokenizertokenizer, dtypetorch.bfloat16, device_mapauto, )[!TIP]dtypetorch.bfloat16以低精度加载权重以节省显存device_mapauto由 accelerate 自动分配设备。这两个参数对本地跑大模型非常关键。提示词的最佳实践清单以下是本仓库文档整理出的、有助于改善提示词效果的最佳实践清单选择最新且功能最全的模型同时注意 base 与 instruct/chat 变体之别。从简短、简单的提示开始然后在此基础上迭代优化。把指令放在提示词的开头或结尾。处理大上下文时模型会应用各种优化来防止注意力二次方膨胀这使模型更关注提示的开头与结尾而非中间部分。清晰区分指令与被处理的文本避免混在一起。对任务与期望输出格式、长度、风格、语言等要具体、有描述性避免含糊的说明与指示。指令应聚焦该做什么而非不该做什么。写出第一个词甚至第一句话把输出引导到正确方向——即用前缀词/起始句引导生成。尝试少样本Few-shot与思维链Chain-of-thought等高级技巧。用不同模型测试提示词评估其鲁棒性。对提示词做版本管理并跟踪其性能。从工程角度看第 10 条对应把提示词当作代码一样维护记录每个版本的输入、模型与输出便于回归对比。从零样本到少样本Few-shot Prompting前文基础提示均属于零样本zero-shot提示只给模型指令与上下文不给任何含解答的示例。经指令数据微调的 LLM 通常在这些任务上表现良好。但当任务更复杂、或输出有指令难以传达的隐含要求时可以尝试少样本few-shot提示——在提示中提供若干示例为模型补充上下文让模型按示例的模式生成输出。例如下面的 1-shot 示例给出一个文本 → 日期的映射样例模型就学会了 MM/DD/YYYY 的输出格式。from transformers import pipeline import torch torch.manual_seed(0) # doctest: IGNORE_RESULT pipe pipeline(text-generation, modeltiiuae/falcon-7b-instruct, tokenizerAutoTokenizer.from_pretrained(tiiuae/falcon-7b-instruct), dtypetorch.bfloat16, device_mapauto) prompt Text: The first human went into space and orbited the Earth on April 12, 1961. Date: 04/12/1961 Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon. Date: sequences pipe( prompt, max_new_tokens8, do_sampleTrue, top_k10, ) for seq in sequences: print(fResult: {seq[generated_text]}) # Result: ... Date: 09/28/1960可以尝试 2、4、8 个等不同数量的示例来观察对性能的影响示例数量越多效果通常越好但成本也随之上升。少样本提示的局限LLM 虽能理解示例模式但该方法在复杂推理任务上效果欠佳。少样本提示需要构造更长的提示大量 token 会增加计算量与延迟且提示长度本身有限制。示例过多时模型可能学到你并不想让它学的模式例如第三篇影评总是负面的。用 Chat Template 优化少样本提示针对现代指令微调 LLM建议使用模型专属的 chat template。这些模型以用户-助手多轮对话数据训练将提示按对话结构组织可提升表现。在 Transformers 中通过 [apply_chat_template] 方法完成 tokenize 与格式化from transformers import pipeline import torch pipeline pipeline(modelmistralai/Mistral-7B-Instruct-v0.1, dtypetorch.bfloat16, device_mapauto) messages [ {role: user, content: Text: The first human went into space and orbited the Earth on April 12, 1961.}, {role: assistant, content: Date: 04/12/1961}, {role: user, content: Text: The first-ever televised presidential debate in the United States took place on September 28, 1960, between presidential candidates John F. Kennedy and Richard Nixon.} ] prompt pipeline.tokenizer.apply_chat_template(messages, tokenizeFalse, add_generation_promptTrue) outputs pipeline(prompt, max_new_tokens12, do_sampleTrue, top_k10) for output in outputs: print(fResult: {output[generated_text]})相比把示例塞进单个字符串的基础少样本方式chat template 有两个好处模型能更好识别用户输入与助手输出的角色与模式理解力可能更强由于提示结构与训练时的输入一致模型输出目标格式的一致性更高。在源码层面apply_chat_template定义于 tokenization_utils_base.py。此外TextGenerationPipeline的preprocess方法text_generation.py对Chat输入会自动调用apply_chat_templateadd_generation_promptnot continue_final_message并默认在末条消息为assistant角色时按续写prefill处理。使用某个指令模型前应查阅其文档了解 chat template 的具体格式以正确组织少样本提示。Chain-of-thought思维链让模型逐步推理思维链CoT提示通过引导模型生成中间推理步骤改善复杂推理任务的结果。有两种方式促使模型生成推理步骤少样本式给出带详细解答过程的示例示范如何逐步处理问题指令式在提示中加入Lets think step by step或深呼吸一步步解决问题等短语直接指示模型推理。本文档对推理部分的 muffin 示例应用 CoT 技术并配合更大的模型例如可在 HuggingChat 中体验的tiiuae/falcon-180B-chat推理结果显著改善。下面是一段逐步推理的输出样例Lets go through this step-by-step: 1. You start with 15 muffins. 2. You eat 2 muffins, leaving you with 13 muffins. 3. You give 5 muffins to your neighbor, leaving you with 8 muffins. 4. Your partner buys 6 more muffins, bringing the total number of muffins to 14. 5. Your partner eats 2 muffins, leaving you with 12 muffins. Therefore, you now have 12 muffins.与少样本类似CoT 的代价是需要花更多精力设计一组能引导模型推理的提示且提示变长会增加延迟。实战示例用 Falcon-7b-instruct 解决六类 NLP 任务以下示例沿用前文的pipeFalcon-7b-instructbfloat16 device_mapauto逐一展示不同任务的提示词结构。文本分类情感分析指令放最前随后给出待分类文本并在响应开头预置Sentiment: 引导模型输出标签torch.manual_seed(0) # doctest: IGNORE_RESULT prompt Classify the text into neutral, negative or positive. Text: This movie is definitely one of my favorite movies of its kind. The interaction between respectable and morally strong characters is an ode to chivalry and the honor code amongst thieves and policemen. Sentiment: sequences pipe(prompt, max_new_tokens10) for seq in sequences: print(fResult: {seq[generated_text]}) # Result: ... Sentiment: # Positive输出包含了指令中给出的分类标签之一且是正确标签。这里传入的max_new_tokens控制模型生成的 token 数量是众多文本生成参数之一。命名实体识别NER改变指令即可让 LLM 完成 NER同时设置return_full_textFalse让输出不含提示原文torch.manual_seed(1) # doctest: IGNORE_RESULT prompt Return a list of named entities in the text. Text: The Golden State Warriors are an American professional basketball team based in San Francisco. Named entities: sequences pipe( prompt, max_new_tokens15, return_full_textFalse, ) for seq in sequences: print(f{seq[generated_text]}) # - Golden State Warriors # - San Francisco模型正确识别出了给定文本中的两个命名实体。翻译翻译可用 encoder-decoder 模型但为了示例简洁这里继续使用 Falcon-7b-instruct通过基本提示将英文翻译为意大利语torch.manual_seed(2) # doctest: IGNORE_RESULT prompt Translate the English text to Italian. Text: Sometimes, Ive believed as many as six impossible things before breakfast. Translation: sequences pipe( prompt, max_new_tokens20, do_sampleTrue, top_k10, return_full_textFalse, ) for seq in sequences: print(f{seq[generated_text]}) # A volte, ho creduto a sei impossibili cose prima di colazione.这里通过do_sampleTrue与top_k10让模型在生成时更灵活。文本摘要摘要同样是输出高度依赖输入的生成任务encoder-decoder 模型可能是更好选择但 decoder 风格模型也可胜任。之前我们把指令放在提示开头实际上提示末尾同样是放置指令的好位置通常建议放在两端之一torch.manual_seed(3) # doctest: IGNORE_RESULT prompt Permaculture is a design process mimicking the diversity, functionality and resilience of natural ecosystems. The principles and practices are drawn from traditional ecological knowledge of indigenous cultures combined with modern scientific understanding and technological innovations. Permaculture design provides a framework helping individuals and communities develop innovative, creative and effective strategies for meeting basic needs while preparing for and mitigating the projected impacts of climate change. Write a summary of the above text. Summary: sequences pipe( prompt, max_new_tokens30, do_sampleTrue, top_k10, return_full_textFalse, ) for seq in sequences: print(f{seq[generated_text]}) # Permaculture is an ecological design mimicking natural ecosystems to meet basic needs and prepare for climate change. It is based on traditional knowledge and scientific understanding.问答Question Answering问答提示可拆分为逻辑组件指令、上下文、问题、起始词/短语如Answer:用起始词驱动模型开始生成答案torch.manual_seed(4) # doctest: IGNORE_RESULT prompt Answer the question using the context below. Context: Gazpacho is a cold soup and drink made of raw, blended vegetables. Most gazpacho includes stale bread, tomato, cucumbers, onion, bell peppers, garlic, olive oil, wine vinegar, water, and salt. Northern recipes often include cumin and/or pimentón (smoked sweet paprika). Traditionally, gazpacho was made by pounding the vegetables in a mortar with a pestle; this more laborious method is still sometimes used as it helps keep the gazpacho cool and avoids the foam and silky consistency of smoothie versions made in blenders or food processors. Question: What modern tool is used to make gazpacho? Answer: sequences pipe( prompt, max_new_tokens10, do_sampleTrue, top_k10, return_full_textFalse, ) for seq in sequences: print(fResult: {seq[generated_text]}) # Result: Modern tools are used, such as immersion blenders推理Reasoning推理是 LLM 最具挑战性的任务之一好的结果往往需要 CoT 等高级技巧。先用基础提示尝试简单算术torch.manual_seed(5) # doctest: IGNORE_RESULT prompt There are 5 groups of students in the class. Each group has 4 students. How many students are there in the class? sequences pipe( prompt, max_new_tokens30, do_sampleTrue, top_k10, return_full_textFalse, ) for seq in sequences: print(fResult: {seq[generated_text]}) # Result: # There are a total of 5 groups, so there are 5 x 420 students in the class.答案正确。再提高复杂度torch.manual_seed(6) # doctest: IGNORE_RESULT prompt I baked 15 muffins. I ate 2 muffins and gave 5 muffins to a neighbor. My partner then bought 6 more muffins and ate 2. How many muffins do we now have? sequences pipe( prompt, max_new_tokens10, do_sampleTrue, top_k10, return_full_textFalse, ) for seq in sequences: print(fResult: {seq[generated_text]}) # Result: # The total number of muffins now is 21这次答案是错的应为 12。原因可能是提示过于基础或采样选择所致另外这里选用的是 Falcon 最小版本——各种规模的模型推理都很困难但更大模型的表现通常会更好。这正是需要 CoT 等高级技巧的场景。生成参数的底层原理上述示例反复出现max_new_tokens、do_sample、top_k等参数它们最终进入模型的generate流程。从 configuration_utils.py 的文档可以看到解码策略的对应关系num_beams1且do_sampleFalse贪心解码greedy decodingnum_beams1且do_sampleTrue多项式采样multinomial samplingnum_beams1且do_sampleFalsebeam search 解码num_beams1且do_sampleTruebeam-search 多项式采样。其中max_new_tokens见 configuration_utils.py推荐用于控制模型生成多少个新 tokendo_sample控制是否采样top_k限制采样时只从概率最高的 k 个 token 中选择。默认值上do_sample为False、top_k为 50configuration_utils.py。任务示例中do_sampleTrue, top_k10的组合本质上是在贪心解码之外引入随机性从而让翻译、摘要等开放生成任务更灵活。提示 vs 微调何时应该转向微调优化提示能取得不错的结果但你可能仍会纠结微调是否更适合自己的场景。以下是倾向于微调较小模型的几种情形领域差异极大你的领域与 LLM 预训练内容差异很大且大量提示优化仍得不到满意结果低资源语言模型需要在低资源语言上表现良好敏感数据需要在受严格监管的敏感数据上训练模型资源受限由于成本、隐私、基础设施等原因必须使用小模型。以上情形还需满足前提你已经拥有或能以合理成本获得足够大的领域数据集用于微调并且有足够的时间与算力资源。若这些条件不满足优化提示词往往是更划算的路径。结语提示词工程是一门需要反复实验的迭代手艺从简单提示起步遵循指令放两端、指令与文本分离、具体描述期望输出、用起始词引导等最佳实践配合少样本与思维链技巧再根据生成参数与模型选型持续调优。本文所有示例均基于本仓库transformers的text-generation管道可直接复制运行深入理解生成策略可继续阅读 文本生成策略与 LLM 生成教程相关任务专章还可见 命名实体识别、翻译、摘要、问答 等文档。【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考