1.下载源码git clone https://github.com/PrismML-Eng/llama.cpp2编译cd llama.cpp # Build with CUDA support cmake -B build -DGGML_CUDAON cmake --build build -j3.启动服务./bin/llama-server -m /mnt/87b26cb3-9627-446a-b048-2f61a945fafa/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf --host 0.0.0.0 --port 8004 -ngl 99 -c 16384 -fa on -np 4 --reasoning off如果遇到CUDA编译问题可以直接指定路径mkdir -p build cd build/ cmake .. -DCMAKE_CUDA_COMPILER/usr/local/cuda-12.9/bin/nvcc -DCUDAToolkit_ROOT/usr/local/cuda-12.9 -DCUDAToolkit_INCLUDE_DIR/usr/local/cuda-12.9/include -DGGML_CUDAON -DGGML_CUDA_FORCE_MMQON make -j $(nproc)Embeding 模型./bin/llama-server \ -m /mnt/87b26cb3-9627-446a-b048-2f61a945fafa/bge-m3-Q4_K_M-GGUF/bge-m3-q4_k_m.gguf \ --host 0.0.0.0 --port 8005 \ --embedding \ -ngl 99 \ -c 8192
1.下载源码git clone https://github.com/PrismML-Eng/llama.cpp2编译cd llama.cpp # Build with CUDA support cmake -B build -DGGML_CUDAON cmake --build build -j3.启动服务./bin/llama-server -m /mnt/87b26cb3-9627-446a-b048-2f61a945fafa/Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf --host 0.0.0.0 --port 8004 -ngl 99 -c 16384 -fa on -np 4 --reasoning off如果遇到CUDA编译问题可以直接指定路径mkdir -p build cd build/ cmake .. -DCMAKE_CUDA_COMPILER/usr/local/cuda-12.9/bin/nvcc -DCUDAToolkit_ROOT/usr/local/cuda-12.9 -DCUDAToolkit_INCLUDE_DIR/usr/local/cuda-12.9/include -DGGML_CUDAON -DGGML_CUDA_FORCE_MMQON make -j $(nproc)Embeding 模型./bin/llama-server \ -m /mnt/87b26cb3-9627-446a-b048-2f61a945fafa/bge-m3-Q4_K_M-GGUF/bge-m3-q4_k_m.gguf \ --host 0.0.0.0 --port 8005 \ --embedding \ -ngl 99 \ -c 8192
)
