Google DeepMind 推出 Gemma 4 12B：免編碼器多模態模型，原生支援音訊，可在 16GB 筆電上執行

Google DeepMind just released Gemma 4 12B, a dense multimodal model that strips out traditional encoders entirely. Vision and audio flow straight into the LLM backbone. The result is a model that runs agentic workflows on a consumer laptop with 16 GB of RAM. It ships under the Apache 2.0 license. Model Overview & Access Gemma 4 12B is a 12-billion-parameter decoder-only transformer. It handles text, images, audio, and video natively. There are no separate vision or audio encoders. The decoder uses the same structure as the Gemma 4 31B Dense model. It bridges the gap between the edge-friendly E4B and the larger 26B Mixture of Experts variant. Architecture: Unified, encoder-free decoder-only transformer. Modalities: Text, image, video, and native audio input — the first mid-sized Gemma with audio. Hardware requirement: 16 GB VRAM or unified memory. Runs on consumer GPU laptops and Apple Silicon Macs. License: Apache 2.0. Weights are open and publicly downloadable. Inference stack: Compatible with llama.cpp, MLX, vLLM, Ollama, SGLang, Unsloth, and LM Studio. Download: Hugging Face and Kaggle. The instruct variant is google/gemma-4-12B-it. Integration: Hugging Face Transformers, LiteRT-LM CLI, and an OpenAI-compatible local API server via litert-lm serve. A dedicated Multi-Token Prediction (MTP) drafter model is also released. It reduces inference latency on local hardware. Architecture: The Encoder-Free Design Every prior mid-sized Gemma model used separate Transformer encoders for vision and audio. Those encoders added latency and parameter overhead. The medium-sized Gemma 4 models carry a 550M-parameter vision encoder. The E2B and E4B models include a 300M-parameter audio encoder. All of that is gone in the 12B. Vision embedder (35M parameters): Raw images are split into 48×48 pixel patches. Each patch is projected to the LLM’s hidden dimension with a single matrix multiplication. There is no attention layer; each patch is processed independently. Spatial position is injected using a factorized coordinate lookup: a learned X matrix and a learned Y matrix. For a patch at (x, y), the model looks up two learned embeddings and adds them to form a position vector. This is added to the patch embedding, followed by normalization. That is the entire vision pipeline. Audio wave projection: Raw 16 kHz audio is sliced into 40 ms frames. Each frame contains 640 values. Those values are linearly projected into the same embedding space as text tokens. There is no feature extraction and no conformer layers. The LLM’s existing Rotary Position Embedding (RoPE) handles the 1-D temporal sequence. The audio encoder in the E2B and E4B used 12 conformer layers. All of that is removed. Importance: The unified weight space means you no longer co-tune separate frozen encoders. Downstream fine-tuning with LoRA or full tuning updates vision, audio, and text processing in a single pass. Hugging Face Transformers and Unsloth already support this. The encoder-free design reduces multimodal latency. The LLM backbone starts processing immediately. No encoder must finish first. Capabilities & Performance Google DeepMind team has not published full benchmark results in the initial launch materials. The official release notes state the 12B model performs nearing the 26B MoE model on standard benchmarks, at less than half the total memory footprint. https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12b/ The model’s demonstrated capabilities include: Automatic speech recognition. Transcribes audio natively without an external ASR pipeline. Agentic reasoning. Runs multi-step workflows locally, with performance approaching the 26B MoE model. Diarization. Distinguishes speakers in audio input. Video understanding. Processes video frames alongside audio. A demo analyzed a 5-minute Google I/O keynote segment using 313 frames at 1 FPS with a visual token budget of 70 per frame. Coding. Built a Gradio image-processing app using its own code generation, served locally with llama.cpp. Multimodal agentic workflows. The official Gemma Skills repository at github.com/google-gemma/gemma-skills provides pre-built agent capabilities. In Google’s own Google AI Edge Eloquent app, the switch to Gemma 4 12B produced what Google reports as a 60%+ jump in overall quality, with improved instruction following and scope adherence. Marktechpost’s Visual Explainer #mtp-g4-slider hr, #mtp-g4-slider p:empty, #mtp-g4-slider del, #mtp-g4-slider s { display:none !important; } #mtp-g4-slider *{ box-sizing:border-box !important; margin:0; padding:0; } #mtp-g4-slider{ --g-blue:#4285F4; --g-red:#EA4335; --g-yellow:#FBBC04; --g-green:#34A853; --g-ink:#202124; --g-grey:#5f6368; --g-line:#dadce0; --g-surface:#f8f9fa; font-family:'Google Sans','Product Sans',Roboto,Arial,sans-serif !important; color:var(--g-ink) !important; background:#ffffff !important; border:1px solid var(--g-line) !important; border-radius:20px !important; max-width:840px; margin:24px auto; overflow:hidden; box-shadow:0 1px 3px rgba(60,64,67,.16), 0 8px 28px rgba(60,64,67,.10); } /* four-color brand bar */ #mtp-g4-slider .mtpg4-bar{ display:flex; height:6px; width:100%; } #mtp-g4-slider .mtpg4-bar span{ flex:1; } #mtp-g4-slider .mtpg4-bar span:nth-child(1){ background:var(--g-blue) !important; } #mtp-g4-slider .mtpg4-bar span:nth-child(2){ background:var(--g-red) !important; } #mtp-g4-slider .mtpg4-bar span:nth-child(3){ background:var(--g-yellow) !important; } #mtp-g4-slider .mtpg4-bar span:nth-child(4){ background:var(--g-green) !important; } #mtp-g4-slider .mtpg4-viewport{ overflow:hidden; position:relative; } #mtp-g4-slider .mtpg4-track{ display:flex; transition:transform .45s cubic-bezier(.4,0,.2,1); } #mtp-g4-slider .mtpg4-slide{ min-width:100%; padding:38px 44px 30px; } #mtp-g4-slider .mtpg4-eyebrow{ display:inline-block; font-size:12px; font-weight:700; letter-spacing:.10em; text-transform:uppercase; padding:5px 12px; border-radius:999px; background:var(--g-surface) !important; color:var(--g-grey) !important; border:1px solid var(--g-line) !important; margin-bottom:16px; } #mtp-g4-slider .mtpg4-slide h2{ font-size:27px; line-height:1.2; font-weight:700; color:var(--g-ink) !important; margin-bottom:8px; letter-spacing:-.01em; } #mtp-g4-slider .mtpg4-slide h3{ font-size:15px; font-weight:500; color:var(--g-grey) !important; margin-bottom:20px; } #mtp-g4-slider .mtpg4-slide p.lead{ font-size:16px; line-height:1.6; color:var(--g-ink) !important; max-width:62ch; margin-bottom:18px; } #mtp-g4-slider ul.mtpg4-list{ list-style:none !important; padding:0 !important; } #mtp-g4-slider ul.mtpg4-list li{ position:relative; padding:11px 14px 11px 18px; margin-bottom:10px; font-size:15px; line-height:1.5; color:var(--g-ink) !important; background:var(--g-surface) !important; border-radius:10px; border-left:4px solid var(--accent) !important; } #mtp-g4-slider ul.mtpg4-list li b{ font-weight:700; } #mtp-g4-slider code{ font-family:'Roboto Mono',ui-monospace,monospace !important; font-size:13px; padding:2px 6px; border-radius:5px; background:#ffffff !important; color:var(--g-blue) !important; border:1px solid var(--g-line) !important; } /* per-slide accent */ #mtp-g4-slider .mtpg4-slide[data-accent="blue"]{ --accent:var(--g-blue); } #mtp-g4-slider .mtpg4-slide[data-accent="red"]{ --accent:var(--g-red); } #mtp-g4-slider .mtpg4-slide[data-accent="yellow"]{ --accent:var(--g-yellow); } #mtp-g4-slider .mtpg4-slide[data-accent="green"]{ --accent:var(--g-green); } #mtp-g4-slider .mtpg4-slide .mtpg4-eyebrow{ color:var(--accent) !important; border-color:var(--accent) !important; } /* controls */ #mtp-g4-slider .mtpg4-nav{ display:flex; align-items:center; justify-content:space-between; padding:14px 24px; border-top:1px solid var(--g-line) !important; background:#ffffff !important; } #mtp-g4-slider .mtpg4-dots{ display:flex; gap:8px; } #mtp-g4-slider .mtpg4-dot{ width:8px; height:8px; border-radius:999px; border:none !important; backgrou