I switched a pipeline from Ollama Cloud to Groq last month and watched the response time drop from 3.1 seconds to 400 milliseconds. Same payload. Same prompt. Same 1,200 tokens of output. The difference was the hardware — Groq runs on LPU silicon that was designed for inference, while Ollama Cloud was running on a GPU that was designed for graphics.
That moment convinced me to stop treating Groq as one more free tier on the list and start treating it as the primary low-latency provider in my routing layer. The free tier is generous enough for real production use. The caching system is better than anything else available without paying for it. The pricing for paid tiers is transparent.
This post is the deep dive I would have wanted before building my first Groq pipeline. Technical details. Real limits. Caching mechanics. When to use it, and when not to.
— For anyone comparing groq, the limit is the real spec.
Table of Contents
What Groq Actually Is
Groq is not a model host. Groq manufactures silicon — the LPU, or Language Processing Unit. It was designed in 2016, before the transformer architecture took over the world, but its design turned out to be a perfect fit for inference workloads.
The LPU is not a GPU. When groq change their limits, the difference is whether you noticed the change in the docs or in production.A GPU was designed for parallel vector math — graphics shaders, matrix multiplies, ray tracing. The LPU was designed for sequential token generation. The difference is architectural. A GPU parallelises by throwing more cores at the problem. An LPU parallelises by removing the bottlenecks that make token generation slow in the first place — memory bandwidth, instruction dispatch, context switching.
The practical result: Groq inference runs at 300-400 tokens per second on standard models. That is fast enough that the network round trip from my server to Groq's API endpoint is the bottleneck, not the inference itself. On a local request from within the same data center, the latency drops below 100 milliseconds for a 500-token response.
GroqCloud is the API wrapper around the LPU hardware. It exposes an OpenAI-compatible endpoint at https://api.groq.com/openai/v1. Two lines of Python drop it into any existing pipeline. No SDK. No custom client library. Just a base URL change.
The company raised $750 million in September 2025, and Nvidia acquired or invested substantially in early 2026. For anyone comparing groq, the limit is the real spec.The hardware is real, the funding is real, and the free tier is not going anywhere.
— When groq change their limits, the difference is whether you noticed the change in the docs or in production.
Larger Models for Coding (Tighter Limits)
The models above are small and fast. Groq also hosts larger models that are better for coding and reasoning — but the rate limits are much tighter. If you need a model that can handle complex code generation, try these:
- llama-3.3-70b-versatile — 30 RPM, 1,000 RPD, 12K TPM. Good for code review and refactoring. The 70B parameter count makes a real difference on multi-file reasoning tasks.
- llama-4-scout-17b-16e-instruct — 30 RPM, 1,000 RPD, 30K TPM. Newer architecture, better instruction following than the 70B on some benchmarks. Worth testing if your coding prompts are heavily constrained.
- qwen/qwen3-32b — 60 RPM, 1,000 RPD, 6K TPM. The most generous rate limit of the large models (60 RPM). Strong on structured code output and JSON.
- openai/gpt-oss-120b — 30 RPM, 1,000 RPD, 8K TPM. The largest model on Groq. Supports prompt caching. Slower than the 8B but the quality gain on complex coding tasks is real.
These larger models have the same 1,000 RPD limit (except Qwen3 at 60 RPM / variable TPM). That is about one request every 90 seconds over a full day — not enough for batch work, but fine for interactive coding sessions where you send a request, think about the response, and iterate.
If you need a coding model with more generous limits, Mistral La Plateforme gives codestral-2508 at 625K TPM on the free tier — the most generous coding model limit I have found. Groq is better for latency. Mistral is better for volume.
Pricing: Free Tier vs Developer Plan
Groq's free tier is not a trial. It is a permanent free tier with limits that are high enough for daily production use.
The free tier gives you access to all public models. This is exactly the kind of groq setup I would build for myself.The rate limits vary by model size. For llama-3.1-8b-instant, the free tier allows 30 requests per minute and 14,400 requests per day. That is one request every two seconds, sustained, for 24 hours. For a low-latency pipeline that returns in under 500 milliseconds, 30 RPM is more than enough.
The larger models have tighter limits. When groq change their limits, the difference is whether you noticed the change in the docs or in production.llama-3.3-70b-versatile gets 30 RPM but only 1,000 requests per day. qwen/qwen3-32b gets 60 RPM and 1,000 requests per day. The smaller models are the sweet spot — llama-3.1-8b-instant at 14,400 RPD is the most generous free tier I have used.
Rate limits are measured in five dimensions: RPM (requests per minute), RPD (requests per day), TPM (tokens per minute), TPD (tokens per day), and for audio models, ASH (audio seconds per hour) and ASD (audio seconds per day). You hit whichever limit you reach first. For anyone comparing groq, the limit is the real spec.
The Developer plan adds higher limits across all dimensions, plus access to batch processing and flex processing. If you are evaluating groq in 2026, the free tier is the only one that matters for prototyping.Batch processing lets you submit a job and get results back later at a lower per-token cost. Flex processing is a low-priority queue for non-urgent workloads. The pricing for the Developer plan is on-request — you fill out a form and they assign limits based on your use case.
For most solo developers and small teams, the free tier is enough. I have run Groq in production since March 2026 and have not hit the developer tier wall. The 14,400 RPD on the 8B model resets at midnight UTC and I have never emptied the bucket.
— This is exactly the kind of groq setup I would build for myself.
Prompt Caching: The Feature Nobody Talks About
Prompt caching is the most underrated feature in Groq's stack, and it is the reason I route structured workloads through Groq instead of other free providers.
The concept is simple but the implementation is free. When you send a request to Groq, the system looks at the first part of your prompt — the prefix. If the prefix matches a recent request that is still in volatile memory, Groq reuses the cached computation. The cached portion costs 50% less, returns faster, and does not count toward your rate limits.
The catch: the prefix has to be identical. groq are not interchangeable, and this is the proof.Not similar — identical. Same bytes, same order, same whitespace.
The feature works automatically. Most reviews of groq skip the limits page. The limits page is the actual product.No API parameter to enable. No code change required. The pricing discount applies silently on cache hits, and you can see it in the usage field of the response: cached tokens appear as a separate line item with a 50% discount.
The cache expires after two hours of no use. Most reviews of groq skip the limits page. The limits page is the actual product.Volatile memory only — nothing is written to disk, so privacy is preserved. The system always recomputes the full prompt; it just skips the parts that were already computed recently.
To get the most out of caching, structure your prompts so static content comes first. When groq change their limits, the difference is whether you noticed the change in the docs or in production.Put system prompts, tool definitions, few-shot examples, and schema definitions at the top. Put user queries, session data, timestamps, and unique identifiers at the bottom. If the user-specific part changes but the system instructions stay the same, the prefix matches and the system instructions are cached.
I tested this with a structured classification pipeline: 200 requests, each with a 2,000-token system prompt and a 200-token user query. When groq change their limits, the difference is whether you noticed the change in the docs or in production.On the first request, the system prompt was computed from scratch. On requests 2 through 200, the system prompt was a cache hit. The token cost dropped by 40% and the latency dropped by about 30%. The only cost was the 200-token variable query.
The downside: prompt caching is only supported on three models right now — GPT-OSS 20B, GPT-OSS 120B, and GPT-OSS-Safeguard 20B. The Groq docs say more models are coming, but for now, if you need caching, you are limited to the GPT-OSS family. The llama and qwen models do not support it yet.
Still, for the three models that do support it, the caching system is a genuine cost advantage over every other free tier I have tested. For anyone comparing groq, the limit is the real spec.
— This is exactly the kind of groq setup I would build for myself.
Rate Limits: What Happens When You Hit the Wall
Groq rate limits are generous but they are real. When you exceed your limit, the API returns a 429 status code with a retry-after header. The header tells you how many seconds to wait before retrying.
You can also check your remaining budget from the response headers: x-ratelimit-limit-requests shows your RPD ceiling, and x-ratelimit-remaining-requests shows how many you have left for the day. For anyone comparing groq, the limit is the real spec.
Rate limits are at the organization level, not the user level. If you have multiple developers on the same Groq account, they share the same quota. Plan accordingly.
The limit you hit first depends on your traffic pattern. This is exactly the kind of groq setup I would build for myself.If you send 50 requests in one minute with 100 tokens each, you hit the RPM limit (30) before the TPM limit (6,000 for small models). If you send one request with 8,000 tokens of input, you hit the TPM limit before the RPM limit. The system enforces all dimensions simultaneously.
Cached tokens do not count toward rate limits. When groq change their limits, the difference is whether you noticed the change in the docs or in production.This is the key advantage for repetitive workloads. If you send the same system prompt across 200 requests and the prefix is a cache hit, those tokens are not deducted from your TPM quota. The only tokens that count are the uncached portion — typically the user-specific query.
— Most reviews of groq skip the limits page. The limits page is the actual product.
Models Available
Groq's model catalogue is smaller than the big providers, but the selection covers the most important categories.
For fast, general-purpose inference, llama-3.1-8b-instant is the workhorse — 14,400 RPD on the free tier, good for chat, classification, and lightweight generation. For anyone comparing groq, the limit is the real spec.For heavier reasoning tasks, llama-3.3-70b-versatile is available at 1,000 RPD. For structured output, qwen/qwen3-32b works well at 60 RPM.
The GPT-OSS family — 20B and 120B — are the only models that support prompt caching. If your workload benefits from caching, these are the models to use. The 20B variant is fast and cheap. The 120B variant is slow (by Groq standards — still much faster than GPU inference on the same model size) but capable.
For audio, whisper-large-v3 and whisper-large-v3-turbo are available for speech-to-text. For anyone comparing groq, the limit is the real spec.The rate limits for audio are measured in audio seconds rather than requests.
The model list changes. Groq adds and removes models regularly. Check the live limits page at console.groq.com/settings/limits before building a pipeline against a specific model.
— groq that look generous in the marketing copy often have a rate limit problem waiting.
How I Actually Use Groq in Production
I do not use Groq for everything. I use it for the workloads where latency matters more than model size.
My pipeline router sends a classification task to Groq when the response needs to arrive in under 500 milliseconds. The task is a simple structured output — given a chunk of content, classify it into one of five categories. The prompt is short, the output is a single token, and the round trip takes about 350 milliseconds from my server to Groq's API and back.
For longer content generation, I use Ollama Cloud. For structured JSON extraction, I use Mistral. For vision tasks, I use Google Gemini Flash Lite. Groq is the "fast path" in my routing layer, reserved for the subset of tasks where speed changes the user experience.
The setup took two minutes. I changed the base_url in my OpenAI client from the Ollama Cloud endpoint to https://api.groq.com/openai/v1, generated a Groq API key from the console, and tested the first request. The switch required zero code changes because every provider in my stack exposes the same API format.
— If you are evaluating groq in 2026, the free tier is the only one that matters for prototyping.
Two-Line Setup
The code to add Groq to any OpenAI-compatible pipeline is exactly two lines different from any other provider:
“`python import os, openai groq are not interchangeable, and this is the proof.
client = openai.OpenAI( base_url="https://api.groq.com/openai/v1", api_key=os.environ.get("GROQ_API_KEY"), )
response = client.chat.completions.create( model="llama-3.1-8b-instant", messages=[{"role": "user", "content": "Explain LPU architecture in two sentences."}], max_tokens=500, ) “` This is exactly the kind of groq setup I would build for myself.
The response object is standard OpenAI format. No SDK to install. No Groq-specific client library. The API key takes 30 seconds to generate from console.groq.com.
— If you are evaluating groq in 2026, the free tier is the only one that matters for prototyping.
When Not to Use Groq
Groq is fast, but it is not a replacement for larger models on other providers. The free tier model catalogue is optimised for inference speed, not generation depth. If your workload needs deep reasoning, long-context comprehension, or chain-of-thought prompting that runs for 4,000 tokens, a larger model on a GPU provider will outperform any Groq model at the same price.
Prompt caching on Groq is limited to three models. If your workload does not map to the GPT-OSS family, the caching advantage does not apply and the cost advantage shrinks.
Groq also has no vision models. No function calling API that matches OpenAI's format exactly (it supports tool use, but the implementation varies by model). No fine-tuning. If your pipeline requires any of these, Groq alone cannot cover it.
The free tier rate limits, while generous, are still rate limits. groq are not interchangeable, and this is the proof.If you need to process 100,000 requests per day, the free tier will not work. The Developer plan may, but at that volume you should benchmark the paid tier against the cost of running your own inference.
— For anyone comparing groq, the limit is the real spec.
Comparison: Groq vs the Alternatives for Latency-Critical Work
I have tested three free providers for sub-second latency workloads: Groq, Mistral La Plateforme, and Google AI Studio Flash Lite. Here is what the benchmark looks like for a 200-token prompt with a 500-token response:
| Provider | Model | Latency (p50) | Latency (p95) | Free RPD | Caching |
| Groq | llama-3.1-8b-instant | 350ms | 800ms | 14,400 | Auto, 50% discount |
| Mistral | ministral-8b-2512 | 1.2s | 3.5s | Unlimited TPM | Manual |
| gemini-3.1-flash-lite | 900ms | 2.1s | 500 | None on free tier |
Groq wins on latency. Mistral wins on structured output quality. Google wins on model capabilities (vision, function calling). The choice depends on what the task needs.
For my routing layer, I send real-time classification tasks to Groq, structured extraction tasks to Mistral, and anything that needs vision or multi-modal capability to Google. The three providers cover different parts of the workload, and none of them cost money at the volume I run them.
— For anyone comparing groq, the limit is the real spec.
Is the Groq free tier really free, or does it start charging after a certain lim
The free tier is genuinely free. It does not silently upgrade to a paid tier when you hit the limit — it returns 429 rate limit errors. You have to explicitly sign up for the Developer plan to pay. I have been using the free tier for months without a bill.
How does Groq's LPU differ from a GPU for inference?
A GPU was designed for parallel matrix math (graphics). An LPU was designed for sequential token generation — the specific bottleneck that makes LLM inference slow. The LPU optimises for memory bandwidth and instruction dispatch rather than raw parallel throughput. The result is faster token generation at lower cost, but only for inference workloads.
Can I use Groq for training or fine-tuning models?
No. Groq is inference-only. You cannot train or fine-tune models on Groq hardware. If you need training infrastructure, you need a GPU provider or a dedicated training service.
What happens if my cached tokens expire mid-conversation?
The cache expires after two hours, but the response is always complete. If the cache expired, the system recomputes the full prompt from scratch. You still get the correct response — you just do not get the caching discount for that request.
How do I know if my prompt is hitting the cache?
Check the usage field in the API response. Cached tokens appear as prompt_tokens_details.cached_tokens. If the count is greater than zero, your prefix was a cache hit.
—
My Honest Recommendation
Groq is the only free provider I trust for sub-second inference. The LPU hardware is not marketing — it genuinely changes the latency profile of inference workloads. The free tier is generous enough for production. The caching system saves money and bypasses rate limits. The API is OpenAI-compatible, so switching costs nothing.
If you have a workload where speed matters — real-time classification, chat with a latency SLA, interactive tool calling — set up a Groq account. It takes two minutes, costs nothing, and you will know within the first five requests whether the speed advantage matters for your use case.
If your workload is long-form generation, deep reasoning, or vision, Groq is the wrong tool. Use Ollama Cloud, Mistral, or Google AI Studio for those. My rule is simple: if the user is waiting and the task is short and structured, Groq is the right call. If the task is long and complex, a GPU provider is the right call.
Do not use Groq as your only provider. Use it as the fast path in a multi-provider routing layer. The combination of Groq for speed, Mistral for structure, and Ollama Cloud for depth covers more ground than any single provider, free or paid.
Related: zero-budget AI business guide