OSLabs developers have created PennyWyze, a new open-source tool designed to help you save money on Claude AI models. It identifies the cheapest Claude tier that still meets your quality requirements.
Good news for anyone using Claude AI models: a new tool called PennyWyze can help you stop overpaying and save money! This means you no longer have to guess which Claude tier is right for you, or automatically default to the most expensive option.
Three clever engineers from OSLabs, including Maia, have built PennyWyze, an open-source command-line interface (CLI). Its whole purpose is to audit your Claude usage and tell you which tier—Opus, Sonnet, or Haiku—provides the best value for your specific needs.
It’s quite straightforward. You give PennyWyze two things: first, the exact prompt you're already using with Claude in your production environment. Second, you provide a 'golden dataset'—a small set of real-world examples where you already know the correct answers. For instance, if your input is «I was charged twice this month,» your expected output might be «billing.»
PennyWyze then takes your prompt and dataset and sends each example to all three Claude tiers (Opus, Sonnet, and Haiku) using the real Anthropic API. It carefully grades each model's response against your expected answers. Crucially, it calculates the actual API cost based on token counts, not just an estimate.
After running its tests, PennyWyze generates a report. This report shows you each model's accuracy on your specific task and projects your potential monthly cost at your actual usage volume. It then gives you a clear verdict, often recommending a switch to a cheaper tier.
Imagine an audit where PennyWyze tested 50 questions. It found that Opus and Haiku performed equally well, both getting 49 out of 50 correct. Sonnet actually scored lower while costing almost three times as much as Haiku. The amazing part? PennyWyze suggested switching to claude-haiku, saving that user nearly $180 per month for an audit cost of just $0.15! This shows that sometimes, the most expensive model isn’t necessary to meet your quality standards. Instead of assuming, you can now test and optimize.
Three clever engineers from OSLabs, including Maia, have built PennyWyze, an open-source command-line interface (CLI). Its whole purpose is to audit your Claude usage and tell you which tier—Opus, Sonnet, or Haiku—provides the best value for your specific needs.
It’s quite straightforward. You give PennyWyze two things: first, the exact prompt you're already using with Claude in your production environment. Second, you provide a 'golden dataset'—a small set of real-world examples where you already know the correct answers. For instance, if your input is «I was charged twice this month,» your expected output might be «billing.»
PennyWyze then takes your prompt and dataset and sends each example to all three Claude tiers (Opus, Sonnet, and Haiku) using the real Anthropic API. It carefully grades each model's response against your expected answers. Crucially, it calculates the actual API cost based on token counts, not just an estimate.
After running its tests, PennyWyze generates a report. This report shows you each model's accuracy on your specific task and projects your potential monthly cost at your actual usage volume. It then gives you a clear verdict, often recommending a switch to a cheaper tier.
Imagine an audit where PennyWyze tested 50 questions. It found that Opus and Haiku performed equally well, both getting 49 out of 50 correct. Sonnet actually scored lower while costing almost three times as much as Haiku. The amazing part? PennyWyze suggested switching to claude-haiku, saving that user nearly $180 per month for an audit cost of just $0.15! This shows that sometimes, the most expensive model isn’t necessary to meet your quality standards. Instead of assuming, you can now test and optimize.
