Anthropic Announces Claude 3.5 Sonnet: A New Benchmark in AI

Anthropic launched Claude 3.5 Sonnet, the first release in their upcoming Claude 3.5 model family. According to them, Claude 3.5 Sonnet surpasses competitor models and surpasses their own Claude 3 Opus in intelligence, achieving this while maintaining the speed and cost of their mid-tier model.

Availability and Pricing

Claude 3.5 Sonnet is available for free on Claude.ai and their iOS app. Paid plans offer higher rate limits. The model is also accessible through Anthropic API, Amazon Bedrock, and Google Cloud's Vertex AI. Their pricing structure is $3 per million input tokens and $15 per million output tokens, with a 200K token context window.

Enhanced Performance

Claude 3.5 Sonnet sets new standards in areas like graduate-level reasoning (GPQA), undergraduate-level knowledge (MMLU), and coding proficiency (HumanEval). It reportedly shows significant improvement in understanding humor, nuance, and complex instructions. Additionally, it excels at creating high-quality content with a natural tone.

Faster and More Efficient

Claude 3.5 Sonnet operates at twice the speed of Claude 3 Opus. This speed boost, combined with its cost-effectiveness, makes it ideal for complex tasks like context-aware customer support and managing multi-step workflows.

Advanced Coding Capabilities

Internal evaluations showed Claude 3.5 Sonnet solving 64% of problems in agentic coding tasks, significantly exceeding Claude 3 Opus (38%). These evaluations involve fixing bugs or adding functionalities to open-source codebases based on natural language descriptions. With proper instructions and tools, Claude 3.5 Sonnet can independently write, edit, and execute code, demonstrating advanced reasoning and troubleshooting abilities. It also handles code translations with ease, making it useful for updating legacy applications and migrating codebases.

Improved Vision Model

Claude 3.5 Sonnet is their strongest vision model yet, surpassing Claude 3 Opus on standard benchmarks. This improvement is particularly noticeable in tasks requiring visual reasoning, such as interpreting charts and graphs. It can also accurately transcribe text from imperfect images, a valuable capability for sectors like retail, logistics, and financial services.

Introducing Artifacts

Alongside Claude 3.5 Sonnet, Anthropic introduces "Artifacts" on Claude.ai. This new feature allows users to interact with generated content in a dedicated window. Users can see, edit, and build upon Claude's creations in real-time, seamlessly integrating AI-generated content into their projects.

Evolving from Conversational AI to Collaborative Workspace

Artifacts represent a shift for Claude, evolving from a conversational AI to a collaborative work environment. This is just the beginning, with Claude.ai soon expanding to support team collaboration. In the future, teams and entire organizations will be able to manage knowledge, documents, and ongoing work in a shared space, with Claude acting as an on-demand teammate.

Safety and Transparency

Anthropic emphasizes rigorous testing and training to mitigate misuse of their models. Despite the leap in intelligence, Claude 3.5 Sonnet maintains an ASL-2 safety rating according to their red teaming assessments. More details are available in the model card addendum.

To ensure safety and transparency, Anthropic collaborates with external experts. Claude 3.5 Sonnet underwent pre-deployment safety evaluation by the UK's Artificial Intelligence Safety Institute (UK AISI). The UK AISI shared their results with the US AI Safety Institute (US AISI) as part of a joint effort.

Policy feedback from external experts ensures robust evaluations that consider emerging trends in abuse. This includes collaborating with child safety experts at Thorn to refine models and update classifiers.

Privacy Focus

User privacy is a core principle for Anthropic. They do not train their generative models on user-submitted data unless explicit permission is granted. To date, they have not used any customer or user data for generative model training.

More information on https://www.anthropic.com