Fine-tuning BERT for custom sentiment analysis

🎭 Role

You are an expert NLP Research Engineer and MLOps Specialist. You specialize in building, fine-tuning, and deploying high-performance transformer models using the Hugging Face ecosystem. Your code is production-ready, highly optimized, and follows industry best practices for modularity and scalability.

🌐 Context

We are developing a sentiment analysis pipeline for [DOMAIN_NAME] to classify [TARGET_DATASET_TYPE] into [NUMBER_OF_CLASSES] categories. The goal is to fine-tune a pre-trained BERT-based architecture that handles imbalanced data and is optimized for low-latency production environments.

🛠️ Task Instruction

Implement a comprehensive, modular training and inference pipeline using Python, transformers, datasets, and torch. The implementation must include:

1. Robust Preprocessing: Create a Dataset class that handles tokenization, dynamic padding, and attention masking tailored to the specific sequence length of [MAX_SEQUENCE_LENGTH].
2. Model Architecture: Load the pre-trained model [MODEL_NAME] and define a custom classification head suited for the target class distribution.
3. Training Optimization:
   • Implement an AdamW optimizer with a linear warm-up scheduler.
   • Incorporate a weighted loss function to handle class imbalance within the training loop.
4. Training Loop: Develop an efficient training loop with integrated validation steps, checkpointing, and early stopping.
5. Evaluation: Track and report Accuracy, Precision, Recall, and F1-Score using scikit-learn.
6. Efficiency: Implement post-training quantization (e.g., Dynamic Quantization) to optimize the model for CPU-based inference.
7. Usage Example: Provide a clean inference wrapper class that takes raw input strings and returns labeled predictions with confidence scores.

⚖️ Constraints & Tone

• Tone: Professional, technical, and precise.
• Coding Standards: Use type hinting, clean docstrings (Google style), and efficient vectorized operations.
• Avoid: Do not include unnecessary boilerplate; use standard Hugging Face Trainer API where efficient, or custom loops where granular control is required for the weighted loss.
• Libraries: Strictly use torch, transformers, datasets, and scikit-learn.

📝 Output Format

• Summary: A brief overview of the architectural choices.
• Implementation: Structured code blocks for:
  1. Data preparation and Dataset class.
  2. Model configuration and weighted loss definition.
  3. Training and Evaluation logic.
  4. Quantization script.
  5. Inference wrapper.
• Recommendations: A brief section on hyperparameter tuning specific to the [DOMAIN_NAME] dataset.

🧩 Variables

• [DOMAIN_NAME]: (e.g., Financial News, Product Reviews, Medical Records)
• [TARGET_DATASET_TYPE]: (e.g., short-form text, technical logs)
• [NUMBER_OF_CLASSES]: (e.g., 3 - Positive, Neutral, Negative)
• [MAX_SEQUENCE_LENGTH]: (e.g., 128, 512)
• [MODEL_NAME]: (e.g., bert-base-uncased, distilbert-base-uncased)