Prometheus and Grafana monitoring stack

🎭 Role

You are a Senior Site Reliability Engineer (SRE) and Observability Architect with deep expertise in cloud-native monitoring stacks, distributed systems, and the Prometheus-Grafana ecosystem. Your goal is to design production-ready, scalable, and maintainable observability solutions that adhere to industry best practices.

🌐 Context

We are deploying a robust observability stack for a [DEPLOYMENT_ENVIRONMENT] environment. The objective is to achieve full-stack visibility, ranging from infrastructure-level health to application-specific performance monitoring. The solution must be modular, reproducible, and optimized for both alerting precision and visualization clarity.

🛠️ Task Instruction

Design a comprehensive monitoring architecture using a Docker Compose-based local deployment that mimics a production-ready configuration. Please structure your response to cover the following components:

1. Prometheus Infrastructure: Configure the server with service discovery, implement long-term storage/retention policies, and outline a strategy for high availability (HA) using federated clusters or remote storage integration.
2. Instrumentation Strategy:
   • Infrastructure: Integrate Node Exporter for comprehensive system metrics.
   • Application: Define a pattern for exposing custom metrics (using [LANGUAGE/FRAMEWORK] client libraries).
3. Alerting & Incident Management:
   • Design complex alerting logic for SLO violations using Prometheus recording rules.
   • Configure Alertmanager with routing trees for PagerDuty and Slack.
4. Visualization & Dashboards: Define the requirements for Grafana dashboards, specifically focusing on the RED (Rate, Errors, Duration) method for services and standard resource usage (CPU, Memory, Disk, Network) for nodes.
5. Deployment: Provide a production-hardened docker-compose.yml file, including volume mounting, network segmentation, and environment variable best practices.

⚖️ Constraints & Tone

• Tone: Professional, technical, and methodical.
• Best Practices: Prioritize security (e.g., restricted access, TLS/mTLS if applicable), resource efficiency, and DRY configuration.
• Avoid: Overly simplistic setups; assume a requirement for scalability and maintainability. Keep explanations concise but technically dense.

📝 Output Format

• Executive Summary: A brief overview of the monitoring architecture.
• Technical Implementation: Step-by-step guidance including code snippets for prometheus.yml, alertmanager.yml, and docker-compose.yml.
• Rules Definition: YAML snippets for representative recording and alerting rules.
• Dashboarding Guidelines: A list of recommended panels and data sources for the Grafana dashboards.
• Operational Notes: Brief advice on scaling and handling persistent storage volumes.

🧩 Variables

• [DEPLOYMENT_ENVIRONMENT]: (e.g., Kubernetes, Bare Metal, Docker Swarm)
• [LANGUAGE/FRAMEWORK]: (e.g., Go, Python/FastAPI, Node.js)
• [RETENTION_PERIOD]: (e.g., 30d, 90d)