When everything in your monitoring dashboard looks green but no data reaches your backend, you know something is off. That is exactly what happened while trying to self-host SigNoz to validate how an OpenTelemetry instrumentation library captures errors in Model Context Protocol (MCP) tools. The goal was simple: demonstrate that an MCP server can return a logical error (isError: true) within a successful JSON-RPC response, and that the trace correctly reflects that failure as an ERROR span with the attribute error.type=tool_error. The library worked; spans were generated as expected, but the SigNoz UI remained empty. The cause was not in the code, but in the infrastructure.
Self-hosting observability tools like SigNoz is a decision many companies make to keep control over their data and avoid recurring cloud costs. However, the path is not always obstacle‑free. Recent documentation shows SigNoz has moved its deployment to Foundry, an approach that simplifies centralized configuration through casting.yaml files and the OpAMP agent. This allows managing fleets of collectors without redeployment, a powerful feature that, when it fails, can leave the operator with no clues. In our case, the OTLP receiver on port 4318 never bound because the effective configuration was downloaded via OpAMP with an incompatible schema, while the local ingester.yaml file, though correct, was ignored.
The problem worsens when the host machine has tight resources. With 7.28 GB total RAM and Docker Desktop allocating 3.47 GB to the WSL2 VM, the SigNoz container barely met the recommended minimum of 4 GB. Database migrations failed with 'unexpected EOF', and the lock persisted because the migration_lock row survived restarts. Manually deleting that row via psql was required. These silent failures are the hardest to diagnose, especially when every health indicator responds with HTTP 200.
From the perspective of a software development company like Q2BSTUDIO, specialized in custom software, this case reinforces a key lesson: observability depends not only on tools but on how they integrate into the broader ecosystem. When we build AI, cybersecurity, or cloud AWS/Azure solutions for our clients, reliable telemetry is a non‑negotiable requirement. A system that appears healthy while ignoring real errors can lead to wrong decisions. That is why in the BI / Power BI projects we undertake, we always prioritize validating data pipelines before trusting the dashboards.
The incident also highlighted a common pitfall in multilanguage environments: dependency duplication. Running npm install from the example directory instead of the repository root created two copies of @opentelemetry/api and @modelcontextprotocol/sdk, completely silencing the spans. The result was not an error but an informational void, the worst scenario for any debugging task. The fix was to register the example as a workspace member, sharing a single instance of each package.
Another critical point emerged at the end of the process: OpenTelemetry's BatchSpanProcessor uses an unref() timer that, in short‑lived scripts, allows the process to exit before the batch is exported. Spans were lost without reaching the backend. An explicit flush() call on stdin close solved the issue. This subtlety is easy to overlook when testing a library, but in production it can mean the difference between an observable system and one that hides its failures.
The final outcome was bittersweet: the opentel-mcp library indeed captures the logical error in MCP and transforms it into an ERROR span with the tool_error attribute. However, we were never able to see it in the SigNoz UI because the ingester was not listening. The most valuable lesson was learning to check the actual listener (cat /proc/net/tcp inside the container) instead of trusting configuration files or green panels. In a world where more and more companies adopt AI agents and MCP‑based systems, having end‑to‑end observability tools that work is essential. And if you decide to self‑host them, remember: green does not always mean everything is fine.
