<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xsl" href="atom.xsl"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <id>https://avuruobs.io/blog</id>
    <title>Avuru Obs Blog</title>
    <updated>2026-07-19T00:00:00.000Z</updated>
    <generator>https://github.com/jpmonette/feed</generator>
    <link rel="alternate" href="https://avuruobs.io/blog"/>
    <subtitle>Avuru Obs Blog</subtitle>
    <icon>https://avuruobs.io/img/favicon.ico</icon>
    <entry>
        <title type="html"><![CDATA[One ClickHouse, four signals: the schema behind avuru obs]]></title>
        <id>https://avuruobs.io/blog/one-clickhouse-four-signals</id>
        <link href="https://avuruobs.io/blog/one-clickhouse-four-signals"/>
        <updated>2026-07-19T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[A tour of the table schema that lets avuru obs keep traces, logs, metrics and profiling in a single ClickHouse engine — and why that makes cross-signal correlation just another SQL query.]]></summary>
        <content type="html"><![CDATA[<p>The pitch for <strong>avuru obs</strong> is that every signal lives in one engine. That is easy
to say on a landing page. This post is the part underneath: the actual ClickHouse
tables, the columns they share, and why "one store" turns cross-signal correlation
into a plain SQL <code>JOIN</code> instead of a wiring exercise across four systems.</p>
<!-- -->
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="one-database-one-engine">One database, one engine<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#one-database-one-engine" class="hash-link" aria-label="Direct link to One database, one engine" title="Direct link to One database, one engine" translate="no">​</a></h2>
<p>Everything lands in a single ClickHouse database, <code>otel</code>, and almost every table is
a plain <code>MergeTree</code>. There is no per-signal store to size, scale and back up
separately — traces, logs, metrics and profiles are tables next to each other, on
the same disks, in the same query engine.</p>
<p>The trace and log tables aren't hand-rolled. Their column contract is <strong>frozen
verbatim from the OpenTelemetry Collector's ClickHouse exporter</strong> (pinned at
<code>0.154.0</code>), so the exporter can <code>INSERT</code> with an explicit column list and avuru's
<code>hub</code> can read the exact same shape. That's the quiet superpower of building on
OTLP: the storage schema is the OpenTelemetry data model, not a proprietary
re-encoding of it.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="the-four-signals-table-by-table">The four signals, table by table<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#the-four-signals-table-by-table" class="hash-link" aria-label="Direct link to The four signals, table by table" title="Direct link to The four signals, table by table" translate="no">​</a></h2>
<p><strong>Traces</strong> live in <code>otel.otel_traces</code>:</p>
<div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token keyword" style="color:rgb(86, 156, 214)">CREATE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">TABLE</span><span class="token plain"> otel</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">otel_traces</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token plain">          DateTime64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">9</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    TraceId            String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    SpanId             String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    ParentSpanId       String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    SpanName           LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    ServiceName        LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    ResourceAttributes Map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    SpanAttributes     Map</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    Duration           UInt64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    StatusCode         LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token comment" style="color:rgb(106, 153, 85)">-- Events.*, Links.*, skip indexes on TraceId / attrs / Duration …</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ENGINE</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> MergeTree</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">PARTITION</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> toDate</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ORDER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">Tenant</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> ServiceName</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> SpanName</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> toDateTime</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>A companion table plus a materialized view keep a <code>TraceId → [start, end]</code> index so
"open this trace" is a narrow time-bounded read instead of a scan. <code>TraceId</code> and
<code>SpanId</code> carry <code>bloom_filter</code> skip indexes; <code>Duration</code> gets a <code>minmax</code> index so
"spans slower than X" prunes granules early.</p>
<p><strong>Logs</strong> live in <code>otel.otel_logs</code>, and they carry the same <code>TraceId</code>, <code>SpanId</code> and
<code>ServiceName</code> columns as spans do. On ClickHouse 26.x the log body and attribute
maps get full-text <code>text</code> indexes, and the common Kubernetes resource attributes
(<code>k8s.namespace.name</code>, <code>k8s.pod.name</code>, …) are materialized into their own columns
so filtering by pod or namespace stays cheap.</p>
<p><strong>Metrics</strong> are five tables — <code>otel_metrics_gauge</code>, <code>_sum</code>, <code>_histogram</code>,
<code>_exponential_histogram</code>, <code>_summary</code> — one per OTLP metric type. All five are
created even when only some are in use, because with a fixed schema a <em>missing</em>
table means silently dropped inserts the day a new app starts emitting histograms.
Each row keeps an <code>Exemplars.TraceId</code> / <code>Exemplars.SpanId</code> array — the thread that
sews a metric data point back to the exact trace that produced it.</p>
<p><strong>Profiling</strong> — the newest, opt-in signal — is the one schema avuru fully owns,
because OTLP Profiles is still alpha and the exporter has no profiles support yet.
It uses a two-table split so repeated stacks cost almost nothing:</p>
<div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token comment" style="color:rgb(106, 153, 85)">-- Unique stacks stored once, keyed by a 64-bit hash.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">CREATE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">TABLE</span><span class="token plain"> otel</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">profiling_stacks</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    Tenant    LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    StackHash UInt64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    Frames    Array</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    LastSeen  </span><span class="token keyword" style="color:rgb(86, 156, 214)">DateTime</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">DEFAULT</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">now</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ENGINE</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> ReplacingMergeTree</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">LastSeen</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ORDER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">Tenant</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> StackHash</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain" style="display:inline-block"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token comment" style="color:rgb(106, 153, 85)">-- Samples reference a stack by hash instead of repeating frames.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">CREATE</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">TABLE</span><span class="token plain"> otel</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">profiling_samples</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token plain">   DateTime64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">9</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    Tenant      LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    ServiceName LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    SampleType  LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    StackHash   UInt64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    </span><span class="token keyword" style="color:rgb(86, 156, 214)">Value</span><span class="token plain">       UInt64</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    NodeName    LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    PodName     String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">    ContainerName LowCardinality</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">String</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ENGINE</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> MergeTree</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ORDER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> </span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">Tenant</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> ServiceName</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="errors-a-fifth-signal-derived-not-ingested">Errors: a fifth signal derived, not ingested<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#errors-a-fifth-signal-derived-not-ingested" class="hash-link" aria-label="Direct link to Errors: a fifth signal derived, not ingested" title="Direct link to Errors: a fifth signal derived, not ingested" translate="no">​</a></h2>
<p>Error tracking isn't a separate pipeline — it's a <strong>materialized view</strong> on top of
the logs table. Any <code>ERROR</code>/<code>FATAL</code> record (OTLP <code>SeverityNumber &gt;= 17</code>) is
projected into an <code>error_events</code> table at insert time, with a stack-trace
<strong>fingerprint</strong> computed in SQL: normalize hex addresses and line numbers, hash the
top frames, and group identical crashes into one issue. Exceptions ingested from a
Sentry-compatible SDK land in the same logs table (tagged at the gateway) and flow
through the very same view — so browser errors and backend panics dedupe together,
for free, with no extra store.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="what-makes-them-one-store">What makes them <em>one</em> store<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#what-makes-them-one-store" class="hash-link" aria-label="Direct link to what-makes-them-one-store" title="Direct link to what-makes-them-one-store" translate="no">​</a></h2>
<p>Look at the sort keys and you'll notice the same columns over and over:
<code>ServiceName</code>, <code>Timestamp</code>, and above all <code>TraceId</code>. Because every signal is keyed
on the OpenTelemetry resource and trace identity, correlation is a query, not an
integration:</p>
<div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token comment" style="color:rgb(106, 153, 85)">-- From a slow span straight to the logs that explain it.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">SELECT</span><span class="token plain"> l</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> l</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">SeverityText</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> l</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">Body</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">FROM</span><span class="token plain"> otel</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">otel_logs </span><span class="token keyword" style="color:rgb(86, 156, 214)">AS</span><span class="token plain"> l</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">WHERE</span><span class="token plain"> l</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">TraceId </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> {trace_id:String}</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ORDER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> l</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<div class="language-sql codeBlockContainer_Ckt0 theme-code-block" style="--prism-color:#9CDCFE;--prism-background-color:#1E1E1E"><div class="codeBlockContent_QJqH"><pre tabindex="0" class="prism-code language-sql codeBlock_bY9V thin-scrollbar" style="color:#9CDCFE;background-color:#1E1E1E"><code class="codeBlockLines_e6Vv"><div class="token-line" style="color:#9CDCFE"><span class="token comment" style="color:rgb(106, 153, 85)">-- p99 latency per endpoint for one service, in the last hour.</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">SELECT</span><span class="token plain"> SpanName</span><span class="token punctuation" style="color:rgb(212, 212, 212)">,</span><span class="token plain"> quantile</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token number" style="color:rgb(181, 206, 168)">0.99</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token plain">Duration</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">/</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token plain">e6 </span><span class="token keyword" style="color:rgb(86, 156, 214)">AS</span><span class="token plain"> p99_ms</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">FROM</span><span class="token plain"> otel</span><span class="token punctuation" style="color:rgb(212, 212, 212)">.</span><span class="token plain">otel_traces</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">WHERE</span><span class="token plain"> ServiceName </span><span class="token operator" style="color:rgb(212, 212, 212)">=</span><span class="token plain"> </span><span class="token string" style="color:rgb(206, 145, 120)">'checkout'</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain">  </span><span class="token operator" style="color:rgb(212, 212, 212)">AND</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">Timestamp</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">&gt;</span><span class="token plain"> </span><span class="token function" style="color:rgb(220, 220, 170)">now</span><span class="token punctuation" style="color:rgb(212, 212, 212)">(</span><span class="token punctuation" style="color:rgb(212, 212, 212)">)</span><span class="token plain"> </span><span class="token operator" style="color:rgb(212, 212, 212)">-</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">INTERVAL</span><span class="token plain"> </span><span class="token number" style="color:rgb(181, 206, 168)">1</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">HOUR</span><span class="token plain"></span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">GROUP</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> SpanName</span><br></div><div class="token-line" style="color:#9CDCFE"><span class="token plain"></span><span class="token keyword" style="color:rgb(86, 156, 214)">ORDER</span><span class="token plain"> </span><span class="token keyword" style="color:rgb(86, 156, 214)">BY</span><span class="token plain"> p99_ms </span><span class="token keyword" style="color:rgb(86, 156, 214)">DESC</span><span class="token punctuation" style="color:rgb(212, 212, 212)">;</span><br></div></code></pre></div></div>
<p>A latency histogram's exemplar carries the <code>TraceId</code> of a representative request,
so a spike on a dashboard links directly to the trace behind it — same store, one
join. No <code>trace_id</code>-to-<code>trace_id</code> bridge to configure between separate systems.</p>
<p>And it's all <strong>one query language</strong>. There's no PromQL for metrics, a log query
language for logs, and a third dialect for traces — it's SQL for all four, over
tables you can inspect with <code>SHOW CREATE TABLE</code>.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="a-few-deliberate-choices">A few deliberate choices<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#a-few-deliberate-choices" class="hash-link" aria-label="Direct link to A few deliberate choices" title="Direct link to A few deliberate choices" translate="no">​</a></h2>
<ul>
<li class=""><strong>Partition by day, drop by part.</strong> Every table is <code>PARTITION BY toDate(...)</code>
with <code>ttl_only_drop_parts = 1</code>, so retention is applied by dropping whole daily
parts (fast) rather than row-level TTL mutations (slow). Retention windows are
set at migrate time from the environment, not frozen into the DDL.</li>
<li class=""><strong>The right index for the shape.</strong> Bloom filters for high-cardinality id/attribute
lookups on traces and metrics; ClickHouse <code>text</code> indexes for full-text log search.</li>
<li class=""><strong>A multi-tenancy seam that's free in single-tenant mode.</strong> Every table leads its
sort key with a <code>Tenant</code> column that defaults to <code>'default'</code>. With one tenant it's
a single constant value — zero storage and sort cost — but the column that a
hosted, multi-tenant deployment needs is already there.</li>
</ul>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="read-the-tables-yourself">Read the tables yourself<a href="https://avuruobs.io/blog/one-clickhouse-four-signals#read-the-tables-yourself" class="hash-link" aria-label="Direct link to Read the tables yourself" title="Direct link to Read the tables yourself" translate="no">​</a></h2>
<p>None of this is hidden: the migrations that create these tables ship in the engine
repo, and a running instance is one <code>SHOW CREATE TABLE</code> away. If you want to see the
whole thing light up over OTLP, <a class="" href="https://avuruobs.io/docs/getting-started/30-seconds">install in 30
seconds</a>, skim the
<a class="" href="https://avuruobs.io/guides/architecture">architecture</a>, or read how avuru obs
<a class="" href="https://avuruobs.io/docs/compare">compares to the tools you already run</a>.</p>]]></content>
        <author>
            <name>avuru obs team</name>
            <uri>https://github.com/avuruvision</uri>
        </author>
        <category label="Engineering" term="Engineering"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Your errors, finally next to your traces]]></title>
        <id>https://avuruobs.io/blog/error-tracking</id>
        <link href="https://avuruobs.io/blog/error-tracking"/>
        <updated>2026-07-16T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[avuru obs now does error tracking — issues derived from the OTLP telemetry you already send, plus Sentry-SDK-compatible ingest, all correlated with traces and logs in one ClickHouse store.]]></summary>
        <content type="html"><![CDATA[<p>Error tracking is the newest signal in <strong>avuru obs</strong>. It captures exceptions two
ways — <strong>derived automatically</strong> from the OTLP telemetry you already send, and
<strong>ingested directly from Sentry SDKs</strong> — then stores those errors right next to
the traces and logs that explain them, in one ClickHouse engine.</p>
<!-- -->
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="two-ways-to-capture-errors-no-rewrite">Two ways to capture errors, no rewrite<a href="https://avuruobs.io/blog/error-tracking#two-ways-to-capture-errors-no-rewrite" class="hash-link" aria-label="Direct link to Two ways to capture errors, no rewrite" title="Direct link to Two ways to capture errors, no rewrite" translate="no">​</a></h2>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="derived-from-telemetry-you-already-send">Derived from telemetry you already send<a href="https://avuruobs.io/blog/error-tracking#derived-from-telemetry-you-already-send" class="hash-link" aria-label="Direct link to Derived from telemetry you already send" title="Direct link to Derived from telemetry you already send" translate="no">​</a></h3>
<p>avuru obs reads the exceptions already flowing in over eBPF and OTLP — span
<code>exception</code> events, error spans and ERROR/FATAL logs — and groups them into
deduplicated <strong>issues</strong> at insert time in ClickHouse. For a backend service that
already exports to avuru obs, error tracking needs <strong>zero instrumentation</strong>: no
SDK, no code change, no extra agent.</p>
<h3 class="anchor anchorTargetStickyNavbar_Vzrq" id="sentry-sdk-compatible-ingest">Sentry-SDK-compatible ingest<a href="https://avuruobs.io/blog/error-tracking#sentry-sdk-compatible-ingest" class="hash-link" aria-label="Direct link to Sentry-SDK-compatible ingest" title="Direct link to Sentry-SDK-compatible ingest" translate="no">​</a></h3>
<p>For the errors eBPF can't see — browser exceptions above all — the gateway runs
a Sentry-protocol receiver. Point an existing Sentry SDK's <strong>DSN</strong> at the avuru
obs gateway and its events flow into the same store. You keep the SDKs you
already run and change a single config value.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="one-store-correlated-by-default">One store, correlated by default<a href="https://avuruobs.io/blog/error-tracking#one-store-correlated-by-default" class="hash-link" aria-label="Direct link to One store, correlated by default" title="Direct link to One store, correlated by default" translate="no">​</a></h2>
<p>Because every error shares a <code>trace_id</code> with the rest of your telemetry, an
issue links straight back to the trace that produced it, the logs around it, and
the metrics and profiles for that service — no jump to a separate tool. It's one
ClickHouse to size, back up and query, instead of a separate error database
bolted on beside your traces and logs.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="triage-and-regression-detection">Triage and regression detection<a href="https://avuruobs.io/blog/error-tracking#triage-and-regression-detection" class="hash-link" aria-label="Direct link to Triage and regression detection" title="Direct link to Triage and regression detection" translate="no">​</a></h2>
<p>Each issue carries what you need to triage it: a grouped stack trace, first- and
last-seen timestamps, an occurrence count and a triage state. A resolved issue
that reappears is flagged as a <strong>regression</strong>, so a fix that quietly breaks again
doesn't slip back in unnoticed.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="where-it-is-today">Where it is today<a href="https://avuruobs.io/blog/error-tracking#where-it-is-today" class="hash-link" aria-label="Direct link to Where it is today" title="Direct link to Where it is today" translate="no">​</a></h2>
<p>Error tracking is the youngest module in avuru obs. Derivation from your existing
telemetry runs by default; the Sentry ingest port is opt-in. Some things you'd
expect from a mature error tracker — deminified JavaScript frames from source
maps, release tracking and alerting on new or spiking issues — aren't here yet.
Check <a class="" href="https://avuruobs.io/docs/status">Feature status</a> for exactly what's live before you rely on
it.</p>
<h2 class="anchor anchorTargetStickyNavbar_Vzrq" id="learn-more">Learn more<a href="https://avuruobs.io/blog/error-tracking#learn-more" class="hash-link" aria-label="Direct link to Learn more" title="Direct link to Learn more" translate="no">​</a></h2>
<ul>
<li class=""><a class="" href="https://avuruobs.io/docs/signals/errors">Errors signal</a> — how derivation and Sentry ingest work.</li>
<li class=""><a class="" href="https://avuruobs.io/docs/compare/sentry">avuru obs vs. Sentry</a> — derived issues, DSN ingest and
no per-event meter.</li>
<li class=""><a class="" href="https://avuruobs.io/docs/compare/glitchtip">avuru obs vs. GlitchTip</a> — the same Sentry protocol,
unified with your other signals.</li>
</ul>]]></content>
        <author>
            <name>avuru obs team</name>
            <uri>https://github.com/avuruvision</uri>
        </author>
        <category label="Announcement" term="Announcement"/>
    </entry>
    <entry>
        <title type="html"><![CDATA[Hello, avuru obs docs]]></title>
        <id>https://avuruobs.io/blog/hello-avuru-obs</id>
        <link href="https://avuruobs.io/blog/hello-avuru-obs"/>
        <updated>2026-06-22T00:00:00.000Z</updated>
        <summary type="html"><![CDATA[Welcome to the avuru obs docs — all-in-one, eBPF-native observability with zero code changes. Start with the 30-second guide.]]></summary>
        <content type="html"><![CDATA[<p>Welcome to the <strong>avuru obs</strong> documentation site.</p>
<p>avuru obs is all-in-one observability — traces, metrics, logs and continuous
profiling — in a single install, on one storage engine, with <strong>zero code
changes</strong>. eBPF discovers your services; OpenTelemetry carries the detail.</p>
<!-- -->
<p>This site is where the onboarding funnel, guides, reference and integrations
live. The fastest way in is the
<a class="" href="https://avuruobs.io/docs/getting-started/30-seconds">30-second guide</a>: one <code>helm install</code> and the
service map lights up.</p>
<p>The docs are open and bilingual (English + French). Found a typo or a gap? Hit
<strong>"Edit this page"</strong> on any doc — it's a three-click pull request from there.</p>]]></content>
        <author>
            <name>avuru obs team</name>
            <uri>https://github.com/avuruvision</uri>
        </author>
        <category label="Announcement" term="Announcement"/>
    </entry>
</feed>