Product Analytics

Product analytics can be defined as the way companies analyze what their users are doing in their product.

• Organizations want visibility into how their users use their products to make educated decisions on what they should be improving and building.
• SQL is not the ideal tool for analyzing event data. Explorations into why users are doing a particular action requires diving into the granular events themselves.
• Start small (ant iterate), stay small (limit metrics and dashboards), and keep it simple (you can explain the metric in plain language).
• Product analytics should be part of the product life-cycle the same way writing unit tests is. That is, without a middleman and having data involved from the start.
• Have one clear owner for: event implementation, QA, stakeholder buy-in on metrics from events. For each of these areas, ownership extends forever to make sure there are no regressions.
• Measure tracking quality and treat all tracking bugs as production incidents.
• Data trust issues come when data producers and data consumers doesn't work together. Aim to shorten the data loop. E.g: The feedback loop for the performance engineer is much faster and reliable (same person tracks and analyzes) than for the standard engineer that doesn't look at the data to see the impact they made with their changes.
• Product Analytics is hard because when you instrument the events you don't know what you want to analyze. When you analyze them you often didn't capture what you want to measure. Also, Ad-Blockers.
• Even tracking is seen as an after-the-fact activity instead of something integrated into day-to-day design and development.
  • No one gets promoted for doing a good tracking work. People get promoted for delivering features.
• Some of hard things: building a common nomenclature, knowing what exists, what's reusable/equivalent in new features, testing, aligning across different platforms…
• Don't treat all events the same. Missing some data in key events is not great. Keep in mind that many events are ephemeral.
• If a stakeholder wants to know if users behave a certain way you can apply this heuristic: assume they do - what decision would we make? What's the risk of just making that decision now? Risk is low? Just make it. Risk is uncertain? Let's run an experiment.
• Developers should be involved in analytics. They work on building things, they will want to know the impact.